DagsterDocs
Integrations#
Dagster is flexible and allows incremental adoption. It provides add-on libraries to integrate with your existing tools and infrastructure.
Guides#
This section includes guides on how to use Dagster with other tools.
| Name | Description |
|---|---|
| Dagster with dbt | This guide shows how to orchestrate dbt from Dagster. |
| Dagster with Airflow | This guide shows how to compile an Airflow DAG into a Dagster pipeline. |
| Dagster with Great Expectations | This guide shows how to run data quality tests using Great Expectations in a Dagster pipeline. |
| Dagster with PySpark | This guide shows how to define and execute spark jobs in Dagster. |
| Dagster with Pandas | This guide shows how Dagster works with Pandas. |
| Dagster with Jupyter/Papermill | This guide shows how to orchestrate Jupyter notebooks from Dagster. |
Libraries#
Here is a complete list of Dagster's integration libraries. See full documentation in API Reference.
| Integration | Library |
|---|---|
| Airflow | dagster-airflow |
| AWS | dagster-aws |
| Azure | dagster-azure |
| Celery | dagster-celery |
| Celery + Docker | dagster-celery-docker |
| Cron | dagster-cron |
| Dask | dagster-dask |
| Databricks | dagster-databricks |
| Datadog | dagster-datadog |
| Docker | dagster-docker |
| dbt | dagster-dbt |
| GCP | dagster-gcp |
| Great Expectations | dagster-ge |
| Github | dagster-github |
| Kubernetes | dagster-k8s |
| MySQL | dagster-mysql |
| PagerDuty | dagster-pagerduty |
| Pandas | dagster-pandas |
| Papermill | dagstermill |
| Papertrail | dagster-papertrail |
| PostgreSQL | dagster-postgres |
| Prometheus | dagster-prometheus |
| Pyspark | dagster-pyspark |
| Shell | dagster-shell |
| Slack | dagster-slack |
| Snowflake | dagster-snowflake |
| Spark | dagster-spark |
| SSH / SFTP | dagster-ssh |
| Twilio | dagster-twilio |