Dagster is a data orchestrator for machine learning, analytics, and ETL
New to Dagster? Learn all about the library in a short tutorial.
Take the TutorialOr read about:
To install Dagster and Dagit into an existing Python environment, run:
pip install dagster
This will install the latest stable version of the core Dagster packages in your current Python environment.
Let's get your first pipeline up and running.
from dagster import pipeline, solid
@solid
def get_name(_):
return "dagster"
@solid
def hello(context, name: str):
context.log.info("Hello, {name}!".format(name=name))
@pipeline
def hello_pipeline():
hello(get_name())
Save the code above in a file named hello_world.py
.
You can execute the pipeline in three different ways: Dagit, Dagster Python API, or Dagster CLI.
It's highly recommended to use Dagit with Dagster. Dagit is a web-based interface for viewing and interacting with Dagster objects.
pip install dagit
To visualize your pipeline in Dagit, run the following command:
dagit -f hello_world.py
Then navigate to http://localhost:3000 to start using Dagit:
Click on the "Playground" tab, then press the "Launch Execution" button to execute the pipeline. You will then see Dagit launches a pipeline run:
You can also execute the pipeline without the UI in the following methods:
Dagster Python API
from dagster import execute_pipeline
if __name__ == "__main__":
result = execute_pipeline(hello_pipeline)
Dagster CLI
dagster pipeline execute -f hello_world.py
If you have questions on getting started, we'd love to hear from you: