Pipeline Modes#

Modes (along with Resources) provide a way to control the behavior of multiple solids at pipeline execution time.

Relevant APIs#

Overview#

For those familiar with dependency injection, modes and resources offer a similar capability, using pythonic idioms and Dagster’s configuration system.

Each pipeline can be defined with multiple modes, but each pipeline run uses exactly one mode. Each mode exposes:

• A set of Resources
• A Logger
• An Executor

A typical usage for modes is to vary pipeline behavior between different deployment environments. For example, you might define a local_dev mode for running pipelines on a laptop against synthetic data and a “prod” mode for running pipelines against production data in the cloud.

Modes affect solid behavior by providing resources to the solids. For example, the local_dev mode might reference a SQLiteDatabase resource that solids can execute queries against. The prod mode might instead reference an PostgresDatabase resource.

Defining a Mode Definition#

To define a mode, we use the ModeDefinition class:

ModeDefinition(name="prod", resources={...}, loggers={...}, executor_defs=[...])

Modes with Resources#

In this example, we define two modes that contain a database resource:

@pipeline(
    mode_defs=[
        ModeDefinition("local_dev", resource_defs={"database": sqlite_database}),
        ModeDefinition("prod", resource_defs={"database": postgres_database}),
    ],
)
def query_pipeline():
    do_query()

We can access the resource by resource name in the solid:

@solid(required_resource_keys={"database"})
def generate_table_1(context):
    context.resources.database.execute_query("create table_1 as select * from table_0")

When the pipeline is run with the local_dev mode, context.resources.database will be an instance of SqliteDatabase. When running in prod mode, the same resource will be a PostgresDatabase

Modes with Loggers#

Modes with Executors#

Selecting a Mode during execution#

In Dagit#

When Launching the pipeline via the Dagit Playground, you can select a mode from the mode selector dropdown:

Dagster CLI#

When launching a pipeline via the CLI, you can use the -d option to specify the mode.

$ dagster pipeline execute -d prod_mode my_pipeline