kick off a job run after another job completes without using Assets

[yuhan]

Question was originally asked in Dagster Slack. Reposting it here for posterity and discoverability.

What is the appropriate way to construct a sensor that yields a job_2 run after any completed job_1 run, if i am not using Assets?
is this the right approach?

@op
def do_a_thing():
    print("foo")

@job
def do_the_first_thing():
    do_a_thing()

@job
def do_the_second_thing():
    do_a_thing()

@run_status_sensor(
    pipeline_run_status=PipelineRunStatus.SUCCESS, 
    job_selection=[do_the_first_thing], 
)
def thing_sensor(context):
    yield RunRequest(
        job_name=do_the_second_thing
    )

[yuhan]

Run status sensors currently don’t support yielding RunRequest. So, I'd recommend to write a custom sensor that listens to the run storage and yields a RunRequest when a job_1 run finishes.

Something like:

from dagster import RunsFilter, sensor, DagsterRunStatus

@sensor(job=do_the_second_thing)
def thing_sensor(context):
    run_records = context.instance.get_run_records(
        filters=RunsFilter(
            job_name="do_the_first_thing",
            statuses=[DagsterRunStatus.SUCCESS],
            updated_after=..., # can also filter by timestamp to do more efficient fetching
        ),
        order_by="update_timestamp",
        ascending=False,
    )
    for run_record in run_records:
        yield RunRequest(
            run_key=run_record.dagster_run.run_id,  # avoid double firing for the same run
        )

[nvinhphuc]

This gives error: AttributeError: 'RunRecord' object has no attribute 'run_id'

[alangenfeld]

run_record.pipeline_run.run_id should work

[hec10r]

Hi @yuhan, thanks for the examples. Question: what is the problem with the first approach? Or why the second one is preferred? Thanks!

[tav1nald3rs0n]

Same question. Was run_status_sensor designed only for telemetry purpose?

[yuhan]

Mainly because run_status_sensor currently doesn't support yielding RunRequest so the first approach won't work. I updated the answer to make that clear.

[CamFromStar]

@yuhan in the first approach, could yield not be changed to return to make it work? Is there a reason in this case for why yield is desired over return?