Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep #11010

Closed
wants to merge 2 commits into from
Closed

[AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep #11010

wants to merge 2 commits into from
+158 −100

Commits on Oct 26, 2020

  1. [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (apa… 

    …che#4751)

This decreases scheduler delay between tasks by about 20% for larger DAGs,
sometimes more for larger or more complex DAGs.

The delay between tasks can be a major issue, especially when we have dags with
many subdags, figures out that the scheduling process spends plenty of time in
dependency checking, we took the trigger rule dependency which calls the db for
each task instance, we made it call the db just once for each dag_run

(cherry picked from commit 50efda5)
    @amichai07 @yuqian90
    amichai07 authored and yuqian90 committed Oct 26, 2020
    Configuration menu
    Copy the full SHA
    903a2ca View commit details
    Browse the repository at this point in the history

  2. [AIRFLOW-3607] Optimize dep checking when depends on past set and con… 

    …currency limit
    @amichai07 @yuqian90
    amichai07 authored and yuqian90 committed Oct 26, 2020
    Configuration menu
    Copy the full SHA
    435c225 View commit details
    Browse the repository at this point in the history