[AIRFLOW-3607] fix scheduler bug related to task concurrency and depends on past #7402
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
the
area:Scheduler
houqp
changed the title
ashb
reviewed
Feb 12, 2020
…n past commit 50efda5 introduced a bug that prevents scheduler from scheduling tasks with the following properties: * has depends on past set to True * has custom concurrency limit
|
@ashb updated PR |
Codecov Report
@@ Coverage Diff @@
## master #7402 +/- ##
==========================================
- Coverage 86.61% 86.15% -0.46%
==========================================
Files 873 874 +1
Lines 40757 40873 +116
==========================================
- Hits 35300 35213 -87
- Misses 5457 5660 +203
Continue to review full report at Codecov.
|
turbaszek
approved these changes
Feb 14, 2020
|
I've re-targeted this PR against AIRFLOW-3607 as it's a bug fix to a commit that has not yet been included in any release. |
ashb
changed the title
amichai07
reviewed
Feb 20, 2020
| else: | ||
| # slow path | ||
| for ti in scheduleable_tasks: | ||
| if ti.are_dependencies_met( |
There was a problem hiding this comment.
why not use get_ready_tis?
6 tasks
galuszkak
pushed a commit
to FlyrInc/apache-airflow
that referenced
this pull request
Mar 5, 2020
…n past (apache#7402) commit 50efda5 introduced a bug that prevents scheduler from scheduling tasks with the following properties: * has depends on past set to True * has custom concurrency limit
turbaszek
pushed a commit
to PolideaInternal/airflow
that referenced
this pull request
Oct 1, 2020
…n past (apache#7402) commit 50efda5 introduced a bug that prevents scheduler from scheduling tasks with the following properties: * has depends on past set to True * has custom concurrency limit
turbaszek
pushed a commit
to PolideaInternal/airflow
that referenced
this pull request
Oct 1, 2020
[AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (apache#4751) This decreases scheduler delay between tasks by about 20% for larger DAGs, sometimes more for larger or more complex DAGs. The delay between tasks can be a major issue, especially when we have dags with many subdags, figures out that the scheduling process spends plenty of time in dependency checking, we took the trigger rule dependency which calls the db for each task instance, we made it call the db just once for each dag_run [AIRFLOW-3607] fix scheduler bug related to concurrency and depends on past (apache#7402) commit 50efda5 introduced a bug that prevents scheduler from scheduling tasks with the following properties: * has depends on past set to True * has custom concurrency limit [AIRFLOW-3607] Optimize dep checking when depends on past set and concurrency limit (apache#7503)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
commit 50efda5 introduced a bug that
prevents scheduler from scheduling tasks with the following properties:
Issue link: AIRFLOW-3607
Make sure to mark the boxes below before creating PR: [x]
[AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID** For document-only changes commit message can start with
[AIRFLOW-XXXX].In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.