You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Looking at the documentation about trigger rules, it is not clear how the upstream_failed status is transmitted downstream.
The documentation contains several details about how the rules behave with skipped tasks, but no information (that I could find) is dedicated to upstream_failed, which is equally important in my opinion.
For instance, from past experience I understand that:
a task with multiple upstream dependencies and all_success is immediately executed as soon as one of the upstream tasks fails (or upstream-fails due to its own upstream dependencies), and it goes into a failed state
a task with multiple upstream dependencies and all_done does wait for all upstream dependencies, regardless of the fact that one of them has upstream_failed, and it is fully executed, it does not go automatically into a failed state.
My use case is: I have a task with multiple upstream dependencies, and if one of those fails or it receives an upstream failure (and goes into upstream_failed), it should wait for all the other dependencies to arrive at some state before being executed (and failing as well, so all_done would not be enough).
I have created a dummy example (please ignore the fact that there are a couple of custom functions called in the dag object creation, they simply set a few parameters): this simulates the structure of an existing DAG that runs on AWS EMR; if the task then_2a fails (for any reason, it can be simulated via the UI by marking it as failed), the task checkpoint_standard cascading fails, without waiting for the other branch to finish its tasks. It is correct for me that it cascading fails if one of the two branches fails, but it should wait for all branches to terminate before being executed. I tried other trigger rules for the checkpoint, without success.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello everyone,
Looking at the documentation about trigger rules, it is not clear how the
upstream_failed
status is transmitted downstream.The documentation contains several details about how the rules behave with skipped tasks, but no information (that I could find) is dedicated to
upstream_failed
, which is equally important in my opinion.For instance, from past experience I understand that:
all_success
is immediately executed as soon as one of the upstream tasks fails (or upstream-fails due to its own upstream dependencies), and it goes into a failed stateall_done
does wait for all upstream dependencies, regardless of the fact that one of them hasupstream_failed
, and it is fully executed, it does not go automatically into a failed state.My use case is: I have a task with multiple upstream dependencies, and if one of those fails or it receives an upstream failure (and goes into upstream_failed), it should wait for all the other dependencies to arrive at some state before being executed (and failing as well, so all_done would not be enough).
I have created a dummy example (please ignore the fact that there are a couple of custom functions called in the
dag
object creation, they simply set a few parameters): this simulates the structure of an existing DAG that runs on AWS EMR; if the taskthen_2a
fails (for any reason, it can be simulated via the UI by marking it as failed), the taskcheckpoint_standard
cascading fails, without waiting for the other branch to finish its tasks. It is correct for me that it cascading fails if one of the two branches fails, but it should wait for all branches to terminate before being executed. I tried other trigger rules for the checkpoint, without success.What am I missing?
Beta Was this translation helpful? Give feedback.
All reactions