Replies: 2 comments
-
Can't you just query for task details and see when the task started? That should be possible and this is a recommended solution also described as an example in https://www.bilibili.com/video/BV1zC4y1D73h/?spm_id_from=333.788.recommend_more_video.7 on how you can handle SLAs. The current SLA "feature" of Airflow is generally broken and there are some ideas (including AIP - Airflow Improvement Proposl) to rewrite it for Airflow 3 https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Workstreams - but it does not seem it has a clear owner, so not sure if it's going to happen, but generally speaking this is not something that can be treated as "feature" - it's more of complete overhaul of SLA approach for Airflow - so it's definitely not a "feature" to implement. You can read more details on why this is a complex thing in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-57+Refactor+SLA+Feature and if you, or your company would like to take a lead on discussing, re-approving the AIP and getting it implemented for Airlfow 3, that would be great - we look for someone who has enouhg of an interest and implementation / engineering powers to commit on leading and implementing it. |
Beta Was this translation helpful? Give feedback.
-
Converting to discussion if more is needed. |
Beta Was this translation helpful? Give feedback.
-
Description
As of Airflow 2.5.2, the airflow does not store SLA for a dag in the metadata database. We need to have it in the database table to check it and compare it periodically using an external monitoring tool like Grafana.
Use case/motivation
We have long-running tasks that run for more than 20 hours, and some time is left running. Since SLA is checked at the daily end, The callback does not run to notify us.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions