Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run the task with the configured dag bundle #44752

Merged
merged 2 commits into from
Jan 15, 2025

Conversation

dstandish
Copy link
Contributor

@dstandish dstandish commented Dec 6, 2024

Here we ensure that when a dag run is created, we stamp the current bundle version on the dag run so that we can ensure the run is run with the right bundle version. Then i make sure it flows through properly to the task run context where it's checked out and loaded.

I also rename 'path' or 'file' to dag_rel_path in some parts of the execution machinery, so that it's clear where we are using the relative path and the absolute path.

@boring-cyborg boring-cyborg bot added area:Executors-core LocalExecutor & SequentialExecutor area:task-sdk labels Dec 6, 2024
@dstandish dstandish force-pushed the run-the-configured-bundle branch from 5fd9bfe to 6553836 Compare December 6, 2024 23:03
airflow/models/dag.py Outdated Show resolved Hide resolved
@dstandish dstandish force-pushed the run-the-configured-bundle branch 2 times, most recently from 6724f00 to 45b68d0 Compare December 17, 2024 18:04
@dstandish dstandish force-pushed the run-the-configured-bundle branch from 45b68d0 to 042fb04 Compare December 30, 2024 21:05
@dstandish dstandish force-pushed the run-the-configured-bundle branch 5 times, most recently from ff2b6d2 to 0a7e0ad Compare January 8, 2025 19:01
@dstandish dstandish force-pushed the run-the-configured-bundle branch 2 times, most recently from bfa70db to c9ccedd Compare January 13, 2025 15:34
@dstandish dstandish marked this pull request as ready for review January 13, 2025 21:05
task_sdk/src/airflow/sdk/execution_time/supervisor.py Outdated Show resolved Hide resolved
task_sdk/src/airflow/sdk/execution_time/task_runner.py Outdated Show resolved Hide resolved
task_sdk/src/airflow/sdk/execution_time/task_runner.py Outdated Show resolved Hide resolved
tests/dag_processing/bundles/test_dag_bundle_manager.py Outdated Show resolved Hide resolved
airflow/models/dagbundle.py Outdated Show resolved Hide resolved
@dstandish dstandish force-pushed the run-the-configured-bundle branch from 5487556 to 80000c1 Compare January 13, 2025 23:16
@dstandish dstandish force-pushed the run-the-configured-bundle branch from 80000c1 to 472f578 Compare January 14, 2025 18:31
@dstandish dstandish requested a review from jscheffl as a code owner January 14, 2025 18:31
@dstandish dstandish added the full tests needed We need to run full set of tests for this PR to merge label Jan 14, 2025
@dstandish dstandish force-pushed the run-the-configured-bundle branch from c9757db to f1d744f Compare January 14, 2025 22:45
@dstandish dstandish merged commit 6d048c4 into apache:main Jan 15, 2025
91 checks passed
@dstandish dstandish deleted the run-the-configured-bundle branch January 15, 2025 00:19
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
Ensures that dag runs are created with a reference to the bundle that was in effect at the time.  And when a dag run has  bundle info, the task will be run with that dag bundle version.
dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
Ensures that dag runs are created with a reference to the bundle that was in effect at the time.  And when a dag run has  bundle info, the task will be run with that dag bundle version.
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
Ensures that dag runs are created with a reference to the bundle that was in effect at the time.  And when a dag run has  bundle info, the task will be run with that dag bundle version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AIP-66: DAG Bundle/Manifest area:Executors-core LocalExecutor & SequentialExecutor area:task-sdk full tests needed We need to run full set of tests for this PR to merge
Projects
Development

Successfully merging this pull request may close these issues.

2 participants