Watcher depends on past by johnhoran · Pull Request #2615 · astronomer/astronomer-cosmos

johnhoran · 2026-04-27T11:53:26Z

Description

Adds a wait_for_downstream to all tasks in the DbtTaskGroup if depends_on_past is set to true. If depends_on_past is set to true, then the aim is to consider the DbtTaskGroup as a single atomic step. Most of the time this is fine, but as outlined in #2596, if a run fails and is marked as skipped, then the producer task from a subsequent run can start, even if the watcher consumer tasks from the initial run haven't completed.

This PR aims to fix that by moving the leaves of the task group so they are upstream of the watcher done task, and have wait_for_downstream set to true, in which case airflow will only start a task group if the previous run watcher done task has completed.
Before:

After this change when using TestBehavior.AFTER_ALL

After this change when using TestBehavior.AFTER_EACH

As an aside, I also put the test added for TestBehavior.AFTER_ALL task_map, mostly so I'm only modifying the wait_for_downstream in a single method, but also I suspect there was a bug when using it with ExecutionConfig.AIRFLOW_ASYNC.

astronomer-cosmos/cosmos/airflow/graph.py

Lines 881 to 883 in c919104

    
           for task_id, task in tasks_map.items(): 
        
               if len(task.downstream_list) == 0: 
        
                   task >> teardown_airflow_task

I haven't used that mode, so I can't really test that.

Related Issue(s)

closes #2596

Breaking Change?

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

Co-authored-by: Copilot <copilot@github.com>

Copilot

Pull request overview

This PR addresses a race condition in ExecutionMode.WATCHER when depends_on_past=True by making the DbtTaskGroup behave like a single atomic unit across DAG runs (so a later run’s producer can’t start while a prior run’s watcher consumers are still finishing).

Changes:

Wire task-group “leaf” tasks to be upstream of the watcher *_done gate task when depends_on_past=True.
Set wait_for_downstream=True across watcher tasks (when applicable) to enforce cross-run downstream completion before starting the next run.
Add a parametrized unit test to validate dependency wiring across TestBehavior variants and depends_on_past settings.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`cosmos/airflow/graph.py`	Adds watcher done task into `tasks_map`, applies `wait_for_downstream` behavior when `depends_on_past=True`, and rewires leaves to the watcher done gate.
`tests/airflow/test_graph.py`	Adds coverage validating watcher dependency wiring and `wait_for_downstream` behavior across test behaviors.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <copilot@github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <copilot@github.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

codecov · 2026-05-01T09:19:48Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.01%. Comparing base (2ee41a6) to head (cdf688f).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2615   +/-   ##
=======================================
  Coverage   98.01%   98.01%           
=======================================
  Files         104      104           
  Lines        7796     7803    +7     
=======================================
+ Hits         7641     7648    +7     
  Misses        155      155

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    child_2b = DbtNode(
+        unique_id=f"{DbtResourceType.MODEL.value}.{SAMPLE_PROJ_PATH.stem}.child2.v2_b",
+        resource_type=DbtResourceType.MODEL,
+        depends_on=[parent_node.unique_id],
+        path_base=SAMPLE_PROJ_PATH,
+        original_file_path=Path("gen3/models/child2_v2.sql"),
+        tags=["nightly"],
+        config={"materialized": "table", "meta": {"cosmos": {"operator_kwargs": {"pool": "custom_pool"}}}},
+        has_test=True,
+        has_non_detached_test=True,
+    )
+    child_2b_test = DbtNode(
+        unique_id=f"{DbtResourceType.TEST.value}.{SAMPLE_PROJ_PATH.stem}.child2.test_v2_b",
+        resource_type=DbtResourceType.TEST,
+        depends_on=[child_2b.unique_id],
+        path_base=Path("."),
+        original_file_path=Path("."),
+    )
+
+    build_airflow_graph(
+        nodes={child_2b.unique_id: child_2b, child_2b_test.unique_id: child_2b_test, **sample_nodes},
+        dag=dag,
+        execution_mode=ExecutionMode.WATCHER,
+        test_indirect_selection=TestIndirectSelection.EAGER,
+        task_args=task_args,
+        render_config=RenderConfig(
+            test_behavior=test_behavior,
+        ),
+        dbt_project_name="astro_shop",
+        task_group=TaskGroup("tg", dag=dag),
+    )
+    if not depends_on_past:
+        assert dag.task_dict["tg.dbt_producer_watcher_done"].upstream_task_ids == {"tg.dbt_producer_watcher"}
+        assert all(task.wait_for_downstream is False for task in dag.tasks)
+        return
+
+    assert all(task.wait_for_downstream is True for task in dag.tasks if task.task_id != "tg.dbt_producer_watcher_done")
+    if test_behavior == TestBehavior.NONE:
+        assert dag.task_dict["tg.dbt_producer_watcher_done"].upstream_task_ids == {
+            "tg.child_run",
+            "tg.dbt_producer_watcher",
+            "tg.child2_v2_run",
+            "tg.child2_v2_b_run",
+        }
+    if test_behavior == TestBehavior.AFTER_EACH:
+        assert dag.task_dict["tg.dbt_producer_watcher_done"].upstream_task_ids == {
+            "tg.child_run",
+            "tg.dbt_producer_watcher",
+            "tg.child2_v2_run",
+            "tg.child2_v2_b.test",
+        }
+    if test_behavior == TestBehavior.AFTER_ALL:
+        assert dag.task_dict["tg.dbt_producer_watcher_done"].upstream_task_ids == {
+            "tg.dbt_producer_watcher",
+            "tg.astro_shop_test",
+        }


tatiana · 2026-05-19T12:29:13Z

+        assert dag.task_dict["tg.dbt_producer_watcher_done"].upstream_task_ids == {
+            "tg.dbt_producer_watcher",
+            "tg.astro_shop_test",
+        }


The new test covers the dependency topology well. A small extra assertion that tg.dbt_producer_watcher_done.wait_for_downstream is False would make the intended gate behaviour explicit.

tatiana

@johnhoran Thanks a lot for working on this, and I'm sorry for the delay in reviewing. The approach makes sense to me, and I’m comfortable moving forward so we can release the fix.

A few non-blocking follow-ups that would be helpful, in a separate PR after this lands:

As we agreed, this only addresses the race for DbtTaskGroup, since the dbt_producer_watcher_done gate only exists when task_group is None. Could we explicitly:

Document/call out that DbtDag is still not covered, and
Create a follow-up issue or PR to either raise an exception on DbtDag or properly handle it there as well?

The watcher docs still describe max_active_runs=1 as the workaround for the depends_on_past=True race. Once this lands, that section should be updated to explain the new DbtTaskGroup behaviour and any remaining limitations for DbtDag.
Adding the AFTER_ALL test task to tasks_map looks useful, but it is also a behavioural side-change: tasks_map now includes an additional entry. In the future, it would be great to split this up into a separate PR. If, for now, we could document this change of behaviour, it would be great.
Minor inline comment (#2615 (comment))

I’m okay with the current test staying as wiring/topology coverage rather than a full cross-run scheduler integration test, since reproducing the actual concurrent-run race cleanly is much harder and probably not worth blocking this fix.

Place the watcher-related fixes (#2683, #2684, #2615) immediately after the related XCom-backup-key sanitization (#2629) so the watcher work reads as a coherent block in the 1.14.2 entry.

Four watcher-mode changes in this release adjust observable behaviour without breaking the public API. Flagging them under a new 'Behaviour Changes' heading (intentionally not 'Breaking Changes' since this is a patch release) so users relying on undocumented internals can review before upgrading: - #2615: producer-done gateway task is now downstream of every consumer in DbtTaskGroup when depends_on_past=True. Default (depends_on_past=False) topology unchanged. - #2684: downstream models skipped after upstream-failure now retry when the upstream task recovers (previously remained skipped for the run). - #2713: consumer retry no longer inherits the producer's --log-format json flag. Retry logs default to text format; JSON opt-in via dbt_cmd_flags. - #2629/#2683: XCom backup Variable key scheme changed (now includes task-group path and sanitises run-id). Monitoring scripts matching the old pattern need updating.

@MichaelRBlack

## CHANGELOG entry 1.14.2 (2026-05-21) ------------------- Behaviour Changes These changes adjust observable behaviour of the ``ExecutionMode.WATCHER`` execution mode. None of them breaks the public Cosmos API, but users relying on undocumented internals (graph wiring assertions, XCom backup Variable names, retry-on-recovery semantics, or retry log format) should review before upgrading. * ``ExecutionMode.WATCHER`` + ``depends_on_past=True``: when the producer task has ``depends_on_past=True`` (typically set via ``default_args``), the producer-done gateway task inside ``DbtTaskGroup`` is now wired downstream of every consumer task, in addition to the producer. This is required so that ``wait_for_downstream`` gating behaves correctly across DAG runs and the task group acts as a single unit that must fully succeed before the next run starts. Users with ``depends_on_past=False`` (the default) see no topology change. See #2615. * ``ExecutionMode.WATCHER`` downstream retry on upstream recovery: dbt models that were skipped after an upstream-failure event are now retried in the same DAG run when the upstream task succeeds on retry. Previously these models remained skipped for the run. See #2684. * ``ExecutionMode.WATCHER`` consumer-retry log format: the consumer's fallback ``dbt`` invocation no longer inherits the producer's internal ``--log-format json`` flag, so retry task logs now default to dbt's normal text format. Users who relied on JSON output in retry logs can opt in via ``operator_args={"dbt_cmd_flags": ["--log-format", "json"]}``. See #2713. * ``ExecutionMode.WATCHER`` XCom-backup Variable key scheme: the per-model XCom backup Variable key now includes the full task-group path and sanitises disallowed characters (``+`` / ``:``) from ``run_id``. External monitoring or cleanup scripts that match the old key pattern will need updating. See #2629 and #2683. Bug Fixes * Sanitize disallowed characters from XCom backup variable key by @MichaelRBlack in #2629 * Prevent watcher producers from colliding on one XCom-backup key by @tatiana in #2683 * Retry watcher downstream models on upstream-failure recovery by @tatiana in #2684 * Fix ``ExecutionMode.WATCHER`` interaction with ``depends_on_past`` by @johnhoran in #2615 * Strip ``--log-format`` from producer flags on watcher consumer retry by @tatiana in #2713 * Fix duplicate ``deferrable`` kwarg in ``DbtRunAirflowAsyncBigqueryOperator`` by @pankajastro in #2616 * Fix dbt docs iframe ``src`` missing deployment path prefix by @pankajastro in #2640 * Defer ``TaskInstance`` import in cluster policy to fix Sentry init crash by @pankajastro in #2662 * Restore type hints broken by lazy imports in ``cosmos/__init__.py`` by @pankajastro in #2647 * Fix ``ExecutionMode.WATCHER`` non-dbt stdout being suppressed from logs by @pankajastro in #2654 * Fix test sensor retry behaviour in ``ExecutionMode.WATCHER`` by @pankajkoti in #2658 * Fix watcher fallback selector for versioned dbt models by @pankajkoti in #2659 * Break out of iframe from Airflow 2 dbt Docs 404 link by @pankajastro in #2685 Docs * Document source freshness aware execution for ``ExecutionMode.WATCHER`` by @pankajastro in #2617 * Add reference docs for ``DbtRunLocalOperator``, ``DbtTestLocalOperator``, ``DbtSnapshotLocalOperator`` and ``DbtBuildLocalOperator`` by @pankajastro in #2643 * Add watcher retry behaviour history documentation by @tatiana in #2600 * Add Apache Airflow® trademark on first prominent mention by @pankajkoti in #2624 * Sentence-case section headings by @pankajkoti in #2630 * Use ``-`` for bullet points by @pankajkoti in #2631 * Drop decorative separator lines by @pankajkoti in #2632 * Normalize heading underlines in ``docs/guides/`` and ``docs/index.rst`` by @pankajkoti in #2664 * Fix broken cross-directory doc links by @pankajastro in #2694 * Fix broken external links in hand-written docs by @pankajastro in #2696 * Document support for Airflow 3.2 in the compatibility policy by @pankajastro in #2652 * Refresh the dbt/Airflow conflicts table to match the compatibility policy by @pankajastro in #2653 * Document incremental model limitation for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajastro in #2642 Others * Import ``ParamValidationError`` from ``airflow.sdk`` to silence deprecation warning by @pankajastro in #2645 * Import ``DAG`` from ``airflow.sdk`` to silence deprecation warning by @pankajastro in #2644 * Enforce docs style guide via pre-commit hook by @pankajkoti and @tatiana in #2633 * Add Airflow 3.2 to the test matrix in ``CLAUDE.md`` by @pankajastro in #2646 * Document the lazy-logging standard in ``CLAUDE.md`` by @pankajastro in #2679 * Extract watcher XCom-key helpers and inline single-use bindings by @pankajastro in #2673 * Remove leftover ``scripts/airflow3`` directory by @pankajastro in #2661 * Fix ``altered_jaffle_shop`` seed-dep CTE references by @pankajastro in #2690 * Skip Airflow 3.0 integration test stuck on ``example_watcher_with_freshness`` by @pankajastro in #2692 * Fix typo "constrantis" → "constraints" in tests env comment by @pankajastro in #2669 ## Summary Drafts the **Cosmos 1.14.2** release. Latest alpha cut is **1.14.2a4** — refreshed from `1.14.2a3` after maintainers (@pankajastro, @pankajkoti) added 11 more PRs to the milestone: - @pankajastro: #2646, #2652, #2653, #2661, #2669, #2673, #2679, #2690, #2692 - @pankajkoti: #2658, #2659 All 11 picks applied cleanly on top of the existing branch — no additional manual conflict resolution needed. **Excluded:** #2618 ("Improve glossary") — modifies `docs/reference/glossary.rst`, which doesn't exist on `release-1.14` (added on main by #2461, never backported). Deferred to 1.15.0. 33 PRs cherry-picked total (22 in the initial a3 cut + 11 in this a4 refresh); two PRs (#2575, #2618) deliberately held back as 1.15.0 content. ## Milestone [Cosmos 1.14.2](https://github.com/astronomer/astronomer-cosmos/milestone/48) — 33 merged PRs across Bug Fixes, Docs, and Others. ## Inclusion provenance | Path | PRs | Notes | |---|---|---| | **Originally in milestone — a3 cut (12)** | #2629, #2616, #2640, #2662, #2654, #2683, #2684, #2615, #2694, #2696, #2645, #2644 | Assigned by maintainers before the initial release-draft run | | **Pulled in via closed-issue link — a3 cut (1)** | #2647 | Closes milestone issue #2634 ("Typehinting broken with lazy imports") but the PR itself was never assigned to the milestone — included via `closedByPullRequestsReferences` | | **Added during cherry-pick conflict resolution — a3 cut (9)** | #2631, #2624, #2630, #2632, #2664, #2633, #2617, #2600, #2643 | Docs PRs whose absence caused `release-1.14` ↔ `main` textual drift. #2631 caused #2696's conflict; the rest were transitive dependencies (especially #2664, on top of the bullet/heading/trademark sweeps). #2643 was added to unblock #2664's operator-docs conflict | | **Added to milestone after a3 — included in a4 (11)** | #2646, #2652, #2653, #2658, #2659, #2661, #2669, #2673, #2679, #2690, #2692 | Added by @pankajastro and @pankajkoti after the initial draft. All applied cleanly on top of the a3 cherry-picks | | **Deliberately excluded (2)** | #2575, #2618 | #2575: documents `DbtDocsS3KubernetesOperator` with `.. versionadded:: 1.15.0` (already in the `Cosmos 1.15.0` milestone). #2618: improves a glossary file that doesn't exist on `release-1.14` (the stub was added by #2461, not backported) | ### Manual conflict resolution Cherry-picks that needed manual fix-up. Reviewers should double-check the files listed below: | PR | File(s) | Resolution | |---|---|---| | **#2664** | `docs/guides/dbt_docs/generating-docs.rst` | **Substantive exclusion** — manually removed the entire `Upload to S3 from Kubernetes` section (lines ~46–77 of the incoming diff) that documents `DbtDocsS3KubernetesOperator` (1.15.0 feature, PR #2575). Kept HEAD (no S3-from-Kubernetes section). | | **#2664** | `docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst` | Took incoming side for "Example 1" / "Example 2" heading underlines (`++++` style — matches #2664's normalization across the rest of the file). Also took incoming for em-dash → colon ("Example 1 —" → "Example 1:"). | | **#2664** | `docs/guides/run_dbt/operators/operators.rst` | Auto-resolved once #2643 was cherry-picked first (added missing Run/Test/Snapshot/Build operator reference docs that #2664 expected to be present). | | **#2633** | `docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst` | Initially took `'''` (Example 1/2 underlines) per #2633's incoming side — this broke the file's heading hierarchy and prevented sphinx from registering the `_watcher-source-freshness:` label, causing the docs build to fail. **Fixed in a follow-up commit** by reverting to `+++` to match main and the rest of the file's level-3 sections. | | **#2617** | `docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst` | **Substantive trim** — #2617 documented both the 1.14.0 source-freshness execution path AND the 1.15.0 `freshness_callback` override (which ships with #2586, not in this release line). Removed the `literalinclude` of `dev/dags/watcher_with_freshness_check.py` (1.15.0 example DAG, missing on release-1.14) and the surrounding override section so the 1.14.2 docs cover only what the 1.14.x line supports. Surfaced as a `-W` (warnings-as-errors) docs build failure on the first CI run. | | **#2684** | `cosmos/operators/_watcher/state.py` | Took incoming side — added two new frozensets (`DBT_UPSTREAM_FAILURE_SKIP_EVENT_NAMES`, `DBT_SOURCE_FRESHNESS_STALE_STATUSES`) at lines 28–37. HEAD had nothing; incoming had the additions. | | **#2615** | `tests/airflow/test_graph.py` | Took incoming side — added two new tests at the end of the file: `test_add_watcher_producer_task_passes_freshness_callback_via_setup_operator_args` and `test_watcher_dependency_wiring`. HEAD had nothing; incoming had the additions. | All a4 cherry-picks (11) applied without manual intervention. ## Test plan > Long-term goal: automate. For now, please pick the slice relevant to your environment and report deviations as a comment on this PR. ### Watcher mode (Postgres) - [x] @pankajkoti `example_watcher` (`dev/dags/example_watcher.py`) — default watcher run; exercises **#2629** (XCom backup key sanitization triggers on the `+` in any default Airflow run_id) - [x] (@tatiana) **[NEW]** `example_watcher_xcom_collision` (`dev/failed_dags/example_watcher_xcom_collision.py`) — validates **#2683** - [x] @tatiana **[NEW]** `example_watcher_recovers_skipped_downstream` (`dev/failed_dags/example_watcher_recovers_skipped_downstream.py`) — validates **#2684** - [x] @tatiana `example_watcher` with `default_args={"depends_on_past": True}` and ≥2 consecutive runs — validates **#2615** (manual edit; no dedicated example DAG) - [x] @pankajkoti watcher DAG with at least one model that has tests + Airflow retries enabled on the test sensor — validates **#2658** (test sensor retry path) - [x] @pankajkoti watcher DAG referencing a versioned dbt model (e.g. `models/foo_v2.sql`) and triggering the fallback selector path — validates **#2659** ### BigQuery (async) - [x] @tatiana `simple_dag_async` (`dev/dags/simple_dag_async.py`) — validates **#2616** (duplicate `deferrable` kwarg fix) ### dbt docs plugin - [x] (@pankajkoti) `docs_dag` (`dev/dags/dbt_docs.py`) + open the Cosmos dbt docs URL in the Airflow UI under a non-root deployment path — validates **#2640** (iframe `src` deployment prefix) ### Cross-cutting scenarios (no dedicated DAG) - [x] @pankajastro **#2662** — boot Airflow with Cosmos cluster policy + Sentry init; verify no `TaskInstance`-import crash on startup - [x] @pankajastro **#2654** — run a watcher DAG that prints non-dbt stdout (e.g., Snowflake `externalbrowser` auth URL); confirm output reaches task logs - [x] (@pankajkoti) **#2647** — `mypy` / IDE inspection of `from cosmos import DbtDag, ProjectConfig, ProfileConfig, RenderConfig, ExecutionConfig` resolves attributes - [x] @pankajastro **#2645, #2644** — boot Airflow 3 with `airflow.sdk` available; no `ParamValidationError` / `DAG` deprecation warnings from Cosmos imports - [x] @pankajastro **#2673** — verify watcher XCom-key extraction left no behavioural drift (run the existing watcher integration suite end-to-end on Postgres + Airflow 2.10) ### Docs build (covers all docs PRs) - [x] (@pankajkoti) `sphinx-build -W -b html docs docs/_build` succeeds with no warnings (#2617, #2624, #2630, #2631, #2632, #2643, #2600, #2664, #2694, #2696, #2652, #2653) ### Tooling - [x] (@pankajkoti) `pre-commit run check-docs-style --all-files` passes (#2633) ## Reviewer checklist - [x] CHANGELOG section assignments reviewed - [x] Entry wording reviewed - [x] `cosmos/__init__.py` bumped to `1.14.2a4` - [x] Cosmetic docs PRs (#2624, #2630, #2631, #2632, #2633, #2664) confirmed acceptable in patch line - [x] **Manual conflict resolutions** (table above) reviewed file-by-file - [x] Test plan executed on at least Postgres + one warehouse - [x] Ready to mark non-draft --------- Co-authored-by: Michael Black <4128408+MichaelRBlack@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: John Horan <jhoran@zendesk.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

tatiana · 2026-05-21T16:36:33Z

🚀 Released in Cosmos 1.14.2 (PyPI).

Two consistency / accuracy fixes pointed out in review: 1. "Does the Airflow state match dbt's?" — 1.14.1 row no longer contradicts the new 1.14.2 false-green note. It was previously a bare "Yes", but #2684 (added in 1.14.2) closes a real false-green gap that existed in 1.14.1: upstream-failure-skipped downstream nodes were silently marked SKIPPED while the DAG completed success. The 1.14.1 row now reads "Mostly, except upstream- failure-skipped downstream nodes could produce a 'false green' -- fixed in 1.14.2 by #2684", with the explicit forward pointer. 2. "Avoid duplicate or concurrent runs" — 1.14.2 row corrected to describe the actual topology landed by #2615. Previously read "the producer-done gateway is wired downstream of every consumer task and the task-group leaves carry wait_for_downstream=True", which overstates both halves: only leaf consumer tasks (those without downstream tasks) get wired upstream of the gateway (see cosmos/airflow/graph.py:795-796), and wait_for_downstream=True is set on the producer plus all watcher consumer tasks, not just task group leaves (see graph.py:777, 793). Reworded to: "leaf consumer tasks are wired upstream of the producer-done gateway, and the producer plus watcher tasks carry wait_for_downstream=True." Verified: hatch run docs:sphinx-build -W -b html docs docs/_build/html succeeds. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

johnhoran and others added 4 commits April 27, 2026 10:33

update graph

70b4415

Co-authored-by: Copilot <copilot@github.com>

add test

b54a38c

Co-authored-by: Copilot <copilot@github.com>

add test

4d52358

Co-authored-by: Copilot <copilot@github.com>

use task_map

e6b22ed

Copilot AI review requested due to automatic review settings April 27, 2026 11:53

johnhoran requested review from a team, corsettigyg, dwreeves and jbandoro as code owners April 27, 2026 11:53

johnhoran requested review from pankajkoti and tatiana April 27, 2026 11:53

johnhoran had a problem deploying to external April 27, 2026 11:53 — with GitHub Actions Error

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

271f4f1

Copilot started reviewing on behalf of johnhoran April 27, 2026 11:53 View session

pre-commit-ci Bot had a problem deploying to external April 27, 2026 11:54 Error

Copilot AI reviewed Apr 27, 2026

View reviewed changes

Comment thread cosmos/airflow/graph.py

Comment thread tests/airflow/test_graph.py

refactor around mcabe limit

9731143

Co-authored-by: Copilot <copilot@github.com>

johnhoran had a problem deploying to external April 27, 2026 13:51 — with GitHub Actions Error

mypy

a5436e0

Copilot AI review requested due to automatic review settings April 27, 2026 13:55

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

061d706

Copilot started reviewing on behalf of johnhoran April 27, 2026 13:56 View session

pre-commit-ci Bot had a problem deploying to external April 27, 2026 13:56 Error

Copilot AI reviewed Apr 27, 2026

View reviewed changes

Comment thread cosmos/airflow/graph.py

update comment

4c0931c

Co-authored-by: Copilot <copilot@github.com>

johnhoran temporarily deployed to external April 27, 2026 14:10 — with GitHub Actions Inactive

resolve test failure

ec12577

Copilot AI review requested due to automatic review settings April 28, 2026 16:33

johnhoran had a problem deploying to external April 28, 2026 16:33 — with GitHub Actions Error

Copilot started reviewing on behalf of johnhoran April 28, 2026 16:34 View session

Copilot AI reviewed Apr 28, 2026

View reviewed changes

Merge branch 'main' into 2596_fix-watcher-depends-on-past

0ed111d

johnhoran temporarily deployed to external April 30, 2026 09:47 — with GitHub Actions Inactive

Merge branch 'main' into 2596_fix-watcher-depends-on-past

441cf87

Copilot AI review requested due to automatic review settings May 5, 2026 08:40

johnhoran temporarily deployed to external May 5, 2026 08:41 — with GitHub Actions Inactive

Copilot started reviewing on behalf of johnhoran May 5, 2026 08:41 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Merge branch 'main' into 2596_fix-watcher-depends-on-past

cdf688f

johnhoran temporarily deployed to external May 11, 2026 15:23 — with GitHub Actions Inactive

tatiana added this to the Cosmos 1.14.2 milestone May 19, 2026

tatiana reviewed May 19, 2026

View reviewed changes

tatiana approved these changes May 19, 2026

View reviewed changes

tatiana merged commit b0b5bc3 into astronomer:main May 19, 2026
125 checks passed

tatiana mentioned this pull request May 20, 2026

Release 1.14.2 #2708

Merged

22 tasks

tatiana mentioned this pull request May 28, 2026

Add Cosmos 1.14.2 row to the watcher-retry behavior history #2742

Open

4 tasks

00yhj22-debug mentioned this pull request Jun 1, 2026

Refactor build_airflow_graph to satisfy the C901 limit #2689

Open

2 tasks

pankajkoti mentioned this pull request Jun 2, 2026

refactor: extract _add_test_tasks to reduce build_airflow_graph complexity #2734

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Watcher depends on past#2615

Watcher depends on past#2615
tatiana merged 13 commits into
astronomer:mainfrom
johnhoran:2596_fix-watcher-depends-on-past

johnhoran commented Apr 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

codecov Bot commented May 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

tatiana May 19, 2026

Uh oh!

tatiana left a comment •

edited

Loading

Uh oh!

Uh oh!

tatiana commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	for task_id, task in tasks_map.items():
	if len(task.downstream_list) == 0:
	task >> teardown_airflow_task

Conversation

johnhoran commented Apr 27, 2026

Description

Related Issue(s)

Breaking Change?

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

codecov Bot commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

tatiana May 19, 2026

Choose a reason for hiding this comment

Uh oh!

tatiana left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tatiana commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented May 1, 2026 •

edited

Loading

tatiana left a comment •

edited

Loading