Add initial documentation for `ExecutionMode.WATCHER` by tatiana · Pull Request #2046 · astronomer/astronomer-cosmos

tatiana · 2025-10-21T13:59:26Z

Introduce the documentaiton for the recently introduced high-performance execution mode, named ExecutionMode.WATCHER, following the implementation in PRs #1982, #1993 and #1998. #1999.

Closes: #1965
Closes: https://github.com/astronomer/oss-integrations-private/issues/245

codecov · 2025-10-21T14:16:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.77%. Comparing base (eda990d) to head (fa3fce2).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2046   +/-   ##
=======================================
  Coverage   97.77%   97.77%           
=======================================
  Files          91       91           
  Lines        5830     5830           
=======================================
  Hits         5700     5700           
  Misses        130      130

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

yetudada

This is great work @tatiana! I've left some feedback but I'm happy with how the limitations are presented.

pankajkoti

Looks great to me, thanks for working on this. Happy to re-review if you'd like me to -- once out of draft and if we make additional updates by then.

pankajastro

Looks good to me. Since the CI isn’t passing, I can’t see how it’s rendering. Happy for this to be merged once the CI passes and the pending feedback is addressed.

Co-authored-by: Yetunde Dada <43755008+yetudada@users.noreply.github.com>

…ull/2046/files\#r2454901842

…/pull/2046/files\#r2466253122

…/2046/files\#r2455041189

…/2046/files\#r2455045660

…-cosmos/pull/2046/files\#r2466262778

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Copilot

Pull Request Overview

This PR introduces comprehensive documentation for the new experimental ExecutionMode.WATCHER feature in Cosmos 1.11.0, which dramatically improves dbt pipeline performance in Airflow by running dbt as a single command while maintaining model-level observability through a producer-consumer pattern.

Key changes:

Documents performance improvements of up to 80% reduction in DAG runtime compared to ExecutionMode.LOCAL
Explains the producer/watcher architecture using DbtProducerWatcherOperator and DbtConsumerWatcherSensor
Details known limitations including installation requirements, operator support, and sensor implementation

Reviewed Changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 5 comments.

File	Description
docs/getting_started/watcher-execution-mode.rst	New comprehensive documentation page covering the WATCHER execution mode concept, performance benchmarks, usage examples, and known limitations
docs/getting_started/execution-modes.rst	Updates the execution modes overview to include the new WATCHER mode in the list and comparison table

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…ull/2046\#discussion_r2469655920

yetudada

All happy with this! Thank you so much @tatiana!

@tatiana

**Features** * Introduce ``ExecutionMode.WATCHER`` to reduce DAG run time by 1/5 in several PRs. Learn more about it [here](https://astronomer.github.io/astronomer-cosmos/getting_started/watcher-execution-mode.html#watcher-execution-mode). This feature was implemented via multiple PRs, including: * Expose new execution mode by @tatiana @pankajastro @pankajkoti in #1999 * Add ``DbtProducerWatcherOperator`` for the proposed ``ExecutionMode.WATCHER`` by @pankajkoti in #1982 * Add ``DbtConsumerWatcherSensor`` for the proposed ``ExecutionMode.WATCHER`` by @pankajastro in #1998 * Push producer's task completion status to XCOM by @pankajkoti in #2000 * Add default priority_weight for ``DbtProducerWatcherOperator`` by @pankajkoti in #1995 * Add sample dbt events for the dbt watcher execution mode by @pankajkoti in #1952 * Add ``compiled_sql`` as a template fields on ```ExecutionMode.WATCHER``` when using ``run_results.json`` by @pankajastro in #2070 * Set ``push_run_results_to_xcom`` kwargs correctly for invocation mode subprocess and Watcher mode by @pankajastro in #2067 * Store compiled SQL as template field for dbt callback events in ``ExecutionMode.WATCHER`` by @pankajkoti in #2068 * Add initial documentation for ``ExecutionMode.WATCHER`` by @tatiana in #2046 * Support running ``State.UPSTREAM_FAILED`` tasks when WATCHER consumer upstream tasks fail by @tatiana in #2062 * Fail sensor tasks immediately if the ``ExecutionMode.WATCHER`` producer task fails by @pankajastro in #2040 * Add ``WATCHER``` to GitHub issue template by @tatiana in #2056 * Add support for ``TestBehavior.AFTER_ALL`` with ``ExecutionMode.WATCHER`` by @pankajastro in #2049 * Add support for ``TestBehavior.NONE`` with ``ExecutionMode.WATCHER`` by @pankajastro in #2047 * Fix ``ExecutionMode.WATCHER`` behaviour with ``DbtTaskGroup`` by @tatiana in #2044 * Fix Cosmos behaviour when using watcher with ``InvocationMode.DBT_RUNNER`` by @tatiana in #2048 * Add Airflow 3 plugin for dbt docs with multiple dbt projects support by @pankajkoti in #2009, check the [documentation](https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html). * Initial support to ``dbt Fusion`` by @tatiana in #1803. More details [here](https://astronomer.github.io/astronomer-cosmos/configuration/dbt-fusion). * Support to prune sources without downstream references in dbt projects by @corsettigyg in #1988 * Allow to set task display name as a user-defined function by @corsettigyg in #1761 * Add dbt project's hash to dag docs to support dag versioning in Airflow 3 by @pankajkoti in #1907 * feat: Add Jinja templating support for ``dbt_cmd_flags`` by @skillicinski in #1899 * Add Scarf metric to collect the execution mode uses by @pankajastro in #1981 * Support Airflow 3.1 by @tatiana in #1980 * Add MySQL profile mapping by @Lee2532 in #1977 * Add sqlserver profile mapping by @pankajastro in #1737 **Enhancement** * Use XCom to store sql when using ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajastro in #1934 * Refactor ``AIRFLOW_ASYNC`` teardown so it doesn't install the virtualenv by @pankajastro in #1938 * Reuse the virtual env for ``AIRFLOW_ASYNC`` setup task by @pankajastro in #1939 * Improve dataset/asset experience in Cosmos by @tatiana in #2030 * Add ``downstreams`` to ``DbtNode`` by @wornjs in #2028 **Bug fixes** * Fix tags extraction by @ms32035 in #1915 * Fix task flow operator args by @anyapriya in #2024 **Documentation** * Add documentation for Airflow 3 Plugin supporting dbt docs for multiple dbt projects by @pankajkoti in #2063 * Add Cosmos Deferrable Operator Guide by @pankajastro in #1922 * Add dbt Fusion documentation by @tatiana in #1824 #1830 * Update dbt-fusion.rst to explicitly highlight it is in alpha by @tatiana in #1838 * Fix a bunch of docs build errors and warnings by @pankajkoti in #1886 * Add docs note for param virtualenv_dir for async execution mode by @pankajastro in #1969 * Use pepy.tech downloads badge in README by @pankajkoti in #1920 * Correct the default value of ``cache_dir`` by @seokyun.ha in #2027 **Others** * Promote @corsettigyg to committer by @tatiana in #1985 * Add @pankajkoti and @pankajastro to ``contributors.rst`` by @tatiana in #1983 * Update setup script for airflow3 script by @dwreeves in #2023 * Prevent pytest from trying to test classes that aren't actually tests by @anyapriya in #2032 * Fix ``dag.test()`` for Airflow 3.1+ by syncing DAG to database bby @kaxil in #2037 * Disable Scarf in CI by @pankajastro in #2016 * Fix failing dbt Fusion tests when run in parallel in CI by @pankajkoti in #1896 * Fix MyPy issues related to ``ObjectStoragePath`` in main branch by @tatiana in #2012 * Cleanup example dbt event JSON dictionaries kept for XCOM referencby @pankajkoti in #1997 * Bump min hatch version that includes fixes for click>=8.3.0 by @pankajkoti in #1996 * Use official postgres image from Docker hub for kubernetes setup by @pankajkoti in #1986 * Use click<8.3.0 for hatch as click 8.3 breaks hatch by @pankajkoti in #1987 * Pin Airflow version in type check CI job by @pankajastro in #2003 * Improve comments after feedback on #1948 by @tatiana in #1963 * Fix running tests with dbt Fusion 2.0.0 preview versions by @tatiana in #1948 * Test hardening of dbt node having tags as unset or missing by @pankajkoti in #1918 * Fix Sphinx issue in the main branch by @tatiana in #2064 * pre-commit autoupdate in #2065, #2043, #2033, #2019, #1990, #2019, #2008, #1941, #1935, #1924 * GitHub dependabot update in #2051, #2050, #2038, #2022, #1947, #1955, #1946, #1944, #1945, #1928, #1921, #1917 Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io> Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

tatiana temporarily deployed to internal October 21, 2025 13:59 — with GitHub Actions Inactive

pankajastro mentioned this pull request Oct 21, 2025

Fail sensor tasks immediately if the ExecutionMode.WATCHER producer task fails #2040

Merged

tatiana changed the title ~~Add initial documentation for ExecutionMode.WATCHER~~ Add initial documentation for ExecutionMode.WATCHER Oct 22, 2025

tatiana temporarily deployed to internal October 22, 2025 10:30 — with GitHub Actions Inactive

tatiana temporarily deployed to internal October 22, 2025 12:31 — with GitHub Actions Inactive

tatiana temporarily deployed to internal October 22, 2025 13:16 — with GitHub Actions Inactive

tatiana temporarily deployed to internal October 22, 2025 14:33 — with GitHub Actions Inactive

tatiana temporarily deployed to internal October 22, 2025 14:52 — with GitHub Actions Inactive

tatiana requested review from pankajastro, pankajkoti and yetudada October 22, 2025 14:52

yetudada requested changes Oct 23, 2025

View reviewed changes

tatiana requested a deployment to internal October 23, 2025 16:59 — with GitHub Actions In progress

tatiana temporarily deployed to internal October 23, 2025 16:59 — with GitHub Actions Inactive

tatiana had a problem deploying to internal October 23, 2025 17:01 — with GitHub Actions Error

tatiana temporarily deployed to internal October 23, 2025 17:01 — with GitHub Actions Inactive

tatiana temporarily deployed to internal October 23, 2025 17:07 — with GitHub Actions Inactive

pankajkoti approved these changes Oct 27, 2025

View reviewed changes

pankajastro reviewed Oct 28, 2025

View reviewed changes

tatiana temporarily deployed to internal October 28, 2025 10:33 — with GitHub Actions Inactive

tatiana and others added 9 commits October 28, 2025 10:35

Add initial documentation for ExecutionMode.WATCHER

52c8c0c

Improve docs

ad0a37c

Improve docs, add test behaviour

256cda3

Add virtualenv limitation

58ee4bc

Add virtualenv limitation

5d023a0

Add synchronous x asynchronous limitation

40073ad

Add images

96f1801

Apply suggestion from @yetudada

d9186c6

Co-authored-by: Yetunde Dada <43755008+yetudada@users.noreply.github.com>

Apply suggestion from @yetudada

c7fba86

Co-authored-by: Yetunde Dada <43755008+yetudada@users.noreply.github.com>

tatiana temporarily deployed to internal October 28, 2025 12:20 — with GitHub Actions Inactive

Address PR feedback https://github.com/astronomer/astronomer-cosmos/p…

4a0ca7a

…ull/2046/files\#r2454901842

tatiana temporarily deployed to internal October 28, 2025 12:23 — with GitHub Actions Inactive

Address feedback from https://github.com/astronomer/astronomer-cosmos…

636881e

…/pull/2046/files\#r2466253122

tatiana temporarily deployed to internal October 28, 2025 12:25 — with GitHub Actions Inactive

Address feedback https://github.com/astronomer/astronomer-cosmos/pull…

2e652aa

…/2046/files\#r2455041189

tatiana temporarily deployed to internal October 28, 2025 12:28 — with GitHub Actions Inactive

Address feedback https://github.com/astronomer/astronomer-cosmos/pull…

29f6f3d

…/2046/files\#r2455045660

tatiana temporarily deployed to internal October 28, 2025 12:31 — with GitHub Actions Inactive

Address Code Review feedback https://github.com/astronomer/astronomer…

dae36fc

…-cosmos/pull/2046/files\#r2466262778

tatiana temporarily deployed to internal October 28, 2025 13:41 — with GitHub Actions Inactive

Apply suggestion from @pankajkoti

1834271

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

tatiana temporarily deployed to internal October 28, 2025 13:43 — with GitHub Actions Inactive

tatiana requested a review from yetudada October 28, 2025 13:46

tatiana marked this pull request as ready for review October 28, 2025 13:48

Copilot AI review requested due to automatic review settings October 28, 2025 13:48

Copilot AI reviewed Oct 28, 2025

View reviewed changes

tatiana and others added 2 commits October 28, 2025 13:50

Update docs/getting_started/watcher-execution-mode.rst

fab9a32

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/getting_started/watcher-execution-mode.rst

966809b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

tatiana temporarily deployed to internal October 28, 2025 13:50 — with GitHub Actions Inactive

Update docs/getting_started/watcher-execution-mode.rst

9a5982f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

tatiana temporarily deployed to internal October 28, 2025 13:51 — with GitHub Actions Inactive

Address code review https://github.com/astronomer/astronomer-cosmos/p…

fa3fce2

…ull/2046\#discussion_r2469655920

tatiana temporarily deployed to internal October 28, 2025 13:53 — with GitHub Actions Inactive

yetudada approved these changes Oct 28, 2025

View reviewed changes

tatiana added this to the Cosmos 1.11.0 milestone Oct 28, 2025

tatiana merged commit 9953f96 into main Oct 28, 2025
81 checks passed

tatiana deleted the docs-watcher branch October 28, 2025 15:16

tatiana mentioned this pull request Oct 29, 2025

Release 1.11.0 #2020

Merged

Conversation

tatiana commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yetudada left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajastro left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yetudada left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tatiana commented Oct 21, 2025 •

edited

Loading

codecov Bot commented Oct 21, 2025 •

edited

Loading