Fix the watcher queue precedence and add doc by pankajastro · Pull Request #2391 · astronomer/astronomer-cosmos

pankajastro · 2026-02-21T18:58:22Z

We had a few outdated and redundant pieces of information. This PR aims to clean up the watcher documentation so it is accurate.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

codecov · 2026-02-23T15:33:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.96%. Comparing base (6162a0b) to head (f97813b).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2391   +/-   ##
=======================================
  Coverage   97.96%   97.96%           
=======================================
  Files         102      102           
  Lines        7022     7022           
=======================================
  Hits         6879     6879           
  Misses        143      143

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tatiana

Looks great, thank you very much, @pankajastro !

pankajkoti

LGTM

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-25T08:58:30Z

@@ -133,7 +133,7 @@ def test_dbt_producer_watcher_operator_queue(queue_override, expected_queue):
 def test_producer_queue_from_setup_operator_args_when_both_set():


The test name test_producer_queue_from_setup_operator_args_when_both_set no longer matches its behavior/assertion: it now verifies that watcher_dbt_execution_queue takes precedence when both are set. Renaming the test (and optionally aligning the docstring’s first line) would prevent confusion and make failures easier to interpret.

Suggested change

def test_producer_queue_from_setup_operator_args_when_both_set():

def test_producer_queue_prefers_watcher_queue_when_both_set():

@jedcunningham

1.14.0 (2026-04-07) --------------------- Breaking Changes * Drop support for Airflow versions earlier than **2.9** by @jedcunningham in #2288 * Fix inclusion of package models and selection/exclusion behavior by @pankajkoti in #2357 * ``ExecutionMode.WATCHER``: The per-node ``*_status`` XCom value is now a dict (``{"status": "<status>", "outlet_uris": [...]}``) instead of a plain string. Any custom code that reads these internal XCom keys directly will need to be updated by @pankajkoti in #2507 Features * Add cluster policy support for ``ExecutionMode.WATCHER`` sensor retries by @astro-anand in #2293 * Add debug mode to track memory utilization by @tatiana in #2327 * Add FQN selection support for ``LoadMode.DBT_MANIFEST`` by @pankajastro in #2375 * Introduce interceptors for Cosmos tasks by @tatiana in #2419 * Add config to allow disabling dag versioning by @pankajkoti in #2470 * Implement TaskGroups by models folder by @maximilianoarcieri and @tatiana in #1566, #2469, and #2420 * feat: implement DbtTestWatcherOperator by @michal-mrazek in #2447 * Add source freshness aware execution for ``ExecutionMode.WATCHER`` by @pankajastro and @tatiana in #2467 * Note: Like ``ExecutionMode.WATCHER``, this feature is experimental and its interface and implementation can change in the future. * Add Airflow 3.2 support by @pankajastro and @pankajkoti in #2472 Enhancements * Add watcher mode support for dbt test node states by @michal-mrazek in #2318 * Rename watcher-mode sensor retry queue and reuse it for producer tasks by @pankajastro in #2331 * Fix leaked semaphore warnings in Airflow 3 by resetting dbt adapters by @pankajkoti in #2335 * Improve dbt Fusion support and related tests by @tatiana in #2356 * Default Snowflake profile mappings to four threads by @tatiana in #2374 * Attempt to remove Pydantic as a dependency by @tatiana in #2377 * Log dbt-core and adapter versions in watcher consumer tasks by @pankajastro in #2412 * Log model errors in watcher consumer on dbt node failure by @pankajastro in #2431 * Reduce XCom read/write for tracking node state and errors in ConsumerWatcher task by @pankajastro in #2471 * Remove duplicate debug log in watcher subprocess path by @tatiana in #2494 * Simplify and unify WATCHER implementation regardless of InvocationMode by @tatiana in #2498 * Switch to lazy imports in cosmos/__init__.py by @pankajkoti in #2531 Bug Fixes * Handle invalid YAML errors with ``LoadMode.DBT_MANIFEST`` and ``RenderConfig.selector`` by @YourRoyalLinus in #2316 * Populate ``compiled_sql`` for ``InvocationMode.SUBPROCESS`` in ``ExecutionMode.WATCHER`` by @pankajkoti in #2319 * Fix select/exclude type mismatch by @tatiana in #2364 * Set ``emit_datasets=False`` for ``DbtTest*`` operators by @pankajastro in #2365 * Set correct queue priority for watcher producer tasks by @pankajastro in #2372 * Preserve ``extra_context`` for watcher consumer task instances by @pankajkoti in #2381 * Respect ``deferrable=False`` from ``operator_args`` on watcher consumer sensors by @pankajkoti in #2384 * Fix watcher queue precedence and add documentation by @pankajastro in #2391 * Do not set ``compiled_sql`` on ``ExecutionMode.WATCHER`` producers by @pankajkoti in #2440 * Remove const attribute for ``__cosmos_telemetry_metadata__`` dag param by @pankajkoti in #2466 * Remove timeout override from Cosmos watcher sensors by @tatiana and @claude in #2478 * Remove forced ``retries=0`` from watcher producer operators by @tatiana in #2479 * RFC: Add patch for newer versions of amazon provider when running dbt on EKS by @aoelvp94 in #2481 * Fix ``cosmos_debug_max_memory_mb`` XCom not pushed in Watcher sensor tasks by @tatiana in #2503 * Fix ``TestBehavior.NONE`` and ``TestBehavior.AFTER_ALL`` exclude ignored with selectors in ``ExecutionMode.WATCHER`` by @pankajkoti in #2511 * Move dataset emission for ``ExecutionMode.WATCHER`` from producer to consumer sensors by @pankajkoti in #2507 Docs * Document cluster policy configuration for ``ExecutionMode.WATCHER`` sensor tasks by @pankajastro in #2315 * Remove outdated docs for the dbt docs plugin with Airflow 3 by @pankajastro in #2353 * Make Watcher DBT Execution Queue heading clickable by @pankajastro in #2354 * Update ``ExecutionMode.WATCHER`` documentation regarding test node implementation by @jroachgolf84 in #2355 * Fix ``pre_dbt_fusion`` configuration rendering by @pankajastro in #2369 * Add documentation for including/excluding nodes based on FQN by @pankajastro in #2371 * Update watcher execution mode documentation by @tatiana in #2380 * Add documentation for ``DbtSeedLocalOperator`` by @jroachgolf84 in #2383 * Fix miscellaneous Sphinx warnings by @pankajastro in #2395 * Improve contributing documentation by @lzdanski in #2397 * Add **Get Started in 5 Minutes** guide by @lzdanski in #2398 * Add Sphinx redirects package for documentation redirects by @lzdanski in #2407 * Restructure **Getting Started** and **Guides** sections by @lzdanski in #2418 * Add open-source quickstart by @lzdanski in #2439 * Fix documentation redirects by @lzdanski in #2442 * Restructure and refactor reference documentation by @lzdanski in #2443 * Add execution modes decision documentation by @lzdanski in #2444 * Add **Core Concepts** page to Getting Started by @lzdanski in #2448 * Add guide: *How Cosmos Works* by @lzdanski in #2449 * Update **Getting Started** overview and index pages by @lzdanski in #2452 * Add guide: *How Cosmos Runs dbt* by @lzdanski in #2453 * Fix miscellaneous documentation links by @lzdanski in #2454 * Add Mermaid diagrams and execution mode diagrams by @lzdanski and @tatiana in #2459 * Add documentation for memory optimization options by @pankajastro in #2340 * Fix typo in watcher execution mode docs by @evanvolgas in #2485 * Fix minor documentation issues by @evanvolgas in #2489 * Add troubleshooting note for dbt debug logs in ExecutionMode.WATCHER by @tatiana in #2491 * docs: unify RST header styles across documentation by @jigangz in #2473 * docs: fix env var for rich logging by @vricciardulli in #2514 * docs: update dbt project path example for Airflow 3 Astro compatibility by @yeoreums in #2512 * Document missing Cosmos Airflow config settings in cosmos-conf.rst by @tatiana in #2515 * Split security-privacy policy doc and add dependency cooldown by @pankajkoti in #2519 * Add performance optimization and troubleshooting docs by @pankajkoti in #2521 * Update copyright year to 2026 by @tayloramurphy in #2527 * docs: Updating "Project Policies" to "Policies" in menu bar by @jroachgolf84 in #2526 Others * Fix tests after removing support for Airflow versions earlier than 2.9 by @tatiana in #2321 * Enable listener tests for Airflow 3.1 by @pankajastro in #2348 * Accept ``int`` or ``float`` for ``cosmos_debug_max_memory_mb`` in integration tests by @pankajkoti in #2352 * Update ``CODEOWNERS`` to prioritize ``oss-integrations`` by @tatiana in #2359 * Fix automatic reviewer assignment in GitHub by @tatiana and @phanikumv in #2360 * Improve PyPI tagging by @tatiana in #2363 * Add integration tests for dbt Fusion and ``ExecutionMode.WATCHER`` by @tatiana in #2373 * Fix Zizmor check by @tatiana in #2376 * Remove ``methodtools`` dependency by @tatiana in #2378 * Improve comments on #2389 by @evanvolgas in #2394 * Refactor ``load_from_dbt_manifest`` to reduce code complexity by @pankajkoti in #2399 * Refactor ``_handle_no_precursors_or_descendants`` to reduce complexity by @pankajkoti in #2400 * Improve issue templates by @tatiana in #2401 * Avoid running tests when only docs change by @tatiana in #2402 * Add ``no-reload`` target for serving docs locally by @pankajkoti in #2405 * Fix test hash checks on macOS by @tatiana in #2406 * Attempt deterministic dbt project copy in test fixtures by @pankajkoti in #2409 * Pin ``virtualenv <21`` due to hatch incompatibility in CI by @pankajkoti in #2410 * Revert virtualenv pin for hatch installation in CI by @pankajkoti in #2426 * Add version comments for commit SHA pinned GitHub Actions by @pankajkoti in #2436 * Fix ``hatch run docs:build`` issues by @tatiana in #2437 * Minor code improvements by @dnskr in #2446 * Pre-commit autoupdate by @pre-commit-ci in #2367, #2396, #2422, #2451, #2468, #2495, and #2516 * Add file to support Claude understanding the Cosmos repository by @tatiana in #2458 * Dependency updates by @dependabot in #2368, #2425, #2435, #2465, #2475, #2504, #2518, and #2528 * Isolate Scarf telemetry integration test into its own CI job by @pankajkoti and @claude in #2477 * ci: upgrade Airflow version to 3.1 in MyPy type-check job by @yeoreums in #2506 * Add commit message guidelines to CLAUDE.md by @pankajkoti in #2509 * Extend skipping tests in CI for more non-code file changes by @pankajkoti in #2510 * Add Dependabot pre-commit support with 7-day cooldown by @pankajkoti in #2517 * Enforce zero warnings policy for documentation by @dnskr in #2513 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

tatiana and others added 18 commits February 18, 2026 17:25

Update watcher execution mode docs

4568d4e

We had a few outdated and redundant pieces of information. This PR aims to clean up the watcher documentation so it is accurate.

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

ceb596f

Apply suggestion from @tatiana

bf44803

Merge branch 'main' into tatiana-patch-1

b14b121

Update docs/getting_started/watcher-execution-mode.rst

bb6c17a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/getting_started/watcher-execution-mode.rst

d3b175b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/getting_started/watcher-execution-mode.rst

4876098

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/getting_started/watcher-execution-mode.rst

d745230

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/getting_started/watcher-execution-mode.rst

27668fc

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @tatiana

ea62472

Apply suggestion from @tatiana

d0ec73e

Apply suggestion from @Copilot

4a35ad9

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @Copilot

f95d585

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @Copilot

dceead7

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @Copilot

5e7ed83

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/getting_started/watcher-execution-mode.rst

c3afcee

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/getting_started/watcher-execution-mode.rst

1057baa

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Add docs note for watcher queue precedence

a2428c4

tatiana mentioned this pull request Feb 23, 2026

Update watcher execution mode docs #2380

Merged

Base automatically changed from tatiana-patch-1 to main February 23, 2026 14:01

Merge branch 'main' into watcher_queue_note

847efcd

pankajastro temporarily deployed to internal February 23, 2026 14:34 — with GitHub Actions Inactive

pankajastro marked this pull request as ready for review February 23, 2026 17:31

Copilot AI review requested due to automatic review settings February 23, 2026 17:31

pankajastro requested review from a team, corsettigyg, dwreeves and jbandoro as code owners February 23, 2026 17:31

pankajastro requested a review from pankajkoti February 23, 2026 17:31

Copilot started reviewing on behalf of pankajastro February 24, 2026 11:31 View session

pankajastro temporarily deployed to internal February 24, 2026 11:31 — with GitHub Actions Inactive

pankajastro changed the title ~~Add docs note for watcher queue precedence~~ Fix the watcher queue precedence and add doc Feb 24, 2026

Copilot AI reviewed Feb 24, 2026

View reviewed changes

Comment thread cosmos/operators/watcher.py

tatiana reviewed Feb 24, 2026

View reviewed changes

Comment thread docs/getting_started/watcher-execution-mode.rst Outdated

Adjust tests

c21505e

pankajastro temporarily deployed to internal February 24, 2026 13:36 — with GitHub Actions Inactive

Fix note

55f2704

Copilot AI review requested due to automatic review settings February 24, 2026 13:40

Copilot started reviewing on behalf of pankajastro February 24, 2026 13:40 View session

pankajastro temporarily deployed to internal February 24, 2026 13:40 — with GitHub Actions Inactive

Merge branch 'main' into watcher_queue_note

921d11d

pankajastro temporarily deployed to internal February 24, 2026 13:42 — with GitHub Actions Inactive

Copilot AI reviewed Feb 24, 2026

View reviewed changes

Comment thread tests/operators/test_watcher.py

Comment thread tests/operators/test_watcher.py

Comment thread docs/getting_started/watcher-execution-mode.rst Outdated

Fix tests

f97813b

pankajastro temporarily deployed to internal February 24, 2026 13:52 — with GitHub Actions Inactive

tatiana reviewed Feb 24, 2026

View reviewed changes

Comment thread docs/getting_started/watcher-execution-mode.rst Outdated

tatiana approved these changes Feb 24, 2026

View reviewed changes

pankajkoti approved these changes Feb 24, 2026

View reviewed changes

Apply suggestion from @tatiana

b9ceb0c

Copilot AI review requested due to automatic review settings February 25, 2026 08:55

tatiana temporarily deployed to internal February 25, 2026 08:55 — with GitHub Actions Inactive

Merge branch 'main' into watcher_queue_note

495fb87

Copilot started reviewing on behalf of tatiana February 25, 2026 08:56 View session

tatiana temporarily deployed to internal February 25, 2026 08:56 — with GitHub Actions Inactive

tatiana merged commit c8a7e21 into main Feb 25, 2026
5 of 6 checks passed

tatiana deleted the watcher_queue_note branch February 25, 2026 08:56

Copilot AI reviewed Feb 25, 2026

View reviewed changes

pankajastro mentioned this pull request Mar 16, 2026

Release 1.14.0 #2314

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix the watcher queue precedence and add doc#2391

Fix the watcher queue precedence and add doc#2391
tatiana merged 29 commits into
mainfrom
watcher_queue_note

pankajastro commented Feb 21, 2026

Uh oh!

codecov Bot commented Feb 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

pankajkoti left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -133,7 +133,7 @@ def test_dbt_producer_watcher_operator_queue(queue_override, expected_queue):
		def test_producer_queue_from_setup_operator_args_when_both_set():

	def test_producer_queue_from_setup_operator_args_when_both_set():
	def test_producer_queue_prefers_watcher_queue_when_both_set():

Conversation

pankajastro commented Feb 21, 2026

Uh oh!

codecov Bot commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Feb 23, 2026 •

edited

Loading