Document the cluster policy config for ExecutionMode.WATCHER sensor task by pankajastro · Pull Request #2315 · astronomer/astronomer-cosmos

pankajastro · 2026-02-02T16:38:00Z

Adds documentation for the watcher_retry_queue Airflow config to route WATCHER consumer sensor retries onto a different worker queue with more resources.

Changes:

Document watcher_retry_queue behavior and configuration in the WATCHER execution mode guide.
Add watcher_retry_queue to the Cosmos configuration reference, including env var and default value.

Related PR: #2293

netlify · 2026-02-02T16:38:07Z

✅ Deploy Preview for astronomer-cosmos ready!

Name	Link
🔨 Latest commit	`85bba6f`
🔍 Latest deploy log	https://app.netlify.com/projects/astronomer-cosmos/deploys/698daa8e85cf8900086edd53
😎 Deploy Preview	https://deploy-preview-2315--astronomer-cosmos.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Copilot

Pull request overview

Adds documentation for the watcher_retry_queue Airflow config to route WATCHER consumer sensor retries onto a different worker queue with more resources.

Changes:

Document watcher_retry_queue behavior and configuration in the WATCHER execution mode guide.
Add watcher_retry_queue to the Cosmos configuration reference, including env var and default value.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
docs/getting_started/watcher-execution-mode.rst	Adds a new “Watcher retry queue” section describing why/when to use `watcher_retry_queue` and how to set it.
docs/configuration/cosmos-conf.rst	Adds `watcher_retry_queue` to the Cosmos config reference list with behavior details, default, and env var.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

codecov · 2026-02-03T08:01:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.90%. Comparing base (98dc74b) to head (87e228e).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2315   +/-   ##
=======================================
  Coverage   97.90%   97.90%           
=======================================
  Files         102      102           
  Lines        6818     6818           
=======================================
  Hits         6675     6675           
  Misses        143      143

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tatiana

@pankajastro, while reviewing the docs, I noticed we can significantly improve the feature. I suggest we improve the feature before publishing any documentation, and only add the documentation after we review the feature's behaviour.

The main feature changes I'm thinking about are:

Rename watcher_retry_queue to watcher_dbt_execution_queue
Use this setting not only for the consumer, during retries, but also to set the default node pool for the producer, if defined

This woudl alllow users to be able to use their "default" queue for running watcher and only about setting the "dbt execution" node queue in a single place - and Cosmos would handle which tasks need to use it, regardless if they are consumer or producer tasks.

pankajastro · 2026-02-05T18:02:03Z

@pankajastro, while reviewing the docs, I noticed we can significantly improve the feature. I suggest we improve the feature before publishing any documentation, and only add the documentation after we review the feature's behaviour.

The main feature changes I'm thinking about are:

Rename watcher_retry_queue to watcher_dbt_execution_queue

Use this setting not only for the consumer, during retries, but also to set the default node pool for the producer, if defined

This woudl alllow users to be able to use their "default" queue for running watcher and only about setting the "dbt execution" node queue in a single place - and Cosmos would handle which tasks need to use it, regardless if they are consumer or producer tasks.

Created PR: #2331 to implement the suggestions

#2331) Implement the [suggestion](#2315 (review)) from PR: #2315 This PR renames the `watcher_retry_queue` configuration setting to `watcher_dbt_execution_queue` to better reflect its expanded purpose: not only handling retries for consumer watcher sensors, but also for running the producer task itself. **Changes:** - Renamed `watcher_retry_queue` to `watcher_dbt_execution_queue` in settings and all references - Added queue assignment logic to `DbtProducerWatcherOperator` to use the configured queue - Updated all test patches and test names to reflect the new setting name ---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

tatiana

Thanks a lot for improving the behaviour and updating the documentation in this area, @pankajastro . This is a very important feature that will further support the adoption of ExecutionMode.WATCHER

PR: #2315 added docs for `watcher_dbt_execution_queue`, but the heading is not clickable. This PR make the heading clickable and improve heading

@jedcunningham

1.14.0 (2026-04-07) --------------------- Breaking Changes * Drop support for Airflow versions earlier than **2.9** by @jedcunningham in #2288 * Fix inclusion of package models and selection/exclusion behavior by @pankajkoti in #2357 * ``ExecutionMode.WATCHER``: The per-node ``*_status`` XCom value is now a dict (``{"status": "<status>", "outlet_uris": [...]}``) instead of a plain string. Any custom code that reads these internal XCom keys directly will need to be updated by @pankajkoti in #2507 Features * Add cluster policy support for ``ExecutionMode.WATCHER`` sensor retries by @astro-anand in #2293 * Add debug mode to track memory utilization by @tatiana in #2327 * Add FQN selection support for ``LoadMode.DBT_MANIFEST`` by @pankajastro in #2375 * Introduce interceptors for Cosmos tasks by @tatiana in #2419 * Add config to allow disabling dag versioning by @pankajkoti in #2470 * Implement TaskGroups by models folder by @maximilianoarcieri and @tatiana in #1566, #2469, and #2420 * feat: implement DbtTestWatcherOperator by @michal-mrazek in #2447 * Add source freshness aware execution for ``ExecutionMode.WATCHER`` by @pankajastro and @tatiana in #2467 * Note: Like ``ExecutionMode.WATCHER``, this feature is experimental and its interface and implementation can change in the future. * Add Airflow 3.2 support by @pankajastro and @pankajkoti in #2472 Enhancements * Add watcher mode support for dbt test node states by @michal-mrazek in #2318 * Rename watcher-mode sensor retry queue and reuse it for producer tasks by @pankajastro in #2331 * Fix leaked semaphore warnings in Airflow 3 by resetting dbt adapters by @pankajkoti in #2335 * Improve dbt Fusion support and related tests by @tatiana in #2356 * Default Snowflake profile mappings to four threads by @tatiana in #2374 * Attempt to remove Pydantic as a dependency by @tatiana in #2377 * Log dbt-core and adapter versions in watcher consumer tasks by @pankajastro in #2412 * Log model errors in watcher consumer on dbt node failure by @pankajastro in #2431 * Reduce XCom read/write for tracking node state and errors in ConsumerWatcher task by @pankajastro in #2471 * Remove duplicate debug log in watcher subprocess path by @tatiana in #2494 * Simplify and unify WATCHER implementation regardless of InvocationMode by @tatiana in #2498 * Switch to lazy imports in cosmos/__init__.py by @pankajkoti in #2531 Bug Fixes * Handle invalid YAML errors with ``LoadMode.DBT_MANIFEST`` and ``RenderConfig.selector`` by @YourRoyalLinus in #2316 * Populate ``compiled_sql`` for ``InvocationMode.SUBPROCESS`` in ``ExecutionMode.WATCHER`` by @pankajkoti in #2319 * Fix select/exclude type mismatch by @tatiana in #2364 * Set ``emit_datasets=False`` for ``DbtTest*`` operators by @pankajastro in #2365 * Set correct queue priority for watcher producer tasks by @pankajastro in #2372 * Preserve ``extra_context`` for watcher consumer task instances by @pankajkoti in #2381 * Respect ``deferrable=False`` from ``operator_args`` on watcher consumer sensors by @pankajkoti in #2384 * Fix watcher queue precedence and add documentation by @pankajastro in #2391 * Do not set ``compiled_sql`` on ``ExecutionMode.WATCHER`` producers by @pankajkoti in #2440 * Remove const attribute for ``__cosmos_telemetry_metadata__`` dag param by @pankajkoti in #2466 * Remove timeout override from Cosmos watcher sensors by @tatiana and @claude in #2478 * Remove forced ``retries=0`` from watcher producer operators by @tatiana in #2479 * RFC: Add patch for newer versions of amazon provider when running dbt on EKS by @aoelvp94 in #2481 * Fix ``cosmos_debug_max_memory_mb`` XCom not pushed in Watcher sensor tasks by @tatiana in #2503 * Fix ``TestBehavior.NONE`` and ``TestBehavior.AFTER_ALL`` exclude ignored with selectors in ``ExecutionMode.WATCHER`` by @pankajkoti in #2511 * Move dataset emission for ``ExecutionMode.WATCHER`` from producer to consumer sensors by @pankajkoti in #2507 Docs * Document cluster policy configuration for ``ExecutionMode.WATCHER`` sensor tasks by @pankajastro in #2315 * Remove outdated docs for the dbt docs plugin with Airflow 3 by @pankajastro in #2353 * Make Watcher DBT Execution Queue heading clickable by @pankajastro in #2354 * Update ``ExecutionMode.WATCHER`` documentation regarding test node implementation by @jroachgolf84 in #2355 * Fix ``pre_dbt_fusion`` configuration rendering by @pankajastro in #2369 * Add documentation for including/excluding nodes based on FQN by @pankajastro in #2371 * Update watcher execution mode documentation by @tatiana in #2380 * Add documentation for ``DbtSeedLocalOperator`` by @jroachgolf84 in #2383 * Fix miscellaneous Sphinx warnings by @pankajastro in #2395 * Improve contributing documentation by @lzdanski in #2397 * Add **Get Started in 5 Minutes** guide by @lzdanski in #2398 * Add Sphinx redirects package for documentation redirects by @lzdanski in #2407 * Restructure **Getting Started** and **Guides** sections by @lzdanski in #2418 * Add open-source quickstart by @lzdanski in #2439 * Fix documentation redirects by @lzdanski in #2442 * Restructure and refactor reference documentation by @lzdanski in #2443 * Add execution modes decision documentation by @lzdanski in #2444 * Add **Core Concepts** page to Getting Started by @lzdanski in #2448 * Add guide: *How Cosmos Works* by @lzdanski in #2449 * Update **Getting Started** overview and index pages by @lzdanski in #2452 * Add guide: *How Cosmos Runs dbt* by @lzdanski in #2453 * Fix miscellaneous documentation links by @lzdanski in #2454 * Add Mermaid diagrams and execution mode diagrams by @lzdanski and @tatiana in #2459 * Add documentation for memory optimization options by @pankajastro in #2340 * Fix typo in watcher execution mode docs by @evanvolgas in #2485 * Fix minor documentation issues by @evanvolgas in #2489 * Add troubleshooting note for dbt debug logs in ExecutionMode.WATCHER by @tatiana in #2491 * docs: unify RST header styles across documentation by @jigangz in #2473 * docs: fix env var for rich logging by @vricciardulli in #2514 * docs: update dbt project path example for Airflow 3 Astro compatibility by @yeoreums in #2512 * Document missing Cosmos Airflow config settings in cosmos-conf.rst by @tatiana in #2515 * Split security-privacy policy doc and add dependency cooldown by @pankajkoti in #2519 * Add performance optimization and troubleshooting docs by @pankajkoti in #2521 * Update copyright year to 2026 by @tayloramurphy in #2527 * docs: Updating "Project Policies" to "Policies" in menu bar by @jroachgolf84 in #2526 Others * Fix tests after removing support for Airflow versions earlier than 2.9 by @tatiana in #2321 * Enable listener tests for Airflow 3.1 by @pankajastro in #2348 * Accept ``int`` or ``float`` for ``cosmos_debug_max_memory_mb`` in integration tests by @pankajkoti in #2352 * Update ``CODEOWNERS`` to prioritize ``oss-integrations`` by @tatiana in #2359 * Fix automatic reviewer assignment in GitHub by @tatiana and @phanikumv in #2360 * Improve PyPI tagging by @tatiana in #2363 * Add integration tests for dbt Fusion and ``ExecutionMode.WATCHER`` by @tatiana in #2373 * Fix Zizmor check by @tatiana in #2376 * Remove ``methodtools`` dependency by @tatiana in #2378 * Improve comments on #2389 by @evanvolgas in #2394 * Refactor ``load_from_dbt_manifest`` to reduce code complexity by @pankajkoti in #2399 * Refactor ``_handle_no_precursors_or_descendants`` to reduce complexity by @pankajkoti in #2400 * Improve issue templates by @tatiana in #2401 * Avoid running tests when only docs change by @tatiana in #2402 * Add ``no-reload`` target for serving docs locally by @pankajkoti in #2405 * Fix test hash checks on macOS by @tatiana in #2406 * Attempt deterministic dbt project copy in test fixtures by @pankajkoti in #2409 * Pin ``virtualenv <21`` due to hatch incompatibility in CI by @pankajkoti in #2410 * Revert virtualenv pin for hatch installation in CI by @pankajkoti in #2426 * Add version comments for commit SHA pinned GitHub Actions by @pankajkoti in #2436 * Fix ``hatch run docs:build`` issues by @tatiana in #2437 * Minor code improvements by @dnskr in #2446 * Pre-commit autoupdate by @pre-commit-ci in #2367, #2396, #2422, #2451, #2468, #2495, and #2516 * Add file to support Claude understanding the Cosmos repository by @tatiana in #2458 * Dependency updates by @dependabot in #2368, #2425, #2435, #2465, #2475, #2504, #2518, and #2528 * Isolate Scarf telemetry integration test into its own CI job by @pankajkoti and @claude in #2477 * ci: upgrade Airflow version to 3.1 in MyPy type-check job by @yeoreums in #2506 * Add commit message guidelines to CLAUDE.md by @pankajkoti in #2509 * Extend skipping tests in CI for more non-code file changes by @pankajkoti in #2510 * Add Dependabot pre-commit support with 7-day cooldown by @pankajkoti in #2517 * Enforce zero warnings policy for documentation by @dnskr in #2513 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

pankajastro temporarily deployed to internal February 2, 2026 16:38 — with GitHub Actions Inactive

pankajastro added this to the Cosmos 1.14.0 milestone Feb 2, 2026

pankajastro marked this pull request as ready for review February 2, 2026 16:38

Copilot AI review requested due to automatic review settings February 2, 2026 16:38

pankajastro marked this pull request as draft February 2, 2026 16:38

Copilot started reviewing on behalf of pankajastro February 2, 2026 16:38 View session

Copilot AI reviewed Feb 2, 2026

View reviewed changes

pankajastro temporarily deployed to internal February 2, 2026 16:49 — with GitHub Actions Inactive

pankajastro requested a review from Copilot February 2, 2026 16:49

Copilot started reviewing on behalf of pankajastro February 2, 2026 16:49 View session

pankajastro temporarily deployed to internal February 2, 2026 16:54 — with GitHub Actions Inactive

Copilot AI reviewed Feb 2, 2026

View reviewed changes

Comment thread docs/getting_started/watcher-execution-mode.rst Outdated

pankajastro temporarily deployed to internal February 2, 2026 16:56 — with GitHub Actions Inactive

pankajastro marked this pull request as ready for review February 2, 2026 16:59

pankajastro temporarily deployed to internal February 2, 2026 16:59 — with GitHub Actions Inactive