Add docs for deferrable ``DbtConsumerWatcherSensor`` by pankajastro · Pull Request #2115 · astronomer/astronomer-cosmos

pankajastro · 2025-11-12T08:49:47Z

closes: #2106

This PR

Document the deferrable support for DbtConsumerWatcherSensor
Add an example DAG to run it in synchronous mode and enable it in CI

netlify · 2025-11-12T08:49:54Z

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name	Link
🔨 Latest commit	`f943fd9`
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/6916119d480af0000757a7b1
😎 Deploy Preview	https://deploy-preview-2115--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

pankajkoti

I know it's in draft but can we have a more specific/descriptive PR title :) ?

codecov · 2025-11-12T12:11:17Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.81%. Comparing base (fa645d8) to head (76543a3).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2115   +/-   ##
=======================================
  Coverage   97.81%   97.81%           
=======================================
  Files          92       92           
  Lines        5961     5961           
=======================================
  Hits         5831     5831           
  Misses        130      130

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull Request Overview

This PR adds documentation for the deferrable (asynchronous) execution mode support in DbtConsumerWatcherSensor, which was introduced in Cosmos 1.12.0. The key changes include:

Documentation explaining the new deferrable execution mode and its limitations
Example DAG demonstrating how to disable deferrable mode (run in synchronous mode)

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
docs/getting_started/watcher-execution-mode.rst	Adds documentation section for asynchronous sensor execution with deferrable mode and its limitations
dev/dags/example_watcher.py	Adds example DAG showing how to configure synchronous mode by setting `deferrable: False`

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tatiana

It looks great, @pankajastro , thank you!

@pankajastro

Breaking changes * Introduced in the PR #2080. The following functions are expected to be used internally only to Cosmos, so we hope these won't impact end-users, but we are documenting the changes just in case: - ``generate_task_or_group`` receives ``render_config`` instead of its individual configurations, such as ``test_behavior``, ``source_rendering_behavior`` and ``enable_owner_inheritance`` - ``create_task_metadata`` receives ``render_config`` instead of its individual configurations, such as ``test_behavior``, ``source_rendering_behavior`` and ``enable_owner_inheritance`` - ``create_task_metadata`` now expects the ``node_converters`` argument * Drop Python 3.9 support by @pankajastro in #2118 * Drop Airflow 2.4 support by @pankajastro in #2161 * Drop Airflow 2.5 support by @pankajastro in #2165 Features * Support applying ``node_converter`` at a task level instead of task group level by @anyapriya in #1759 * Allow overriding ``DbtProducerWatcherOperator`` parameters via ``ExecutionConfig.setup_operator_args`` by @pankajastro in #2133 * Use deferrable sensors by default in ``ExecutionMode.WATCHER`` by @pankajastro in #2084 * Support real-time consumer updates when using ``ExecutionMode.WATCHER`` and ``InvocationMode.SUBPROCESS`` by @pankajastro in #2152 * Update telemetry to v3 format with query parameters by @pankajkoti in #2192 * Add initial set of telemetry task listener metrics for Cosmos operators by @pankajkoti in #2195 Enhancements * Unify Airflow version handling into ``constants.py`` by @tatiana in #2089 * Refactor ``airflow/graph.py`` to simplify the code base by @tatiana in #2080 * Force watcher producer retries to zero by @pankajkoti in #2114 * Fail ``ExecutionMode.WATCHER`` consumer sensors immediately when the producer fails using Airflow context by @pankajkoti in #2126 * ``ExecutonMode.WATCHER``: fetch producer status asynchronously from the Airflow runtime so deferrable sensors fail immediately when the producer task fails by @pankajkoti in #2144 * Refactor ``ExecutionMode.WATCHER`` ``InvocationMode.SUBPROCESS`` log parser by @tatiana in #2183 * Replace map_index with is_mapped_task boolean in task telemetry metrics by @pankajkoti in #2210 * Collect cosmos profile metrics in task telemetry metrics by @pankajastro in #2198 * Remove unnecessary information from telemetry by @tatiana in #2211 Bug fixes * Clarify ``ExecutionMode.WATCHER`` deferrable failure messaging by @pankajkoti in #2124 * Remove empty test tasks when all tests are detached by @anyapriya in #2010 * Fix forwarding ``DbtProducerWatcherOperator`` ``dbt build`` flags by @michal-mrazek in #2127 * Add databricks oauth mock profile by @fjmacagno in #2164 * Register listeners in Airflow 3 plugin implementation by @pankajastro in #2187 * Fix resolution of ``packages-install-path`` when it uses ``env_var`` by @tatiana in #2194 * Fix ``template_fields`` in ``DbtConsumerWatcherSensor`` to include ``DbtRunLocalOperator`` template_fields`` by @tiovader and @emanuel-luis in #2209 * Emit asset events in ExecutionMode.AIRFLOW_ASYNC mode by @pankajastro in #2184 * Remove dag_run_id from telemetry tests by @tatiana in #2213 Docs * Document dataset-event limitation when using ``ExecutionMode.AIRFLOW_ASYNC`` by @varaprasadregani in #2143 * Expand ``ExecutionMode.KUBERNETES`` guidance by @tatiana in #2139 * Add docs for deferrable ``DbtConsumerWatcherSensor`` by @pankajastro in #2115 * Fix reStructuredText formatting by @dnskr in #2132 * Add docs for ``setup_operator_args`` param by @pankajastro in #2136 * Remove experimental flag for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajastro in #2153 * Clarify ``ExecutionMode.AIRFLOW_ASYNC`` dataset limits by @pankajkoti in #2167 * Update PRIVACY_NOTICE.rst by @tatiana in #2212 Others * Drop Python 3.9 support by @pankajastro in #2118 * Drop Airflow 2.4 support by @pankajastro in #2161 * Drop Airflow 2.5 support by @pankajastro in #2165 * Improve example DAG ``jaffle_shop_kubernetes.py`` by @tatiana in #2140 * Enable tests for Python 3.13 by @pankajastro in #2154 * Add Python 3.12 to CI integration tests matrix by @pankajastro in #2168 * Retry flaky Telemetry success test to stabilise CI by @pankajkoti in #2138 * Drop unused producer state xcom handling in ``ExecutionMode.WATCHER`` by @pankajkoti in #2145 * Remove unused Python3.9 uses from Github action CI by @pankajastro in #2117 * Run pre-commit on ``ExecutionMode.WATCHER`` modules by @pankajkoti in #2150 * Refactor: Use shared airflow version constant by @pankajkoti in #2157 * Pin ``pydantic<2.0`` for Airflow 2.6 compatibility by @pankajastro in #2172 * Remove duplicate ``dbt-duckdb`` dependency by @pankajastro in #2170 * Add targeted ``type: ignore`` for untyped decorators to fix ``mypy`` errors by @pankajastro in #2174 * Replace Legacy typing Aliases with Built-in Types for Python 3.10+ by @pankajastro in #2175 * Refactor to reuse ``load_method_from_module`` from ``_utils/importer.py`` by @pankajastro in #2176 * Remove try except block for cache import and unused python_version variable by @pankajastro in #2186 * Unpin Airflow to satisfy GitHub Security tab requirements by @pankajastro in #2171 * Update Python version for ``pyupgrade`` in ``pre-commit`` config by @pankajastro in #2190 * Add cooldown config in ``dependabot`` config by @pankajastro in #2189 * Adjust pre-commit so Python 3.10 or higher can be used by @tatiana in #2196 * Remove empty variables emission from telemetry metrics by @pankajkoti in #2197 * Reformat documented comments for historical URL formats by @pankajkoti in #2199 * Bump ``actions/checkout`` from ``5.0.0`` to ``5.0.1`` by @dependabot in #2135 * Bump ``actions/checkout`` to ``6.0.0`` in GitHub workflows by @dependabot in #2147 * Bump ``zizmorcore/zizmor-action`` from ``0.2.0`` to ``0.3.0`` by @dependabot in #2156 * Bump ``actions/checkout`` from ``5.0.1`` to ``6.0.0`` by @dependabot in #2155 * Bump ``actions/checkout`` from ``6.0.0`` to ``6.0.1`` by @dependabot in #2178 * Bump ``codecov/codecov-action`` from ``5.5.1`` to ``5.5.2`` by @dependabot in #2208 * pre-commit autoupdate by @pre-commit-ci[bot] in #2134, #2162, #2173, #2191, #2202 closes: astronomer/oss-integrations-private#275

pankajastro temporarily deployed to internal November 12, 2025 08:49 — with GitHub Actions Inactive

pankajastro force-pushed the fix_2106 branch from 1c59624 to 60fc237 Compare November 12, 2025 08:51

pankajastro temporarily deployed to internal November 12, 2025 08:51 — with GitHub Actions Inactive

pre-commit-ci Bot temporarily deployed to internal November 12, 2025 08:52 Inactive

pankajastro temporarily deployed to internal November 12, 2025 09:02 — with GitHub Actions Inactive

tatiana reviewed Nov 12, 2025

View reviewed changes

Comment thread dev/dags/example_watcher_deferrable.py Outdated

pankajkoti reviewed Nov 12, 2025

View reviewed changes

pankajastro temporarily deployed to internal November 12, 2025 11:42 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal November 12, 2025 11:44 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal November 12, 2025 12:14 — with GitHub Actions Inactive

pankajastro changed the title ~~Document Async ExecutionMode.WATCHER~~ Add docs for deferrable DbtConsumerWatcherSensor Nov 12, 2025

pankajastro temporarily deployed to internal November 12, 2025 12:22 — with GitHub Actions Inactive

pankajastro marked this pull request as ready for review November 12, 2025 12:32

Copilot AI review requested due to automatic review settings November 12, 2025 12:32

Copilot started reviewing on behalf of pankajastro November 12, 2025 12:32 View session

Copilot finished reviewing on behalf of pankajastro November 12, 2025 12:34

Copilot AI reviewed Nov 12, 2025

View reviewed changes

Comment thread docs/getting_started/watcher-execution-mode.rst Outdated

Comment thread dev/dags/example_watcher.py

pankajastro temporarily deployed to internal November 12, 2025 12:51 — with GitHub Actions Inactive

pankajastro force-pushed the fix_2106 branch from 9cfea9d to 95a7b0a Compare November 12, 2025 12:52

pankajastro temporarily deployed to internal November 12, 2025 12:52 — with GitHub Actions Inactive

tatiana reviewed Nov 12, 2025

View reviewed changes

Comment thread docs/getting_started/watcher-execution-mode.rst Outdated

pankajastro temporarily deployed to internal November 12, 2025 15:01 — with GitHub Actions Inactive

pankajastro force-pushed the fix_2106 branch from 04381a1 to 76543a3 Compare November 12, 2025 15:07

pankajastro temporarily deployed to internal November 12, 2025 15:07 — with GitHub Actions Inactive

tatiana approved these changes Nov 13, 2025

View reviewed changes

pankajkoti reviewed Nov 13, 2025

View reviewed changes

Comment thread docs/getting_started/watcher-execution-mode.rst Outdated

pankajastro requested a review from pankajkoti November 13, 2025 15:10

pankajastro temporarily deployed to internal November 13, 2025 17:12 — with GitHub Actions Inactive

pankajastro and others added 13 commits November 13, 2025 22:42

Document Async ExecutionMode.WATCHER

1e6c8e7

Document Async ExecutionMode.WATCHER

f02f885

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

8918fb8

cleanup

406794a

Keep sync and async dag in same file

c181f80

Use default invocation mode

b788546

Fix some airflow param

c9b2042

Fix some airflow param

ab30e0c

Add docs for async execution

d30cd09

Fix rendering

03c668a

Fix try_no

9466054

Address feedback

03eb0a4

Address feedback

f943fd9

pankajastro force-pushed the fix_2106 branch from 760e082 to f943fd9 Compare November 13, 2025 17:12

pankajastro temporarily deployed to internal November 13, 2025 17:13 — with GitHub Actions Inactive

pankajastro merged commit 0304a81 into main Nov 13, 2025
82 checks passed

pankajastro deleted the fix_2106 branch November 13, 2025 17:34

pankajkoti mentioned this pull request Nov 21, 2025

Release 1.12.0 #2149

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add docs for deferrable `DbtConsumerWatcherSensor`#2115

Add docs for deferrable `DbtConsumerWatcherSensor`#2115
pankajastro merged 13 commits into
mainfrom
fix_2106

pankajastro commented Nov 12, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Nov 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

pankajkoti left a comment

Uh oh!

codecov Bot commented Nov 12, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pankajastro commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify Bot commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Uh oh!

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pankajastro commented Nov 12, 2025 •

edited

Loading

netlify Bot commented Nov 12, 2025 •

edited

Loading

codecov Bot commented Nov 12, 2025 •

edited

Loading