Store compiled SQL as template field for dbt callback events in `ExecutionMode.WATCHER` by pankajkoti · Pull Request #2068 · astronomer/astronomer-cosmos

pankajkoti · 2025-10-28T12:25:03Z

Compiled SQL rendered in UI with the change in the PR:

related: #1968
related: https://github.com/astronomer/oss-integrations-private/issues/243

codecov · 2025-10-28T15:19:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.77%. Comparing base (9953f96) to head (d4a98f8).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2068   +/-   ##
=======================================
  Coverage   97.77%   97.77%           
=======================================
  Files          91       91           
  Lines        5830     5845   +15     
=======================================
+ Hits         5700     5715   +15     
  Misses        130      130

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull Request Overview

This PR enhances the dbt watcher execution mode by storing compiled SQL as a template field in callback events. When a dbt model finishes execution, the producer operator now extracts the compiled SQL from the dbt target directory and includes it in the XCom payload. The consumer sensor retrieves this compiled SQL and exposes it as a templated field for downstream use.

Key Changes:

Added logic to extract and store compiled SQL for model nodes during event handling
Modified the consumer sensor to retrieve compiled SQL from events and expose it as a template field
Added test coverage for compiled SQL injection and absence scenarios

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
cosmos/operators/watcher.py	Added `_extract_compiled_sql_for_node_event` method to extract compiled SQL, updated `_handle_node_finished` to include compiled SQL in event payload, modified `_get_status_from_events` to extract and store compiled SQL, and added `compiled_sql` to template fields
tests/operators/test_watcher.py	Extended `_fake_event` helper to support resource_type and node_path parameters, added tests for compiled SQL injection and absence scenarios, updated existing tests to pass context parameter

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tatiana

Thanks for finding a solution to this problem so quickly, @pankajkoti !
Minor feedback inline, happy for this feature to be merged once the checks pass.

…en using `run_results.json` (#2070) The PR #2068 exposes dbt SQL files as templated fields when end-users have installed dbt and Cosmos/Airflow in the same Python virtual environment. The current PR implements the same feature, but for the case when dbt and Cosmos/Airflow are not being installed in the same Python virtual environment, and Cosmos uses the `run_results.json` to update the status of downstream tasks. closes: #1968 closes: astronomer/oss-integrations-private#243 <img width="1683" height="986" alt="Screenshot 2025-10-29 at 3 21 53 PM" src="https://github.com/user-attachments/assets/cb3b8710-49e5-4170-9e93-91dc0d008ff7" /> <img width="1688" height="993" alt="Screenshot 2025-10-29 at 3 22 36 PM" src="https://github.com/user-attachments/assets/ca5e0f47-6c87-4ae0-befd-7f76eaa63fea" /> <img width="1695" height="993" alt="Screenshot 2025-10-29 at 3 23 02 PM" src="https://github.com/user-attachments/assets/27c8aec3-ded9-471f-9113-1a22af23ad1c" /> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

@tatiana

**Features** * Introduce ``ExecutionMode.WATCHER`` to reduce DAG run time by 1/5 in several PRs. Learn more about it [here](https://astronomer.github.io/astronomer-cosmos/getting_started/watcher-execution-mode.html#watcher-execution-mode). This feature was implemented via multiple PRs, including: * Expose new execution mode by @tatiana @pankajastro @pankajkoti in #1999 * Add ``DbtProducerWatcherOperator`` for the proposed ``ExecutionMode.WATCHER`` by @pankajkoti in #1982 * Add ``DbtConsumerWatcherSensor`` for the proposed ``ExecutionMode.WATCHER`` by @pankajastro in #1998 * Push producer's task completion status to XCOM by @pankajkoti in #2000 * Add default priority_weight for ``DbtProducerWatcherOperator`` by @pankajkoti in #1995 * Add sample dbt events for the dbt watcher execution mode by @pankajkoti in #1952 * Add ``compiled_sql`` as a template fields on ```ExecutionMode.WATCHER``` when using ``run_results.json`` by @pankajastro in #2070 * Set ``push_run_results_to_xcom`` kwargs correctly for invocation mode subprocess and Watcher mode by @pankajastro in #2067 * Store compiled SQL as template field for dbt callback events in ``ExecutionMode.WATCHER`` by @pankajkoti in #2068 * Add initial documentation for ``ExecutionMode.WATCHER`` by @tatiana in #2046 * Support running ``State.UPSTREAM_FAILED`` tasks when WATCHER consumer upstream tasks fail by @tatiana in #2062 * Fail sensor tasks immediately if the ``ExecutionMode.WATCHER`` producer task fails by @pankajastro in #2040 * Add ``WATCHER``` to GitHub issue template by @tatiana in #2056 * Add support for ``TestBehavior.AFTER_ALL`` with ``ExecutionMode.WATCHER`` by @pankajastro in #2049 * Add support for ``TestBehavior.NONE`` with ``ExecutionMode.WATCHER`` by @pankajastro in #2047 * Fix ``ExecutionMode.WATCHER`` behaviour with ``DbtTaskGroup`` by @tatiana in #2044 * Fix Cosmos behaviour when using watcher with ``InvocationMode.DBT_RUNNER`` by @tatiana in #2048 * Add Airflow 3 plugin for dbt docs with multiple dbt projects support by @pankajkoti in #2009, check the [documentation](https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html). * Initial support to ``dbt Fusion`` by @tatiana in #1803. More details [here](https://astronomer.github.io/astronomer-cosmos/configuration/dbt-fusion). * Support to prune sources without downstream references in dbt projects by @corsettigyg in #1988 * Allow to set task display name as a user-defined function by @corsettigyg in #1761 * Add dbt project's hash to dag docs to support dag versioning in Airflow 3 by @pankajkoti in #1907 * feat: Add Jinja templating support for ``dbt_cmd_flags`` by @skillicinski in #1899 * Add Scarf metric to collect the execution mode uses by @pankajastro in #1981 * Support Airflow 3.1 by @tatiana in #1980 * Add MySQL profile mapping by @Lee2532 in #1977 * Add sqlserver profile mapping by @pankajastro in #1737 **Enhancement** * Use XCom to store sql when using ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajastro in #1934 * Refactor ``AIRFLOW_ASYNC`` teardown so it doesn't install the virtualenv by @pankajastro in #1938 * Reuse the virtual env for ``AIRFLOW_ASYNC`` setup task by @pankajastro in #1939 * Improve dataset/asset experience in Cosmos by @tatiana in #2030 * Add ``downstreams`` to ``DbtNode`` by @wornjs in #2028 **Bug fixes** * Fix tags extraction by @ms32035 in #1915 * Fix task flow operator args by @anyapriya in #2024 **Documentation** * Add documentation for Airflow 3 Plugin supporting dbt docs for multiple dbt projects by @pankajkoti in #2063 * Add Cosmos Deferrable Operator Guide by @pankajastro in #1922 * Add dbt Fusion documentation by @tatiana in #1824 #1830 * Update dbt-fusion.rst to explicitly highlight it is in alpha by @tatiana in #1838 * Fix a bunch of docs build errors and warnings by @pankajkoti in #1886 * Add docs note for param virtualenv_dir for async execution mode by @pankajastro in #1969 * Use pepy.tech downloads badge in README by @pankajkoti in #1920 * Correct the default value of ``cache_dir`` by @seokyun.ha in #2027 **Others** * Promote @corsettigyg to committer by @tatiana in #1985 * Add @pankajkoti and @pankajastro to ``contributors.rst`` by @tatiana in #1983 * Update setup script for airflow3 script by @dwreeves in #2023 * Prevent pytest from trying to test classes that aren't actually tests by @anyapriya in #2032 * Fix ``dag.test()`` for Airflow 3.1+ by syncing DAG to database bby @kaxil in #2037 * Disable Scarf in CI by @pankajastro in #2016 * Fix failing dbt Fusion tests when run in parallel in CI by @pankajkoti in #1896 * Fix MyPy issues related to ``ObjectStoragePath`` in main branch by @tatiana in #2012 * Cleanup example dbt event JSON dictionaries kept for XCOM referencby @pankajkoti in #1997 * Bump min hatch version that includes fixes for click>=8.3.0 by @pankajkoti in #1996 * Use official postgres image from Docker hub for kubernetes setup by @pankajkoti in #1986 * Use click<8.3.0 for hatch as click 8.3 breaks hatch by @pankajkoti in #1987 * Pin Airflow version in type check CI job by @pankajastro in #2003 * Improve comments after feedback on #1948 by @tatiana in #1963 * Fix running tests with dbt Fusion 2.0.0 preview versions by @tatiana in #1948 * Test hardening of dbt node having tags as unset or missing by @pankajkoti in #1918 * Fix Sphinx issue in the main branch by @tatiana in #2064 * pre-commit autoupdate in #2065, #2043, #2033, #2019, #1990, #2019, #2008, #1941, #1935, #1924 * GitHub dependabot update in #2051, #2050, #2038, #2022, #1947, #1955, #1946, #1944, #1945, #1928, #1921, #1917 Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io> Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Store compiled SQL as template fielf for Watcher mode events

3be6bbc

pankajkoti temporarily deployed to internal October 28, 2025 12:25 — with GitHub Actions Inactive

Merge branch 'main' into watcher-compiled-sql

8c56189

pankajkoti changed the title ~~Store compiled SQL as template fielf for Watcher mode events~~ Store compiled SQL as template field for dbt callback events in Watcher execution mode Oct 28, 2025

pankajkoti temporarily deployed to internal October 28, 2025 12:26 — with GitHub Actions Inactive

pankajkoti commented Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py Outdated

pankajkoti commented Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py Outdated

pankajkoti added 2 commits October 28, 2025 17:57

Apply suggestion from @pankajkoti

9c7cae3

Apply suggestion from @pankajkoti

5a30727

pankajkoti temporarily deployed to internal October 28, 2025 12:27 — with GitHub Actions Inactive

tatiana added this to the Cosmos 1.11.0 milestone Oct 28, 2025

pankajkoti added 2 commits October 28, 2025 20:21

Refactor

ef7790c

Add tests

0de4180

pankajkoti temporarily deployed to internal October 28, 2025 14:57 — with GitHub Actions Inactive

pankajkoti commented Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py Outdated

Apply suggestion from @pankajkoti

a10f14a

pankajkoti temporarily deployed to internal October 28, 2025 15:21 — with GitHub Actions Inactive

Add test for missing coverage

61ef046

pankajkoti temporarily deployed to internal October 28, 2025 15:27 — with GitHub Actions Inactive

Merge branch 'main' into watcher-compiled-sql

df64cb5

pankajkoti temporarily deployed to internal October 28, 2025 15:28 — with GitHub Actions Inactive

pankajkoti marked this pull request as ready for review October 28, 2025 16:08

Copilot AI review requested due to automatic review settings October 28, 2025 16:08

Copilot AI reviewed Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py Outdated

Comment thread cosmos/operators/watcher.py Outdated

Comment thread cosmos/operators/watcher.py Outdated

Comment thread cosmos/operators/watcher.py

pankajkoti requested review from pankajastro and tatiana October 28, 2025 16:10

tatiana reviewed Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py Outdated

tatiana reviewed Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py Outdated

tatiana reviewed Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py

tatiana reviewed Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py Outdated

tatiana reviewed Oct 28, 2025

View reviewed changes

Comment thread cosmos/operators/watcher.py Outdated

tatiana approved these changes Oct 28, 2025

View reviewed changes

tatiana changed the title ~~Store compiled SQL as template field for dbt callback events in Watcher execution mode~~ Store compiled SQL as template field for dbt callback events in ExecutionMode.WATCHER Oct 28, 2025

pankajkoti temporarily deployed to internal October 29, 2025 08:52 — with GitHub Actions Inactive

Address review comments

d4a98f8

pankajkoti force-pushed the watcher-compiled-sql branch from f0be3b7 to d4a98f8 Compare October 29, 2025 08:56

pankajkoti temporarily deployed to internal October 29, 2025 08:56 — with GitHub Actions Inactive

pankajkoti merged commit f1d4aa7 into main Oct 29, 2025
81 checks passed

pankajkoti deleted the watcher-compiled-sql branch October 29, 2025 09:21

tatiana mentioned this pull request Oct 29, 2025

Add compiled_sql as a template fields on ExecutionMode.WATCHER when using run_results.json #2070

Merged

tatiana mentioned this pull request Oct 29, 2025

Release 1.11.0 #2020

Merged

tatiana added the roadmap:P1 BOSS roadmap-committed work (priority P1) label Jun 6, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store compiled SQL as template field for dbt callback events in `ExecutionMode.WATCHER`#2068

Store compiled SQL as template field for dbt callback events in `ExecutionMode.WATCHER`#2068
pankajkoti merged 10 commits into
mainfrom
watcher-compiled-sql

pankajkoti commented Oct 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pankajkoti commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pankajkoti commented Oct 28, 2025 •

edited

Loading

codecov Bot commented Oct 28, 2025 •

edited

Loading