Add sample dbt events for the dbt watcher execution mode by pankajkoti · Pull Request #1952 · astronomer/astronomer-cosmos

pankajkoti · 2025-09-11T19:10:11Z

related: #1950

netlify · 2025-09-11T19:10:18Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`d511243`
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/68c8183d213f9c00090660f1

Copilot

Pull Request Overview

This PR adds sample dbt event data structures to support the dbt watcher execution mode. The changes provide example event payloads that represent different types of dbt execution events that the watcher would need to process.

Adds sample dbt event structures for NodeFinished, MainReportVersion, and AdapterRegistered events
Provides realistic event data with timing information, execution results, and metadata
Includes BigQuery-specific adapter response data for the sample events

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

codecov · 2025-09-11T19:29:14Z

Codecov Report

❌ Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.99%. Comparing base (e3c8fad) to head (d511243).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cosmos/operators/watcher.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1952      +/-   ##
==========================================
- Coverage   98.04%   97.99%   -0.06%     
==========================================
  Files          86       87       +1     
  Lines        5321     5324       +3     
==========================================
  Hits         5217     5217              
- Misses        104      107       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tatiana

Thanks for adding the messages that we'll use for the interfaces of the operators, @pankajkoti !
Happy for this PR to be merged once the tests are green.

…1997) Now that PR #1982 is merged and we have the XCOM getting populated, we can clean up the example dbt event payload dictionaries kept in the watcher.py module. If needed to refer again, we can refer to the previous PR #1952 that added these examples. related: #1952 related: #1950

@tatiana

**Features** * Introduce ``ExecutionMode.WATCHER`` to reduce DAG run time by 1/5 in several PRs. Learn more about it [here](https://astronomer.github.io/astronomer-cosmos/getting_started/watcher-execution-mode.html#watcher-execution-mode). This feature was implemented via multiple PRs, including: * Expose new execution mode by @tatiana @pankajastro @pankajkoti in #1999 * Add ``DbtProducerWatcherOperator`` for the proposed ``ExecutionMode.WATCHER`` by @pankajkoti in #1982 * Add ``DbtConsumerWatcherSensor`` for the proposed ``ExecutionMode.WATCHER`` by @pankajastro in #1998 * Push producer's task completion status to XCOM by @pankajkoti in #2000 * Add default priority_weight for ``DbtProducerWatcherOperator`` by @pankajkoti in #1995 * Add sample dbt events for the dbt watcher execution mode by @pankajkoti in #1952 * Add ``compiled_sql`` as a template fields on ```ExecutionMode.WATCHER``` when using ``run_results.json`` by @pankajastro in #2070 * Set ``push_run_results_to_xcom`` kwargs correctly for invocation mode subprocess and Watcher mode by @pankajastro in #2067 * Store compiled SQL as template field for dbt callback events in ``ExecutionMode.WATCHER`` by @pankajkoti in #2068 * Add initial documentation for ``ExecutionMode.WATCHER`` by @tatiana in #2046 * Support running ``State.UPSTREAM_FAILED`` tasks when WATCHER consumer upstream tasks fail by @tatiana in #2062 * Fail sensor tasks immediately if the ``ExecutionMode.WATCHER`` producer task fails by @pankajastro in #2040 * Add ``WATCHER``` to GitHub issue template by @tatiana in #2056 * Add support for ``TestBehavior.AFTER_ALL`` with ``ExecutionMode.WATCHER`` by @pankajastro in #2049 * Add support for ``TestBehavior.NONE`` with ``ExecutionMode.WATCHER`` by @pankajastro in #2047 * Fix ``ExecutionMode.WATCHER`` behaviour with ``DbtTaskGroup`` by @tatiana in #2044 * Fix Cosmos behaviour when using watcher with ``InvocationMode.DBT_RUNNER`` by @tatiana in #2048 * Add Airflow 3 plugin for dbt docs with multiple dbt projects support by @pankajkoti in #2009, check the [documentation](https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html). * Initial support to ``dbt Fusion`` by @tatiana in #1803. More details [here](https://astronomer.github.io/astronomer-cosmos/configuration/dbt-fusion). * Support to prune sources without downstream references in dbt projects by @corsettigyg in #1988 * Allow to set task display name as a user-defined function by @corsettigyg in #1761 * Add dbt project's hash to dag docs to support dag versioning in Airflow 3 by @pankajkoti in #1907 * feat: Add Jinja templating support for ``dbt_cmd_flags`` by @skillicinski in #1899 * Add Scarf metric to collect the execution mode uses by @pankajastro in #1981 * Support Airflow 3.1 by @tatiana in #1980 * Add MySQL profile mapping by @Lee2532 in #1977 * Add sqlserver profile mapping by @pankajastro in #1737 **Enhancement** * Use XCom to store sql when using ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajastro in #1934 * Refactor ``AIRFLOW_ASYNC`` teardown so it doesn't install the virtualenv by @pankajastro in #1938 * Reuse the virtual env for ``AIRFLOW_ASYNC`` setup task by @pankajastro in #1939 * Improve dataset/asset experience in Cosmos by @tatiana in #2030 * Add ``downstreams`` to ``DbtNode`` by @wornjs in #2028 **Bug fixes** * Fix tags extraction by @ms32035 in #1915 * Fix task flow operator args by @anyapriya in #2024 **Documentation** * Add documentation for Airflow 3 Plugin supporting dbt docs for multiple dbt projects by @pankajkoti in #2063 * Add Cosmos Deferrable Operator Guide by @pankajastro in #1922 * Add dbt Fusion documentation by @tatiana in #1824 #1830 * Update dbt-fusion.rst to explicitly highlight it is in alpha by @tatiana in #1838 * Fix a bunch of docs build errors and warnings by @pankajkoti in #1886 * Add docs note for param virtualenv_dir for async execution mode by @pankajastro in #1969 * Use pepy.tech downloads badge in README by @pankajkoti in #1920 * Correct the default value of ``cache_dir`` by @seokyun.ha in #2027 **Others** * Promote @corsettigyg to committer by @tatiana in #1985 * Add @pankajkoti and @pankajastro to ``contributors.rst`` by @tatiana in #1983 * Update setup script for airflow3 script by @dwreeves in #2023 * Prevent pytest from trying to test classes that aren't actually tests by @anyapriya in #2032 * Fix ``dag.test()`` for Airflow 3.1+ by syncing DAG to database bby @kaxil in #2037 * Disable Scarf in CI by @pankajastro in #2016 * Fix failing dbt Fusion tests when run in parallel in CI by @pankajkoti in #1896 * Fix MyPy issues related to ``ObjectStoragePath`` in main branch by @tatiana in #2012 * Cleanup example dbt event JSON dictionaries kept for XCOM referencby @pankajkoti in #1997 * Bump min hatch version that includes fixes for click>=8.3.0 by @pankajkoti in #1996 * Use official postgres image from Docker hub for kubernetes setup by @pankajkoti in #1986 * Use click<8.3.0 for hatch as click 8.3 breaks hatch by @pankajkoti in #1987 * Pin Airflow version in type check CI job by @pankajastro in #2003 * Improve comments after feedback on #1948 by @tatiana in #1963 * Fix running tests with dbt Fusion 2.0.0 preview versions by @tatiana in #1948 * Test hardening of dbt node having tags as unset or missing by @pankajkoti in #1918 * Fix Sphinx issue in the main branch by @tatiana in #2064 * pre-commit autoupdate in #2065, #2043, #2033, #2019, #1990, #2019, #2008, #1941, #1935, #1924 * GitHub dependabot update in #2051, #2050, #2038, #2022, #1947, #1955, #1946, #1944, #1945, #1928, #1921, #1917 Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io> Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Copilot AI review requested due to automatic review settings September 11, 2025 19:10

pankajkoti temporarily deployed to internal September 11, 2025 19:10 — with GitHub Actions Inactive

Copilot AI reviewed Sep 11, 2025

View reviewed changes

pankajkoti temporarily deployed to internal September 15, 2025 09:16 — with GitHub Actions Inactive

pankajkoti force-pushed the dbt-event-samples branch from 4707d10 to 0fa5da4 Compare September 15, 2025 09:16

pankajkoti temporarily deployed to internal September 15, 2025 09:17 — with GitHub Actions Inactive

tatiana temporarily deployed to internal September 15, 2025 13:36 — with GitHub Actions Inactive

tatiana approved these changes Sep 15, 2025

View reviewed changes

pankajkoti added 2 commits September 15, 2025 07:44

Add sample dbt events for the dbt watcher execution mode

764c11c

Rename dbt_watcher.py to watcher.py

d511243

pankajkoti force-pushed the dbt-event-samples branch from afd6a95 to d511243 Compare September 15, 2025 13:44

pankajkoti temporarily deployed to internal September 15, 2025 13:44 — with GitHub Actions Inactive

tatiana merged commit c1b77d5 into main Sep 15, 2025
94 of 96 checks passed

tatiana deleted the dbt-event-samples branch September 15, 2025 14:30

tatiana mentioned this pull request Sep 16, 2025

Evaluate dbt events for callbacks registered and finalise XCOM content schema [ExecutionMode.WATCHER] #1957

Closed

pankajkoti mentioned this pull request Sep 25, 2025

Cleanup example dbt event JSON dictionaries kept for XCOM reference #1997

Merged

tatiana mentioned this pull request Oct 7, 2025

Release 1.11.0 #2020

Merged

tatiana added this to the Cosmos 1.11.0 milestone Oct 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add sample dbt events for the dbt watcher execution mode#1952

Add sample dbt events for the dbt watcher execution mode#1952
tatiana merged 2 commits into
mainfrom
dbt-event-samples

pankajkoti commented Sep 11, 2025

Uh oh!

netlify Bot commented Sep 11, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

codecov Bot commented Sep 11, 2025 •

edited

Loading

Uh oh!

tatiana left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pankajkoti commented Sep 11, 2025

Uh oh!

netlify Bot commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

codecov Bot commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tatiana left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

netlify Bot commented Sep 11, 2025 •

edited

Loading

codecov Bot commented Sep 11, 2025 •

edited

Loading

tatiana left a comment •

edited

Loading