logs dbt-core and adapter in watcher consumer task by pankajastro · Pull Request #2412 · astronomer/astronomer-cosmos

pankajastro · 2026-02-26T19:35:25Z

Related to: #2072

codecov · 2026-02-26T23:58:33Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.00%. Comparing base (d91ea8d) to head (9b1b9f7).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2412      +/-   ##
==========================================
+ Coverage   97.99%   98.00%   +0.01%     
==========================================
  Files         103      103              
  Lines        7086     7131      +45     
==========================================
+ Hits         6944     6989      +45     
  Misses        142      142

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Adds support for surfacing dbt-core and adapter version information in ExecutionMode.WATCHER consumer tasks by persisting dbt startup events into XCom and having the triggerer wait for (and log) those events.

Changes:

Introduce DBT_STARTUP_EVENTS_XCOM_KEY constant and use it consistently instead of a hard-coded XCom key string.
Capture startup events (MainReportVersion, AdapterRegistered) from both dbtRunner event callbacks and subprocess JSON logs, and push them to XCom.
Add triggerer logic/tests to poll for startup events and log dbt/adaptor version messages in the consumer task context.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`cosmos/constants.py`	Adds `DBT_STARTUP_EVENTS_XCOM_KEY` constant for startup events XCom.
`cosmos/operators/watcher.py`	Pushes startup events to XCom early during dbtRunner callbacks and switches to the constant key.
`cosmos/operators/_watcher/base.py`	Merges startup events from subprocess JSON logs into the same XCom key.
`cosmos/operators/_watcher/triggerer.py`	Adds `_wait_and_log_startup_versions()` and calls it at trigger start.
`tests/operators/test_watcher.py`	Updates watcher tests for the constant key; adds tests for log-merge helper and trigger wait/logging.
`tests/operators/_watcher/test_triggerer.py`	Adjusts triggerer tests to account for startup events XCom polling.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tatiana

@pankajastro Thanks a lot for this, it will be great to add more visibility to the consumer logs! I left some feedback inline. My last question is:

Does the current implementation work with both InvocationMode.SUBPROCESS and InvocationMode.DBT_RUNNER?

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

netlify · 2026-03-10T20:11:06Z

✅ Deploy Preview for astronomer-cosmos canceled.

Name	Link
🔨 Latest commit	`9b1b9f7`
🔍 Latest deploy log	https://app.netlify.com/projects/astronomer-cosmos/deploys/69b10f8dcfcd5f0009a86db7

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tatiana

Thanks a lot for the implementation and for addressing the feedback, @pankajastro ! I believe this is already a significant improvement - and if we notice it is not behaving as expected, we can always improve the implementation over time.

@jedcunningham

1.14.0 (2026-04-07) --------------------- Breaking Changes * Drop support for Airflow versions earlier than **2.9** by @jedcunningham in #2288 * Fix inclusion of package models and selection/exclusion behavior by @pankajkoti in #2357 * ``ExecutionMode.WATCHER``: The per-node ``*_status`` XCom value is now a dict (``{"status": "<status>", "outlet_uris": [...]}``) instead of a plain string. Any custom code that reads these internal XCom keys directly will need to be updated by @pankajkoti in #2507 Features * Add cluster policy support for ``ExecutionMode.WATCHER`` sensor retries by @astro-anand in #2293 * Add debug mode to track memory utilization by @tatiana in #2327 * Add FQN selection support for ``LoadMode.DBT_MANIFEST`` by @pankajastro in #2375 * Introduce interceptors for Cosmos tasks by @tatiana in #2419 * Add config to allow disabling dag versioning by @pankajkoti in #2470 * Implement TaskGroups by models folder by @maximilianoarcieri and @tatiana in #1566, #2469, and #2420 * feat: implement DbtTestWatcherOperator by @michal-mrazek in #2447 * Add source freshness aware execution for ``ExecutionMode.WATCHER`` by @pankajastro and @tatiana in #2467 * Note: Like ``ExecutionMode.WATCHER``, this feature is experimental and its interface and implementation can change in the future. * Add Airflow 3.2 support by @pankajastro and @pankajkoti in #2472 Enhancements * Add watcher mode support for dbt test node states by @michal-mrazek in #2318 * Rename watcher-mode sensor retry queue and reuse it for producer tasks by @pankajastro in #2331 * Fix leaked semaphore warnings in Airflow 3 by resetting dbt adapters by @pankajkoti in #2335 * Improve dbt Fusion support and related tests by @tatiana in #2356 * Default Snowflake profile mappings to four threads by @tatiana in #2374 * Attempt to remove Pydantic as a dependency by @tatiana in #2377 * Log dbt-core and adapter versions in watcher consumer tasks by @pankajastro in #2412 * Log model errors in watcher consumer on dbt node failure by @pankajastro in #2431 * Reduce XCom read/write for tracking node state and errors in ConsumerWatcher task by @pankajastro in #2471 * Remove duplicate debug log in watcher subprocess path by @tatiana in #2494 * Simplify and unify WATCHER implementation regardless of InvocationMode by @tatiana in #2498 * Switch to lazy imports in cosmos/__init__.py by @pankajkoti in #2531 Bug Fixes * Handle invalid YAML errors with ``LoadMode.DBT_MANIFEST`` and ``RenderConfig.selector`` by @YourRoyalLinus in #2316 * Populate ``compiled_sql`` for ``InvocationMode.SUBPROCESS`` in ``ExecutionMode.WATCHER`` by @pankajkoti in #2319 * Fix select/exclude type mismatch by @tatiana in #2364 * Set ``emit_datasets=False`` for ``DbtTest*`` operators by @pankajastro in #2365 * Set correct queue priority for watcher producer tasks by @pankajastro in #2372 * Preserve ``extra_context`` for watcher consumer task instances by @pankajkoti in #2381 * Respect ``deferrable=False`` from ``operator_args`` on watcher consumer sensors by @pankajkoti in #2384 * Fix watcher queue precedence and add documentation by @pankajastro in #2391 * Do not set ``compiled_sql`` on ``ExecutionMode.WATCHER`` producers by @pankajkoti in #2440 * Remove const attribute for ``__cosmos_telemetry_metadata__`` dag param by @pankajkoti in #2466 * Remove timeout override from Cosmos watcher sensors by @tatiana and @claude in #2478 * Remove forced ``retries=0`` from watcher producer operators by @tatiana in #2479 * RFC: Add patch for newer versions of amazon provider when running dbt on EKS by @aoelvp94 in #2481 * Fix ``cosmos_debug_max_memory_mb`` XCom not pushed in Watcher sensor tasks by @tatiana in #2503 * Fix ``TestBehavior.NONE`` and ``TestBehavior.AFTER_ALL`` exclude ignored with selectors in ``ExecutionMode.WATCHER`` by @pankajkoti in #2511 * Move dataset emission for ``ExecutionMode.WATCHER`` from producer to consumer sensors by @pankajkoti in #2507 Docs * Document cluster policy configuration for ``ExecutionMode.WATCHER`` sensor tasks by @pankajastro in #2315 * Remove outdated docs for the dbt docs plugin with Airflow 3 by @pankajastro in #2353 * Make Watcher DBT Execution Queue heading clickable by @pankajastro in #2354 * Update ``ExecutionMode.WATCHER`` documentation regarding test node implementation by @jroachgolf84 in #2355 * Fix ``pre_dbt_fusion`` configuration rendering by @pankajastro in #2369 * Add documentation for including/excluding nodes based on FQN by @pankajastro in #2371 * Update watcher execution mode documentation by @tatiana in #2380 * Add documentation for ``DbtSeedLocalOperator`` by @jroachgolf84 in #2383 * Fix miscellaneous Sphinx warnings by @pankajastro in #2395 * Improve contributing documentation by @lzdanski in #2397 * Add **Get Started in 5 Minutes** guide by @lzdanski in #2398 * Add Sphinx redirects package for documentation redirects by @lzdanski in #2407 * Restructure **Getting Started** and **Guides** sections by @lzdanski in #2418 * Add open-source quickstart by @lzdanski in #2439 * Fix documentation redirects by @lzdanski in #2442 * Restructure and refactor reference documentation by @lzdanski in #2443 * Add execution modes decision documentation by @lzdanski in #2444 * Add **Core Concepts** page to Getting Started by @lzdanski in #2448 * Add guide: *How Cosmos Works* by @lzdanski in #2449 * Update **Getting Started** overview and index pages by @lzdanski in #2452 * Add guide: *How Cosmos Runs dbt* by @lzdanski in #2453 * Fix miscellaneous documentation links by @lzdanski in #2454 * Add Mermaid diagrams and execution mode diagrams by @lzdanski and @tatiana in #2459 * Add documentation for memory optimization options by @pankajastro in #2340 * Fix typo in watcher execution mode docs by @evanvolgas in #2485 * Fix minor documentation issues by @evanvolgas in #2489 * Add troubleshooting note for dbt debug logs in ExecutionMode.WATCHER by @tatiana in #2491 * docs: unify RST header styles across documentation by @jigangz in #2473 * docs: fix env var for rich logging by @vricciardulli in #2514 * docs: update dbt project path example for Airflow 3 Astro compatibility by @yeoreums in #2512 * Document missing Cosmos Airflow config settings in cosmos-conf.rst by @tatiana in #2515 * Split security-privacy policy doc and add dependency cooldown by @pankajkoti in #2519 * Add performance optimization and troubleshooting docs by @pankajkoti in #2521 * Update copyright year to 2026 by @tayloramurphy in #2527 * docs: Updating "Project Policies" to "Policies" in menu bar by @jroachgolf84 in #2526 Others * Fix tests after removing support for Airflow versions earlier than 2.9 by @tatiana in #2321 * Enable listener tests for Airflow 3.1 by @pankajastro in #2348 * Accept ``int`` or ``float`` for ``cosmos_debug_max_memory_mb`` in integration tests by @pankajkoti in #2352 * Update ``CODEOWNERS`` to prioritize ``oss-integrations`` by @tatiana in #2359 * Fix automatic reviewer assignment in GitHub by @tatiana and @phanikumv in #2360 * Improve PyPI tagging by @tatiana in #2363 * Add integration tests for dbt Fusion and ``ExecutionMode.WATCHER`` by @tatiana in #2373 * Fix Zizmor check by @tatiana in #2376 * Remove ``methodtools`` dependency by @tatiana in #2378 * Improve comments on #2389 by @evanvolgas in #2394 * Refactor ``load_from_dbt_manifest`` to reduce code complexity by @pankajkoti in #2399 * Refactor ``_handle_no_precursors_or_descendants`` to reduce complexity by @pankajkoti in #2400 * Improve issue templates by @tatiana in #2401 * Avoid running tests when only docs change by @tatiana in #2402 * Add ``no-reload`` target for serving docs locally by @pankajkoti in #2405 * Fix test hash checks on macOS by @tatiana in #2406 * Attempt deterministic dbt project copy in test fixtures by @pankajkoti in #2409 * Pin ``virtualenv <21`` due to hatch incompatibility in CI by @pankajkoti in #2410 * Revert virtualenv pin for hatch installation in CI by @pankajkoti in #2426 * Add version comments for commit SHA pinned GitHub Actions by @pankajkoti in #2436 * Fix ``hatch run docs:build`` issues by @tatiana in #2437 * Minor code improvements by @dnskr in #2446 * Pre-commit autoupdate by @pre-commit-ci in #2367, #2396, #2422, #2451, #2468, #2495, and #2516 * Add file to support Claude understanding the Cosmos repository by @tatiana in #2458 * Dependency updates by @dependabot in #2368, #2425, #2435, #2465, #2475, #2504, #2518, and #2528 * Isolate Scarf telemetry integration test into its own CI job by @pankajkoti and @claude in #2477 * ci: upgrade Airflow version to 3.1 in MyPy type-check job by @yeoreums in #2506 * Add commit message guidelines to CLAUDE.md by @pankajkoti in #2509 * Extend skipping tests in CI for more non-code file changes by @pankajkoti in #2510 * Add Dependabot pre-commit support with 7-day cooldown by @pankajkoti in #2517 * Enforce zero warnings policy for documentation by @dnskr in #2513 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

pankajastro temporarily deployed to internal February 26, 2026 19:35 — with GitHub Actions Inactive

logs dbt-core and adapter in watcher consumer task

566146b

pankajastro force-pushed the log_dbt_version branch from 8850f0f to 566146b Compare February 26, 2026 19:39

pankajastro temporarily deployed to internal February 26, 2026 19:40 — with GitHub Actions Inactive

pankajastro mentioned this pull request Feb 26, 2026

Improve visibility of dbt command logs in ExecutionMode.WATCHER consumer tasks #2072

Closed

Fix tests

f15c60b

pankajastro temporarily deployed to internal February 26, 2026 19:48 — with GitHub Actions Inactive

pankajastro marked this pull request as ready for review February 27, 2026 03:03

pankajastro requested a review from jbandoro as a code owner February 27, 2026 03:03

Copilot AI review requested due to automatic review settings February 27, 2026 03:03

pankajastro requested review from a team, corsettigyg and dwreeves as code owners February 27, 2026 03:03

pankajastro requested review from pankajkoti and tatiana February 27, 2026 03:03

Copilot started reviewing on behalf of pankajastro February 27, 2026 03:04 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

Comment thread cosmos/operators/_watcher/triggerer.py

Comment thread cosmos/operators/_watcher/base.py Outdated

Comment thread tests/operators/test_watcher.py

tatiana reviewed Mar 3, 2026

View reviewed changes

Comment thread cosmos/operators/_watcher/triggerer.py Outdated

tatiana reviewed Mar 3, 2026

View reviewed changes

Comment thread cosmos/operators/_watcher/triggerer.py Outdated

tatiana reviewed Mar 3, 2026

View reviewed changes

pankajkoti reviewed Mar 5, 2026

View reviewed changes

Comment thread cosmos/operators/_watcher/triggerer.py Outdated

Comment thread cosmos/operators/_watcher/triggerer.py Outdated

Comment thread cosmos/operators/watcher.py Outdated

pankajkoti reviewed Mar 5, 2026

View reviewed changes

Comment thread cosmos/operators/_watcher/triggerer.py

Merge branch 'main' into log_dbt_version

1fa32dc

pankajastro temporarily deployed to internal March 9, 2026 05:44 — with GitHub Actions Inactive

Address review feedback

8b2d44a

Copilot AI review requested due to automatic review settings March 9, 2026 07:22

pankajastro temporarily deployed to internal March 9, 2026 07:22 — with GitHub Actions Inactive

Copilot started reviewing on behalf of pankajastro March 9, 2026 07:23 View session

Copilot AI reviewed Mar 9, 2026

View reviewed changes

Comment thread cosmos/operators/_watcher/base.py Outdated

Comment thread tests/operators/_watcher/test_triggerer.py

Comment thread cosmos/operators/_watcher/triggerer.py Outdated

Copilot AI review requested due to automatic review settings March 10, 2026 20:10

pankajastro temporarily deployed to internal March 10, 2026 20:10 — with GitHub Actions Inactive

Copilot started reviewing on behalf of pankajastro March 10, 2026 20:10 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

Comment thread tests/operators/test_watcher_kubernetes_unit.py Outdated

Comment thread cosmos/operators/_watcher/triggerer.py

Comment thread tests/operators/_watcher/test_triggerer.py Outdated

Fix tests

b755771

pankajastro temporarily deployed to internal March 10, 2026 20:17 — with GitHub Actions Inactive

Fix tests

d482beb

Copilot AI review requested due to automatic review settings March 10, 2026 20:18

Copilot started reviewing on behalf of pankajastro March 10, 2026 20:18 View session

pankajastro temporarily deployed to internal March 10, 2026 20:20 — with GitHub Actions Inactive

Copilot AI reviewed Mar 10, 2026

View reviewed changes

Comment thread cosmos/operators/_watcher/base.py

Merge branch 'main' into log_dbt_version

dc319e3

pankajastro temporarily deployed to internal March 10, 2026 21:11 — with GitHub Actions Inactive

pankajastro requested review from pankajkoti and tatiana March 10, 2026 22:35

Add tests

9b1b9f7

Copilot AI review requested due to automatic review settings March 11, 2026 06:45

pankajastro temporarily deployed to internal March 11, 2026 06:45 — with GitHub Actions Inactive

Copilot started reviewing on behalf of pankajastro March 11, 2026 06:45 View session

Copilot AI reviewed Mar 11, 2026

View reviewed changes

Comment thread cosmos/operators/_watcher/base.py

Comment thread cosmos/operators/_watcher/triggerer.py

Comment thread cosmos/operators/_watcher/triggerer.py

This was referenced Mar 11, 2026

Enhance consumer task logging for non-deferrable mode #2455

Open

Ship DBT logs to consumer task in WATCHER_KUBERNETES mode #2457

Open

tatiana approved these changes Mar 12, 2026

View reviewed changes

tatiana merged commit f62ecbc into main Mar 12, 2026
83 checks passed

tatiana deleted the log_dbt_version branch March 12, 2026 09:43

pankajastro mentioned this pull request Mar 16, 2026

Release 1.14.0 #2314

Merged

tatiana added this to the Cosmos 1.14.0 milestone Apr 28, 2026

tatiana added the roadmap:P1 BOSS roadmap-committed work (priority P1) label Jun 6, 2026

Conversation

pankajastro commented Feb 26, 2026

Uh oh!

codecov Bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

netlify Bot commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for astronomer-cosmos canceled.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Feb 26, 2026 •

edited

Loading

netlify Bot commented Mar 10, 2026 •

edited

Loading