feat: handle tests for WATCHER_KUBERNETES mode#2529
Conversation
WATCHER_KUBERNETES mode
There was a problem hiding this comment.
Pull request overview
Implements dbt test handling for ExecutionMode.WATCHER_KUBERNETES by turning the previously-placeholder test operator into a watcher sensor and updating docs/tests to reflect the supported TestBehavior options in this mode.
Changes:
- Implement
DbtTestWatcherKubernetesOperatoras aDbtConsumerWatcherKubernetesSensorthat watches aggregated test results via XCom. - Add/adjust unit and graph-construction tests to cover watcher-kubernetes test behavior (
NONE,AFTER_EACH,AFTER_ALL). - Update watcher and watcher-kubernetes documentation to describe the supported test behaviors and operators.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
cosmos/operators/watcher_kubernetes.py |
Replaces placeholder test operator with a real watcher sensor implementation. |
tests/operators/test_watcher_kubernetes_unit.py |
Adds unit tests for the new test watcher sensor; simplifies producer callback test. |
tests/airflow/test_graph.py |
Verifies task generation for watcher-kubernetes across TestBehavior modes. |
docs/guides/run_dbt/container/watcher-kubernetes-execution-mode.rst |
Updates limitations/behavior docs to reflect test support in watcher-kubernetes. |
docs/guides/run_dbt/airflow-worker/watcher-execution-mode.rst |
Updates watcher-mode test behavior documentation to match current behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tatiana
left a comment
There was a problem hiding this comment.
Thank you very much @vricciardulli , this looks great!
|
Awesome, thanks! :) |
|
Not sure if it's due to this PR, but it's evident on the merge of this PR, that the kubernetes tests are getting stuck https://github.com/astronomer/astronomer-cosmos/actions/runs/24081355329/job/70242570483 |
This reverts commit 9611fa5.
|
@vricciardulli @michal-mrazek we unfortunately had to revert (#2535) this PR since it caused the K8S integration tests to get stuck for some reason, and we do not have time to fix it right now. I made the mistake of merging it without realising that those tests were stuck. @pankajkoti created a PR to revert. Please, if you could open a new PR with the change and fix the integration tests - we can make sure this will be shipped as part of Cosmos 1.15.0. |
|
@tatiana sorry for the inconvenience! |
|
@vricciardulli don't worry, it was my fault - I should have realised the tests weren't running as expected, and that would have avoided this situation. It would be amazing if you could create a new PR with the feature, and we could support on checking what may be hanging. Did you test the feature locally, using Airflow standalone or some other way? |
…#2529)" (astronomer#2535) This reverts commit f4fd7df.
Description
Following #2441 and #2447, this PR implements test handling for
WATCHER_KUBERNETESmode.Related Issue(s)
closes #2427
Breaking Change?
Checklist