Introduce ExecutionMode.WATCHER_KUBERNETES to use watcher with KubernetesPodOperator

[tatiana]

Introduce ExecutionMode.WATCHER_KUBERNETES, allowing users to use the Producer/Consumer approach without requiring dbt to be installed in the same environment as Airflow.

This PR includes an example DAG illustrating the use-case.

Given an existing bug in the latest releases of apache-airflow-providers-cncf-kubernetes (apache/airflow#59366), we had to subclass the KubernetesPodManager class and override a method. This may not be needed once the fix is merged and the new provider version is released. Given this issue, we have tested and successfully validated the usage of this execution mode with the following versions of apache-airflow-providers-cncf-kubernetes

• 10.8.0
• 10.9.0
• 10.10.0
• 10.11.0
• 10.12.2 (tested on 30 January 2026, after this PR was merged)

Why

There is significant overhead each time a K8S Pod is spawned and when the dbt project is parsed.

Cosmos ExecutionMode.WATCHER (docs) has delivered significant performance improvements for users who were previously using ExecutionMode.LOCAL; however, some users need to keep Airflow and dbt dependencies isolated.

This PR extends the ExecutionMode.WATCHER performance gains to Kubernetes.

Comparison of the same dbt project DAG run with Jaffle Shop (reproducible by modifying the DAG jaffle_shop_watcher_kubernetes included in this PR):

• ExecutionMode.KUBERNETES: 00:00:32.155
• ExecutionMode.WATCHER_KUBERNETES: 00:00:11.783

Limitations

• Does not work with older versions of the apache-airflow-providers-cncf-kubernetes provider (<=10.7.0)
• Only works withdeferrable=False.

** Pending work**

• The documentation will come in a follow-up PR.

How to use

• Users previously using ExecutionMode.KUBERNETES can just replace the execution_mode to use ExecutionMode.WATCHER_KUBERNETES.

Example DAG

This PR includes an example DAG that runs with our integration tests, jaffle_shop_watcher_kubernetes:

The producer task runs dbt build using the KubernetesPodOperator:

This task will populate XCom with status updates for each dbt node that is completed.

By default, the consumer tasks will be native Airflow deferrable sensors:

And, in case of retries, the consumer tasks use Cosmos ExecutionMode.KUBERNETES KPO operators:

Closes: #2042

[netlify]

✅ Deploy Preview for astronomer-cosmos canceled.

Name	Link
🔨 Latest commit	1f18858
🔍 Latest deploy log	https://app.netlify.com/projects/astronomer-cosmos/deploys/695e67c643be610008310299

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.03%. Comparing base (345aab8) to head (c6c156e).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2207      +/-   ##
==========================================
+ Coverage   97.97%   98.03%   +0.05%     
==========================================
  Files          97       98       +1     
  Lines        6273     6352      +79     
==========================================
+ Hits         6146     6227      +81     
+ Misses        127      125       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tatiana]

Thanks a lot for the reviews, @johnhoran @pankajkoti @pankajastro - I think I addressed most, if not all, of them

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pankajastro]

LGTM

[pankajkoti]

LGTM. Couple of questions inline