Skip to content

Fix source freshness warnings in kubernetes em#1859

Merged
tatiana merged 7 commits into
astronomer:mainfrom
Pawel-Drabczyk:fix-source-freshness-warnings-in-kubernetes-em
Jul 11, 2025
Merged

Fix source freshness warnings in kubernetes em#1859
tatiana merged 7 commits into
astronomer:mainfrom
Pawel-Drabczyk:fix-source-freshness-warnings-in-kubernetes-em

Conversation

@Pawel-Drabczyk
Copy link
Copy Markdown
Contributor

@Pawel-Drabczyk Pawel-Drabczyk commented Jul 2, 2025

Description

Improved handling of warnings from DBT freshness tests in Kubernetes Execution Mode.

Related Issue(s)

closes #1797

Breaking Change?

No

Checklist

  • [No need] I have made corresponding changes to the documentation (if required)
  • [Done] I have added tests that prove my fix is effective or that my feature works

Copilot AI review requested due to automatic review settings July 2, 2025 16:04
@netlify
Copy link
Copy Markdown

netlify Bot commented Jul 2, 2025 •
edited
Loading

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 70896be
🔍 Latest deploy log https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/68655b73c4ad240008479ed0

Copilot AI reviewed Jul 2, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Improves detection and handling of DBT freshness warnings in Kubernetes Execution Mode by extending the warning handler and adding comprehensive tests.

  • Adds _detect_standard_warnings and _detect_source_freshness_warnings to parse different warning patterns
  • Updates on_pod_completion logic to invoke callbacks only when warnings are detected
  • Introduces parametric tests for both standard and source freshness warning scenarios

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
tests/operators/test_kubernetes.py New tests for standard DBT warnings and source freshness warnings
cosmos/operators/kubernetes.py Enhanced DbtTestWarningHandler with detection methods and logic
Comments suppressed due to low confidence (2)

tests/operators/test_kubernetes.py:307

  • Consider adding a test case for logs containing only the simple freshness warning pattern (e.g., WARN freshness of my_source.my_table) without timestamps to verify the secondary regex branch.
        (

cosmos/operators/kubernetes.py:282

  • Update the docstring to list all keys present in the returned warning dicts, including timestamp and execution_time, for clearer API documentation.
    def _detect_source_freshness_warnings(self, log_text: str) -> List[Dict[str, Any]]:

Comment thread cosmos/operators/kubernetes.py Outdated
Comment thread cosmos/operators/kubernetes.py Outdated
Comment thread cosmos/operators/kubernetes.py Outdated
Pawel-Drabczyk and others added 3 commits July 2, 2025 18:14
@Pawel-Drabczyk
fixing freshness warning in Kubernetes EM
9f3da2a 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@Pawel-Drabczyk
fixing freshness warning in Kubernetes EM
5a7dbf4 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@Pawel-Drabczyk
fixing freshness warning in Kubernetes EM
70896be 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@Pawel-Drabczyk Pawel-Drabczyk mentioned this pull request Jul 2, 2025
Closed
1 task
tatiana
tatiana approved these changes Jul 11, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, @Pawel-Drabczyk , thank you very much for reporting the isssue and implementing the fix - with tests that will help us avoiding the bug from happening again.

@tatiana tatiana merged commit 460e58c into astronomer:main Jul 11, 2025
5 checks passed
@pankajkoti pankajkoti mentioned this pull request Jul 17, 2025
Merged
tatiana pushed a commit that referenced this pull request Jul 18, 2025
@pankajkoti
Remove commented code and fix mypy failures (#1876)
0635e2e 
Some commented code was pushed as part of
#1859 and also
responsible for mypy failures. Removing that unused commented code fixes
the [type-check
failures](https://github.com/astronomer/astronomer-cosmos/actions/runs/16322249943/job/46169121034#step:5:595)
observed in main branch CI.

Successful type check job run from this PR:
https://github.com/astronomer/astronomer-cosmos/actions/runs/16343955179/job/46172890065?pr=1876#step:5:591

related: #1859
related: #1873
@tatiana tatiana mentioned this pull request Aug 7, 2025
Merged
tatiana pushed a commit that referenced this pull request Aug 8, 2025
@Pawel-Drabczyk @tatiana
Fix source freshness warnings in kubernetes em (#1859)
81b1fb8 
Improved handling of warnings from DBT freshness tests in Kubernetes
Execution Mode.

Closes #1797

(cherry picked from commit 460e58c)
tatiana pushed a commit that referenced this pull request Aug 8, 2025
@pankajkoti @tatiana
Remove commented code and fix mypy failures (#1876)
f8b0832 
Some commented code was pushed as part of
#1859 and also
responsible for mypy failures. Removing that unused commented code fixes
the [type-check
failures](https://github.com/astronomer/astronomer-cosmos/actions/runs/16322249943/job/46169121034#step:5:595)
observed in main branch CI.

Successful type check job run from this PR:
https://github.com/astronomer/astronomer-cosmos/actions/runs/16343955179/job/46172890065?pr=1876#step:5:591

related: #1859
related: #1873
(cherry picked from commit 0635e2e)
tatiana added a commit that referenced this pull request Aug 8, 2025
@tatiana @pankajkoti
Release 1.10.2 (#1911)
59b71ae 
**Bug Fixes**

* Fix task instance ``try_number`` attribute for Airflow 3 compatibility
by @pankajkoti in #1781
* Fix rendered template override logic when
``should_store_compiled_sql=False`` to restore pre-refactor behaviour by
@pankajkoti in #1777
* Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by
@ramonvermeulen in #1783
* Fix dbt Docs page height by @1cadumagalhaes in #1793
* Add support to base64 encoded pem in Snowflake profiles by
@brunocmartins in #1801
* Allow to disable owner inheritance from dbt into airflow DAG owners by
@CorsettiS in #1787
* Fix Kubernetes Pod Operator conversion of ``container_resources`` to
``resources`` by @johnhoran in #1821
* Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in
#1822
* Fix source freshness warnings in kubernetes execution mode by
@Pawel-Drabczyk in #1859
* Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877
* Fix cache behaviour when DAG name contains "." by @tatiana in #1908

**Documentation**

* Fix ``contributing.rst`` docs by @tatiana in #1785
* Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790
* Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814
* Update testing behavior file with ``ExecutionMode.KUBERNETES`` by
@LuigiCerone in #1813
* Add step to fork repo in contributing guide by @pankajastro in #1808
* Fix ``depends_on`` attribute by @benedikt-buchert in #1837
* Fix character name by @ThePsyjo in #1860
* Update suggested MWAA startup script by @jaklan in #1884
* Make implementation & docs consistent regarding
``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878

**Others**

* Set retries to 0 in example DAGs by @pankajkoti in #1782
* Fix ``test_async_example_dag_without_setup_task`` tests by
@pankajastro in #1788
* Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in
#1786
* Upgrade Python and Airflow used to run MyPy checks by @tatiana in
#1796
* Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and
@tatiana in #1778
* Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9
& 1.10 by @tatiana in #1820
* Fix broken CI due to Pydantic conflicts by @tatiana in #1809
* Drop Python 3.8 Support by @pankajastro in #1852
* Add Airflow 2.11 to the test matrix by @tatiana in #1807
* Require Authorize for all jobs on pull requests from external
contributors in CI by @pankajkoti in #1861
* Leverage Trusted Publisher Management when publishing PyPI package by
@tatiana in #1862
* CI: Add back accidentally deleted python-version matrix for running
unit tests by @pankajkoti in #1872
* Remove commented code and fix mypy failures by @pankajkoti in #1876
* Add Zizmor analysis GitHub action by @pankajkoti in #1870
* Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti
in #1880
* Add deprecation warning for ``LoadMode.CUSTOM`` parser by
@duongphannamhung in #1885
* CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in
#1871
* Resolve 'credential persistence through GitHub Actions artifacts'
warnings from Zizmor by @pankajkoti in #1890
* Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti
in #1889
* Resolve Zizmor error alerts for unpinned image references; mark alert
for pull_request_target ignored by @pankajkoti in #1888
* Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by
@tatiana in #1902
* Add dbt-core 1.10 to test matrix by @tatiana in #1767
* Pin package dbt-databricks by @pankajastro in #1909
* Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests
in CI by @pankajastro in #1900
* Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892,
#1901
* Dependabot updates: #1904

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
@pankajkoti pankajkoti mentioned this pull request Aug 8, 2025
Merged
tatiana added a commit that referenced this pull request Aug 8, 2025
@pankajkoti @tatiana
Release 1.10.2 (#1912)
f698166 
**Bug Fixes**

* Fix task instance ``try_number`` attribute for Airflow 3 compatibility
by @pankajkoti in #1781
* Fix rendered template override logic when
``should_store_compiled_sql=False`` to restore pre-refactor behaviour by
@pankajkoti in #1777
* Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by
@ramonvermeulen in #1783
* Fix dbt Docs page height by @1cadumagalhaes in #1793
* Add support to base64 encoded pem in Snowflake profiles by
@brunocmartins in #1801
* Allow to disable owner inheritance from dbt into airflow DAG owners by
@CorsettiS in #1787
* Fix Kubernetes Pod Operator conversion of ``container_resources`` to
``resources`` by @johnhoran in #1821
* Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in
#1822
* Fix source freshness warnings in kubernetes execution mode by
@Pawel-Drabczyk in #1859
* Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877
* Fix cache behaviour when DAG name contains "." by @tatiana in #1908

**Documentation**

* Fix ``contributing.rst`` docs by @tatiana in #1785
* Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790
* Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814
* Update testing behavior file with ``ExecutionMode.KUBERNETES`` by
@LuigiCerone in #1813
* Add step to fork repo in contributing guide by @pankajastro in #1808
* Fix ``depends_on`` attribute by @benedikt-buchert in #1837
* Fix character name by @ThePsyjo in #1860
* Update suggested MWAA startup script by @jaklan in #1884
* Make implementation & docs consistent regarding
``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878

**Others**

* Set retries to 0 in example DAGs by @pankajkoti in #1782
* Fix ``test_async_example_dag_without_setup_task`` tests by
@pankajastro in #1788
* Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in
#1786
* Upgrade Python and Airflow used to run MyPy checks by @tatiana in
#1796
* Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and
@tatiana in #1778
* Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9
& 1.10 by @tatiana in #1820
* Fix broken CI due to Pydantic conflicts by @tatiana in #1809
* Drop Python 3.8 Support by @pankajastro in #1852
* Add Airflow 2.11 to the test matrix by @tatiana in #1807
* Require Authorize for all jobs on pull requests from external
contributors in CI by @pankajkoti in #1861
* Leverage Trusted Publisher Management when publishing PyPI package by
@tatiana in #1862
* CI: Add back accidentally deleted python-version matrix for running
unit tests by @pankajkoti in #1872
* Remove commented code and fix mypy failures by @pankajkoti in #1876
* Add Zizmor analysis GitHub action by @pankajkoti in #1870
* Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti
in #1880
* Add deprecation warning for ``LoadMode.CUSTOM`` parser by
@duongphannamhung in #1885
* CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in
#1871
* Resolve 'credential persistence through GitHub Actions artifacts'
warnings from Zizmor by @pankajkoti in #1890
* Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti
in #1889
* Resolve Zizmor error alerts for unpinned image references; mark alert
for pull_request_target ignored by @pankajkoti in #1888
* Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by
@tatiana in #1902
* Add dbt-core 1.10 to test matrix by @tatiana in #1767
* Pin package dbt-databricks by @pankajastro in #1909
* Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests
in CI by @pankajastro in #1900
* Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892,
#1901
* Dependabot updates: #1904

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
@tatiana tatiana added this to the Cosmos 1.10.2 milestone Oct 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@tatiana tatiana tatiana approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

Cosmos 1.10.2

Development

Successfully merging this pull request may close these issues.

[Bug] source freshness warning notifications in Kubernetes Execution mode

3 participants

@Pawel-Drabczyk @tatiana