Assert example DAGs' `DagRunState` and fix issues by pankajkoti · Pull Request #1778 · astronomer/astronomer-cosmos

pankajkoti · 2025-05-26T12:49:12Z

We identified that, although we executed our example test DAGs during our integration tests, we were not verifying that they were successful. Many of them were failing.

This PR confirms that the DAGs are being successful and aims to address those that were not.

Co-authored-by: Tatiana Al-Chueyr tatiana.alchueyr@gmail.com

netlify · 2025-05-26T12:49:17Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`408c568`
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/684b0881995ccc0008c7c39f

cloudflare-workers-and-pages · 2025-05-26T12:49:52Z

Deploying astronomer-cosmos with Cloudflare Pages

Latest commit:	`408c568`
Status:	✅ Deploy successful!
Preview URL:	https://64b28467.astronomer-cosmos.pages.dev
Branch Preview URL:	https://assert-example-dagrun-state.astronomer-cosmos.pages.dev

View logs

tatiana

Looks great, happy for us to merge this change once we've fixed the associated failing DAGs

…1781) This PR updates the way we access the task instance try number attribute in Cosmos io module's callback utility to ensure compatibility with both Airflow 2 and Airflow 3. The private `_try_number` attribute was removed in Airflow 3, making it necessary to use the public `try_number` property instead. This change ensures that Cosmos works correctly with both Airflow 2 and Airflow 3 versions. #### Testing - Updated test cases to reflect the new attribute access - Verified compatibility with both Airflow 2 and Airflow 3 closes: #1712 related: #1778 related: astronomer/oss-integrations-private#124

This PR removes default retries (previously set to 2) from example DAGs to speed up CI and surface failures faster. While we can add retries to specific DAGs if needed, having them enabled by default adds unnecessary delays to our test suite. related: astronomer/oss-integrations-private#124 related: #1778

Handles this exception being faced when running in AF2.5 and AF2.6: ``` def test_dag(dag, conn_file_path: str | None = None, custom_tester: bool = False) -> DagRun: dr = None if custom_tester: dr = test_old_dag(dag, conn_file_path) assert dr.state == DagRunState.SUCCESS, f"Dag {dag.dag_id} did not run successfully. State: {dr.state}. " elif AIRFLOW_VERSION >= version.Version("2.5"): if AIRFLOW_VERSION not in (Version("2.10.0"), Version("2.10.1"), Version("2.10.2")): dr = dag.test() > assert dr.state == DagRunState.SUCCESS, f"Dag {dag.dag_id} did not run successfully. State: {dr.state}. " E AttributeError: 'NoneType' object has no attribute 'state' tests/utils.py:42: AttributeError ```

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

https://github.com/astronomer/astronomer-cosmos/actions/runs/15613004782/job/43978485756

codecov · 2025-06-12T17:23:21Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.00%. Comparing base (d805662) to head (408c568).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1778      +/-   ##
==========================================
- Coverage   98.02%   98.00%   -0.02%     
==========================================
  Files          85       85              
  Lines        5262     5266       +4     
==========================================
+ Hits         5158     5161       +3     
- Misses        104      105       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pankajkoti

Thanks for fixing those bunch of issues and logging the tickets, @tatiana , appreciate the efforts!

pankajkoti · 2025-06-13T09:11:41Z

-def test_dag(dag, conn_file_path: str | None = None, custom_tester: bool = False) -> DagRun:
+def check_dag_success(dag_run: DagRun | None, expect_success: bool = True) -> bool:
+    """Check if a DAG was successful, if that Airflow version allows it."""
+    if dag_run is not None:


do we have a hint here on when/why the dag_run could be None?

…1781) This PR updates the way we access the task instance try number attribute in Cosmos io module's callback utility to ensure compatibility with both Airflow 2 and Airflow 3. The private `_try_number` attribute was removed in Airflow 3, making it necessary to use the public `try_number` property instead. This change ensures that Cosmos works correctly with both Airflow 2 and Airflow 3 versions. #### Testing - Updated test cases to reflect the new attribute access - Verified compatibility with both Airflow 2 and Airflow 3 closes: #1712 related: #1778 related: astronomer/oss-integrations-private#124 (cherry picked from commit 1f63202)

This PR removes default retries (previously set to 2) from example DAGs to speed up CI and surface failures faster. While we can add retries to specific DAGs if needed, having them enabled by default adds unnecessary delays to our test suite. related: astronomer/oss-integrations-private#124 related: #1778 (cherry picked from commit 9519324)

We identified that, although we executed our example test DAGs during our integration tests, we were not verifying that they were successful. Many of them were failing. This PR confirms that the DAGs are being successful and aims to address those that were not. Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com> (cherry picked from commit 67be637)

@pankajkoti

**Bug Fixes** * Fix task instance ``try_number`` attribute for Airflow 3 compatibility by @pankajkoti in #1781 * Fix rendered template override logic when ``should_store_compiled_sql=False`` to restore pre-refactor behaviour by @pankajkoti in #1777 * Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by @ramonvermeulen in #1783 * Fix dbt Docs page height by @1cadumagalhaes in #1793 * Add support to base64 encoded pem in Snowflake profiles by @brunocmartins in #1801 * Allow to disable owner inheritance from dbt into airflow DAG owners by @CorsettiS in #1787 * Fix Kubernetes Pod Operator conversion of ``container_resources`` to ``resources`` by @johnhoran in #1821 * Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in #1822 * Fix source freshness warnings in kubernetes execution mode by @Pawel-Drabczyk in #1859 * Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877 * Fix cache behaviour when DAG name contains "." by @tatiana in #1908 **Documentation** * Fix ``contributing.rst`` docs by @tatiana in #1785 * Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790 * Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814 * Update testing behavior file with ``ExecutionMode.KUBERNETES`` by @LuigiCerone in #1813 * Add step to fork repo in contributing guide by @pankajastro in #1808 * Fix ``depends_on`` attribute by @benedikt-buchert in #1837 * Fix character name by @ThePsyjo in #1860 * Update suggested MWAA startup script by @jaklan in #1884 * Make implementation & docs consistent regarding ``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878 **Others** * Set retries to 0 in example DAGs by @pankajkoti in #1782 * Fix ``test_async_example_dag_without_setup_task`` tests by @pankajastro in #1788 * Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in #1786 * Upgrade Python and Airflow used to run MyPy checks by @tatiana in #1796 * Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and @tatiana in #1778 * Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9 & 1.10 by @tatiana in #1820 * Fix broken CI due to Pydantic conflicts by @tatiana in #1809 * Drop Python 3.8 Support by @pankajastro in #1852 * Add Airflow 2.11 to the test matrix by @tatiana in #1807 * Require Authorize for all jobs on pull requests from external contributors in CI by @pankajkoti in #1861 * Leverage Trusted Publisher Management when publishing PyPI package by @tatiana in #1862 * CI: Add back accidentally deleted python-version matrix for running unit tests by @pankajkoti in #1872 * Remove commented code and fix mypy failures by @pankajkoti in #1876 * Add Zizmor analysis GitHub action by @pankajkoti in #1870 * Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti in #1880 * Add deprecation warning for ``LoadMode.CUSTOM`` parser by @duongphannamhung in #1885 * CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in #1871 * Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #1890 * Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #1889 * Resolve Zizmor error alerts for unpinned image references; mark alert for pull_request_target ignored by @pankajkoti in #1888 * Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by @tatiana in #1902 * Add dbt-core 1.10 to test matrix by @tatiana in #1767 * Pin package dbt-databricks by @pankajastro in #1909 * Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests in CI by @pankajastro in #1900 * Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892, #1901 * Dependabot updates: #1904 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

@pankajkoti

**Bug Fixes** * Fix task instance ``try_number`` attribute for Airflow 3 compatibility by @pankajkoti in #1781 * Fix rendered template override logic when ``should_store_compiled_sql=False`` to restore pre-refactor behaviour by @pankajkoti in #1777 * Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by @ramonvermeulen in #1783 * Fix dbt Docs page height by @1cadumagalhaes in #1793 * Add support to base64 encoded pem in Snowflake profiles by @brunocmartins in #1801 * Allow to disable owner inheritance from dbt into airflow DAG owners by @CorsettiS in #1787 * Fix Kubernetes Pod Operator conversion of ``container_resources`` to ``resources`` by @johnhoran in #1821 * Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in #1822 * Fix source freshness warnings in kubernetes execution mode by @Pawel-Drabczyk in #1859 * Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877 * Fix cache behaviour when DAG name contains "." by @tatiana in #1908 **Documentation** * Fix ``contributing.rst`` docs by @tatiana in #1785 * Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790 * Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814 * Update testing behavior file with ``ExecutionMode.KUBERNETES`` by @LuigiCerone in #1813 * Add step to fork repo in contributing guide by @pankajastro in #1808 * Fix ``depends_on`` attribute by @benedikt-buchert in #1837 * Fix character name by @ThePsyjo in #1860 * Update suggested MWAA startup script by @jaklan in #1884 * Make implementation & docs consistent regarding ``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878 **Others** * Set retries to 0 in example DAGs by @pankajkoti in #1782 * Fix ``test_async_example_dag_without_setup_task`` tests by @pankajastro in #1788 * Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in #1786 * Upgrade Python and Airflow used to run MyPy checks by @tatiana in #1796 * Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and @tatiana in #1778 * Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9 & 1.10 by @tatiana in #1820 * Fix broken CI due to Pydantic conflicts by @tatiana in #1809 * Drop Python 3.8 Support by @pankajastro in #1852 * Add Airflow 2.11 to the test matrix by @tatiana in #1807 * Require Authorize for all jobs on pull requests from external contributors in CI by @pankajkoti in #1861 * Leverage Trusted Publisher Management when publishing PyPI package by @tatiana in #1862 * CI: Add back accidentally deleted python-version matrix for running unit tests by @pankajkoti in #1872 * Remove commented code and fix mypy failures by @pankajkoti in #1876 * Add Zizmor analysis GitHub action by @pankajkoti in #1870 * Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti in #1880 * Add deprecation warning for ``LoadMode.CUSTOM`` parser by @duongphannamhung in #1885 * CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in #1871 * Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #1890 * Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #1889 * Resolve Zizmor error alerts for unpinned image references; mark alert for pull_request_target ignored by @pankajkoti in #1888 * Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by @tatiana in #1902 * Add dbt-core 1.10 to test matrix by @tatiana in #1767 * Pin package dbt-databricks by @pankajastro in #1909 * Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests in CI by @pankajastro in #1900 * Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892, #1901 * Dependabot updates: #1904 Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

saurav-rty · 2025-10-14T11:21:14Z

Hi @pankajkoti,

Is it possible to add the Airflow 3 and Airflow 2 compatibility for upload_to_azure_wasb as well

# Airflow 3 and Airflow 2 compatibility, respectively: try_number = getattr(context["task_instance"], "try_number") or getattr(context["task_instance"], "_try_number")

astronomer-cosmos/cosmos/io.py

Line 126 in e76d295

f"/{context['task_instance']._try_number}"

pankajkoti temporarily deployed to internal May 26, 2025 12:49 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal May 26, 2025 13:34 — with GitHub Actions Inactive

pankajkoti force-pushed the assert-example-dagrun-state branch from 6cb222c to 8ae3287 Compare May 26, 2025 13:49

pankajkoti temporarily deployed to internal May 26, 2025 13:49 — with GitHub Actions Inactive

This was referenced May 27, 2025

Fix task instance try_number attribute for Airflow 3 compatibility #1781

Merged

Set retries to 0 in example DAGs #1782

Merged

pankajkoti commented May 27, 2025

View reviewed changes

Comment thread cosmos/io.py

Comment thread dev/dags/cosmos_callback_dag.py

Comment thread dev/dags/example_cosmos_dbt_build.py

Comment thread dev/dags/example_source_rendering.py

Comment thread tests/operators/test_local.py

Comment thread tests/test_io.py

pankajkoti temporarily deployed to internal May 27, 2025 09:29 — with GitHub Actions Inactive

tatiana approved these changes May 27, 2025

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label May 27, 2025

pankajkoti temporarily deployed to internal May 27, 2025 09:54 — with GitHub Actions Inactive

pankajastro approved these changes May 27, 2025

View reviewed changes

pankajkoti temporarily deployed to internal May 28, 2025 08:41 — with GitHub Actions Inactive

pankajkoti and others added 3 commits May 28, 2025 16:47

Assert Example DAGs' DagRunState

40bcd48

Apply suggestions from code review

64ed30a

tatiana force-pushed the assert-example-dagrun-state branch from 14bb654 to 353214e Compare May 28, 2025 15:47

tatiana temporarily deployed to internal May 28, 2025 15:47 — with GitHub Actions Inactive

pankajkoti commented May 28, 2025

View reviewed changes

Merge branch 'main' into assert-example-dagrun-state

c1206cd

tatiana temporarily deployed to internal June 3, 2025 14:02 — with GitHub Actions Inactive

tatiana and others added 5 commits June 3, 2025 15:04

Update dev/dags/cosmos_callback_dag.py

4c52924

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Update cosmos/io.py

3269a8c

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Update tests/test_io.py

73ae439

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Update tests/operators/test_local.py

05ba199

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Update dev/dags/example_cosmos_dbt_build.py

a01ff5c

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

tatiana temporarily deployed to internal June 12, 2025 12:52 — with GitHub Actions Inactive

Troubleshoot in the CI

a9c6e25

tatiana temporarily deployed to internal June 12, 2025 13:59 — with GitHub Actions Inactive

Troubleshoot in the CI

6c121c3

tatiana temporarily deployed to internal June 12, 2025 14:15 — with GitHub Actions Inactive

Attempt to avoid error seen in

9306913

https://github.com/astronomer/astronomer-cosmos/actions/runs/15613004782/job/43978485756

tatiana temporarily deployed to internal June 12, 2025 14:58 — with GitHub Actions Inactive

Fix unittest

521881e

tatiana temporarily deployed to internal June 12, 2025 15:07 — with GitHub Actions Inactive

Give more visibility of what is going on

38b02c6

tatiana temporarily deployed to internal June 12, 2025 16:33 — with GitHub Actions Inactive

Attempt to solve CI issue with Py2.6

408c568

tatiana temporarily deployed to internal June 12, 2025 17:04 — with GitHub Actions Inactive

tatiana marked this pull request as ready for review June 13, 2025 08:36

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. area:testing Related to testing, like unit tests, integration tests, etc labels Jun 13, 2025

tatiana changed the title ~~Assert Example DAGs' DagRunState~~ Assert example DAGs' DagRunState and fix issues Jun 13, 2025

tatiana merged commit 67be637 into main Jun 13, 2025
92 of 93 checks passed

tatiana deleted the assert-example-dagrun-state branch June 13, 2025 08:37

pankajkoti commented Jun 13, 2025

View reviewed changes

pankajkoti added this to the Cosmos 1.10.2 milestone Jun 13, 2025

tatiana mentioned this pull request Aug 7, 2025

Release 1.10.2 #1911

Merged

pankajkoti mentioned this pull request Aug 8, 2025

Release 1.10.2 #1912

Merged

Conversation

pankajkoti commented May 26, 2025 • edited by tatiana Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify Bot commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

cloudflare-workers-and-pages Bot commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying astronomer-cosmos with Cloudflare Pages

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

pankajkoti Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

saurav-rty commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pankajkoti commented May 26, 2025 •

edited by tatiana

Loading

netlify Bot commented May 26, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented May 26, 2025 •

edited

Loading

codecov Bot commented Jun 12, 2025 •

edited

Loading

saurav-rty commented Oct 14, 2025 •

edited

Loading