Skip to content

Use a different way to get tasks count for asserting test_perf_dag#1714

Merged
pankajkoti merged 37 commits into
mainfrom
issue-1710-enable-af3-perf-dag
Apr 28, 2025
Merged

Use a different way to get tasks count for asserting test_perf_dag#1714
pankajkoti merged 37 commits into
mainfrom
issue-1710-enable-af3-perf-dag

Conversation

@pankajkoti
Copy link
Copy Markdown
Contributor

The PR makes a change in the assert of test_perf_dag to use len(dag.tasks) which works with both AF2 and AF3 for getting the task count in the DAG. We observed that task_count attribute is unavailable on DAG object in AF3 like it's available in AF3.

closes: #1710

@dosubot dosubot Bot added the area:testing Related to testing, like unit tests, integration tests, etc label Apr 28, 2025
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR aims to support Airflow 3 compatibility by updating test assertions and skipping tests that are currently failing or unsupported with Airflow 3. Key changes include:

  • Replacing the DAG task count check with len(dag.tasks) in test_perf_dag.
  • Adding skip markers and TODOs in multiple tests for Airflow 3 compatibility.
  • Updating CI configurations and project metadata (pyproject.toml and GitHub workflows) to include Airflow 3.0.

Reviewed Changes

Copilot reviewed 12 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/test_example_k8s_dags.py Adds skip markers for Airflow 3 compatibility.
tests/test_example_dags.py Updates DAG file ignores and DBT version handling for AF3.
tests/test_cache.py Skips cache tests on Airflow 3.0+.
tests/perf/test_performance.py Changes task count assertion to len(dag.tasks).
tests/operators/test_virtualenv.py Adjusts patching and assertions for virtualenv execution in AF3.
tests/operators/test_local.py Introduces skip markers for various tests under AF3.
tests/operators/test_kubernetes.py Adds skip markers for AF3 compatibility.
tests/listeners/test_dag_run_listener.py Applies skip markers for AF3 in listener tests.
tests/dbt/parser/test_output.py Updates imports to handle Airflow 3 conditional imports.
pyproject.toml Adds Airflow 3.0 to the testing matrix.
dev/dags/example_tests_multiple_parents.py Uses Path.resolve() for clearer file resolution.
.github/workflows/test.yml Updates test workflows to include Airflow 3.0 in the matrix.
Files not reviewed (5)
  • scripts/test/integration-dbt-1-5-4.sh: Language not supported
  • scripts/test/integration-dbt-async.sh: Language not supported
  • scripts/test/integration-setup.sh: Language not supported
  • scripts/test/performance-setup.sh: Language not supported
  • scripts/test/pre-install-airflow.sh: Language not supported
Comments suppressed due to low confidence (2)

tests/operators/test_virtualenv.py:112

  • [nitpick] The command prefix 'uv' used in the virtualenv execution assertions is not self-explanatory; consider adding inline documentation or renaming the expected value for improved clarity.
assert virtualenv_call[0][0][0] == "uv"

tests/test_example_dags.py:91

  • Review the DBT version check logic to ensure that the change does not unintentionally allow async DAGs to run on unsupported DBT versions.
if DBT_VERSION < Version("1.6.0"):

Comment thread .github/workflows/test.yml
@pankajkoti
Copy link
Copy Markdown
Contributor Author

pankajkoti commented Apr 28, 2025

TODO: Rebase after #1646 is merged. The branch is cut from the branch in that PR.

Main is merged back in the branch.

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Apr 28, 2025

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4ca57c9
Status:⚡️  Build in progress...

View logs

@dosubot dosubot Bot removed the size:L This PR changes 100-499 lines, ignoring generated files. label Apr 28, 2025
@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Apr 28, 2025
@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Apr 28, 2025
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks, @pankajkoti !

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Apr 28, 2025
Comment thread .github/workflows/test.yml Outdated
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 28, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.50%. Comparing base (6762ebd) to head (4ca57c9).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1714   +/-   ##
=======================================
  Coverage   97.50%   97.50%           
=======================================
  Files          83       83           
  Lines        5049     5049           
=======================================
  Hits         4923     4923           
  Misses        126      126           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Apr 28, 2025
@pankajkoti pankajkoti merged commit 25dd549 into main Apr 28, 2025
27 of 28 checks passed
@pankajkoti pankajkoti deleted the issue-1710-enable-af3-perf-dag branch April 28, 2025 13:52
@pankajkoti pankajkoti mentioned this pull request Apr 30, 2025
@tatiana tatiana added this to the Cosmos 1.10.0 milestone Apr 30, 2025
tatiana added a commit that referenced this pull request May 1, 2025
Features

* Airflow 3 support
* Support running ``dbt deps`` incrementally to pre-defined
``dbt_packages`` by @tatiana in #1668 and #1670
* Add ``DuckDB`` profile mapping by @prithvijitguha and @pankajastro in
#1553
* Implement DBT exposure selector by ghjklw #1717

Bug Fixes

* Fix ``test_indirect_selection`` flag to be propagated in case of
``TestBehavior.BUILD`` by @corsettigyg in #1663
* Fix ``select`` clause in the case of detached tests by @anyapriya in
#1680
* Operator argument fixes by @johnhoran in #1648


Airflow 3 Support

* Support rendering DbtDag in Airflow 3 by @tatiana and @ashb in #1657
* Refactor Rendered Task Instance Fields (RTIF) handling for Airflow 2.x
and 3.x by @pankajkoti in #1661
* Run cosmos operator in Airflow 3 by @pankajastro in #1642
* Fix ``python_virtualenv.prepare_env`` top-level import for Airflow 3
by @pankajkoti in #1678
* Fix Variable not found issue in Airflow 3 by @tatiana in #1684
* Disable CosmosPlugin on Airflow 3 setup by @pankajkoti in #1692, #1698
* Use ``schedule`` param in example DAGs instead of the 2.10 deprecated
and 3.0 removed ``schedule_interval`` by @pankajkoti in #1701
* Ensure ``virtualenv_dir`` path exists by @pankajkoti in #1724
* Support emitting Assets with Airflow 3 by @tatiana in #1713
* Add docs on Airflow 3 compatibility by @pankajkoti and @tatiana in
#1731
* Introduce, test and document asset/dataset breaking change by @tatiana
in #1672
* Improve dataset/asset driven scheduling documentation by @tatiana in
#1729

Enhancements

* Allow multiple callbacks by @corsettigyg #1693
* Refactor kubernetes warning callback handling by @canbekley in #1681

Documentation

* Add documentation related to ``copy_dbt_packages`` by @tatiana in
#1671
* Make wording and command consistent in the contributing doc by
@pankajkoti in #1697
* Add MonteCarlo callback example for importing dbt artifacts by
@corsettigyg #1695
* Change async feature to be non-experimental by @tatiana in #1732

Others

* Add sample ``dbt_packages`` to validate incremental ``dbt deps`` by
@tatiana in #1669
* Add kubernetes execution mode example in Airflow 3 by @pankajastro in
#1667
* Check only major version until Airflow 3 stable release by
@pankajastro in #1665
* Install Airflow from main branch by @pankajastro in #1660
* Add dev tool for Airflow 3 by @pankajastro and @tatiana in #1627
* Improve Airflow 3 tooling by @pankajastro in #1656
* Skip associating ``openlineage_events_completes`` to ``ti`` in Airflow
3 by @pankajkoti in #1662
* Add .gitignore file for the scripts/airflow3 directory by @pankajkoti
in #1658
* Remove ``original_jaffle_shop`` dbt project by @pankajkoti in #1676
* Fix or ignore type check error by @pankajastro in #1687
* Run virtualenv example with Airflow 3 tooling by @pankajastro in #1686
* Enable running setup/teardown tasks with Async execution DAG with
Airflow 3 tooling by @pankajastro in #1696
* Enable integration tests for the DuckDB adapter by @pankajastro in
#1699
* Add Airflow 3 tests matrix entries in CI by @pankajkoti in #1646
* Use a different way to get tasks count for asserting test_perf_dag by
@pankajkoti in #1714
* Reinstall Airflow 3 dependency on ``pydantic>=2.11`` for dbt adapter
versions 1.6 & 1.9 by @pankajkoti in #1715
* Fix outdated ``echo`` in Airflow 3 tooling script #1700
* Add files not needed for git tracking to .gitignore by @pankajkoti in
#1723
* Use latest minor versions for dbt adapters to get in compatibility
fixes by @pankajkoti in #1719
* Fix Airflow 3 tests raising generate_run_id() takes 0 positional
arguments by @tatiana in #1725
* Fix dataset tests failing in Airflow 3 by @tatiana in #1716
* Enable example DAGs to run in CI that were disabled in PR #1646 by
@pankajkoti in #1726
* Pre-commit updates: #1666, #1653, #1641, #1682, #1720


Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Pankaj Singh
<98807258+pankajastro@users.noreply.github.com>

---------

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:testing Related to testing, like unit tests, integration tests, etc lgtm This PR has been approved by a maintainer size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance test failing AttributeError: 'DbtDag' object has no attribute 'task_count' AF 3 CI

3 participants