Skip to content

Fix dataset tests failing in Airflow 3#1716

Merged
tatiana merged 14 commits into
mainfrom
issue-1704
Apr 29, 2025
Merged

Fix dataset tests failing in Airflow 3#1716
tatiana merged 14 commits into
mainfrom
issue-1704

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Apr 28, 2025

Previously, the following tests were failing:

FAILED tests/operators/test_local.py::test_run_operator_dataset_inlets_and_outlets_airflow_210_onwards - ModuleNotFoundError: No module named 'airflow.models.dataset'

Details:

_______ test_run_operator_dataset_inlets_and_outlets_airflow_210_onwards _______

caplog = <_pytest.logging.LogCaptureFixture object at 0x7f1234faf3a0>

    @pytest.mark.skipif(
        version.parse(airflow_version) < version.parse("2.10"),
        reason="From Airflow 2.10 onwards, we started using DatasetAlias, which changed this behaviour.",
    )
    @pytest.mark.integration
    def test_run_operator_dataset_inlets_and_outlets_airflow_210_onwards(caplog):
>       from airflow.models.dataset import DatasetAliasModel
E       ModuleNotFoundError: No module named 'airflow.models.dataset'

tests/operators/test_local.py:471: ModuleNotFoundError

The test test_run_operator_dataset_url_encoded_names will be handled in the PR #1713

Closes: #1704

@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 28, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 3168623
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6810d8a0d958f400080d5527

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Apr 28, 2025

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: 3168623
Status: ✅  Deploy successful!
Preview URL: https://998af9d3.astronomer-cosmos.pages.dev
Branch Preview URL: https://issue-1704.astronomer-cosmos.pages.dev

View logs

@tatiana tatiana marked this pull request as ready for review April 29, 2025 08:24
Copilot AI review requested due to automatic review settings April 29, 2025 08:24
@dosubot dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Apr 29, 2025
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes failing dataset tests under Airflow 3 by updating import references and assertions to use the Asset alias model instead of the old Dataset alias model, and by removing a skip marker that prevented testing for Airflow 3.

  • Updated import logic for AssetAliasModel in test_run_operator_dataset_airflow_210_onwards
  • Updated test assertions to use AssetAliasModel instead of DatasetAliasModel
  • Removed a skip marker to allow testing dataset emission under Airflow 3
Comments suppressed due to low confidence (1)

tests/operators/test_local.py:576

  • The removal of the skip marker for Airflow 3 raises a concern about the test's compatibility; please verify that the updated logic behaves as expected under Airflow 3 conditions.
# TODO: Make test compatible with Airflow 3.0. Issue:https://github.com/astronomer/astronomer-cosmos/issues/1705

Comment thread tests/operators/test_local.py
@dosubot dosubot Bot added area:datasets Related to the Airflow datasets feature/module area:testing Related to testing, like unit tests, integration tests, etc labels Apr 29, 2025
@tatiana
Copy link
Copy Markdown
Collaborator Author

tatiana commented Apr 29, 2025

The only tests that are failing will work once #1725 is merged and this PR is rebased.

@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Apr 29, 2025
tatiana and others added 3 commits April 29, 2025 13:37
Airflow3 pre-emptively tries to validate rendered fields and also
validates that the path exists in case of Path like objects specified in
rendered fields. Hence, we ensure that the path specified is created
first.


closes: #1707 

Co-authored-by: Pankaj Singh 98807258+pankajastro@users.noreply.github.com

Co-authored-by: Pankaj Singh
<98807258+pankajastro@users.noreply.github.com>
@dosubot dosubot Bot removed the size:M This PR changes 30-99 lines, ignoring generated files. label Apr 29, 2025
@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Apr 29, 2025
tatiana added a commit that referenced this pull request Apr 29, 2025
…uments` (#1725)

Example of Action Link:
https://github.com/astronomer/astronomer-cosmos/actions/runs/14703034847/job/41256451294?pr=1646#step:7:4023

A few tests in the tests/operators/test_local.py failing with the below
error
```
FAILED tests/operators/test_local.py::test_run_operator_dataset_url_encoded_names - TypeError: generate_run_id() takes 0 positional arguments but 2 were given
FAILED tests/operators/test_local.py::test_run_operator_caches_partial_parsing - TypeError: generate_run_id() takes 0 positional arguments but 2 were given
FAILED tests/operators/test_local.py::test_run_test_operator_with_callback[InvocationMode.SUBPROCESS] - TypeError: generate_run_id() takes 0 positional arguments but 2 were given
FAILED tests/operators/test_local.py::test_run_test_operator_with_callback[InvocationMode.DBT_RUNNER] - TypeError: generate_run_id() takes 0 positional arguments but 2 were given
FAILED tests/operators/test_local.py::test_run_test_operator_without_callback[InvocationMode.SUBPROCESS] - TypeError: generate_run_id() takes 0 positional arguments but 2 were given
FAILED tests/operators/test_local.py::test_run_test_operator_without_callback[InvocationMode.DBT_RUNNER] - TypeError: generate_run_id() takes 0 positional arguments but 2 were given
```

The only test this PR is not uncommenting is
`test_run_operator_dataset_url_encoded_names` - that is being addressed in PR #1716

Closes: #1705
@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Apr 29, 2025
Copy link
Copy Markdown
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for fixing this!

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 29, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.56%. Comparing base (6eaf032) to head (3168623).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1716   +/-   ##
=======================================
  Coverage   97.56%   97.56%           
=======================================
  Files          83       83           
  Lines        5176     5176           
=======================================
  Hits         5050     5050           
  Misses        126      126           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tatiana tatiana merged commit 21b36eb into main Apr 29, 2025
93 checks passed
@tatiana tatiana deleted the issue-1704 branch April 29, 2025 16:09
@pankajkoti pankajkoti mentioned this pull request Apr 30, 2025
@tatiana tatiana added this to the Cosmos 1.10.0 milestone Apr 30, 2025
tatiana added a commit that referenced this pull request May 1, 2025
Features

* Airflow 3 support
* Support running ``dbt deps`` incrementally to pre-defined
``dbt_packages`` by @tatiana in #1668 and #1670
* Add ``DuckDB`` profile mapping by @prithvijitguha and @pankajastro in
#1553
* Implement DBT exposure selector by ghjklw #1717

Bug Fixes

* Fix ``test_indirect_selection`` flag to be propagated in case of
``TestBehavior.BUILD`` by @corsettigyg in #1663
* Fix ``select`` clause in the case of detached tests by @anyapriya in
#1680
* Operator argument fixes by @johnhoran in #1648


Airflow 3 Support

* Support rendering DbtDag in Airflow 3 by @tatiana and @ashb in #1657
* Refactor Rendered Task Instance Fields (RTIF) handling for Airflow 2.x
and 3.x by @pankajkoti in #1661
* Run cosmos operator in Airflow 3 by @pankajastro in #1642
* Fix ``python_virtualenv.prepare_env`` top-level import for Airflow 3
by @pankajkoti in #1678
* Fix Variable not found issue in Airflow 3 by @tatiana in #1684
* Disable CosmosPlugin on Airflow 3 setup by @pankajkoti in #1692, #1698
* Use ``schedule`` param in example DAGs instead of the 2.10 deprecated
and 3.0 removed ``schedule_interval`` by @pankajkoti in #1701
* Ensure ``virtualenv_dir`` path exists by @pankajkoti in #1724
* Support emitting Assets with Airflow 3 by @tatiana in #1713
* Add docs on Airflow 3 compatibility by @pankajkoti and @tatiana in
#1731
* Introduce, test and document asset/dataset breaking change by @tatiana
in #1672
* Improve dataset/asset driven scheduling documentation by @tatiana in
#1729

Enhancements

* Allow multiple callbacks by @corsettigyg #1693
* Refactor kubernetes warning callback handling by @canbekley in #1681

Documentation

* Add documentation related to ``copy_dbt_packages`` by @tatiana in
#1671
* Make wording and command consistent in the contributing doc by
@pankajkoti in #1697
* Add MonteCarlo callback example for importing dbt artifacts by
@corsettigyg #1695
* Change async feature to be non-experimental by @tatiana in #1732

Others

* Add sample ``dbt_packages`` to validate incremental ``dbt deps`` by
@tatiana in #1669
* Add kubernetes execution mode example in Airflow 3 by @pankajastro in
#1667
* Check only major version until Airflow 3 stable release by
@pankajastro in #1665
* Install Airflow from main branch by @pankajastro in #1660
* Add dev tool for Airflow 3 by @pankajastro and @tatiana in #1627
* Improve Airflow 3 tooling by @pankajastro in #1656
* Skip associating ``openlineage_events_completes`` to ``ti`` in Airflow
3 by @pankajkoti in #1662
* Add .gitignore file for the scripts/airflow3 directory by @pankajkoti
in #1658
* Remove ``original_jaffle_shop`` dbt project by @pankajkoti in #1676
* Fix or ignore type check error by @pankajastro in #1687
* Run virtualenv example with Airflow 3 tooling by @pankajastro in #1686
* Enable running setup/teardown tasks with Async execution DAG with
Airflow 3 tooling by @pankajastro in #1696
* Enable integration tests for the DuckDB adapter by @pankajastro in
#1699
* Add Airflow 3 tests matrix entries in CI by @pankajkoti in #1646
* Use a different way to get tasks count for asserting test_perf_dag by
@pankajkoti in #1714
* Reinstall Airflow 3 dependency on ``pydantic>=2.11`` for dbt adapter
versions 1.6 & 1.9 by @pankajkoti in #1715
* Fix outdated ``echo`` in Airflow 3 tooling script #1700
* Add files not needed for git tracking to .gitignore by @pankajkoti in
#1723
* Use latest minor versions for dbt adapters to get in compatibility
fixes by @pankajkoti in #1719
* Fix Airflow 3 tests raising generate_run_id() takes 0 positional
arguments by @tatiana in #1725
* Fix dataset tests failing in Airflow 3 by @tatiana in #1716
* Enable example DAGs to run in CI that were disabled in PR #1646 by
@pankajkoti in #1726
* Pre-commit updates: #1666, #1653, #1641, #1682, #1720


Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Pankaj Singh
<98807258+pankajastro@users.noreply.github.com>

---------

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:datasets Related to the Airflow datasets feature/module area:testing Related to testing, like unit tests, integration tests, etc size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataset test failing on no module named error AF 3 CI

3 participants