Reinstall airflow dependency on pydantic>=2.11 for dbt adapater versions 1.6 & 1.9 by pankajkoti · Pull Request #1715 · astronomer/astronomer-cosmos

pankajkoti · 2025-04-28T14:25:18Z

apache-airflow-core 3.0.0 requires pydantic>=2.11.0, but the installed dbt adapters in case of version 1.6 and 1.9 install
pydantic 1.10.22 which make it incompatible.
With pydantic 1.10.22 we get the below error

File "<...>/hatch/env/virtual/astronomer-cosmos/D9FI7Men/tests.py3.11-3.0-1.6/lib/python3.11/site-packages/cadwyn/_utils.py", line 5, in <module>
     from pydantic._internal._decorators import unwrap_wrapped_function
ModuleNotFoundError: No module named 'pydantic._internal'

Hence, we re-install pydantic with the required minimum version after installing dbt adapters in case of 1.6 & 1.9 to be compatible with Airflow 3 dependencies.

closes: #1708

netlify · 2025-04-28T14:25:36Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`4a06a4d`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/680fa91a6726d90008a8bcfe

…ons 1.6 & 1.9

cloudflare-workers-and-pages · 2025-04-28T14:26:57Z

Deploying astronomer-cosmos with Cloudflare Pages

Latest commit:	`4a06a4d`
Status:	⚡️ Build in progress...

View logs

pankajkoti · 2025-04-28T15:07:53Z

Successful jobs runs:
dbt 1.6 https://github.com/astronomer/astronomer-cosmos/actions/runs/14710907329/job/41282853864?pr=1715
dbt 1.9 https://github.com/astronomer/astronomer-cosmos/actions/runs/14710907329/job/41282855770?pr=1715

codecov · 2025-04-28T15:19:15Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.50%. Comparing base (25dd549) to head (4a06a4d).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1715   +/-   ##
=======================================
  Coverage   97.50%   97.50%           
=======================================
  Files          83       83           
  Lines        5049     5049           
=======================================
  Hits         4923     4923           
  Misses        126      126

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tatiana

Looks great, thanks, @pankajkoti !
Dependecies resolution can be a nightmare

@tatiana

Features * Airflow 3 support * Support running ``dbt deps`` incrementally to pre-defined ``dbt_packages`` by @tatiana in #1668 and #1670 * Add ``DuckDB`` profile mapping by @prithvijitguha and @pankajastro in #1553 * Implement DBT exposure selector by ghjklw #1717 Bug Fixes * Fix ``test_indirect_selection`` flag to be propagated in case of ``TestBehavior.BUILD`` by @corsettigyg in #1663 * Fix ``select`` clause in the case of detached tests by @anyapriya in #1680 * Operator argument fixes by @johnhoran in #1648 Airflow 3 Support * Support rendering DbtDag in Airflow 3 by @tatiana and @ashb in #1657 * Refactor Rendered Task Instance Fields (RTIF) handling for Airflow 2.x and 3.x by @pankajkoti in #1661 * Run cosmos operator in Airflow 3 by @pankajastro in #1642 * Fix ``python_virtualenv.prepare_env`` top-level import for Airflow 3 by @pankajkoti in #1678 * Fix Variable not found issue in Airflow 3 by @tatiana in #1684 * Disable CosmosPlugin on Airflow 3 setup by @pankajkoti in #1692, #1698 * Use ``schedule`` param in example DAGs instead of the 2.10 deprecated and 3.0 removed ``schedule_interval`` by @pankajkoti in #1701 * Ensure ``virtualenv_dir`` path exists by @pankajkoti in #1724 * Support emitting Assets with Airflow 3 by @tatiana in #1713 * Add docs on Airflow 3 compatibility by @pankajkoti and @tatiana in #1731 * Introduce, test and document asset/dataset breaking change by @tatiana in #1672 * Improve dataset/asset driven scheduling documentation by @tatiana in #1729 Enhancements * Allow multiple callbacks by @corsettigyg #1693 * Refactor kubernetes warning callback handling by @canbekley in #1681 Documentation * Add documentation related to ``copy_dbt_packages`` by @tatiana in #1671 * Make wording and command consistent in the contributing doc by @pankajkoti in #1697 * Add MonteCarlo callback example for importing dbt artifacts by @corsettigyg #1695 * Change async feature to be non-experimental by @tatiana in #1732 Others * Add sample ``dbt_packages`` to validate incremental ``dbt deps`` by @tatiana in #1669 * Add kubernetes execution mode example in Airflow 3 by @pankajastro in #1667 * Check only major version until Airflow 3 stable release by @pankajastro in #1665 * Install Airflow from main branch by @pankajastro in #1660 * Add dev tool for Airflow 3 by @pankajastro and @tatiana in #1627 * Improve Airflow 3 tooling by @pankajastro in #1656 * Skip associating ``openlineage_events_completes`` to ``ti`` in Airflow 3 by @pankajkoti in #1662 * Add .gitignore file for the scripts/airflow3 directory by @pankajkoti in #1658 * Remove ``original_jaffle_shop`` dbt project by @pankajkoti in #1676 * Fix or ignore type check error by @pankajastro in #1687 * Run virtualenv example with Airflow 3 tooling by @pankajastro in #1686 * Enable running setup/teardown tasks with Async execution DAG with Airflow 3 tooling by @pankajastro in #1696 * Enable integration tests for the DuckDB adapter by @pankajastro in #1699 * Add Airflow 3 tests matrix entries in CI by @pankajkoti in #1646 * Use a different way to get tasks count for asserting test_perf_dag by @pankajkoti in #1714 * Reinstall Airflow 3 dependency on ``pydantic>=2.11`` for dbt adapter versions 1.6 & 1.9 by @pankajkoti in #1715 * Fix outdated ``echo`` in Airflow 3 tooling script #1700 * Add files not needed for git tracking to .gitignore by @pankajkoti in #1723 * Use latest minor versions for dbt adapters to get in compatibility fixes by @pankajkoti in #1719 * Fix Airflow 3 tests raising generate_run_id() takes 0 positional arguments by @tatiana in #1725 * Fix dataset tests failing in Airflow 3 by @tatiana in #1716 * Enable example DAGs to run in CI that were disabled in PR #1646 by @pankajkoti in #1726 * Pre-commit updates: #1666, #1653, #1641, #1682, #1720 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

pankajkoti temporarily deployed to internal April 28, 2025 14:25 — with GitHub Actions Inactive

Reinstall airflow dependency on pydantic>=2.11 for dbt adapater versi…

7dc5ef3

…ons 1.6 & 1.9

pankajkoti force-pushed the enable-ci-dbt-async-1.6-1.9-af3 branch from 6bc5b9c to 7dc5ef3 Compare April 28, 2025 14:26

pankajkoti temporarily deployed to internal April 28, 2025 14:26 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal April 28, 2025 14:27 — with GitHub Actions Inactive

Print shell

5fa106c

pankajkoti temporarily deployed to internal April 28, 2025 14:34 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal April 28, 2025 14:35 — with GitHub Actions Inactive

Use bash for dbt-async job

623fb50

pankajkoti temporarily deployed to internal April 28, 2025 14:38 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal April 28, 2025 14:39 — with GitHub Actions Inactive

pankajkoti commented Apr 28, 2025

View reviewed changes

Comment thread scripts/test/integration-dbt-async.sh Outdated

pankajkoti commented Apr 28, 2025

View reviewed changes

Comment thread pyproject.toml Outdated

Revert bash to sh

f313896

pankajkoti temporarily deployed to internal April 28, 2025 14:55 — with GitHub Actions Inactive

Update scripts/test/integration-dbt-async.sh

08bf17f

pankajkoti temporarily deployed to internal April 28, 2025 14:55 — with GitHub Actions Inactive

pankajkoti marked this pull request as ready for review April 28, 2025 14:58

dosubot Bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Apr 28, 2025

pankajkoti requested review from pankajastro and tatiana April 28, 2025 14:58

dosubot Bot added the area:dependencies Related to dependencies, like Python packages, library versions, etc label Apr 28, 2025

pankajkoti commented Apr 28, 2025

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

tatiana approved these changes Apr 28, 2025

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Apr 28, 2025

Update .github/workflows/test.yml

4a06a4d

pankajkoti temporarily deployed to internal April 28, 2025 16:13 — with GitHub Actions Inactive

pankajkoti merged commit f9bdcc1 into main Apr 28, 2025
36 of 38 checks passed

pankajkoti deleted the enable-ci-dbt-async-1.6-1.9-af3 branch April 28, 2025 16:13

pankajkoti mentioned this pull request Apr 30, 2025

Release 1.10.0 #1672

Merged

tatiana added this to the Cosmos 1.10.0 milestone Apr 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reinstall airflow dependency on pydantic>=2.11 for dbt adapater versions 1.6 & 1.9#1715

Reinstall airflow dependency on pydantic>=2.11 for dbt adapater versions 1.6 & 1.9#1715
pankajkoti merged 6 commits into
mainfrom
enable-ci-dbt-async-1.6-1.9-af3

pankajkoti commented Apr 28, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Apr 28, 2025

Uh oh!

codecov Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

pankajkoti commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying astronomer-cosmos with Cloudflare Pages

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Apr 28, 2025

Uh oh!

codecov Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pankajkoti commented Apr 28, 2025 •

edited

Loading

netlify Bot commented Apr 28, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented Apr 28, 2025 •

edited

Loading

codecov Bot commented Apr 28, 2025 •

edited

Loading