Use latest minor versions for dbt adapters to get in compatibility fixes by pankajkoti · Pull Request #1719 · astronomer/astronomer-cosmos

pankajkoti · 2025-04-28T17:28:46Z

While the initial x.x.0 versions of dbt adapters may have compatibility issues with certain dependencies, I observed that these incompatibilities are progressively addressed in later patch releases. The issue reported in #1709 appears to stem from dbt-bigquery accessing a protected member google.cloud.bigquery._helpers._CELLDATA_FROM_JSON, which was removed in google-cloud-bigquery 3.31.0. This access I believe was introduced in dbt-bigquery#974 and later removed in dbt-bigquery#1061, with the fix subsequently back-ported to the 1.7 series via dbt-bigquery#1074.
Therefore, I believe that relying on the latest patch versions is a better approach to avoid such issues, rather than individually resolving discrepancies. This PR aligns with that strategy.

closes: #1709

cloudflare-workers-and-pages · 2025-04-28T17:28:49Z

Deploying astronomer-cosmos with Cloudflare Pages

Latest commit:	`39e0d6c`
Status:	⚡️ Build in progress...

View logs

netlify · 2025-04-28T17:29:02Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`39e0d6c`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/681086682f12590008c9cfa6

…ters

codecov · 2025-04-28T17:58:25Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.50%. Comparing base (fa943d9) to head (39e0d6c).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1719   +/-   ##
=======================================
  Coverage   97.50%   97.50%           
=======================================
  Files          83       83           
  Lines        5049     5049           
=======================================
  Hits         4923     4923           
  Misses        126      126

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ibility

pankajkoti · 2025-04-29T05:31:17Z

Successful job run link: https://github.com/astronomer/astronomer-cosmos/actions/runs/14714551325/job/41294867003

tatiana

Excellent analysis and implementation, @pankajkoti ! Thank you very much.

…xes (#1719) While the initial x.x.0 versions of dbt adapters may have compatibility issues with certain dependencies, I observed that these incompatibilities are progressively addressed in later patch releases. The issue reported in #1709 appears to stem from dbt-bigquery accessing a protected member `google.cloud.bigquery._helpers._CELLDATA_FROM_JSON`, which was removed in `google-cloud-bigquery 3.31.0`. This access I believe was introduced in [dbt-bigquery#974](dbt-labs/dbt-bigquery#974) and later removed in [dbt-bigquery#1061](dbt-labs/dbt-bigquery#1061), with the fix subsequently back-ported to the 1.7 series via [dbt-bigquery#1074](dbt-labs/dbt-bigquery#1074). Therefore, I believe that relying on the latest patch versions is a better approach to avoid such issues, rather than individually resolving discrepancies. This PR aligns with that strategy. closes: #1709 --------- Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

@tatiana

Features * Airflow 3 support * Support running ``dbt deps`` incrementally to pre-defined ``dbt_packages`` by @tatiana in #1668 and #1670 * Add ``DuckDB`` profile mapping by @prithvijitguha and @pankajastro in #1553 * Implement DBT exposure selector by ghjklw #1717 Bug Fixes * Fix ``test_indirect_selection`` flag to be propagated in case of ``TestBehavior.BUILD`` by @corsettigyg in #1663 * Fix ``select`` clause in the case of detached tests by @anyapriya in #1680 * Operator argument fixes by @johnhoran in #1648 Airflow 3 Support * Support rendering DbtDag in Airflow 3 by @tatiana and @ashb in #1657 * Refactor Rendered Task Instance Fields (RTIF) handling for Airflow 2.x and 3.x by @pankajkoti in #1661 * Run cosmos operator in Airflow 3 by @pankajastro in #1642 * Fix ``python_virtualenv.prepare_env`` top-level import for Airflow 3 by @pankajkoti in #1678 * Fix Variable not found issue in Airflow 3 by @tatiana in #1684 * Disable CosmosPlugin on Airflow 3 setup by @pankajkoti in #1692, #1698 * Use ``schedule`` param in example DAGs instead of the 2.10 deprecated and 3.0 removed ``schedule_interval`` by @pankajkoti in #1701 * Ensure ``virtualenv_dir`` path exists by @pankajkoti in #1724 * Support emitting Assets with Airflow 3 by @tatiana in #1713 * Add docs on Airflow 3 compatibility by @pankajkoti and @tatiana in #1731 * Introduce, test and document asset/dataset breaking change by @tatiana in #1672 * Improve dataset/asset driven scheduling documentation by @tatiana in #1729 Enhancements * Allow multiple callbacks by @corsettigyg #1693 * Refactor kubernetes warning callback handling by @canbekley in #1681 Documentation * Add documentation related to ``copy_dbt_packages`` by @tatiana in #1671 * Make wording and command consistent in the contributing doc by @pankajkoti in #1697 * Add MonteCarlo callback example for importing dbt artifacts by @corsettigyg #1695 * Change async feature to be non-experimental by @tatiana in #1732 Others * Add sample ``dbt_packages`` to validate incremental ``dbt deps`` by @tatiana in #1669 * Add kubernetes execution mode example in Airflow 3 by @pankajastro in #1667 * Check only major version until Airflow 3 stable release by @pankajastro in #1665 * Install Airflow from main branch by @pankajastro in #1660 * Add dev tool for Airflow 3 by @pankajastro and @tatiana in #1627 * Improve Airflow 3 tooling by @pankajastro in #1656 * Skip associating ``openlineage_events_completes`` to ``ti`` in Airflow 3 by @pankajkoti in #1662 * Add .gitignore file for the scripts/airflow3 directory by @pankajkoti in #1658 * Remove ``original_jaffle_shop`` dbt project by @pankajkoti in #1676 * Fix or ignore type check error by @pankajastro in #1687 * Run virtualenv example with Airflow 3 tooling by @pankajastro in #1686 * Enable running setup/teardown tasks with Async execution DAG with Airflow 3 tooling by @pankajastro in #1696 * Enable integration tests for the DuckDB adapter by @pankajastro in #1699 * Add Airflow 3 tests matrix entries in CI by @pankajkoti in #1646 * Use a different way to get tasks count for asserting test_perf_dag by @pankajkoti in #1714 * Reinstall Airflow 3 dependency on ``pydantic>=2.11`` for dbt adapter versions 1.6 & 1.9 by @pankajkoti in #1715 * Fix outdated ``echo`` in Airflow 3 tooling script #1700 * Add files not needed for git tracking to .gitignore by @pankajkoti in #1723 * Use latest minor versions for dbt adapters to get in compatibility fixes by @pankajkoti in #1719 * Fix Airflow 3 tests raising generate_run_id() takes 0 positional arguments by @tatiana in #1725 * Fix dataset tests failing in Airflow 3 by @tatiana in #1716 * Enable example DAGs to run in CI that were disabled in PR #1646 by @pankajkoti in #1726 * Pre-commit updates: #1666, #1653, #1641, #1682, #1720 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

Use latest minor versions for dbt adapters to get in compatibility fixes

14cc9f9

pankajkoti temporarily deployed to internal April 28, 2025 17:28 — with GitHub Actions Inactive

pankajkoti commented Apr 28, 2025

View reviewed changes

Comment thread scripts/test/integration-dbt-async.sh

Run tests in CI

0772c2a

pankajkoti temporarily deployed to internal April 28, 2025 17:34 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal April 28, 2025 17:35 — with GitHub Actions Inactive

Correct installation with upper bounds for minor versions of dbt adap…

2b56397

…ters

Restrict google-cloud-bigquery<3.31.0 for dbt-bigquery 1.6 for compat…

86c3751

…ibility

pankajkoti temporarily deployed to internal April 28, 2025 18:00 — with GitHub Actions Inactive

pankajkoti commented Apr 28, 2025

View reviewed changes

Comment thread scripts/test/integration-dbt-async.sh Outdated

pankajkoti marked this pull request as ready for review April 28, 2025 18:50

dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. area:ci Related to CI, Github Actions, or other continuous integration tools area:dependencies Related to dependencies, like Python packages, library versions, etc labels Apr 28, 2025

pankajkoti commented Apr 28, 2025

View reviewed changes

Comment thread scripts/test/integration-dbt-async.sh

pankajkoti commented Apr 28, 2025

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

pankajkoti requested review from pankajastro and tatiana April 28, 2025 19:05

Update .github/workflows/test.yml

bf66c50

pankajkoti temporarily deployed to internal April 29, 2025 05:30 — with GitHub Actions Inactive

Update scripts/test/integration-dbt-async.sh

7d56029

pankajkoti temporarily deployed to internal April 29, 2025 05:30 — with GitHub Actions Inactive

tatiana approved these changes Apr 29, 2025

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Apr 29, 2025

Merge branch 'main' into enable-ci-dbt-async-1.7-af3

39e0d6c

tatiana temporarily deployed to internal April 29, 2025 07:57 — with GitHub Actions Inactive

tatiana approved these changes Apr 29, 2025

View reviewed changes

tatiana merged commit f16afe2 into main Apr 29, 2025
87 of 89 checks passed

tatiana deleted the enable-ci-dbt-async-1.7-af3 branch April 29, 2025 07:58

pankajkoti mentioned this pull request Apr 30, 2025

Release 1.10.0 #1672

Merged

tatiana added this to the Cosmos 1.10.0 milestone Apr 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use latest minor versions for dbt adapters to get in compatibility fixes#1719

Use latest minor versions for dbt adapters to get in compatibility fixes#1719
tatiana merged 7 commits into
mainfrom
enable-ci-dbt-async-1.7-af3

pankajkoti commented Apr 28, 2025 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

codecov Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Apr 29, 2025

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

pankajkoti commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying astronomer-cosmos with Cloudflare Pages

Uh oh!

netlify Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

Uh oh!

codecov Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Apr 29, 2025

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pankajkoti commented Apr 28, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented Apr 28, 2025 •

edited

Loading

netlify Bot commented Apr 28, 2025 •

edited

Loading

codecov Bot commented Apr 28, 2025 •

edited

Loading