Pin dbt-databricks version to < 1.9 by pankajastro · Pull Request #1376 · astronomer/astronomer-cosmos

pankajastro · 2024-12-10T15:02:01Z

Our CI is broken due to a dbt-databricks release with conflicting dependencies with Airflow>=2.8.

The latest release ofdbt-databricks has pinned the Pydantic to "pydantic>=1.10.0, <2" https://github.com/databricks/dbt-databricks/blob/main/pyproject.toml#L33-L33

This change was added in PR: databricks/dbt-databricks#843 and released on 9 December as part of dbt-databricks==1.9.0 (https://pypi.org/project/dbt-databricks/1.9.0/).

The debt-data bricks conflicts with the neAirflow version, which that requires Pydantic 2.0.

For Airflow < 2.8, things work as expected: https://github.com/astronomer/astronomer-cosmos/actions/runs/12259335118/job/34201293598.

In this PR, I'm pinning dbt-databricks version to < 1.9 to make CI green again.

Follow-up PR: #1379

cloudflare-workers-and-pages · 2024-12-10T15:04:01Z

Deploying astronomer-cosmos with Cloudflare Pages

Latest commit:	`487a6fd`
Status:	✅ Deploy successful!
Preview URL:	https://b9b27ba2.astronomer-cosmos.pages.dev
Branch Preview URL:	https://fix-1375.astronomer-cosmos.pages.dev

View logs

codecov · 2024-12-10T15:47:29Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.06%. Comparing base (fddc37f) to head (487a6fd).
Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1376      +/-   ##
==========================================
+ Coverage   96.02%   96.06%   +0.04%     
==========================================
  Files          67       67              
  Lines        4025     4042      +17     
==========================================
+ Hits         3865     3883      +18     
+ Misses        160      159       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tatiana

Thanks for fixing this and creating the follow up ticket, @pankajastro ! Happy for this to be merged once the checks pass.

@wornjs

**New Features** * Support customizing Airflow operator arguments per dbt node by @wornjs in #1339. [More information](https://astronomer.github.io/astronomer-cosmos/getting_started/custom-airflow-properties.html). * Support uploading dbt artifacts to remote cloud storages via callback by @pankajkoti in #1389. [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/callbacks.html). * Add support to ``TestBehavior.BUILD`` by @tatiana in #1377. [Documentation](https://astronomer.github.io/astronomer-cosmos/configuration/testing-behavior.html). * Add support for the "at" operator when using ``LoadMode.DBT_MANIFEST`` or ``CUSTOM`` by @benjy44 in #1372 * Add dbt clone operator by @pankajastro in #1326, as documented in [here](https://astronomer.github.io/astronomer-cosmos/getting_started/operators.html). * Support rendering tasks with non-ASCII characters by @t0momi219 in #1278 [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/task-display-name.html) * Add warning callback on source freshness by @pankajastro in #1400 [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/source-nodes-rendering.html#on-warning-callback-callback) * Add Oracle Profile mapping by @slords and @pankajkoti in #1190 and #1404 * Emit telemetry to Scarf during DAG run by @tatiana in #1397 * Save tasks map as ``DbtToAirflowConverter`` property by @internetcoffeephone and @hheemskerk in #1362 **Bug Fixes** * Fix the mock value of port in ``TrinoBaseProfileMapping`` to be an integer by @dwolfeu #1322 * Fix access to the ``dbt docs`` menu item outside of Astro cloud by @tatiana in #1312 * Add missing ``DbtSourceGcpCloudRunJobOperator`` in module ``cosmos.operators.gcp_cloud_run_job`` by @anai-s in #1290 * Support building ``DbtDag`` without setting paths in ``ProjectConfig`` by @tatiana in #1307 * Fix parsing dbt ls outputs that contain JSONs that are not dbt nodes by @tatiana in #1296 * Fix Snowflake Profile mapping when using AWS default region by @tatiana in #1406 * Fix dag rendering for taskflow + DbtTaskGroup combo by @pankajastro in #1360 **Enhancements** * Improve dbt command execution logs to troubleshoot ``None`` values by @tatiana in #1392 * Add logging of stdout to dbt graph run_command by @KarolGongola in #1390 * Save tasks map as DbtToAirflowConverter property by @internetcoffeephone and @hheemskerk in #1362 * Support rendering build operator task-id with non-ASCII characters by @pankajastro in #1415 **Docs** * Remove extra ` char from docs by @pankajastro in #1345 * Add limitation about copying target dir files to remote by @pankajkoti in #1305 * Generalise example from README by @ReadytoRocc in #1311 * Add security policy by @tatiana, @chaosmaw and @lzdanski in # 1385 * Mention in documentation that the callback functionality is supported in ``ExecutionMode.VIRTUALENV`` by @pankajkoti in #1401 **Others** * Restore Jaffle Shop so that ``basic_cosmos_dag`` works as documented by @tatiana in #1374 * Remove Pytest durations from tests scripts by @tatiana in #1383 * Remove typing-extensions as dependency by @pankajastro in #1381 * Pin dbt-databricks version to < 1.9 by @pankajastro in #1376 * Refactor ``dbt-sqlite`` tests to use ``dbt-postgres`` by @pankajastro in #1366 * Remove 'dbt-core<1.8.9' pin by @tatiana in #1371 * Remove dependency ``eval_type_backport`` by @tatiana in #1370 * Enable kubernetes tests for dbt>=1.8 by @pankajastro #1364 * CI Workaround: Pin dbt-core, Disable SQLite Tests, and Correctly Ignore Clone Test to Pass CI by @pankajastro in #1337 * Enable Azure task in the remote store manifest example DAG by @pankajkoti in #1333 * Enable GCP remote manifest task by @pankajastro in #1332 * Add exempt label option in GH action stale job by @pankajastro in #1328 * Add integration test for source node rendering by @pankajastro in #1327 * Fix vulnerability issue on docs dependency by @tatiana in #1313 * Add postgres pod status check for k8s tests in CI by @pankajkoti in #1320 * [CI] Reduce the amount taking to run tests in the CI from 5h to 11min by @tatiana in #1297 * Enable secret detection precommit check by @pankajastro in #1302 * Fix security vulnerability, by not pinning Airflow 2.10.0 by @tatiana in #1298 * Fix Netlify build timeouts by @tatiana in #1294 * Add stalebot to label/close stale PRs and issues by @tatiana in #1288 * Unpin dbt-databricks version by @pankajastro in #1409 * Fix source resource type tests by @pankajastro in #1405 * Increase performance tests models by @tatiana in #1403 * Drop running 1000 models in the CI by @pankajkoti in #1411 * Fix releasing package to PyPI by @tatiana in #1396 * Pre-commit hook updates in #1394, #1373, #1358, #1340, #1331, #1314, #1301 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io> Closes: #1193 --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

Repoduce-1375

a4331e9

pankajastro temporarily deployed to internal December 10, 2024 15:02 — with GitHub Actions Inactive

Repoduce-1375

a363691

pankajastro temporarily deployed to internal December 10, 2024 15:21 — with GitHub Actions Inactive

Repoduce-1375

c74062c

pankajastro temporarily deployed to internal December 10, 2024 15:26 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal December 10, 2024 15:27 — with GitHub Actions Inactive

pin dbt-databricks

06e631c

pankajastro temporarily deployed to internal December 10, 2024 15:50 — with GitHub Actions Inactive

pin dbt-databricks

cab75f2

pankajastro temporarily deployed to internal December 10, 2024 15:56 — with GitHub Actions Inactive

pankajastro changed the title ~~Repoduce-1375~~ Pin dbt-databricks version to < 1.9 Dec 10, 2024

pankajastro commented Dec 10, 2024

View reviewed changes

Comment thread pyproject.toml Outdated

pankajastro commented Dec 10, 2024

View reviewed changes

Comment thread pyproject.toml Outdated

pankajastro temporarily deployed to internal December 10, 2024 16:03 — with GitHub Actions Inactive

pankajastro commented Dec 10, 2024

View reviewed changes

Comment thread pyproject.toml

pankajastro marked this pull request as ready for review December 10, 2024 16:04

dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. area:ci Related to CI, Github Actions, or other continuous integration tools area:dependencies Related to dependencies, like Python packages, library versions, etc labels Dec 10, 2024

pankajastro mentioned this pull request Dec 10, 2024

Unpin dbt-databricks version #1379

Closed

pankajastro added 2 commits December 10, 2024 21:36

Create follow up issue

2479aae

Create follow up issue

d9dc84f

pankajastro force-pushed the fix-1375 branch from dd76895 to d9dc84f Compare December 10, 2024 16:08

pankajastro temporarily deployed to internal December 10, 2024 16:09 — with GitHub Actions Inactive

pankajkoti reviewed Dec 10, 2024

View reviewed changes

Comment thread pyproject.toml

tatiana approved these changes Dec 10, 2024

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Dec 10, 2024

Restore pydantic

487a6fd

pankajastro temporarily deployed to internal December 10, 2024 16:25 — with GitHub Actions Inactive

tatiana merged commit a735b16 into main Dec 10, 2024

tatiana deleted the fix-1375 branch December 10, 2024 16:47

pankajastro mentioned this pull request Dec 10, 2024

CI failing on cosmos main branch with error cannot load CLI commands from auth manager #1375

Closed

pankajastro added this to the Cosmos 1.8.0 milestone Dec 10, 2024

pankajastro self-assigned this Dec 11, 2024

ghjklw mentioned this pull request Dec 11, 2024

Pydantic restricted to v1 databricks/dbt-databricks#872

Closed

tatiana mentioned this pull request Dec 17, 2024

Release 1.8.0 #1395

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pin dbt-databricks version to < 1.9#1376

Pin dbt-databricks version to < 1.9#1376
tatiana merged 8 commits into
mainfrom
fix-1375

pankajastro commented Dec 10, 2024 •

edited by tatiana

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Dec 10, 2024 •

edited

Loading

Uh oh!

codecov Bot commented Dec 10, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pankajastro commented Dec 10, 2024 • edited by tatiana Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying astronomer-cosmos with Cloudflare Pages

Uh oh!

codecov Bot commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pankajastro commented Dec 10, 2024 •

edited by tatiana

Loading

cloudflare-workers-and-pages Bot commented Dec 10, 2024 •

edited

Loading

codecov Bot commented Dec 10, 2024 •

edited

Loading

tatiana left a comment •

edited

Loading