feature: Add DuckDB profile mapping by prithvijitguha · Pull Request #1553 · astronomer/astronomer-cosmos

prithvijitguha · 2025-02-23T16:45:11Z

Description

Added support for dbt-duckdb
Added jaffle_shop repo for duckdb
Added example for duck db dag usage

Related Issue(s)

Relates to Add support for dbt-duckdb adapter #202

Breaking Change?

Checklist

I have made corresponding changes to the documentation (if required)
I have added tests that prove my fix is effective or that my feature works

…rect project configurations to run

…uckdb profile

netlify · 2025-02-23T16:45:37Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`4717c2c`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67bb509ad03ba5000813cfd2

netlify · 2025-02-23T16:46:02Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`c3c0218`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/680f4401d323e70008011439

…tronomer-cosmos into feature/duckdb

prithvijitguha · 2025-02-24T06:39:51Z

Hey all,

It looks like more files due to the addition of the jaffle_shop_duckdb repo to the dev folder. Its the more updated one replacing the original jaffle_shop demo repo.

If you think this seems like too much for one PR, let me know, maybe I can break this up in other PRs :)

tatiana

Thanks a lot to @prithvijitguha for addressing the feedback and @pankajastro for fixing the integration tests in #1699. I'm approving this PR, it will temporarily break our main branch, but it should be resolved as soon as we merge Pankaj's PR.

Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

Run integration tests for PR: #1553

This PR fix the integration test for PR #1553 Success CI: https://github.com/astronomer/astronomer-cosmos/actions/runs/14693076417/job/41231050280?pr=1699 https://github.com/astronomer/astronomer-cosmos/actions/runs/14704324074/job/41260845188?pr=1699

prithvijitguha · 2025-04-28T10:48:44Z

Hey @pankajastro!

Sorry for the late reply, yes I was able to run jaffle shop locally, I seemed to have some issues running the integration tests, I am using WSL2, I assumed that might be an issue as well.

Thank you so much for fixing the integration tests!

prithvijitguha · 2025-04-28T10:49:31Z

Thanks a lot to @prithvijitguha for addressing the feedback and @pankajastro for fixing the integration tests in #1699. I'm approving this PR, it will temporarily break our main branch, but it should be resolved as soon as we merge Pankaj's PR.

Thank you @tatiana!

Its a great experience to work on this repo. Many thanks for @pankajastro for this fix :)

@tatiana

Features * Airflow 3 support * Support running ``dbt deps`` incrementally to pre-defined ``dbt_packages`` by @tatiana in #1668 and #1670 * Add ``DuckDB`` profile mapping by @prithvijitguha and @pankajastro in #1553 * Implement DBT exposure selector by ghjklw #1717 Bug Fixes * Fix ``test_indirect_selection`` flag to be propagated in case of ``TestBehavior.BUILD`` by @corsettigyg in #1663 * Fix ``select`` clause in the case of detached tests by @anyapriya in #1680 * Operator argument fixes by @johnhoran in #1648 Airflow 3 Support * Support rendering DbtDag in Airflow 3 by @tatiana and @ashb in #1657 * Refactor Rendered Task Instance Fields (RTIF) handling for Airflow 2.x and 3.x by @pankajkoti in #1661 * Run cosmos operator in Airflow 3 by @pankajastro in #1642 * Fix ``python_virtualenv.prepare_env`` top-level import for Airflow 3 by @pankajkoti in #1678 * Fix Variable not found issue in Airflow 3 by @tatiana in #1684 * Disable CosmosPlugin on Airflow 3 setup by @pankajkoti in #1692, #1698 * Use ``schedule`` param in example DAGs instead of the 2.10 deprecated and 3.0 removed ``schedule_interval`` by @pankajkoti in #1701 * Ensure ``virtualenv_dir`` path exists by @pankajkoti in #1724 * Support emitting Assets with Airflow 3 by @tatiana in #1713 * Add docs on Airflow 3 compatibility by @pankajkoti and @tatiana in #1731 * Introduce, test and document asset/dataset breaking change by @tatiana in #1672 * Improve dataset/asset driven scheduling documentation by @tatiana in #1729 Enhancements * Allow multiple callbacks by @corsettigyg #1693 * Refactor kubernetes warning callback handling by @canbekley in #1681 Documentation * Add documentation related to ``copy_dbt_packages`` by @tatiana in #1671 * Make wording and command consistent in the contributing doc by @pankajkoti in #1697 * Add MonteCarlo callback example for importing dbt artifacts by @corsettigyg #1695 * Change async feature to be non-experimental by @tatiana in #1732 Others * Add sample ``dbt_packages`` to validate incremental ``dbt deps`` by @tatiana in #1669 * Add kubernetes execution mode example in Airflow 3 by @pankajastro in #1667 * Check only major version until Airflow 3 stable release by @pankajastro in #1665 * Install Airflow from main branch by @pankajastro in #1660 * Add dev tool for Airflow 3 by @pankajastro and @tatiana in #1627 * Improve Airflow 3 tooling by @pankajastro in #1656 * Skip associating ``openlineage_events_completes`` to ``ti`` in Airflow 3 by @pankajkoti in #1662 * Add .gitignore file for the scripts/airflow3 directory by @pankajkoti in #1658 * Remove ``original_jaffle_shop`` dbt project by @pankajkoti in #1676 * Fix or ignore type check error by @pankajastro in #1687 * Run virtualenv example with Airflow 3 tooling by @pankajastro in #1686 * Enable running setup/teardown tasks with Async execution DAG with Airflow 3 tooling by @pankajastro in #1696 * Enable integration tests for the DuckDB adapter by @pankajastro in #1699 * Add Airflow 3 tests matrix entries in CI by @pankajkoti in #1646 * Use a different way to get tasks count for asserting test_perf_dag by @pankajkoti in #1714 * Reinstall Airflow 3 dependency on ``pydantic>=2.11`` for dbt adapter versions 1.6 & 1.9 by @pankajkoti in #1715 * Fix outdated ``echo`` in Airflow 3 tooling script #1700 * Add files not needed for git tracking to .gitignore by @pankajkoti in #1723 * Use latest minor versions for dbt adapters to get in compatibility fixes by @pankajkoti in #1719 * Fix Airflow 3 tests raising generate_run_id() takes 0 positional arguments by @tatiana in #1725 * Fix dataset tests failing in Airflow 3 by @tatiana in #1716 * Enable example DAGs to run in CI that were disabled in PR #1646 by @pankajkoti in #1726 * Pre-commit updates: #1666, #1653, #1641, #1682, #1720 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com> --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>

prithvijitguha added 15 commits February 22, 2025 21:47

Added profiles for duckdb and example dags

c69bf30

Added profiles connection

6a8e952

Added tests to start

ffad8a2

Updated profiles

7e48d10

Updated user pass docstring for duckdb

b17a29a

Added example for dbt duckdb dag

9f24a64

Removed old dag

810a5a1

Removed repo specific content for duckdb

affc808

Updated duckdb dag with docstrings and profile mapping

8e7a74a

Added dbt-duckdb to references

19f91d9

Updated dependencies for dbt-duckdb airflow

34234eb

Removed duckdb dag

f07a365

Pinned duckdb dependencies

95d9f4a

Updated profiles to remove debug code, updated duckdb project for cor…

8a4d512

…rect project configurations to run

Fixed tests for duckdb profile and for example and user mapping for d…

4717c2c

…uckdb profile

prithvijitguha had a problem deploying to external February 23, 2025 16:45 — with GitHub Actions Error

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

15461c2

pre-commit-ci Bot had a problem deploying to external February 23, 2025 16:45 Error

prithvijitguha added 2 commits February 23, 2025 22:16

Removed duck db

a2213a0

Merge branch 'feature/duckdb' of https://github.com/prithvijitguha/as…

366fa54

…tronomer-cosmos into feature/duckdb

prithvijitguha temporarily deployed to external February 23, 2025 16:52 — with GitHub Actions Inactive

prithvijitguha marked this pull request as ready for review February 23, 2025 17:00

dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc area:testing Related to testing, like unit tests, integration tests, etc labels Feb 23, 2025

tatiana reviewed Feb 24, 2025

View reviewed changes

Comment thread cosmos/profiles/duckdb/user_pass.py

tatiana reviewed Feb 24, 2025

View reviewed changes

Comment thread dev/dags/example_duckdb_dag.py Outdated

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

1311167

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

bc98dd5

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

a32daa5

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

7ee82f2

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

28f2791

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

4cea6b7

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

673c670

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

7f0fdea

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

60cd8cc

pankajastro added a commit that referenced this pull request Apr 26, 2025

Run integration tests for PR: #1553

3216dff

pankajastro mentioned this pull request Apr 27, 2025

Enable integration tests for the DuckDB adaptor (PR #1553) #1699

Merged

Merge branch 'main' into feature/duckdb

39b672e

tatiana temporarily deployed to external April 28, 2025 08:46 — with GitHub Actions Inactive

tatiana approved these changes Apr 28, 2025

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Apr 28, 2025

tatiana changed the title ~~feature: Added duckdb adaptor support~~ feature: Add DuckDB adaptor support Apr 28, 2025

tatiana changed the title ~~feature: Add DuckDB adaptor support~~ feature: Add DuckDB profile mapping Apr 28, 2025

Update cosmos/profiles/duckdb/user_pass.py

44392e4

Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

pankajastro added a commit that referenced this pull request Apr 28, 2025

Run integration tests for PR: #1553

3e97f10

Run integration tests for PR: #1553

tatiana reviewed Apr 28, 2025

View reviewed changes

Comment thread tests/profiles/duckdb/test_duckdb_user_pass.py Outdated

Update tests/profiles/duckdb/test_duckdb_user_pass.py

c3c0218

tatiana had a problem deploying to external April 28, 2025 09:02 — with GitHub Actions Failure

tatiana merged commit 1bcfb1e into astronomer:main Apr 28, 2025
32 of 33 checks passed

pankajastro added a commit that referenced this pull request Apr 28, 2025

Run integration tests for PR: #1553

60525c8

Run integration tests for PR: #1553

prithvijitguha deleted the feature/duckdb branch April 28, 2025 10:48

pankajkoti mentioned this pull request Apr 30, 2025

Release 1.10.0 #1672

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: Add DuckDB profile mapping#1553

feature: Add DuckDB profile mapping#1553
tatiana merged 31 commits into
astronomer:mainfrom
prithvijitguha:feature/duckdb

prithvijitguha commented Feb 23, 2025

Uh oh!

netlify Bot commented Feb 23, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Feb 23, 2025 •

edited

Loading

Uh oh!

prithvijitguha commented Feb 24, 2025

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Uh oh!

Uh oh!

Uh oh!

prithvijitguha commented Apr 28, 2025

Uh oh!

prithvijitguha commented Apr 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

prithvijitguha commented Feb 23, 2025

Description

Related Issue(s)

Breaking Change?

Checklist

Uh oh!

netlify Bot commented Feb 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

netlify Bot commented Feb 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

prithvijitguha commented Feb 24, 2025

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

prithvijitguha commented Apr 28, 2025

Uh oh!

prithvijitguha commented Apr 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

netlify Bot commented Feb 23, 2025 •

edited

Loading

netlify Bot commented Feb 23, 2025 •

edited

Loading