Restore Jaffle Shop so that `basic_cosmos_dag` works as documented by tatiana · Pull Request #1374 · astronomer/astronomer-cosmos

tatiana · 2024-12-10T13:52:54Z

When trying out some changes in Cosmos, I realised that we significantly changed the topology of the jaffle_shop project.

Before, the basic_cosmos_dag rendered this way, as described in our README:

Now, it renders like this:

Some of the changes:

standalone task with a different name (m u l t i b y t e _run)
stg_payments does not depend on raw_payments_seed
stg_orders does not depend on raw_orders_seed`

The dependency issue (2) and (3) has been happening since a89389d

While we need to create new dbt projects / make changes that allow us to validate new features in Cosmos, we should be strict with the Jaffle Shop project used as a reference for our most basic example.

cloudflare-workers-and-pages · 2024-12-10T13:54:58Z

Deploying astronomer-cosmos with Cloudflare Pages

Latest commit:	`f553625`
Status:	✅ Deploy successful!
Preview URL:	https://702162d8.astronomer-cosmos.pages.dev
Branch Preview URL:	https://restore-jaffle-shop.astronomer-cosmos.pages.dev

View logs

codecov · 2024-12-11T16:26:57Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.23%. Comparing base (c66681c) to head (f553625).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1374      +/-   ##
==========================================
+ Coverage   96.06%   96.23%   +0.17%     
==========================================
  Files          67       67              
  Lines        4042     4042              
==========================================
+ Hits         3883     3890       +7     
+ Misses        159      152       -7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pankajkoti

LGTM, a couple of minor comments inline since I'm not aware of and would like to understand what the fail.txt and succeed.txt files are for :)

@wornjs

**New Features** * Support customizing Airflow operator arguments per dbt node by @wornjs in #1339. [More information](https://astronomer.github.io/astronomer-cosmos/getting_started/custom-airflow-properties.html). * Support uploading dbt artifacts to remote cloud storages via callback by @pankajkoti in #1389. [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/callbacks.html). * Add support to ``TestBehavior.BUILD`` by @tatiana in #1377. [Documentation](https://astronomer.github.io/astronomer-cosmos/configuration/testing-behavior.html). * Add support for the "at" operator when using ``LoadMode.DBT_MANIFEST`` or ``CUSTOM`` by @benjy44 in #1372 * Add dbt clone operator by @pankajastro in #1326, as documented in [here](https://astronomer.github.io/astronomer-cosmos/getting_started/operators.html). * Support rendering tasks with non-ASCII characters by @t0momi219 in #1278 [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/task-display-name.html) * Add warning callback on source freshness by @pankajastro in #1400 [Read more](https://astronomer.github.io/astronomer-cosmos/configuration/source-nodes-rendering.html#on-warning-callback-callback) * Add Oracle Profile mapping by @slords and @pankajkoti in #1190 and #1404 * Emit telemetry to Scarf during DAG run by @tatiana in #1397 * Save tasks map as ``DbtToAirflowConverter`` property by @internetcoffeephone and @hheemskerk in #1362 **Bug Fixes** * Fix the mock value of port in ``TrinoBaseProfileMapping`` to be an integer by @dwolfeu #1322 * Fix access to the ``dbt docs`` menu item outside of Astro cloud by @tatiana in #1312 * Add missing ``DbtSourceGcpCloudRunJobOperator`` in module ``cosmos.operators.gcp_cloud_run_job`` by @anai-s in #1290 * Support building ``DbtDag`` without setting paths in ``ProjectConfig`` by @tatiana in #1307 * Fix parsing dbt ls outputs that contain JSONs that are not dbt nodes by @tatiana in #1296 * Fix Snowflake Profile mapping when using AWS default region by @tatiana in #1406 * Fix dag rendering for taskflow + DbtTaskGroup combo by @pankajastro in #1360 **Enhancements** * Improve dbt command execution logs to troubleshoot ``None`` values by @tatiana in #1392 * Add logging of stdout to dbt graph run_command by @KarolGongola in #1390 * Save tasks map as DbtToAirflowConverter property by @internetcoffeephone and @hheemskerk in #1362 * Support rendering build operator task-id with non-ASCII characters by @pankajastro in #1415 **Docs** * Remove extra ` char from docs by @pankajastro in #1345 * Add limitation about copying target dir files to remote by @pankajkoti in #1305 * Generalise example from README by @ReadytoRocc in #1311 * Add security policy by @tatiana, @chaosmaw and @lzdanski in # 1385 * Mention in documentation that the callback functionality is supported in ``ExecutionMode.VIRTUALENV`` by @pankajkoti in #1401 **Others** * Restore Jaffle Shop so that ``basic_cosmos_dag`` works as documented by @tatiana in #1374 * Remove Pytest durations from tests scripts by @tatiana in #1383 * Remove typing-extensions as dependency by @pankajastro in #1381 * Pin dbt-databricks version to < 1.9 by @pankajastro in #1376 * Refactor ``dbt-sqlite`` tests to use ``dbt-postgres`` by @pankajastro in #1366 * Remove 'dbt-core<1.8.9' pin by @tatiana in #1371 * Remove dependency ``eval_type_backport`` by @tatiana in #1370 * Enable kubernetes tests for dbt>=1.8 by @pankajastro #1364 * CI Workaround: Pin dbt-core, Disable SQLite Tests, and Correctly Ignore Clone Test to Pass CI by @pankajastro in #1337 * Enable Azure task in the remote store manifest example DAG by @pankajkoti in #1333 * Enable GCP remote manifest task by @pankajastro in #1332 * Add exempt label option in GH action stale job by @pankajastro in #1328 * Add integration test for source node rendering by @pankajastro in #1327 * Fix vulnerability issue on docs dependency by @tatiana in #1313 * Add postgres pod status check for k8s tests in CI by @pankajkoti in #1320 * [CI] Reduce the amount taking to run tests in the CI from 5h to 11min by @tatiana in #1297 * Enable secret detection precommit check by @pankajastro in #1302 * Fix security vulnerability, by not pinning Airflow 2.10.0 by @tatiana in #1298 * Fix Netlify build timeouts by @tatiana in #1294 * Add stalebot to label/close stale PRs and issues by @tatiana in #1288 * Unpin dbt-databricks version by @pankajastro in #1409 * Fix source resource type tests by @pankajastro in #1405 * Increase performance tests models by @tatiana in #1403 * Drop running 1000 models in the CI by @pankajkoti in #1411 * Fix releasing package to PyPI by @tatiana in #1396 * Pre-commit hook updates in #1394, #1373, #1358, #1340, #1331, #1314, #1301 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <pankaj.singh@astronomer.io> Closes: #1193 --------- Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com> Co-authored-by: Pankaj Singh <98807258+pankajastro@users.noreply.github.com>

- `stg_orders.sql` and `stg_payments.sql` in `altered_jaffle_shop` had a `force_seed_dep` CTE that referenced `ref('raw_customers')` instead of the corresponding `raw_orders` / `raw_payments` seeds — a copy-paste leftover from #1107. The CTE exists to keep tests waiting for the correct seed when `source_node_rendering = none`, so referencing the wrong seed defeated its purpose. - #1374 fixed this exact issue in `jaffle_shop` but missed `altered_jaffle_shop`. - The committed `target/manifest.json` snapshot already reflects the correct edges (regenerated in #2296), so only the two SQL files needed updating. The `cosmos_manifest_selectors_example.py` DAG that explicitly lists `raw_orders` / `raw_payments` in its `select=` workaround is now arguably redundant, but harmless — left as-is. Closes #2599

tatiana temporarily deployed to internal December 10, 2024 13:53 — with GitHub Actions Inactive

tatiana temporarily deployed to internal December 10, 2024 14:10 — with GitHub Actions Inactive

tatiana temporarily deployed to internal December 10, 2024 16:47 — with GitHub Actions Inactive

tatiana force-pushed the restore-jaffle-shop branch from 953ec6c to 667fd19 Compare December 11, 2024 09:07

tatiana temporarily deployed to internal December 11, 2024 09:07 — with GitHub Actions Inactive

tatiana force-pushed the restore-jaffle-shop branch from 667fd19 to 054b69c Compare December 11, 2024 09:08

tatiana temporarily deployed to internal December 11, 2024 09:08 — with GitHub Actions Inactive

tatiana changed the title ~~Restore Jaffle Shop so that the basic Cosmos example works as expected~~ Restore Jaffle Shop so that basic_comsmos_dag works as documented Dec 11, 2024

tatiana temporarily deployed to internal December 11, 2024 09:15 — with GitHub Actions Inactive

tatiana temporarily deployed to internal December 11, 2024 14:16 — with GitHub Actions Inactive

tatiana temporarily deployed to internal December 11, 2024 14:46 — with GitHub Actions Inactive

tatiana temporarily deployed to internal December 11, 2024 15:40 — with GitHub Actions Inactive

tatiana temporarily deployed to internal December 11, 2024 16:09 — with GitHub Actions Inactive

tatiana marked this pull request as ready for review December 11, 2024 16:16

dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. area:rendering Related to rendering, like Jinja, Airflow tasks, etc labels Dec 11, 2024

tatiana changed the title ~~Restore Jaffle Shop so that basic_comsmos_dag works as documented~~ Restore Jaffle Shop so that basic_cosmos_dag works as documented Dec 11, 2024

tatiana mentioned this pull request Dec 11, 2024

Add support to TestBehavior.BUILD #1377

Merged

pankajkoti approved these changes Dec 12, 2024

View reviewed changes

Comment thread dev/dags/dbt/jaffle_shop/fail.txt Outdated

Comment thread dev/dags/dbt/jaffle_shop/succeed.txt Outdated

pankajastro approved these changes Dec 12, 2024

View reviewed changes

Comment thread dev/dags/dbt/altered_jaffle_shop/dbt_project.yml

Comment thread dev/dags/dbt/altered_jaffle_shop/fail.txt Outdated

Comment thread dev/dags/dbt/altered_jaffle_shop/packages_.yml Outdated

Comment thread dev/dags/dbt/altered_jaffle_shop/profiles.yml

tatiana temporarily deployed to internal December 12, 2024 09:04 — with GitHub Actions Inactive

tatiana temporarily deployed to internal December 12, 2024 09:06 — with GitHub Actions Inactive

tatiana temporarily deployed to internal December 12, 2024 09:46 — with GitHub Actions Inactive

tatiana added 4 commits December 12, 2024 09:52

Retore jaffle shop so the basic Cosmos example works as expected

37b6479

Fix unittests

8816168

Fix sources DAG test

16434b3

Try to fix k8s integration test

4e242ed

tatiana added 10 commits December 12, 2024 09:52

Fix tests

caebf42

Fix k8s test

9c39035

Fix unittest

ecdb881

Fix tests for AF 2.4

c3394f7

Delete dev/dags/dbt/altered_jaffle_shop/packages_.yml

9c65e8d

Delete dev/dags/dbt/altered_jaffle_shop/fail.txt

2167fbf

Delete dev/dags/dbt/jaffle_shop/succeed.txt

8bfc3eb

Delete dev/dags/dbt/jaffle_shop/fail.txt

dd8c4a5

Address code review feedback

0ea4efd

Remove dev/dags/dbt/altered_jaffle_shop/succeed.txt

27f3706

tatiana mentioned this pull request Dec 12, 2024

Unify default and postgres_profile in the example profiles.yml #1384

Closed

Fix the dbt project hash after changes to the folder

f553625

tatiana force-pushed the restore-jaffle-shop branch from 1d23cc7 to f553625 Compare December 12, 2024 09:58

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Dec 12, 2024

tatiana temporarily deployed to internal December 12, 2024 09:58 — with GitHub Actions Inactive

tatiana self-assigned this Dec 12, 2024

tatiana merged commit 2fa5d01 into main Dec 12, 2024

tatiana deleted the restore-jaffle-shop branch December 12, 2024 10:24

tatiana added this to the Cosmos 1.8.0 milestone Dec 16, 2024

tatiana mentioned this pull request Dec 17, 2024

Release 1.8.0 #1395

Merged

tatiana mentioned this pull request Apr 17, 2025

Add Airflow 3 compatibility for BiqQuery deferrable support (ExecutionMode.AIRFLOW_ASYNC) #1674

Merged

pankajkoti mentioned this pull request Apr 17, 2025

Consolidate existing dbt projects into a single jaffle shop project #1677

Open

pankajastro mentioned this pull request May 18, 2026

Fix altered_jaffle_shop seed-dep CTE refs #2690

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore Jaffle Shop so that `basic_cosmos_dag` works as documented#1374

Restore Jaffle Shop so that `basic_cosmos_dag` works as documented#1374
tatiana merged 15 commits into
mainfrom
restore-jaffle-shop

tatiana commented Dec 10, 2024 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Dec 10, 2024 •

edited

Loading

Uh oh!

codecov Bot commented Dec 11, 2024 •

edited

Loading

Uh oh!

pankajkoti left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tatiana commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages Bot commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying astronomer-cosmos with Cloudflare Pages

Uh oh!

codecov Bot commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tatiana commented Dec 10, 2024 •

edited

Loading

cloudflare-workers-and-pages Bot commented Dec 10, 2024 •

edited

Loading

codecov Bot commented Dec 11, 2024 •

edited

Loading