Add integration tests for ephemeral models rendered as empty operators#2760
Conversation
Add a centralized EMPTY_OPERATOR_CLASS constant that resolves to airflow.providers.standard.operators.empty.EmptyOperator on Airflow 3 and the legacy airflow.operators.empty.EmptyOperator on Airflow 2. Use it for the source-without-freshness rendering path, which previously hardcoded the legacy path and emitted a DeprecatedImportWarning on Airflow 3.
Ephemeral models are inlined as CTEs into downstream models and are never written to the warehouse, so running them through a dbt operator is a no-op that still spends worker time and issues warehouse metadata queries. Add RenderConfig.ephemeral_models_as_empty_operator (default True) so that ephemeral models render as EmptyOperator tasks. The node stays in the graph, preserving the dependency chain that passes through it, while avoiding the wasted dbt invocation and decluttering the DAG. Set it to False to keep rendering ephemeral models as regular dbt run tasks.
|
The PR is ready for review. Please do not merge this PR. It relies on #2759 to be merged first. |
tatiana
left a comment
There was a problem hiding this comment.
Thanks, @pankajkoti , these look great!
…eature-emptytask-for-ephemeral-materializations # Conflicts: # cosmos/airflow/graph.py # tests/airflow/test_graph.py
Add a customers -> ephemeral_customers -> ephemeral_customers_downstream lineage to the altered_jaffle_shop test project and DbtDag tests (via dbt ls) asserting that the ephemeral model renders as an EmptyOperator by default with the dependency chain preserved, and as a dbt run task when the option is disabled. Update the affected node-count assertions for the two added models. Regenerate the project's committed manifest.json (dbt 1.11.2, schema v12) to reflect the new models; this also drops a stale multibyte model node whose sql file had already been removed from disk.
4052906 to
817c0cc
Compare
There was a problem hiding this comment.
Pull request overview
Adds integration test coverage to ensure dbt ephemeral models render as EmptyOperator by default (while preserving the dependency chain), and as a normal dbt run task when the behavior is disabled. Updates the altered dbt test project and related node-count assertions, and regenerates the committed manifest.json to reflect the new models and remove a stale node.
Changes:
- Add integration tests that assert ephemeral models render as
EmptyOperatorby default and asDbtRunLocalOperatorwhen disabled. - Extend the
altered_jaffle_shopdbt test project withcustomers -> ephemeral_customers -> ephemeral_customers_downstream. - Regenerate the altered project
manifest.jsonand update node-count expectations in graph tests.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_converter.py |
Adds integration tests validating DbtDag task types and dependency preservation for ephemeral models. |
tests/dbt/test_graph.py |
Updates expected node counts for altered dbt project after adding two models. |
cosmos/airflow/graph.py |
Adds (but currently duplicates) logic to render ephemeral models as EmptyOperator. |
dev/dags/dbt/altered_jaffle_shop/models/ephemeral_customers.sql |
New ephemeral dbt model used by integration tests. |
dev/dags/dbt/altered_jaffle_shop/models/ephemeral_customers_downstream.sql |
New downstream model referencing the ephemeral model. |
dev/dags/dbt/altered_jaffle_shop/target/manifest.json |
Regenerated manifest fixture reflecting new models and removal of a stale node. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
test_load_via_dbt_ls_with_exclude selects *customers*, which now also matches the two new ephemeral models, so the expected node count and key list grow from 9 to 11.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2760 +/- ##
=======================================
Coverage 98.28% 98.28%
=======================================
Files 107 107
Lines 7923 7923
=======================================
Hits 7787 7787
Misses 136 136 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Adds integration coverage for #1133.
Adds a
customers -> ephemeral_customers -> ephemeral_customers_downstreamlineage to thealtered_jaffle_shoptest project andDbtDagtests (rendering viadbt ls) asserting that the ephemeral model renders as anEmptyOperatorby default with the dependency chain preserved, and as a dbt run task when the option is disabled. Updates the node-count assertions for the two added models.Regenerates the project's committed
manifest.json(dbt 1.11.2, schema v12) for the new models; this also drops a stale multibyte model node whose.sqlfile had already been removed from disk.🤖 Generated with Claude Code