Skip to content

Add integration tests for ephemeral models rendered as empty operators#2760

Merged
pankajkoti merged 12 commits into
mainfrom
pankajkoti/boss-290-ephemeral-integration-test
Jun 4, 2026
Merged

Add integration tests for ephemeral models rendered as empty operators#2760
pankajkoti merged 12 commits into
mainfrom
pankajkoti/boss-290-ephemeral-integration-test

Conversation

@pankajkoti
Copy link
Copy Markdown
Contributor

Adds integration coverage for #1133.

Adds a customers -> ephemeral_customers -> ephemeral_customers_downstream lineage to the altered_jaffle_shop test project and DbtDag tests (rendering via dbt ls) asserting that the ephemeral model renders as an EmptyOperator by default with the dependency chain preserved, and as a dbt run task when the option is disabled. Updates the node-count assertions for the two added models.

Regenerates the project's committed manifest.json (dbt 1.11.2, schema v12) for the new models; this also drops a stale multibyte model node whose .sql file had already been removed from disk.

Stacked on #2759 — review and merge that first.

🤖 Generated with Claude Code

Add a centralized EMPTY_OPERATOR_CLASS constant that resolves to
airflow.providers.standard.operators.empty.EmptyOperator on Airflow 3 and
the legacy airflow.operators.empty.EmptyOperator on Airflow 2. Use it for
the source-without-freshness rendering path, which previously hardcoded the
legacy path and emitted a DeprecatedImportWarning on Airflow 3.
Ephemeral models are inlined as CTEs into downstream models and are never
written to the warehouse, so running them through a dbt operator is a no-op
that still spends worker time and issues warehouse metadata queries.

Add RenderConfig.ephemeral_models_as_empty_operator (default True) so that
ephemeral models render as EmptyOperator tasks. The node stays in the graph,
preserving the dependency chain that passes through it, while avoiding the
wasted dbt invocation and decluttering the DAG. Set it to False to keep
rendering ephemeral models as regular dbt run tasks.
@pankajkoti pankajkoti changed the title Add integration tests for ephemeral models rendered as empty operators [Ready for review, but DO NOT MERGE] Add integration tests for ephemeral models rendered as empty operators Jun 3, 2026
@pankajkoti pankajkoti marked this pull request as ready for review June 3, 2026 15:29
@pankajkoti pankajkoti requested review from a team, corsettigyg, dwreeves and jbandoro as code owners June 3, 2026 15:29
@pankajkoti pankajkoti requested review from pankajastro and tatiana and removed request for a team June 3, 2026 15:29
@pankajkoti
Copy link
Copy Markdown
Contributor Author

pankajkoti commented Jun 3, 2026

The PR is ready for review. Please do not merge this PR. It relies on #2759 to be merged first.

@pankajkoti pankajkoti changed the title [Ready for review, but DO NOT MERGE] Add integration tests for ephemeral models rendered as empty operators [Ready for review, DO NOT MERGE] Add integration tests for ephemeral models rendered as empty operators Jun 3, 2026
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @pankajkoti , these look great!

…eature-emptytask-for-ephemeral-materializations

# Conflicts:
#	cosmos/airflow/graph.py
#	tests/airflow/test_graph.py
Add a customers -> ephemeral_customers -> ephemeral_customers_downstream
lineage to the altered_jaffle_shop test project and DbtDag tests (via dbt ls)
asserting that the ephemeral model renders as an EmptyOperator by default with
the dependency chain preserved, and as a dbt run task when the option is
disabled. Update the affected node-count assertions for the two added models.

Regenerate the project's committed manifest.json (dbt 1.11.2, schema v12) to
reflect the new models; this also drops a stale multibyte model node whose sql
file had already been removed from disk.
@pankajkoti pankajkoti force-pushed the pankajkoti/boss-290-ephemeral-integration-test branch from 4052906 to 817c0cc Compare June 4, 2026 13:04
Base automatically changed from pankajkoti/boss-290-feature-emptytask-for-ephemeral-materializations to main June 4, 2026 14:13
Copilot AI review requested due to automatic review settings June 4, 2026 14:23
@pankajkoti pankajkoti changed the title [Ready for review, DO NOT MERGE] Add integration tests for ephemeral models rendered as empty operators Add integration tests for ephemeral models rendered as empty operators Jun 4, 2026
Comment thread cosmos/airflow/graph.py Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds integration test coverage to ensure dbt ephemeral models render as EmptyOperator by default (while preserving the dependency chain), and as a normal dbt run task when the behavior is disabled. Updates the altered dbt test project and related node-count assertions, and regenerates the committed manifest.json to reflect the new models and remove a stale node.

Changes:

  • Add integration tests that assert ephemeral models render as EmptyOperator by default and as DbtRunLocalOperator when disabled.
  • Extend the altered_jaffle_shop dbt test project with customers -> ephemeral_customers -> ephemeral_customers_downstream.
  • Regenerate the altered project manifest.json and update node-count expectations in graph tests.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tests/test_converter.py Adds integration tests validating DbtDag task types and dependency preservation for ephemeral models.
tests/dbt/test_graph.py Updates expected node counts for altered dbt project after adding two models.
cosmos/airflow/graph.py Adds (but currently duplicates) logic to render ephemeral models as EmptyOperator.
dev/dags/dbt/altered_jaffle_shop/models/ephemeral_customers.sql New ephemeral dbt model used by integration tests.
dev/dags/dbt/altered_jaffle_shop/models/ephemeral_customers_downstream.sql New downstream model referencing the ephemeral model.
dev/dags/dbt/altered_jaffle_shop/target/manifest.json Regenerated manifest fixture reflecting new models and removal of a stale node.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cosmos/airflow/graph.py Outdated
Comment thread dev/dags/dbt/altered_jaffle_shop/target/manifest.json Outdated
Comment thread dev/dags/dbt/altered_jaffle_shop/target/manifest.json Outdated
Comment thread dev/dags/dbt/altered_jaffle_shop/target/manifest.json Outdated
Comment thread dev/dags/dbt/altered_jaffle_shop/target/manifest.json Outdated
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Copilot AI review requested due to automatic review settings June 4, 2026 14:29
pankajkoti and others added 4 commits June 4, 2026 20:03
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated no new comments.

test_load_via_dbt_ls_with_exclude selects *customers*, which now also
matches the two new ephemeral models, so the expected node count and key
list grow from 9 to 11.
Copilot AI review requested due to automatic review settings June 4, 2026 14:58
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.

Comment thread tests/test_converter.py
@pankajkoti pankajkoti merged commit 4bae5f5 into main Jun 4, 2026
124 checks passed
@pankajkoti pankajkoti deleted the pankajkoti/boss-290-ephemeral-integration-test branch June 4, 2026 15:19
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 4, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.28%. Comparing base (a18e4d9) to head (1ba9a85).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2760   +/-   ##
=======================================
  Coverage   98.28%   98.28%           
=======================================
  Files         107      107           
  Lines        7923     7923           
=======================================
  Hits         7787     7787           
  Misses        136      136           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants