Add configurable seed rendering behavior by pankajkoti · Pull Request #2755 · astronomer/astronomer-cosmos

pankajkoti · 2026-06-02T11:46:18Z

Description

Adds a seed_rendering_behavior option to RenderConfig, giving control over how Cosmos renders and runs dbt seeds (analogous to source_rendering_behavior):

ALWAYS (default): render the seed and run dbt seed on every execution (the original Cosmos behaviour).
WHEN_SEED_CHANGES: render the seed, but only run dbt seed when its CSV content has changed since the last successful run. The checksum is read from dbt's manifest (falling back to hashing the CSV) and the last-seen value is persisted as an Airflow Variable scoped per DbtDag/DbtTaskGroup and seed. Supported for ExecutionMode.LOCAL, VIRTUALENV and AIRFLOW_ASYNC, and incompatible with TestBehavior.BUILD; both raise CosmosValueError.
RENDER_ONLY: render the seed as a no-op EmptyOperator placeholder.
NONE: do not render the seed at all.

The gate runs before the TestBehavior.BUILD branch so NONE/RENDER_ONLY are honoured under every test behaviour. On an unchanged run the seed task succeeds without running dbt seed, without emitting its dataset, and without skip-propagating downstream. Change detection delegates to AbstractDbtBase.execute(), preserving extra_context, debug-mode tracking and **kwargs.

Under ExecutionMode.WATCHER a single dbt build runs all seeds regardless of this setting; only ALWAYS is meaningful there.

Fresh implementation building on the approach explored in #2246 by @tuantran0910.

Related Issue(s)

Closes #1576

🤖 Generated with Claude Code

Add a `seed_rendering_behavior` option to `RenderConfig` that controls how Cosmos renders and runs dbt seeds, mirroring `source_rendering_behavior`: - ALWAYS (default): render the seed and run `dbt seed` on every execution. - WHEN_SEED_CHANGES: render the seed, but only run `dbt seed` when the seed's CSV content has changed since the last successful run. The checksum is read from dbt's manifest (falling back to hashing the CSV) and the last-seen value is persisted as an Airflow Variable scoped per DbtDag/TaskGroup and seed. Supported for ExecutionMode.LOCAL, VIRTUALENV and AIRFLOW_ASYNC, and incompatible with TestBehavior.BUILD; both raise CosmosValueError. - RENDER_ONLY: render the seed as a no-op EmptyOperator placeholder. - NONE: do not render the seed at all. The gate runs before the TestBehavior.BUILD branch so NONE and RENDER_ONLY are honored under every test behavior. On an unchanged run the seed task succeeds without running `dbt seed`, without emitting its dataset, and without skip-propagating to downstream tasks.

codecov · 2026-06-02T12:08:04Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.37%. Comparing base (8ea12fa) to head (b6e818d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2755      +/-   ##
==========================================
+ Coverage   98.36%   98.37%   +0.01%     
==========================================
  Files         107      107              
  Lines        7942     8021      +79     
==========================================
+ Hits         7812     7891      +79     
  Misses        130      130

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR adds a new seed_rendering_behavior option to RenderConfig to control how dbt seed nodes are represented in the generated Airflow graph, including a new “run only when seed CSV changes” mode that persists checksums via Airflow Variables.

Changes:

Introduces SeedRenderingBehavior (ALWAYS, WHEN_SEED_CHANGES, RENDER_ONLY, NONE) and wires it into task-metadata generation for seed nodes.
Implements seed change detection utilities (cosmos/dbt/seed.py) and runtime gating in DbtSeedLocalOperator.execute().
Extends config/runtime validation and adds unit tests covering allowed combinations and operator behavior.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`cosmos/airflow/graph.py`	Applies `seed_rendering_behavior` during task metadata creation (including placeholder/no-render paths).
`cosmos/config.py`	Adds `RenderConfig.seed_rendering_behavior` and validates incompatibility with `TestBehavior.BUILD`.
`cosmos/constants.py`	Adds `EMPTY_OPERATOR_CLASS` and the `SeedRenderingBehavior` enum.
`cosmos/converter.py`	Validates `WHEN_SEED_CHANGES` is only used with worker-filesystem execution modes.
`cosmos/dbt/graph.py`	Adds `checksum` to `DbtNode` and populates it from manifest/dbt-ls parsing.
`cosmos/dbt/seed.py`	New module for seed checksum resolution, Variable keying, and persistence helpers.
`cosmos/operators/local.py`	Adds runtime skip logic to `DbtSeedLocalOperator` for unchanged seeds.
`cosmos/__init__.py`	Exposes `SeedRenderingBehavior` at the package top level.
`tests/airflow/test_graph.py`	Tests seed rendering modes in task metadata (NONE / RENDER_ONLY / WHEN_SEED_CHANGES / ALWAYS).
`tests/dbt/test_graph.py`	Updates expected node context to include `checksum`.
`tests/dbt/test_seed.py`	New unit tests for seed change detection helpers.
`tests/operators/test_local.py`	Tests local seed operator skipping/running/persisting behavior under change detection.
`tests/test_config.py`	Tests config validation and defaulting for `seed_rendering_behavior`.
`tests/test_converter.py`	Tests execution-mode validation for `WHEN_SEED_CHANGES`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pankajkoti · 2026-06-03T15:33:50Z

The PR is ready for review. Please do not merge this PR. It relies on #2758 to be merged first.

tuantran0910 · 2026-06-03T15:39:11Z

Hi @pankajkoti, thank you very much for helping me continue with the PR adding seed rendering behavior. I am so sorry that I don't have time to continue putting effort in my original PR because I was so busy with my work. I am happy to see this PR and looking forward to seeing it merged in the future.

…os-renders-runs-seeds

…ring Drop the duplicate EMPTY_OPERATOR_CLASS constant this branch added to constants.py and reference EMPTY_OPERATOR_CLASS_PATH from cosmos.airflow.compatibility instead, so the RENDER_ONLY seed placeholder shares the version-aware path now on main.

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

pankajkoti · 2026-06-04T12:52:30Z

When seed_rendering_behavior=NONE, no seeds rendered

When seed_rendering_behavior=RENDER_ONLY, rendered as empty operators

When seed_rendering_behavior=WHEN_SEED_CHANGES, and seed has not changed by checking the checksum in Variables, does not re-run seed

When seed_rendering_behavior=WHEN_SEED_CHANGES, and TEST_BEHAVIOR=BUILD, incompatible

TEST_BEHAVIOR=BUILD continues to work with seed_rendering_behavior=ALWAYS (the default)

tatiana

@pankajkoti, thanks a lot for this feature; it will benefit many users.

I left some inline feedback.

Regarding this PR comment:

Under ExecutionMode.WATCHER, a single dbt build runs all seeds regardless of this setting; only ALWAYS is meaningful there.

We could adopt a similar strategy to the one we used to add support for SourceRenderingBehavior in ExecutionMode.WATCHER. The work was implemented/improved throughout a few PRs:

WDYT of logging a follow-up ticket to add support for SeedRenderingBehavior in ExecutionMode.WATCHER?

Compute the seed checksum as a DbtNode property that always streams the SHA256 of the seed CSV, so WHEN_SEED_CHANGES behaves the same under LoadMode.MANIFEST and LoadMode.DBT_LS. Remove cosmos/dbt/seed.py and move the Variable persistence into cache.py. Read the stored checksum best-effort, falling back to running the seed on backend errors. Strip whitespace before to_boolean in _is_full_refresh and rename the render flag to should_run_if_seed_changed.

…eature-improve-how-cosmos-renders-runs-seeds # Conflicts: # cosmos/airflow/graph.py # cosmos/dbt/graph.py # tests/test_converter.py

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

pankajkoti · 2026-06-04T17:09:18Z

Thanks @tatiana for the thorough review. I have pushed changes addressing the inline feedback (replies are in each thread):

checksum is now a property on DbtNode, always computed from the seed CSV (streamed in chunks), so it is consistent across LoadMode.MANIFEST and LoadMode.DBT_LS.
Removed cosmos/dbt/seed.py; the Airflow Variable persistence now lives in cache.py, and reads are best effort (they log and fall back to running the seed on backend errors).
Adopted the cleaner should_run interface in the operator and renamed the render flag to should_run_if_seed_changed.
Also handled the Copilot notes: streaming checksum, and stripping whitespace before to_boolean in _is_full_refresh.

On the watcher: agreed on logging a follow-up. I opened #2772 to add SeedRenderingBehavior support under ExecutionMode.WATCHER, following the SourceRenderingBehavior approach you linked.

When you have a moment, could you please take another look? I have re-requested your review.

tatiana

@pankajkoti thanks a lot for addressing the feedback! Minor feedback inline - non-blocking to merge this PR (you could implement the improvements in a follow-up PR).

Please, rebase before merging

…os-renders-runs-seeds

pankajkoti · 2026-06-05T14:21:29Z

@tatiana Thanks for the review. I have logged a follow-up ticket and scheduled it for the upcoming cycle to address the remaining feedback here.

## Description Adds an example DAG demonstrating `SeedRenderingBehavior.WHEN_SEED_CHANGES` against the `jaffle_shop` project. It is exercised end to end by the example-DAG integration tests (`dag.test()`). Stacked on #2755. ## Related Issue(s) Part of #1576 🤖 Generated with [Claude Code](https://claude.com/claude-code)

pankajkoti mentioned this pull request Jun 2, 2026

Add example DAG for seed rendering behavior #2756

Merged

pankajkoti temporarily deployed to internal June 2, 2026 11:46 — with GitHub Actions Inactive

pankajkoti marked this pull request as ready for review June 3, 2026 15:23

pankajkoti requested a review from jbandoro as a code owner June 3, 2026 15:23

Copilot AI review requested due to automatic review settings June 3, 2026 15:23

pankajkoti requested review from a team, corsettigyg and dwreeves as code owners June 3, 2026 15:23

pankajkoti requested review from pankajastro and tatiana June 3, 2026 15:23

Copilot started reviewing on behalf of pankajkoti June 3, 2026 15:24 View session

Copilot AI reviewed Jun 3, 2026

View reviewed changes

Comment thread cosmos/airflow/graph.py Outdated

Comment thread cosmos/dbt/seed.py Outdated

pankajkoti changed the title ~~Add configurable seed rendering behavior~~ [Ready for review, DO NOT MERGE] Add configurable seed rendering behavior Jun 3, 2026

Merge branch 'main' into pankajkoti/boss-240-feature-improve-how-cosm…

3a4de2a

…os-renders-runs-seeds

pankajkoti temporarily deployed to internal June 4, 2026 12:34 — with GitHub Actions Inactive

Copilot AI review requested due to automatic review settings June 4, 2026 12:43

Copilot started reviewing on behalf of pankajkoti June 4, 2026 12:43 View session

pankajkoti temporarily deployed to internal June 4, 2026 12:44 — with GitHub Actions Inactive

pankajkoti changed the title ~~[Ready for review, DO NOT MERGE] Add configurable seed rendering behavior~~ Add configurable seed rendering behavior Jun 4, 2026

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread cosmos/operators/local.py

Comment thread cosmos/airflow/graph.py

Comment thread cosmos/dbt/seed.py Outdated

Comment thread cosmos/dbt/seed.py Outdated

pankajkoti added the priority:high High priority issues are blocking or critical issues without a workaround and large impact label Jun 4, 2026

pankajkoti assigned tatiana Jun 4, 2026

tatiana reviewed Jun 4, 2026

View reviewed changes

pankajkoti temporarily deployed to internal June 4, 2026 16:47 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin/main' into pankajkoti/boss-240-f…

3822ad5

…eature-improve-how-cosmos-renders-runs-seeds # Conflicts: # cosmos/airflow/graph.py # cosmos/dbt/graph.py # tests/test_converter.py

Copilot AI review requested due to automatic review settings June 4, 2026 16:57

Copilot started reviewing on behalf of pankajkoti June 4, 2026 16:57 View session

pankajkoti temporarily deployed to internal June 4, 2026 16:58 — with GitHub Actions Inactive

pankajkoti mentioned this pull request Jun 4, 2026

Support SeedRenderingBehavior under ExecutionMode.WATCHER #2772

Open

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread cosmos/operators/local.py

Comment thread cosmos/converter.py

pankajkoti requested a review from tatiana June 4, 2026 17:08

tatiana approved these changes Jun 5, 2026

View reviewed changes

Comment thread cosmos/dbt/graph.py

Comment thread cosmos/dbt/graph.py

Comment thread cosmos/operators/local.py

Comment thread cosmos/cache.py

Comment thread cosmos/cache.py

Merge branch 'main' into pankajkoti/boss-240-feature-improve-how-cosm…

aa7e863

…os-renders-runs-seeds

pankajkoti temporarily deployed to internal June 5, 2026 14:15 — with GitHub Actions Inactive

🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

b6e818d

Copilot AI review requested due to automatic review settings June 5, 2026 14:15

pankajkoti review requested due to automatic review settings June 5, 2026 14:15

pre-commit-ci Bot temporarily deployed to internal June 5, 2026 14:16 Inactive

pankajkoti merged commit d4c6785 into main Jun 5, 2026
126 checks passed

pankajkoti deleted the pankajkoti/boss-240-feature-improve-how-cosmos-renders-runs-seeds branch June 5, 2026 14:37

tatiana added this to the Cosmos 1.15.0 milestone Jun 6, 2026

Conversation

pankajkoti commented Jun 2, 2026

Description

Related Issue(s)

Uh oh!

codecov Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Jun 3, 2026

Uh oh!

tuantran0910 commented Jun 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Jun 4, 2026

Uh oh!

tatiana left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Jun 4, 2026

Uh oh!

tatiana left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti commented Jun 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Jun 2, 2026 •

edited

Loading

tatiana left a comment •

edited

Loading