Skip to content

Refactor _default_freshness_callback to return list of (id, state) tuple#2572

Merged
pankajastro merged 3 commits into
mainfrom
refactor-freshness-callback-return-type
Apr 21, 2026
Merged

Refactor _default_freshness_callback to return list of (id, state) tuple#2572
pankajastro merged 3 commits into
mainfrom
refactor-freshness-callback-return-type

Conversation

@pankajastro
Copy link
Copy Markdown
Contributor

@pankajastro pankajastro commented Apr 17, 2026

  • Changed _default_freshness_callback return type from tuple[list[str], str] to list[tuple[str, str]] so each node carries its own state instead of
    sharing a single status string
  • Renamed _skipped_node_token → _apply_node_state_tokens and _push_skipped_xcom_for_model → _push_node_state_xcom; both now accept an explicit state
    argument, enabling custom callbacks to push states other than "skipped"
  • Nodes are added to self.exclude only when their state is "skipped"; other states push XCom but do not affect the dbt run
  • Updated all call sites and tests; added a new test for the non-"skipped" state path

This set would set ground work for #2053

Related-to: #2053

…ples

Change return type from tuple[list[str], str] to list[tuple[str, str]]
so each element carries its own state alongside the node unique_id.
Update the call site and all related tests accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 17, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.05%. Comparing base (37fd06c) to head (fc38684).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2572   +/-   ##
=======================================
  Coverage   98.05%   98.05%           
=======================================
  Files         103      103           
  Lines        7601     7604    +3     
=======================================
+ Hits         7453     7456    +3     
  Misses        148      148           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pankajastro pankajastro marked this pull request as ready for review April 17, 2026 15:59
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors the WATCHER producer’s source-freshness “skip” callback to return per-node results as (unique_id, state) tuples instead of a (list[unique_id], state) pair, as groundwork for future source-node behavior improvements (Issue #2053).

Changes:

  • Updated cosmos/operators/watcher.py:_default_freshness_callback return type to list[tuple[str, str]].
  • Updated the producer call site to consume the new return shape.
  • Updated _default_freshness_callback unit tests to assert against the new structure.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
cosmos/operators/watcher.py Changes callback return type and adapts _apply_source_freshness to the new structure.
tests/operators/test_watcher.py Updates tests to validate the new (unique_id, state) results.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cosmos/operators/watcher.py Outdated
Comment thread cosmos/operators/watcher.py Outdated
- Change freshness callback state value from "skip" to "skipped"
- Rename _skipped_node_token to _apply_node_state_tokens; accept
  list[tuple[str, str]] so each node carries its own state
- Rename _push_skipped_xcom_for_model to _push_node_state_xcom;
  take an explicit state argument instead of hard-coding "skipped"
- Only add nodes to self.exclude when their state is "skipped";
  other states push XCom but leave the dbt exclude list unchanged
- Update all call sites and tests accordingly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 17, 2026 16:23
@pankajastro pankajastro review requested due to automatic review settings April 17, 2026 16:23
@pankajastro pankajastro self-assigned this Apr 21, 2026
@tatiana tatiana changed the title Refactor _default_freshness_callback to return list of (id, state) tuple Refactor _default_freshness_callback to return list of (id, state) tuple Apr 21, 2026
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @pankajastro , it looks good, let's ship this as part of Cosmos 1.15.0

@pankajastro pankajastro merged commit b218d33 into main Apr 21, 2026
124 checks passed
@pankajastro pankajastro deleted the refactor-freshness-callback-return-type branch April 21, 2026 13:57
@pankajastro pankajastro added this to the Cosmos 1.15.0 milestone Apr 21, 2026
pankajastro added a commit that referenced this pull request May 4, 2026
…2617)

Add a new experimental section describing the source freshness aware
execution feature introduced in PRs
#2467,
#2572, and
#2586:

- Update the "Source freshness nodes" limitation to reflect that the
  feature is now supported since Cosmos 1.15.0
- Add a "Source freshness aware execution (Experimental)" section under
  Advanced config covering activation, default callback behaviour,
  consumer state handling, callback signature, and known limitations

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants