Support to prune sources without downstream references in dbt projects

[corsettigyg]

Description

In cosmos, once the sources are rendered into our dag, an issue that might arise is that if we are selecting only a portion of our project or if we have dead-code in the sources.yml file, tasks for these unchecked sources will still be created, which will essentially pollute the UI.

For large projects with hundreds of sources, users often select a fraction of the main project per DbtDag object, but all sources being rendered at once essentially means that the project is unreadable. The idea is to allow users to pass a source_prunning flag to RenderConfig, which will essentially remove source tasks without downstream dependencies

Related Issue(s)

Breaking Change?

Should not be

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	fcd08e8
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/68dd76d7f848980008d98d50

[Copilot]

Pull Request Overview

This PR introduces source pruning functionality to Cosmos, allowing users to automatically remove source nodes from DAGs that have no downstream dependencies. This addresses the issue where unused or dead source nodes clutter DAGs, making large projects with hundreds of sources difficult to read when only selecting a subset of models.

Key changes:

• Added source_prunning parameter to RenderConfig to enable/disable source pruning
• Implemented DownstreamGraph class for efficient downstream dependency computation
• Enhanced DbtNode with downstream relationship tracking capabilities

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
cosmos/config.py	Added source_prunning parameter to RenderConfig
cosmos/dbt/graph.py	Implemented DownstreamGraph class and enhanced DbtNode with downstream tracking
cosmos/airflow/graph.py	Added source pruning logic to task creation pipeline
docs/configuration/source-nodes-rendering.rst	Added documentation for the new source pruning feature
dev/dags/example_source_prunning.py	Added example DAG demonstrating source pruning functionality
dev/dags/dbt/altered_jaffle_shop/models/staging/sources.yml	Added test source node without connections

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[tatiana]

HI @corsettigyg, thanks a lot for contributing this, it seems a valuable addition to Cosmos.

I know it is in draft mode - and I'll avoid adding any further comments right now - but I'd love it if we could get some early thoughts from @arojasb3 , who initially implemented the source-node support.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.81%. Comparing base (ba9b869) to head (fcd08e8).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1988   +/-   ##
=======================================
  Coverage   97.81%   97.81%           
=======================================
  Files          87       87           
  Lines        5526     5536   +10     
=======================================
+ Hits         5405     5415   +10     
  Misses        121      121           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[corsettigyg]

@tatiana ready for review now 💯 tested it out in many different scenarios and it is working fine. my only small concern would be with performance since it is yet a loop for the sources, which can end up being kinda expensive.

[tatiana]

@corsettigyg I had the hope we could get the feedback on this improvement based on other dbt source users - but I think we may not get that quick enough.

I have two feedback points:

1. Since this is a breaking change, perhaps we should make this an opt-in feature (disabled by default) - and have a ticket to make it default in Cosmos 2.0, associating it to that milestone.

2. Do you think it is worth to have this feature enabled/disabled per DAG? Would it be worth to, instead, having a "global" config that would affect all the Cosmos DAGs and TaskGroups, similar to these other configurations:
   https://astronomer.github.io/astronomer-cosmos/configuration/cosmos-conf.html?

Once we align on these two points, I'm happy to approve this PR, and we can have an alpha release exposing this feature, for early feedback as well - WDYT?

[pankajastro]

Overall, this looks good to me, and I'm happy for it to be merged once @tatiana's feedback is addressed.
A couple of questions out of curiosity:

• Is it common for dbt projects to leave dead code in sources.yml?
• Does the current source rendering respect the --select flag?

[corsettigyg]

@tatiana

1. it is already disabled by default 💯 python short-circuits the check if source_pruning is False (default)
2. no strong opinions here. I see the advantage to have it as a global config particularly speaking so i can implement it as such 😄

@pankajastro

1. Yes, especially when you have a big dbt project and you select per tags. You end up with a LOT of not used sources
2. yes

[tatiana]

It looks excellent, @corsettigyg, thanks a lot for addressing the feedback so quickly.

Once our CI main branch issue is resolved by merging #2012, please proceed with merging your change - you should have the necessary super-powers!

[tatiana]

@corsettigyg just saw you're on holidays until the 8th - I'll merge this PR, you'll have lots of opportunity to do the same in the future!