Support cross-referencing models across dbt projects using dbt-loom by pankajkoti · Pull Request #2271 · astronomer/astronomer-cosmos

pankajkoti · 2026-01-15T15:30:36Z

This PR adds support for dbt-loom, enabling Cosmos to work with multi-project dbt architectures where downstream projects reference models from upstream projects.

When using dbt-loom, downstream projects reference upstream models via {{ ref('upstream_project', 'model_name') }}. dbt-loom injects these external model references into the downstream project's namespace by reading the upstream project's manifest.json

Cosmos now automatically detects and skips external nodes (those without original_file_path) during DAG generation, while still creating tasks for the project's own models. This works for both:
LoadMode.DBT_LS - parsing via dbt ls
LoadMode.DBT_MANIFEST - parsing via manifest file

The PR adds the example Projects (in dev/dags/dbt/):
dbt_loom_upstream_platform/ - staging & intermediate models with seeds
dbt_loom_downstream_finance/ - finance fact tables referencing upstream models
dbt_loom_dags.py - combined DAG with chained task groups

The PR also adds a comprehensive guide for multi-project setups in docs/configuration/multi-project.rst

closes: #2107

Co-authored-by: Tatiana Al-Chueyr tatiana.alchueyr@gmail.com

netlify · 2026-01-15T15:30:43Z

✅ Deploy Preview for astronomer-cosmos canceled.

Name	Link
🔨 Latest commit	`328937a`
🔍 Latest deploy log	https://app.netlify.com/projects/astronomer-cosmos/deploys/697b3553086b3b000825e8ac

codecov · 2026-01-21T09:12:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.99%. Comparing base (46d57ee) to head (328937a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2271   +/-   ##
=======================================
  Coverage   97.99%   97.99%           
=======================================
  Files         100      100           
  Lines        6431     6440    +9     
=======================================
+ Hits         6302     6311    +9     
  Misses        129      129

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR adds support for multi-project dbt setups using dbt-loom, enabling Cosmos to handle cross-project references where downstream dbt projects reference models from upstream projects.

Changes:

Cosmos now skips external nodes (those without file paths) injected by dbt-loom during DAG generation
Added comprehensive documentation for multi-project setups with configuration examples
Included test coverage for the external node skipping behavior

Reviewed changes

Copilot reviewed 32 out of 34 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
cosmos/dbt/graph.py	Added logic to skip nodes without file paths in both manifest and dbt ls parsing methods
tests/dbt/test_graph.py	Added test to verify external nodes from dbt-loom are properly skipped
docs/configuration/multi-project.rst	New comprehensive documentation explaining multi-project setups with dbt-loom
docs/configuration/index.rst	Added multi-project documentation to the configuration index
pyproject.toml	Added dbt-loom as an optional dependency
scripts/test/pre-install-airflow.sh	Added dbt-loom installation to test setup
dev/dags/dbt_loom_dags.py	Example DAG demonstrating multi-project setup
dev/dags/dbt/dbt_loom_upstream_platform/*	Example upstream dbt project files
dev/dags/dbt/dbt_loom_downstream_finance/*	Example downstream dbt project files

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tatiana · 2026-01-23T10:07:40Z

Hi @pankajkoti ! This work is very exciting - it's really cool to be able to have a dbt Mesh feature in Cosmos without the need to lock into a proprietary platform.

I'd love it if you could address the following points, as we discussed yesterday:

Inside the dev/dags/dbt folder, put both new dbt projects into the folder cross-project. There, I suggest one of the projects is named upstream (or parent) and the other is named downstream (or child), so it is very clear why we have those projects
Please, could you also confirm that Airflow Assets/Datasets emitted via Cosmos by the upstream DAG could trigger the downstream DAG?
Please, could you confirm if the syntax is exactly the same as dbt Mesh, and if there are any other lacking feaeture parities
Were you able to test the cross-project reference, considering users were using LoadMode.DBT_MANIFEST both on the upstream and downstream projects?
Would it be worth highlighting in our example or in the docs the fact that both projects can be run with different profiles?
Is there a way of running nodes of different projects in the same Cosmos DbtTaskGroup?

Add two minimal dbt projects to demonstrate dbt Loom behavior: - platform_project: upstream with 2 public models, 1 macro, 1 source - finance_project: downstream with cross-project refs and dbt Loom config These projects verify that dbt Loom only injects models (not macros or sources) cross-project.

Add two example DAGs to test Cosmos compatibility with dbt Loom: - dbt_loom_platform_dag.py: Upstream project with public models - dbt_loom_finance_dag.py: Downstream project using dbt Loom cross-project refs

dbt Projects: - Rename platform_project -> dbt_loom_upstream_platform - Rename finance_project -> dbt_loom_downstream_finance - Add comprehensive seed data (customers, orders, order_items, products) - Add staging models with public access for cross-project refs - Add intermediate models (int_orders_enriched, int_customer_orders) - Add finance models (fct_revenue, fct_customer_revenue, dim_payment_methods) - Update profiles to use PostgreSQL - Consolidate into single DAG with chained task groups Documentation: - Add docs/configuration/multi-project.rst with comprehensive guide - Cover cross-project model references using dbt-loom - Document patterns for cross-project sources and macros - Include Cosmos DAG configuration examples - Add troubleshooting and best practices sections

Remove curly braces from ref() example in docstring to prevent Airflow from trying to render it as a Jinja template.

pankajkoti · 2026-01-27T17:56:08Z

Thanks for the detailed review @tatiana . I have addressed the feedback.

Inside the dev/dags/dbt folder, put both new dbt projects into the folder cross-project. There, I suggest one of the projects is named upstream (or parent) and the other is named downstream (or child), so it is very clear why we have those projects

Done so, thanks!

Please, could you also confirm that Airflow Assets/Datasets emitted via Cosmos by the upstream DAG could trigger the downstream DAG?

Yes, verified this, works smoothly. Added snapshots in the docs as we discussed earlier today.

Please, could you confirm if the syntax is exactly the same as dbt Mesh, and if there are any other lacking feaeture parities

Yes, confiremd that the syntax is exactly the same as dbt Mesh.

Were you able to test the cross-project reference, considering users were using LoadMode.DBT_MANIFEST both on the upstream and downstream projects?

Yes, I tested this with both projects using LoadMode.DBT_MANIFEST and it works well. Also have added an example DAG for the same to be run in our CI.

Would it be worth highlighting in our example or in the docs the fact that both projects can be run with different profiles?

Yes. highlighted this in our docs now

Is there a way of running nodes of different projects in the same Cosmos DbtTaskGroup?

No, this is not currently supported. Each DbtTaskGroup (or DbtDag) is configured with a single ProjectConfig that points to one dbt project.
When Cosmos parses a dbt-loom project, it encounters external nodes injected from upstream projects. However, Cosmos intentionally skips these external nodes during DAG generation -- they don't have local file paths and are meant to be run by their own project. Cosmos only creates Airflow tasks for the models that belong to the current project. Supporting multiple projects in a single DbtTaskGroup would require significant changes, I believe.

Requesting re-review, please!

Copilot

Pull request overview

Copilot reviewed 35 out of 41 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

tatiana

Hi @pankajkoti, this looks great, thank you very much!

I have one last concern, discussed in the thread https://github.com/astronomer/astronomer-cosmos/pull/2271/changes#r2736991273.

Once you address this and the checks are passing, please feel free to merge the PR.

…t-loom

@pankajkoti

Features * Support cross-referencing models across dbt projects using dbt-loom by @pankajkoti in #2271 * Support use of YAML selectors when using ``LoadMode.DBT_MANIFEST`` by @YourRoyalLinus in #2261 * Introduce ``ExecutionMode.WATCHER_KUBERNETES`` to use the watcher with ``KubernetesPodOperator`` by @tatiana in #2207 * Add support for StarRocks profile mapping by @kurkim0661 in #2256 * Allow pushing URIs as XComs for Cosmos tasks by @corsettigyg in #2275 * Support defining custom callbacks alongside the ``WATCHER_KUBERNETES`` callback by @johnhoran in #2307 Enhancements * Refactor: remove duplicate ``_construct_dest_file_path`` by @jx2lee in #2077 * Leverage Airflow ``::group::`` to group logs associated with DAG parsing by @tatiana in #2235 * Refactor ``DbtConsumerWatcherSensor`` for reusability by @tatiana in #2245 * Restore plain text output when using ``ExecutionMode.WATCHER`` by @tiovader in #2241 Bug Fixes * Fix running empty models or ephemeral nodes in ``ExecutionMode.WATCHER`` by @tatiana in #2279 * Improve watcher producer task priority in scheduling and the UI by @tatiana in #2237 * Fix typos and formatting issues in documentation by @pankajkoti in #2259 * Allow watcher producer retries without erroring by @tatiana in #2283 * Fix ``TestBehavior.AFTER_ALL`` is missing project_name information when loading project using manifest file by @tuantran0910 in #2242 * Fix duplicate log lines in watcher subprocess execution and format timestamps by @pankajkoti in #2301 Docs * Add Watcher Kubernetes documentation by @tatiana in #2303 * Document newly added telemetry metrics in the privacy notice by @pankajkoti in #2249 * Add compatibility policy document by @pankajastro in #2251 * Improve watcher documentation related to dbt threads by @tatiana in #2273 * Fix link in watcher execution mode documentation by @jedcunningham in #2277 * Update Apache Airflow minimum compatibility policy by @tatiana in #2285 * Clarify Cosmos runtime support until "End of Basic Support" by @jedcunningham in #2286 * Update watcher docs by @tatiana in #2298 * Update watcher kubernetes documentation by @tatiana in #2306 Others * Add Airflow 3 DAG versioning tests for Cosmos by @michal-mrazek in #2177 * Add dbt Core 1.11 to the test matrix by @tatiana in #2230 * Add integration tests using InvocationMode.SUBPROCESS and validate output by @tatiana in #2287 * Fix main branch failing tests by @tatiana in #2296 * Update pre-commit hooks to the latest versions by @jedcunningham in #2289 * Pre-commit autoupdates by @pre-commit in #2222, #2264, #2274 and #2290 * Dependabot updates by @dependabot in #2218, #2219, #2220, #2280 and #2284 * Add Scarf metrics to understand Cosmos feature usage patterns - Add telemetry tracking for dbt docs plugin usage by @pankajkoti in #2240 - Add DAG run telemetry metrics for load mode, invocation, and render_config parameters by @pankajkoti in #2223 - Collect profile metrics for DAG runs by @pankajastro in #2228 - Compress telemetry metadata to reduce serialized DAG size by @pankajkoti in #2252 - Skip storing telemetry metadata when emission is disabled by @pankajkoti in #2278 - Hide telemetry metadata parameters from the Airflow trigger UI by @pankajkoti in #2247 closes: astronomer/oss-integrations-private#317 --------- Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

…ternal nodes When using the `+` (precursor) graph selector with dbt-loom cross-project references, `select_node_precursors` crashes with a `KeyError` because external nodes (injected by dbt-loom) are filtered out during manifest loading but local nodes still reference them in `depends_on`. The dbt-loom support added in astronomer#2271 correctly skips external nodes (those without `original_file_path`) during manifest loading. However, when the `+` graph operator traverses upstream dependencies, it encounters `depends_on` entries pointing to these filtered-out external nodes and raises a `KeyError`. This fix adds bounds checks in two locations: - `GraphSelector.select_node_precursors`: skip node IDs not present in the nodes dict during upstream traversal - `NodeSelector.select_nodes_ids_by_intersection`: skip external node IDs that were collected during graph traversal but are not in the nodes dict This allows the graph traversal to gracefully stop at project boundaries, which is the correct behavior for cross-project setups where external dependencies are managed by their own DAGs/task groups. Closes #<TBD> Co-authored-by: Cursor <cursoragent@cursor.com>

@pankajkoti

#2389) Fixes a `KeyError` when using the `+` (precursor) graph selector on a project that uses dbt-loom for cross-project references. cc @pankajkoti @tatiana — This is a follow-up to your dbt-loom support in #2271. The external node skipping works great for basic rendering, but we hit a `KeyError` when combining it with the `+` graph selector. The `+` operator triggers `select_node_precursors` which traverses `depends_on` entries — and those can point to external nodes that were already filtered out during manifest loading. This code path wasn't exercised by the tests in #2271 since the example DAGs don't use graph selectors. ## Problem The dbt-loom support added in #2271 correctly skips external nodes (those without `original_file_path`) during manifest loading in `load_from_dbt_manifest`. However, local nodes still have `depends_on` entries pointing to these filtered-out external nodes. When the `+` graph operator traverses upstream dependencies via `select_node_precursors`, it does `nodes[node_id]` on these external node IDs and raises a `KeyError`: File "cosmos/dbt/selector.py", line 172, in select_node_precursors new_generation.update(set(nodes[node_id].depends_on)) ~~~~~^^^^^^^^^ KeyError: 'model.upstream_project.external_model' **Reproduction:** Use `select: ["+downstream_model"]` in `RenderConfig` with `LoadMode.DBT_MANIFEST` on a project that uses dbt-loom with cross-project `{{ ref('upstream_project', 'model_name') }}` references. ## Fix Adds bounds checks in two locations in `cosmos/dbt/selector.py`: 1. **`GraphSelector.select_node_precursors`** (line 172): Skip node IDs not present in the `nodes` dict during upstream traversal 2. **`NodeSelector.select_nodes_ids_by_intersection`** (line 552): Skip external node IDs that were collected during graph traversal but don't exist in the `nodes` dict This allows the `+` traversal to gracefully stop at project boundaries — the correct behavior for cross-project setups where external dependencies are managed by their own DAGs/task groups. This is consistent with how `select_node_descendants` already handles missing parents via `defaultdict(set)`. ## Test plan - [x] Added `test_select_nodes_by_precursors_with_external_dependency` — creates a graph where a local node's `depends_on` includes an external node ID not in the `nodes` dict, verifies `+` selector returns local nodes without `KeyError` - [x] All 166 existing selector tests pass - [x] All existing dbt-loom tests in `test_graph.py` pass Co-authored-by: Alex Ward <award@Mac.lan> Co-authored-by: Cursor <cursoragent@cursor.com>

@pankajkoti

#2389) Fixes a `KeyError` when using the `+` (precursor) graph selector on a project that uses dbt-loom for cross-project references. cc @pankajkoti @tatiana — This is a follow-up to your dbt-loom support in #2271. The external node skipping works great for basic rendering, but we hit a `KeyError` when combining it with the `+` graph selector. The `+` operator triggers `select_node_precursors` which traverses `depends_on` entries — and those can point to external nodes that were already filtered out during manifest loading. This code path wasn't exercised by the tests in #2271 since the example DAGs don't use graph selectors. ## Problem The dbt-loom support added in #2271 correctly skips external nodes (those without `original_file_path`) during manifest loading in `load_from_dbt_manifest`. However, local nodes still have `depends_on` entries pointing to these filtered-out external nodes. When the `+` graph operator traverses upstream dependencies via `select_node_precursors`, it does `nodes[node_id]` on these external node IDs and raises a `KeyError`: File "cosmos/dbt/selector.py", line 172, in select_node_precursors new_generation.update(set(nodes[node_id].depends_on)) ~~~~~^^^^^^^^^ KeyError: 'model.upstream_project.external_model' **Reproduction:** Use `select: ["+downstream_model"]` in `RenderConfig` with `LoadMode.DBT_MANIFEST` on a project that uses dbt-loom with cross-project `{{ ref('upstream_project', 'model_name') }}` references. ## Fix Adds bounds checks in two locations in `cosmos/dbt/selector.py`: 1. **`GraphSelector.select_node_precursors`** (line 172): Skip node IDs not present in the `nodes` dict during upstream traversal 2. **`NodeSelector.select_nodes_ids_by_intersection`** (line 552): Skip external node IDs that were collected during graph traversal but don't exist in the `nodes` dict This allows the `+` traversal to gracefully stop at project boundaries — the correct behavior for cross-project setups where external dependencies are managed by their own DAGs/task groups. This is consistent with how `select_node_descendants` already handles missing parents via `defaultdict(set)`. ## Test plan - [x] Added `test_select_nodes_by_precursors_with_external_dependency` — creates a graph where a local node's `depends_on` includes an external node ID not in the `nodes` dict, verifies `+` selector returns local nodes without `KeyError` - [x] All 166 existing selector tests pass - [x] All existing dbt-loom tests in `test_graph.py` pass Co-authored-by: Alex Ward <award@Mac.lan> Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 4d86173)

pankajkoti temporarily deployed to internal January 15, 2026 15:30 — with GitHub Actions Inactive

pankajkoti force-pushed the dbt-loom-poc-projects branch from 6291d2b to fc7a16c Compare January 15, 2026 15:30

pankajkoti temporarily deployed to internal January 15, 2026 15:35 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal January 15, 2026 16:39 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal January 15, 2026 16:50 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal January 15, 2026 17:17 — with GitHub Actions Inactive

pankajkoti force-pushed the dbt-loom-poc-projects branch from 2b7d3e4 to b2dd259 Compare January 15, 2026 17:36

pankajkoti temporarily deployed to internal January 15, 2026 17:36 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal January 15, 2026 18:08 — with GitHub Actions Inactive

pankajkoti requested review from pankajastro and tatiana January 16, 2026 10:57

pankajkoti temporarily deployed to internal January 21, 2026 08:40 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal January 21, 2026 15:20 — with GitHub Actions Inactive

pankajkoti force-pushed the dbt-loom-poc-projects branch from b31e48e to a65c838 Compare January 21, 2026 15:20

pankajkoti temporarily deployed to internal January 21, 2026 15:21 — with GitHub Actions Inactive

pankajkoti marked this pull request as ready for review January 21, 2026 15:25

Copilot AI review requested due to automatic review settings January 21, 2026 15:25

Copilot AI reviewed Jan 21, 2026

View reviewed changes

Comment thread dev/dags/dbt/dbt_loom_downstream_finance/dbt_loom.config.yml Outdated

Comment thread cosmos/dbt/graph.py Outdated

Comment thread dev/dags/dbt/cross_project/downstream/models/fct_revenue.sql Outdated

pankajkoti changed the title ~~Support multiple dbt projects using dbt-loom~~ Support cross-referencing models across dbt projects using dbt-loom Jan 22, 2026

tatiana reviewed Jan 23, 2026

View reviewed changes

Comment thread docs/configuration/multi-project.rst

pankajkoti temporarily deployed to internal January 27, 2026 15:56 — with GitHub Actions Inactive

pankajkoti added 6 commits January 27, 2026 21:27

Add Cosmos DAGs for dbt Loom PoC testing

4ddddc9

Add two example DAGs to test Cosmos compatibility with dbt Loom: - dbt_loom_platform_dag.py: Upstream project with public models - dbt_loom_finance_dag.py: Downstream project using dbt Loom cross-project refs

Fix for cross-ref model

f489b4f

Fix example dag

1e01de1

Fix Jinja template error in DAG docstring

26e6608

Remove curly braces from ref() example in docstring to prevent Airflow from trying to render it as a Jinja template.

Address docs feedback

7ec9445

pankajkoti temporarily deployed to internal January 27, 2026 17:46 — with GitHub Actions Inactive

pankajkoti requested a review from tatiana January 27, 2026 17:56

Address Copilot's review comment

4969e16

pankajkoti temporarily deployed to internal January 27, 2026 18:04 — with GitHub Actions Inactive

pankajkoti requested a review from Copilot January 27, 2026 18:05

Copilot AI reviewed Jan 27, 2026

View reviewed changes

Comment thread cosmos/dbt/graph.py Outdated

Comment thread dev/dags/dbt/cross_project/downstream/dbt_loom.config.yml

Comment thread docs/configuration/multi-project.rst Outdated

Update docs/configuration/multi-project.rst

d843c5d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

pankajkoti temporarily deployed to internal January 27, 2026 18:11 — with GitHub Actions Inactive

tatiana reviewed Jan 28, 2026

View reviewed changes

Comment thread cosmos/dbt/graph.py

tatiana reviewed Jan 28, 2026

View reviewed changes

Comment thread pyproject.toml Outdated

tatiana reviewed Jan 28, 2026

View reviewed changes

Comment thread cosmos/dbt/graph.py Outdated

tatiana approved these changes Jan 28, 2026

View reviewed changes

Add safety check for skipping upstream external models injected by db…

53edaf1

…t-loom

pankajkoti temporarily deployed to internal January 29, 2026 10:20 — with GitHub Actions Inactive

pankajkoti mentioned this pull request Jan 29, 2026

Add identifier for nodes injected by dbt-loom #2299

Open

pankajkoti commented Jan 29, 2026

View reviewed changes

Comment thread pyproject.toml Outdated

Apply suggestion from @pankajkoti

328937a

pankajkoti temporarily deployed to internal January 29, 2026 10:24 — with GitHub Actions Inactive

tatiana merged commit 0ce9954 into main Jan 29, 2026
90 checks passed

tatiana deleted the dbt-loom-poc-projects branch January 29, 2026 11:09

pankajkoti mentioned this pull request Jan 29, 2026

[Feature] PoC with dbt Loom to accomplish dbt Mesh cross-project resolution #2107

Closed

1 task

pankajastro mentioned this pull request Jan 29, 2026

Release 1.13.0 #2297

Merged

tatiana added this to the Cosmos 1.13.0 milestone Feb 19, 2026

award1230 mentioned this pull request Feb 20, 2026

Fix KeyError in graph selector when using + operator with dbt-loom ex… #2389

Merged

3 tasks

Conversation

pankajkoti commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify Bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for astronomer-cosmos canceled.

Uh oh!

codecov Bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana commented Jan 23, 2026

Uh oh!

pankajkoti commented Jan 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tatiana left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pankajkoti commented Jan 15, 2026 •

edited

Loading

netlify Bot commented Jan 15, 2026 •

edited

Loading

codecov Bot commented Jan 21, 2026 •

edited

Loading

tatiana left a comment •

edited

Loading