Skip to content

Fix KeyError in graph selector when using + operator with dbt-loom ex…#2389

Merged
tatiana merged 1 commit into
astronomer:mainfrom
award1230:fix/graph-selector-dbt-loom-external-nodes
Feb 22, 2026
Merged

Fix KeyError in graph selector when using + operator with dbt-loom ex…#2389
tatiana merged 1 commit into
astronomer:mainfrom
award1230:fix/graph-selector-dbt-loom-external-nodes

Conversation

@award1230
Copy link
Copy Markdown
Contributor

Summary

Fixes a KeyError when using the + (precursor) graph selector on a project that uses dbt-loom for cross-project references.

cc @pankajkoti @tatiana — This is a follow-up to your dbt-loom support in #2271. The external node skipping works great for basic rendering, but we hit a KeyError when combining it with the + graph selector. The + operator triggers select_node_precursors which traverses depends_on entries — and those can point to external nodes that were already filtered out during manifest loading. This code path wasn't exercised by the tests in #2271 since the example DAGs don't use graph selectors.

Problem

The dbt-loom support added in #2271 correctly skips external nodes (those without original_file_path) during manifest loading in load_from_dbt_manifest. However, local nodes still have depends_on entries pointing to these filtered-out external nodes.

When the + graph operator traverses upstream dependencies via select_node_precursors, it does nodes[node_id] on these external node IDs and raises a KeyError:

File "cosmos/dbt/selector.py", line 172, in select_node_precursors
    new_generation.update(set(nodes[node_id].depends_on))
                              ~~~~~^^^^^^^^^
KeyError: 'model.upstream_project.external_model'

Reproduction: Use select: ["+downstream_model"] in RenderConfig with LoadMode.DBT_MANIFEST on a project that uses dbt-loom with cross-project {{ ref('upstream_project', 'model_name') }} references.

Fix

Adds bounds checks in two locations in cosmos/dbt/selector.py:

  1. GraphSelector.select_node_precursors (line 172): Skip node IDs not present in the nodes dict during upstream traversal
  2. NodeSelector.select_nodes_ids_by_intersection (line 552): Skip external node IDs that were collected during graph traversal but don't exist in the nodes dict

This allows the + traversal to gracefully stop at project boundaries — the correct behavior for cross-project setups where external dependencies are managed by their own DAGs/task groups. This is consistent with how select_node_descendants already handles missing parents via defaultdict(set).

Test plan

  • Added test_select_nodes_by_precursors_with_external_dependency — creates a graph where a local node's depends_on includes an external node ID not in the nodes dict, verifies + selector returns local nodes without KeyError
  • All 166 existing selector tests pass
  • All existing dbt-loom tests in test_graph.py pass

…ternal nodes

When using the `+` (precursor) graph selector with dbt-loom cross-project
references, `select_node_precursors` crashes with a `KeyError` because
external nodes (injected by dbt-loom) are filtered out during manifest
loading but local nodes still reference them in `depends_on`.

The dbt-loom support added in astronomer#2271 correctly skips external nodes (those
without `original_file_path`) during manifest loading. However, when the
`+` graph operator traverses upstream dependencies, it encounters
`depends_on` entries pointing to these filtered-out external nodes and
raises a `KeyError`.

This fix adds bounds checks in two locations:
- `GraphSelector.select_node_precursors`: skip node IDs not present in
  the nodes dict during upstream traversal
- `NodeSelector.select_nodes_ids_by_intersection`: skip external node IDs
  that were collected during graph traversal but are not in the nodes dict

This allows the graph traversal to gracefully stop at project boundaries,
which is the correct behavior for cross-project setups where external
dependencies are managed by their own DAGs/task groups.

Closes #<TBD>

Co-authored-by: Cursor <cursoragent@cursor.com>
Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@award1230 this is amazing, thank you very much for your contribution. It completely makes sense, and both the implementation & tests are spot on. I left inline minor comment - and I'm happy for us to merge this once you address. We could add this to the 1.13.1 release, planned for Monday

Comment thread cosmos/dbt/selector.py
Comment thread cosmos/dbt/selector.py
@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 21, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.96%. Comparing base (9968648) to head (5e5d3ef).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2389   +/-   ##
=======================================
  Coverage   97.96%   97.96%           
=======================================
  Files         102      102           
  Lines        6980     6983    +3     
=======================================
+ Hits         6838     6841    +3     
  Misses        142      142           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

evanvolgas pushed a commit to evanvolgas/astronomer-cosmos that referenced this pull request Feb 22, 2026
…ternal nodes

When using dbt-loom for cross-project references, external nodes are filtered
out during manifest loading (they have no file path). However, local nodes
may still have depends_on entries pointing to these external nodes.

The + graph operator triggers select_node_precursors which traverses depends_on
entries, and when it encounters these external node IDs that were filtered out,
it raises a KeyError.

This fix adds bounds checks in two locations in cosmos/dbt/selector.py:

1. GraphSelector.select_node_precursors: Skip node IDs not present in the
   nodes dict during upstream traversal
2. NodeSelector.select_nodes_ids_by_intersection: Skip external node IDs that
   were collected during graph traversal but don't exist in the nodes dict

This allows the + traversal to gracefully stop at project boundaries, which
is the correct behavior for cross-project setups where external dependencies
are managed by their own DAGs/task groups.

Original fix by @award1230 in astronomer#2389

Co-Authored-By: award1230 <26311596+award1230@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@evanvolgas
Copy link
Copy Markdown
Contributor

@tatiana I've found Github notifications to be a bit wonky sometimes. Cross posting here, in case:

#2394 is Alex's PR + the comments you asked for. Alex is out til next Monday but my team is hoping we can get his code merged as soon as possible.

Please let me know your thoughts!

@tatiana
Copy link
Copy Markdown
Collaborator

tatiana commented Feb 22, 2026

@evanvolgas thank you very much for following up on this. I really appreciate for the follow up PR with the comment. I'll go ahead and merge this PR, and if you can rebase your after, updating the PR description - it would be great! This way we acknowledge both of your contributions

@tatiana tatiana merged commit 4d86173 into astronomer:main Feb 22, 2026
74 checks passed
@pankajkoti
Copy link
Copy Markdown
Contributor

Nice, thanks a lot for testing out the addition to Cosmos and contribution of the fix @award1230 👏🏽

tatiana pushed a commit that referenced this pull request Feb 23, 2026
Address the feedback from the #2389 code review to help maintain the code in the future.
tatiana pushed a commit that referenced this pull request Feb 23, 2026
#2389)

Fixes a `KeyError` when using the `+` (precursor) graph selector on a
project that uses dbt-loom for cross-project references.

cc @pankajkoti @tatiana — This is a follow-up to your dbt-loom support
in #2271. The external node skipping works great for basic rendering,
but we hit a `KeyError` when combining it with the `+` graph selector.
The `+` operator triggers `select_node_precursors` which traverses
`depends_on` entries — and those can point to external nodes that were
already filtered out during manifest loading. This code path wasn't
exercised by the tests in #2271 since the example DAGs don't use graph
selectors.

## Problem

The dbt-loom support added in #2271 correctly skips external nodes
(those without `original_file_path`) during manifest loading in
`load_from_dbt_manifest`. However, local nodes still have `depends_on`
entries pointing to these filtered-out external nodes.

When the `+` graph operator traverses upstream dependencies via
`select_node_precursors`, it does `nodes[node_id]` on these external
node IDs and raises a `KeyError`:

    File "cosmos/dbt/selector.py", line 172, in select_node_precursors
        new_generation.update(set(nodes[node_id].depends_on))
                                  ~~~~~^^^^^^^^^
    KeyError: 'model.upstream_project.external_model'

**Reproduction:** Use `select: ["+downstream_model"]` in `RenderConfig`
with `LoadMode.DBT_MANIFEST` on a project that uses dbt-loom with
cross-project `{{ ref('upstream_project', 'model_name') }}` references.

## Fix

Adds bounds checks in two locations in `cosmos/dbt/selector.py`:

1. **`GraphSelector.select_node_precursors`** (line 172): Skip node IDs
not present in the `nodes` dict during upstream traversal
2. **`NodeSelector.select_nodes_ids_by_intersection`** (line 552): Skip
external node IDs that were collected during graph traversal but don't
exist in the `nodes` dict

This allows the `+` traversal to gracefully stop at project boundaries —
the correct behavior for cross-project setups where external
dependencies are managed by their own DAGs/task groups. This is
consistent with how `select_node_descendants` already handles missing
parents via `defaultdict(set)`.

## Test plan

- [x] Added `test_select_nodes_by_precursors_with_external_dependency` —
creates a graph where a local node's `depends_on` includes an external
node ID not in the `nodes` dict, verifies `+` selector returns local
nodes without `KeyError`
- [x] All 166 existing selector tests pass
- [x] All existing dbt-loom tests in `test_graph.py` pass

Co-authored-by: Alex Ward <award@Mac.lan>
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit 4d86173)
tatiana pushed a commit that referenced this pull request Feb 23, 2026
Address the feedback from the #2389 code review to help maintain the code in the future.

(cherry picked from commit 78a4358)
@tatiana tatiana mentioned this pull request Feb 23, 2026
@tatiana tatiana added this to the Cosmos 1.13.1 milestone Feb 24, 2026
tatiana added a commit that referenced this pull request Feb 25, 2026
**Enhancements**

* Change Snowflake profile mappings to default to four threads by
@tatiana in #2374
* Refactor to avoid potential future ``UnboundLocalError`` for
``producer_task`` in ``calculate_tasks_map`` by @rin in #2309

**Bug Fixes**

* Fix graph selector when using + selector with ``dbt-loom`` by
@award1230 in #2389
* Populate ``compiled_sql`` for ``InvocationMode.SUBPROCESS`` in
``ExecutionMode.WATCHER`` by @pankajkoti in #2319
* Preserve ``extra_context`` for watcher consumer task instances by
@pankajkoti in #2381
* Fix watcher: respect ``deferrable=False`` from ``operator_args`` on
consumer sensor by @pankajkoti in #2384
* Error handle invalid YAML with ``LoadMode.DBT_MANIFEST`` and
``RenderConfig.selector`` by @jonbillings in #2316
* Fix selecting model when it has the same name as folder by
@pankajastro in #2328
* Handle Param Validation errors by @tatiana in #2358
* Fix cache swap issue by @jonbillings in #2332
* Fix leaked semaphore warnings in Airflow 3 by resetting dbt adapters
by @pankajkoti in #2335

**Docs**

* Document ``ExecutionMode.KUBERNETES`` limitations by @tatiana in #2326

**Others**

* Add .airflow-registry.yaml for Airflow Provider Registry by @kaxil in
#2387
* Improve test coverage for PR #2307 by @tatiana in #2308
* Address feedback from code review #2389 by @evanvolgas in #2394

Closes:
astronomer/oss-integrations-private#333
@tatiana tatiana mentioned this pull request Feb 25, 2026
tatiana added a commit that referenced this pull request Feb 25, 2026
Enhancements

* Change Snowflake profile mappings to default to four threads by
@tatiana in #2374
* Refactor to avoid potential future ``UnboundLocalError`` for
``producer_task`` in ``calculate_tasks_map`` by @rin in #2309

Bug Fixes

* Fix graph selector when using + selector with ``dbt-loom`` by
@award1230 in #2389
* Populate ``compiled_sql`` for ``InvocationMode.SUBPROCESS`` in
``ExecutionMode.WATCHER`` by @pankajkoti in #2319
* Preserve ``extra_context`` for watcher consumer task instances by
@pankajkoti in #2381
* Fix watcher: respect ``deferrable=False`` from ``operator_args`` on
consumer sensor by @pankajkoti in #2384
* Error handle invalid YAML with ``LoadMode.DBT_MANIFEST`` and
``RenderConfig.selector`` by @YourRoyalLinus in #2316
* Fix selecting model when it has the same name as folder by
@pankajastro in #2328
* Handle Param Validation errors by @tatiana in #2358
* Fix cache swap issue by @YourRoyalLinus in #2332
* Fix leaked semaphore warnings in Airflow 3 by resetting dbt adapters
by @pankajkoti in #2335

Docs

* Document ``ExecutionMode.KUBERNETES`` limitations by @tatiana in #2326

Others

* Add .airflow-registry.yaml for Airflow Provider Registry by @kaxil in
#2387
* Improve test coverage for PR #2307 by @tatiana in #2308
* Address feedback from code review #2389 by @evanvolgas in #2394

Additional details on this change:
- The actual release was done from branch 1.13:
https://github.com/astronomer/astronomer-cosmos/releases/tag/astronomer-cosmos-v1.13.1
- This PR aims to update our CHANGELOG, and allow us to give credit to
the username that was being questioned by codespell
@pankajastro pankajastro mentioned this pull request Mar 16, 2026
pankajastro added a commit that referenced this pull request Apr 7, 2026
1.14.0 (2026-04-07)
---------------------

Breaking Changes

* Drop support for Airflow versions earlier than **2.9** by
@jedcunningham in #2288
* Fix inclusion of package models and selection/exclusion behavior by
@pankajkoti in #2357
* ``ExecutionMode.WATCHER``: The per-node ``*_status`` XCom value is now
a dict (``{"status": "<status>", "outlet_uris": [...]}``) instead of a
plain string. Any custom code that reads these internal XCom keys
directly will need to be updated by @pankajkoti in #2507

Features

* Add cluster policy support for ``ExecutionMode.WATCHER`` sensor
retries by @astro-anand in #2293
* Add debug mode to track memory utilization by @tatiana in #2327
* Add FQN selection support for ``LoadMode.DBT_MANIFEST`` by
@pankajastro in #2375
* Introduce interceptors for Cosmos tasks by @tatiana in #2419
* Add config to allow disabling dag versioning by @pankajkoti in #2470
* Implement TaskGroups by models folder by @maximilianoarcieri and
@tatiana in #1566, #2469, and #2420
* feat: implement DbtTestWatcherOperator by @michal-mrazek in #2447
* Add source freshness aware execution for ``ExecutionMode.WATCHER`` by
@pankajastro and @tatiana in #2467

* Note: Like ``ExecutionMode.WATCHER``, this feature is experimental and
its interface and implementation can change in the future.
* Add Airflow 3.2 support by @pankajastro and @pankajkoti in #2472

Enhancements

* Add watcher mode support for dbt test node states by @michal-mrazek in
#2318
* Rename watcher-mode sensor retry queue and reuse it for producer tasks
by @pankajastro in #2331
* Fix leaked semaphore warnings in Airflow 3 by resetting dbt adapters
by @pankajkoti in #2335
* Improve dbt Fusion support and related tests by @tatiana in #2356
* Default Snowflake profile mappings to four threads by @tatiana in
#2374
* Attempt to remove Pydantic as a dependency by @tatiana in #2377
* Log dbt-core and adapter versions in watcher consumer tasks by
@pankajastro in #2412
* Log model errors in watcher consumer on dbt node failure by
@pankajastro in #2431
* Reduce XCom read/write for tracking node state and errors in
ConsumerWatcher task by @pankajastro in #2471
* Remove duplicate debug log in watcher subprocess path by @tatiana in
#2494
* Simplify and unify WATCHER implementation regardless of InvocationMode
by @tatiana in #2498
* Switch to lazy imports in cosmos/__init__.py by @pankajkoti in #2531

Bug Fixes

* Handle invalid YAML errors with ``LoadMode.DBT_MANIFEST`` and
``RenderConfig.selector`` by @YourRoyalLinus in #2316
* Populate ``compiled_sql`` for ``InvocationMode.SUBPROCESS`` in
``ExecutionMode.WATCHER`` by @pankajkoti in #2319
* Fix select/exclude type mismatch by @tatiana in #2364
* Set ``emit_datasets=False`` for ``DbtTest*`` operators by @pankajastro
in #2365
* Set correct queue priority for watcher producer tasks by @pankajastro
in #2372
* Preserve ``extra_context`` for watcher consumer task instances by
@pankajkoti in #2381
* Respect ``deferrable=False`` from ``operator_args`` on watcher
consumer sensors by @pankajkoti in #2384
* Fix watcher queue precedence and add documentation by @pankajastro in
#2391
* Do not set ``compiled_sql`` on ``ExecutionMode.WATCHER`` producers by
@pankajkoti in #2440
* Remove const attribute for ``__cosmos_telemetry_metadata__`` dag param
by @pankajkoti in #2466
* Remove timeout override from Cosmos watcher sensors by @tatiana and
@claude in #2478
* Remove forced ``retries=0`` from watcher producer operators by
@tatiana in #2479
* RFC: Add patch for newer versions of amazon provider when running dbt
on EKS by @aoelvp94 in #2481
* Fix ``cosmos_debug_max_memory_mb`` XCom not pushed in Watcher sensor
tasks by @tatiana in #2503
* Fix ``TestBehavior.NONE`` and ``TestBehavior.AFTER_ALL`` exclude
ignored with selectors in ``ExecutionMode.WATCHER`` by @pankajkoti in
#2511
* Move dataset emission for ``ExecutionMode.WATCHER`` from producer to
consumer sensors by @pankajkoti in #2507

Docs

* Document cluster policy configuration for ``ExecutionMode.WATCHER``
sensor tasks by @pankajastro in #2315
* Remove outdated docs for the dbt docs plugin with Airflow 3 by
@pankajastro in #2353
* Make Watcher DBT Execution Queue heading clickable by @pankajastro in
#2354
* Update ``ExecutionMode.WATCHER`` documentation regarding test node
implementation by @jroachgolf84 in #2355
* Fix ``pre_dbt_fusion`` configuration rendering by @pankajastro in
#2369
* Add documentation for including/excluding nodes based on FQN by
@pankajastro in #2371
* Update watcher execution mode documentation by @tatiana in #2380
* Add documentation for ``DbtSeedLocalOperator`` by @jroachgolf84 in
#2383
* Fix miscellaneous Sphinx warnings by @pankajastro in #2395
* Improve contributing documentation by @lzdanski in #2397
* Add **Get Started in 5 Minutes** guide by @lzdanski in #2398
* Add Sphinx redirects package for documentation redirects by @lzdanski
in #2407
* Restructure **Getting Started** and **Guides** sections by @lzdanski
in #2418
* Add open-source quickstart by @lzdanski in #2439
* Fix documentation redirects by @lzdanski in #2442
* Restructure and refactor reference documentation by @lzdanski in #2443
* Add execution modes decision documentation by @lzdanski in #2444
* Add **Core Concepts** page to Getting Started by @lzdanski in #2448
* Add guide: *How Cosmos Works* by @lzdanski in #2449
* Update **Getting Started** overview and index pages by @lzdanski in
#2452
* Add guide: *How Cosmos Runs dbt* by @lzdanski in #2453
* Fix miscellaneous documentation links by @lzdanski in #2454
* Add Mermaid diagrams and execution mode diagrams by @lzdanski and
@tatiana in #2459
* Add documentation for memory optimization options by @pankajastro in
#2340
* Fix typo in watcher execution mode docs by @evanvolgas in #2485
* Fix minor documentation issues by @evanvolgas in #2489
* Add troubleshooting note for dbt debug logs in ExecutionMode.WATCHER
by @tatiana in #2491
* docs: unify RST header styles across documentation by @jigangz in
#2473
* docs: fix env var for rich logging by @vricciardulli in #2514
* docs: update dbt project path example for Airflow 3 Astro
compatibility by @yeoreums in #2512
* Document missing Cosmos Airflow config settings in cosmos-conf.rst by
@tatiana in #2515
* Split security-privacy policy doc and add dependency cooldown by
@pankajkoti in #2519
* Add performance optimization and troubleshooting docs by @pankajkoti
in #2521
* Update copyright year to 2026 by @tayloramurphy in #2527
* docs: Updating "Project Policies" to "Policies" in menu bar by
@jroachgolf84 in #2526

Others

* Fix tests after removing support for Airflow versions earlier than 2.9
by @tatiana in #2321
* Enable listener tests for Airflow 3.1 by @pankajastro in #2348
* Accept ``int`` or ``float`` for ``cosmos_debug_max_memory_mb`` in
integration tests by @pankajkoti in #2352
* Update ``CODEOWNERS`` to prioritize ``oss-integrations`` by @tatiana
in #2359
* Fix automatic reviewer assignment in GitHub by @tatiana and @phanikumv
in #2360
* Improve PyPI tagging by @tatiana in #2363
* Add integration tests for dbt Fusion and ``ExecutionMode.WATCHER`` by
@tatiana in #2373
* Fix Zizmor check by @tatiana in #2376
* Remove ``methodtools`` dependency by @tatiana in #2378
* Improve comments on #2389 by @evanvolgas in #2394
* Refactor ``load_from_dbt_manifest`` to reduce code complexity by
@pankajkoti in #2399
* Refactor ``_handle_no_precursors_or_descendants`` to reduce complexity
by @pankajkoti in #2400
* Improve issue templates by @tatiana in #2401
* Avoid running tests when only docs change by @tatiana in #2402
* Add ``no-reload`` target for serving docs locally by @pankajkoti in
#2405
* Fix test hash checks on macOS by @tatiana in #2406
* Attempt deterministic dbt project copy in test fixtures by @pankajkoti
in #2409
* Pin ``virtualenv <21`` due to hatch incompatibility in CI by
@pankajkoti in #2410
* Revert virtualenv pin for hatch installation in CI by @pankajkoti in
#2426
* Add version comments for commit SHA pinned GitHub Actions by
@pankajkoti in #2436
* Fix ``hatch run docs:build`` issues by @tatiana in #2437
* Minor code improvements by @dnskr in #2446
* Pre-commit autoupdate by @pre-commit-ci in #2367, #2396, #2422, #2451,
#2468, #2495, and #2516
* Add file to support Claude understanding the Cosmos repository by
@tatiana in #2458
* Dependency updates by @dependabot in #2368, #2425, #2435, #2465,
#2475, #2504, #2518, and #2528
* Isolate Scarf telemetry integration test into its own CI job by
@pankajkoti and @claude in #2477
* ci: upgrade Airflow version to 3.1 in MyPy type-check job by @yeoreums
in #2506
* Add commit message guidelines to CLAUDE.md by @pankajkoti in #2509
* Extend skipping tests in CI for more non-code file changes by
@pankajkoti in #2510
* Add Dependabot pre-commit support with 7-day cooldown by @pankajkoti
in #2517
* Enforce zero warnings policy for documentation by @dnskr in #2513

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants