Repository navigation
#450 — cross-model column trace-to-source + blast-radius (normalized join key) - #481
Merged
Merged
Conversation
…ius (normalized join key) CLL-4 — the v0.2 explorer headline: trace a column to its SOURCE field across ref() boundaries, plus column-grain downstream blast-radius. The project-wide output-column map (cute-dbt's zero-compute catalog-equivalent) is built by running the CLL-2/CLL-3 intra resolver over EVERY model and stitching at ref() boundaries. Normalized join key (refinement 3 — the hardest correctness seam): a (database, schema, identifier) tuple parsed from relation_name AT INGESTION (RelationIndex), case-folded — NOT a raw relation_name string-match. The cross-model stitch is gated on BOTH the normalized identifier AND the model's ACTUAL depends_on producers (DagFacts.lineage / ModelLineage.backward), so a leaf that does not uniquely normalize-join against a real producer degrades to Opaque — NEVER attributed to a wrong upstream. Verified on the real 35-model playground manifest: 0 edges attribute to a non-producer. - C (trace-to-source): forward recursion across ref() boundaries to the source()/seed leaf; terminates Source/Root/Opaque (honest thinning). - B (blast-radius): BFS with a visited-set over the cross-model edge set (the child_map direction) — reads ModelLineage, never a third self-inversion. - Star discipline: select * over a KNOWN modeled upstream resolves via the derived projection map; * over an UNKNOWN external stays Opaque; chains of * compound Opaque (no catalog, no fabrication). Scope-as-parameter (architectural): the project-wide graph is built ONLY in the explorer arm (build_cross_model_columns in render_explore). The per-model report path is untouched — the report goldens are byte-identical; only the playground-based explore goldens change. Verified by diffing a fresh report render against the committed golden. New domain module src/domain/column_lineage.rs (pure POD + std/serde — passes domain_clean_arch); the ColumnScope::Cross arm reserved by CLL-2 becomes live. Dogfood: the existing synthetic playground manifest already exercises a cross-model trace + star-over-ref (975 edges, 144 source-terminal traces) — no new fixture needed, no root_path leak. TDD: 15 domain unit tests + 10 integration tests built through the REAL multi-model parse path (parse_cte_graph then model_outputs then ProjectColumnGraph), never a hand-authored join. crap4rs: column_lineage.rs added to the strict 15 keyed set. Closes #450 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Warning Review limit reached
More reviews will be available in 11 minutes and 42 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses rolling per-developer review limits. Reviews become available again as older review attempts age out of the rolling limit window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughImplements project-wide cross-model column lineage (CLL-4) by adding a new ChangesCross-model column lineage (CLL-4)
Sequence DiagramsequenceDiagram
participant Explorer as render_explore
participant BuildCML as build_cross_model_columns
participant RelIdx as RelationIndex
participant CTE as parse_cte_graph
participant CteGraph
participant ProjGraph as ProjectColumnGraph
participant Trace as trace_to_source
participant Blast as blast_radius
Explorer->>BuildCML: Manifest
BuildCML->>RelIdx: from_manifest
RelIdx-->>BuildCML: normalized keys ↔ NodeId
loop per model
BuildCML->>CTE: compiled SQL
CTE-->>CteGraph: intra-model edges
CteGraph->>CteGraph: model_outputs(terminal)
CteGraph-->>BuildCML: output_columns, leaf_refs, passthrough
end
BuildCML->>ProjGraph: build(manifest, lineage, index, model_outputs)
ProjGraph->>ProjGraph: stitch_enumerable (producer-constrained)
ProjGraph->>ProjGraph: source_name_carry (passthrough-only)
ProjGraph->>ProjGraph: stitch_leaf (never-mis-join via RelIdx)
ProjGraph-->>BuildCML: ProjectColumnGraph {edges, outputs}
BuildCML-->>Explorer: CrossModelColumnsPayload or None
Explorer->>Trace: trace column to source
Trace-->>Explorer: TraceToSource {hops, termination}
Explorer->>Blast: downstream impact
Blast-->>Explorer: Vec~TraceHop~
Estimated code review effort🎯 5 (Critical) | ⏱️ ~100 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
📄 Rendered report previewAll golden examples regenerated cleanly. 🟡 Golden examplesCommitted to
🐶 Live dogfood previewThis PR doesn't touch 🧭 Explore previewThe two-page 🟡 Golden exploreThe committed
🐶 Live exploreThis PR doesn't touch ▶ Open ↗ opens the report or explorer in your browser in one The Pages preview may take ~1 min to update after this comment Alternative: GitHub CLI# gh CLI >= 2.63 extracts into ./report-preview-playground/.
gh run download 28047574558 -R breezy-bays-labs/cute-dbt -n report-preview-playground
open report-preview-playground/playground-report.htmlPosted by |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements project-wide cross-model column lineage (CLL-4) to enable tracing columns back to their source fields and calculating downstream blast-radius. It introduces a new domain module column_lineage.rs for stitching model outputs at ref() boundaries using a normalized relation index, updates the explorer adapter to build and serialize this graph, and adds comprehensive integration tests. The review feedback highlights two optimization opportunities: utilizing the by_identifier index in stitch_leaf to avoid O(M) complexity over the entire relation map, and performing a direct lookup in node_for_bare_leaf to prevent redundant string allocations on already-lowercased identifiers.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…urce name-carry honesty) The cross-model source name-carry (a non-enumerable source's column origin) was over-claiming: a column COMPUTED in-model (current_timestamp as _loaded_at, a surrogate row_number() key) was attributed to the upstream source — a fabricated lineage claim (the column does not ORIGINATE there). On the real playground manifest this produced 9 false _loaded_at -> source edges. Fix: the name-carry now only covers a downstream output column the SQL PROVES flows UNCHANGED to a leaf-reading boundary — a pure pass-through/rename chain (CteGraph::model_outputs now carries source_passthrough_columns, computed by column_reaches_leaf tracing each terminal column's intra chain to an EXTERNAL leaf, never a Derived/computed dead-end and never a sibling-CTE ref). A computed column dead-ends at its Derived expression and is excluded; its trace terminates honestly (not Source). Real playground: 0 _loaded_at -> source edges; genuine pass-through columns still trace. Added a TDD test (computed_column_is_never_attributed_to_a_source) over the real parse path. Decomposed model_outputs into terminal_output_columns / model_leaf_refs / leaf_reading_nodes helpers to keep CRAP under threshold (was 31, now a thin dispatcher). Explore goldens regenerated + byte-verified; report goldens byte-identical (scope separation intact). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/domain/column_lineage.rs`:
- Around line 1143-1149: The graph_serde_round_trips test function only provides
round-trip coverage for ProjectColumnGraph and does not test the new wire types
that implement Serialize and Deserialize boundaries. Convert the existing test
or create additional property-based tests using a property testing framework to
cover NormalizedRelation, RelationIndex, ModelOutputs, StitchOutcome, and
TraceToSource. Each type should have property-based coverage that generates
random instances, serializes them to JSON, deserializes them back, and asserts
equality, following the coding guidelines that require property tests for JSON
serde round-trip coverage on these types.
- Around line 286-294: The ModelOutputs::new constructor stores output_columns
and leaf_refs without normalizing them to lowercase, but the documented behavior
promises lowercased values which are expected by trace_to_source and
blast_radius during matching. Modify the constructor to normalize both
parameters: for output_columns, map over the Option and lowercase each string in
the Vec, and for leaf_refs, lowercase each string in the Vec before storing them
in the struct.
- Around line 485-488: The current implementation uses `.find()` in the edges
iterator to locate the next upstream column, which only returns the first
matching edge. This is problematic because a downstream column can have multiple
incoming cross-model edges (from duplicate star columns or derived columns with
multiple inputs), and picking the first one makes the trace sort-order dependent
and potentially incorrect. Instead of using `.find()`, check if there are
multiple edges matching the criteria (where e.downstream equals current.0 and
e.downstream_column equals current.1), and if multiple matches exist, treat this
as an ambiguous case and handle it appropriately by either returning an error or
None since the API does not yet support branching traces.
In `@src/domain/cte.rs`:
- Around line 854-905: The model_outputs function currently returns only
flattened lists of output_columns and leaf_refs, losing the relationship between
leaves and their corresponding output columns, which prevents downstream from
distinguishing between different lineage scenarios. Refactor the function to
return terminal-reaching edge mappings that preserve the association between
each leaf, the upstream column, downstream column, and the edge type (star or
opaque), then derive leaf_refs from these mappings. This requires changing the
ModelOutputs return type to include these mapping details instead of just
separate lists of output_columns and leaf_refs.
In `@tests/cross_model_column_lineage.rs`:
- Around line 285-309: The test function
star_over_unknown_external_stays_opaque_no_fabrication has a mismatch between
its name/comments (which describe Opaque behavior) and its assertion (which
expects TraceTermination::Root). Update the assertion on the trace.termination
field to expect TraceTermination::Opaque instead of TraceTermination::Root to
align the actual test assertion with the documented contract described in the
test name and comments.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ec118c03-5d9a-4c82-994c-bd0cce1913b2
📒 Files selected for processing (8)
crap4rs.tomlexamples/explore-macro/dag.htmlexamples/explore/dag.htmlsrc/adapters/explore.rssrc/domain/column_lineage.rssrc/domain/cte.rssrc/domain/mod.rstests/cross_model_column_lineage.rs
…aque contract (CodeRabbit) Resolve the bot-flagged honesty issues on the cross-model column trace (floor: never-a-false-claim — degrade over a wrong attribution). - trace_to_source: a downstream column with MORE THAN ONE incoming cross-model edge (duplicate `select *` columns, a derived column with several inputs) no longer picks the first via `.find()` (sort-order- dependent WRONG source). It now `.filter()`s and degrades to Opaque on a fork — the single-chain API cannot represent a branch. (CodeRabbit Major ~488). New lib test multi_upstream_column_traces_opaque_not_first_wins (pre-fix: picks up_a, Root, 2 hops → fails). - column_thins_opaque: a model whose OWN terminal projection is non-enumerable (`*` over an unknown external, outputs[node] == None) now thins to Opaque even with no manifest producer — the documented unknown-external contract. The CODE previously yielded Root (a false "originates here" claim); fixed the code, then the test asserts Opaque. (CodeRabbit Major ~309). - ModelOutputs::new: normalize output_columns/leaf_refs to lowercase at the boundary (docs promise lowercased; trace/blast_radius lowercase query columns). (CodeRabbit ~294). - node_for_bare_leaf: direct lookup first, lowercase only on miss — no happy-path allocation. (gemini ~254). - stitch_leaf fallback: use the by_identifier index (O(log M)) instead of scanning the whole by_relation map (O(M)); behavior-preserving. (gemini HIGH ~729). - new_wire_types_serde_round_trip: exhaustive JSON round-trip over the new wire types (NormalizedRelation, ModelOutputs, StitchOutcome, TraceToSource × every TraceTermination). RelationIndex is documented NOT a JSON wire type (struct-keyed map; in-process index only). (CodeRabbit ~1149). Scope discipline: per-model *-report.html goldens byte-identical; explore goldens regenerate identically (the report path never builds the project graph). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/domain/column_lineage.rs (1)
604-615: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy liftMake opaque thinning column-grained.
Line 614 checks whether any column edge lands on the node. A computed-only column on a model with a producer returns
Opaquewhen no other column resolved, but the same computed column returnsRootonce an unrelated pass-through column adds an inbound edge. Carry opaque-thin facts keyed by(node, column)frombuild, or pass the queried column into this check so trace termination does not depend on unrelated columns.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/domain/column_lineage.rs` around lines 604 - 615, The column_thins_opaque method currently checks if any column edge lands on a node at line 614, making the result node-grained rather than column-grained. This causes inconsistent behavior where a computed-only column returns different results depending on whether unrelated columns add edges to the same node. To fix this, either refactor the method to accept the specific column being queried and check only for edges matching that column, or track opaque-thin facts with (node, column) tuple keys during the build phase and look up the result using both the node and column identifiers instead of only the node. This ensures trace termination depends only on the relevant column's edges, not unrelated columns on the same node.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/domain/column_lineage.rs`:
- Around line 338-346: The with_passthrough constructor in column_lineage.rs is
not normalizing its inputs to lowercase, which causes the output_columns,
leaf_refs, and source_passthrough_columns to bypass the lowercase boundary
enforcement that the new constructor applies. This leads to mixed-case names
missing lookups in trace_to_source and blast_radius. Apply the same lowercase
normalization logic to all three input parameters (output_columns, leaf_refs,
and source_passthrough_columns) in the with_passthrough constructor that is
currently being applied in the new constructor to ensure consistent behavior
across both constructors.
In `@src/domain/cte.rs`:
- Around line 1033-1047: The condition in the matches! macro that filters
ColumnEdgeKind is allowing both PassThrough and Renamed edges, but this can
cause source_name_carry to fabricate column traces for renamed columns without
verifying the original source column exists. Remove ColumnEdgeKind::Renamed from
the matches! condition so that only ColumnEdgeKind::PassThrough is permitted,
preventing false lineage traces for renamed columns like turning select amount
as order_amount into a fabricated source.order_amount mapping.
---
Outside diff comments:
In `@src/domain/column_lineage.rs`:
- Around line 604-615: The column_thins_opaque method currently checks if any
column edge lands on a node at line 614, making the result node-grained rather
than column-grained. This causes inconsistent behavior where a computed-only
column returns different results depending on whether unrelated columns add
edges to the same node. To fix this, either refactor the method to accept the
specific column being queried and check only for edges matching that column, or
track opaque-thin facts with (node, column) tuple keys during the build phase
and look up the result using both the node and column identifiers instead of
only the node. This ensures trace termination depends only on the relevant
column's edges, not unrelated columns on the same node.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f6f435d2-0de0-41f6-a18c-818454661af8
📒 Files selected for processing (5)
examples/explore-macro/dag.htmlexamples/explore/dag.htmlsrc/domain/column_lineage.rssrc/domain/cte.rstests/cross_model_column_lineage.rs
…tputs (no phantom narrowed-away columns) stitch_enumerable flowed the upstream's FULL output column set into downstream cross-model edges without checking whether the downstream actually exposes each column. When a downstream NARROWS its projection (the most common dbt pattern), every dropped column became a phantom cross-model edge — breaking never-a-false-claim: blast_radius(stg,"amount") falsely listed a downstream that dropped amount, and trace_to_source returned a confident chain for a column the downstream never exposes. Fix: intersect each flowed upstream column against the downstream model's own terminal output_columns; emit a cross-model edge only for a column the downstream actually exposes. A non-enumerable downstream terminal (None) cannot bound the set, so the prior pass-through stands. Sound seams preserved: star-over-known→Resolved for exposed columns, unknown-external→Opaque, multi-upstream→Opaque fork, scope separation. Tests through the REAL parse_cte_graph + ProjectColumnGraph::build path (never hand-authored edges): the verifier's stg(order_id,amount)→ dim(select order_id) repro asserts no stg.amount→dim.amount edge, blast_radius omits dim, trace_to_source fabricates no chain; a CTE-import-join narrowing repro; and a regression guard that the exposed column keeps its sound edge + traces to source. Explore goldens regenerated: 943→363 cross-model edges per page (new set a strict subset of old — 580 phantoms removed, 0 added). Per-model *-report.html goldens byte-identical (cross-model is explore-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
Contributor
Author
⏸️ Held for founder review — not merging (honest assessment)CLL-4 is built, CI-green, and architecturally sound — the project graph stays in the explorer arm (per-model But I'm holding it rather than merging. Across four review rounds this slice has surfaced six honesty findings, and it is not converging — each fix-round reveals new Majors, and the dedicated adversarial verifier (which returned
Why hold, not fix-and-merge: the cross-model column logic is intricate enough (1200+ LOC, the project-wide stitch) that it keeps yielding genuine fabrications on each pass, and my verification layers aren't fully catching them. This is the hardest, headline slice — it deserves a fresh, careful pass (your eyes on the Nothing is lost — all work is on |
…malize with_passthrough (CodeRabbit) source_name_carry no longer fabricates a same-name source column for a RENAMED output (e.g. `CODE as condition_code` no longer claims source.condition_code — the source field is CODE). column_reaches_leaf now requires a pure PassThrough chain (Renamed excluded) so only a same-name column may name-carry to its source; a renamed column degrades to no source edge until a terminal→leaf original-column mapping exists. A genuine pass-through still traces correctly. with_passthrough now lowercases output_columns/leaf_refs/passthrough names exactly like new(), so a mixed-case output column (e.g. OrderId) is not silently missed by the lowercased trace_to_source/blast_radius lookups on the real explorer path. Explore goldens regenerated: 68 fabricated rename edges REMOVED from each dag.html (0 added); report goldens byte-identical (report path never builds the project graph). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tests/cross_model_column_lineage.rs (1)
861-866: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winTighten this assertion to avoid silently passing on non-enumerable outputs.
if let Some(cols)makes the key expectation optional; ifdim_outputsregresses toNone, this test still passes and can hide a real projection/output regression. AssertSome(...)(or at leastis_some()) here so the test fails when output enumeration unexpectedly disappears.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/cross_model_column_lineage.rs` around lines 861 - 866, The conditional check using `if let Some(cols) = &dim_outputs` makes the assertion optional, allowing the test to pass silently if `dim_outputs` is `None`, which can hide real regressions in output enumeration. Replace this with an explicit assertion that `dim_outputs` is `Some(...)` using either `unwrap()`, `expect()`, or `assert!(is_some())` to ensure the test fails when output enumeration unexpectedly disappears, rather than passing silently when `dim_outputs` is `None`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/cross_model_column_lineage.rs`:
- Around line 861-866: The conditional check using `if let Some(cols) =
&dim_outputs` makes the assertion optional, allowing the test to pass silently
if `dim_outputs` is `None`, which can hide real regressions in output
enumeration. Replace this with an explicit assertion that `dim_outputs` is
`Some(...)` using either `unwrap()`, `expect()`, or `assert!(is_some())` to
ensure the test fails when output enumeration unexpectedly disappears, rather
than passing silently when `dim_outputs` is `None`.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 3ab2719f-ae1a-46f5-948d-0cbff4c341a4
📒 Files selected for processing (5)
examples/explore-macro/dag.htmlexamples/explore/dag.htmlsrc/domain/column_lineage.rssrc/domain/cte.rstests/cross_model_column_lineage.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- src/domain/cte.rs
…name-tracking (no fabricated source column)
Round-4 closes the last open fabrication shape (the 1047 class): a column
RENAMED in a leaf-reading CTE — the canonical dbt staging shape
`renamed as (select legacy_qty as qty from {{source}}) select qty from
renamed`. `column_reaches_leaf` early-returned `true` the moment it reached
the leaf-reading node, BEFORE inspecting the inbound edge name, so `qty`
entered source-name-carry under its RENAMED name and emitted
`source.raw_orders.qty` — a Resolved trace to a source field that does not
exist (the real field is `legacy_qty`).
ROBUST fix (preferred — preserves the staging-rename trace-to-source
headline): carry the ORIGINAL source column name through the chain.
`column_reaches_leaf` becomes `resolve_leaf_column` returning the resolved
LEAF-side column name (or `None` to degrade): a `Renamed` edge is followed on
its UPSTREAM column, so `legacy_qty as qty` resolves to the real field
`legacy_qty` across any number of pass-through hops above the rename. The
leaf early-return is gated on resolving that name through the originating
edge; a computed column on a leaf-reading node (a `Derived` inbound) no longer
over-resolves; a non-uniquely-resolvable fork degrades (never a coin-flip).
`ModelOutputs.source_passthrough_columns` becomes a map (output column → real
source field); `source_name_carry` names the edge with the resolved field.
Covers every rename shape via the REAL parse_cte_graph + ProjectColumnGraph
path: rename-in-leaf-reading-CTE, direct-terminal rename, multi-hop
pass-through above a rename, chained rename (a→b→c), `select *` over a
renaming leaf CTE, plus the same-name and computed-column regressions. Explore
goldens regen (+68 cross-model source edges restored, each naming the REAL
source field, zero removed); report goldens byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
…ource field (no literal/computed fabrication) The cross-model trace-to-source attributed a no-inbound-edge column on a leaf-reading node to the source under its own name, assuming it flowed through `select *`. But a LITERAL/computed alias (`42 AS magic`, `current_timestamp AS t`, `1 AS x`) on a leaf-reading node ALSO has no inbound edge (the engine emits no edge for a constant), so the fallback fabricated `source.<leaf>.<alias>` for a column the source does not have. The star-passthrough fallback in `LeafResolution::into_name` now fires ONLY when the node ACTUALLY carries a `*` projection over an EXTERNAL leaf (a recorded `*→*` edge landing on `(node, "*")` whose from-side is not a sibling CTE — `CteGraph::has_external_star`). A leaf-reading node with no such star has its edge-less columns degrade to None — a literal/computed expression is never a source field. Every source attribution now requires a PROVABLE chain (pass-through/rename edges, or a genuine star over the leaf) terminating in a column that really exists on the source; every other shape (literal, computed, derived, join-key, ambiguous fork, opaque) degrades to None. The R4 robust-rename trace (`legacy_qty AS qty` → `source.legacy_qty`) is preserved. Tests (real `parse_cte_graph` + `ProjectColumnGraph::build`): a literal alias in a leaf-reading CTE never names a source field; assorted constant/computed aliases in a single model never fabricate a source; the real pass-through still traces. The literal-CTE shape was RED pre-fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
…ce edges read it, never infer from edge-absence
Replace the read-time star-passthrough INFERENCE (no-inbound-edge +
node-carries-an-external-star ⇒ assume pass-through) with POSITIVE
per-output-column provenance recorded at the projection resolver.
New domain types: ColumnProvenance { DirectColumn, Rename, Literal,
Expression, StarCoveredExternal } + ColumnProvenanceEntry, carried as a
#[serde(skip)] fact on CteGraph (byte-stable goldens). The resolver
classifies every output column; resolve_leaf_column READS the marker — a
no-inbound-edge column is name-carried to a source ONLY when the node's
`*` slot is StarCoveredExternal AND the column is not Literal/Expression.
Closes the literal/expr-paired-with-star fabrication class: `42 as magic,
* from src` and `a.x+1 as y, * from src` no longer fabricate
source.magic / source.y. R4 rename trace-to-real-field + every prior
cross-model behavior preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
…rojection dispatcher under the strict CRAP cap The added per-column provenance classification pushed resolve_projection_item to CC 15 (the strict <= 15 boundary). Split the `expr AS alias` arm into a dedicated resolve_aliased_item helper: dispatcher drops to CC 4, helper is CC 8 — both comfortably under budget. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199AmBCec5kyVEEF1Qd7TSF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CLL-4 — the v0.2 explorer founder headline: trace a column to its SOURCE field across
ref()boundaries, plus column-grain downstream blast-radius. Runs the CLL-2/CLL-3 intra resolver over every model to build the project-wide output-column map (cute-dbt's zero-compute catalog-equivalent), then stitches atref()boundaries. TheColumnScope::Crossarm reserved by CLL-2 becomes live.Closes #450.
Normalized join key (refinement 3 — the load-bearing correctness seam)
The cross-model
ref()↔leaf join uses a normalized(database, schema, identifier)tuple built AT INGESTION (RelationIndex::from_manifest), case-folded — NOT a rawrelation_namestring-match (engine leaf refs are bare lowercased leaves; aliasing /identifierconfig / case-folding make a raw string-match a hazard).Never a false claim: the stitch attributes a leaf to an upstream
NodeIdONLY when it is BOTH (a) a realdepends_onproducer (fromDagFacts.lineage/ModelLineage.backward) AND (b) the unique producer whose normalized identifier matches the leaf. Non-unique / no-match → Opaque — never a wrong upstream. Verified on the real 35-model playground manifest: 0 edges attribute to a non-producer.What's built
ref()boundaries to thesource()/seed leaf; terminatesSource/Root/Opaque(honest thinning).child_mapdirection) — readsModelLineage, never a thirddepends_onself-inversion.select *over a KNOWN modeled upstream → Resolved via the derived projection map;*over an UNKNOWN external → Opaque; chains of*compound Opaque (no catalog, no fabrication).build_cross_model_columnsinrender_explore). The per-model report path is untouched — report goldens byte-identical; only the two playground-based exploredag.htmlgoldens change (verified by diffing a fresh report render against the committed golden).Tests (TDD-first, the real parse path)
src/domain/column_lineage.rs(pure POD + std/serde — passesdomain_clean_arch).tests/cross_model_column_lineage.rsbuilt through the REAL multi-model parse path (parse_cte_graph→model_outputs→ProjectColumnGraph), never a hand-authored join: normalized-join-key never-mis-join, cross-model stitch, B blast-radius, C trace-to-source, star-over-known→Resolved / star-over-unknown→Opaque / star-chain→compounding-Opaque, scope-as-parameter, usesDagFacts.lineage.Dogfood
The existing synthetic playground manifest already exercises a cross-model trace + star-over-ref — 975 cross-model edges, 144 source-terminal traces (
dim_organizations→stg_synthea__organizations→ theorganizationssource). No new fixture, noroot_pathleak.Gates (all raw-exit-0 locally)
fmt --check · clippy --all-targets --locked -D warnings · full nextest (2621 pass) · BDD (242 scenarios) · cargo doc -D warnings · cargo deny · domain_clean_arch · lineage_seam · resource_ref_lint · crap4rs (PASS —
column_lineage.rsadded to the strict ≤15 keyed set) · explore goldens regen + byte-verified · report goldens byte-identical.🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
New Features
select *scenarios.Tests