Skip to content

#433 — single-pass config provenance + var-inventory package precompute + filterByModels lookup dedup - #441

Merged
cmbays merged 2 commits into
mainfrom
perf-433-traversal-lookup-dedup
Jun 15, 2026
Merged

cmbays merged 2 commits into
mainfrom
perf-433-traversal-lookup-dedup

Conversation

@cmbays

@cmbays cmbays commented Jun 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Post-v0.1 perf pass for the three redundant-traversal/lookup hotspots gemini flagged on #432 (#270 explorer project pane). All three are correctness-neutral.

  • provenance_for_fqn (src/domain/project_def.rs) — one fqn descent accumulating deepest-match-wins configs (O(K·L) → O(L)), removing the keys_along_fqn helper. A BTreeMap accumulator keeps the output sorted by key, so the rendered ordering is byte-identical to the prior keys_along_fqn + per-key re-resolve. (Carries the maintainability win the issue calls out — simpler algorithm, one fewer helper.)
  • project_var_inventory (src/domain/vars.rs) — resolve each scanned model's package once (O(M)) instead of per var (O(V×M) manifest lookups + a NodeId clone each). inventory_entry now takes the precomputed model_pkgs map; the public API is unchanged.
  • filterByModels (templates/explore-lineage.js) — resolve each Cytoscape element once into a {id, node} list instead of calling cy.getElementById up to 3× per id (filter, union, select). Behavior + zero-egress unchanged.

Behavior-neutral by construction

  • provenance: same key set (every key set on a visited node), same key-sorted order (BTreeMap), same deepest (depth, value) → same path.
  • var inventory: same package resolution (package_name().or(root)), same scope_reaches gate, same per-tier tallies, same var_models pushes/order.
  • filterByModels: same present membership + order; activeFilter.matched is the same array of ids (present.map(p => p.id)).

Every committed golden is byte-identical except the four explore pages that embed the changed JS (explore/dag.html, explore-macro/dag.html, explore-macro/macro.html, explore-seed/dag.html), regenerated here — the diff in each is only the filterByModels region. The unchanged tests.html pages + all report goldens independently confirm the Rust changes are byte-neutral.

On the "don't optimize without a measured baseline" AC

cute-dbt is zero-compute on typically small dbt artifacts and has no established perf budget (the org's aspirational — Q4 leg). So per the issue's AC alternative, the justification here is complexity reduction + the provenance_for_fqn maintainability win, not a profiler-driven hot-path fix — the perf framing is analytical Big-O, not benchmarked. No criterion infra added (it isn't a current dep; adding it solely for this would be scope creep against a zero-compute tool with no regression ceiling).

Gates (all green locally)

  • cargo fmt --check ✓ · cargo clippy --all-targets --locked -- -D warnings ✓ · RUSTDOCFLAGS=-D warnings cargo doc --no-deps --locked ✓ · cargo deny check ✓
  • cargo nextest run --locked → 2367 passed (incl. the headless explore-lineage tests that drive filterByModels)
  • cargo test --test bdd → 31 features / 242 scenarios passed
  • Explore goldens regenerated to match the renderer (byte-identity gate satisfied by construction).

DO NOT MERGE

Christopher's call. Gated to green + left open for review.

Closes #433

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Refactor
    • Optimized reference-filtering logic across explore lineage pages to resolve model IDs once instead of repeated lookups, improving performance.
    • Enhanced project configuration provenance computation to calculate in a single pass instead of multiple iterations.
    • Improved variable inventory processing by precomputing model package mappings to reduce redundant lookups during scope analysis.

…recompute + filterByModels lookup dedup

Three correctness-neutral traversal/lookup dedups on the #262 project-config
surfaces (deferred from #432):

- provenance_for_fqn (src/domain/project_def.rs): one fqn descent
  accumulating deepest-match-wins configs (O(K·L) -> O(L)); removes the
  keys_along_fqn helper. A BTreeMap accumulator keeps the output sorted by
  key, so the rendered ordering is byte-identical to the prior pass.
- project_var_inventory (src/domain/vars.rs): resolve each scanned model's
  package once (O(M)) instead of per var (O(V×M) manifest lookups + a
  NodeId clone each); inventory_entry now takes the precomputed map.
- filterByModels (templates/explore-lineage.js): resolve each Cytoscape
  element once, not 3× per id; behavior + zero-egress unchanged.

Behavior-neutral by construction (same key set / order / values; same
scope-reach + tallies; same matched ids). Every committed golden is
byte-identical except the four explore pages embedding the changed JS,
regenerated here. Per the issue AC these are justified by complexity
reduction + the provenance maintainability win (simpler algorithm, one
fewer helper); no profiler or perf budget exists for this zero-compute
tool, so the perf framing is analytical Big-O, not benchmarked.

Closes #433

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@coderabbitai

coderabbitai Bot commented Jun 15, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7b506b71-ecb2-44a3-9878-d965e17d60fb

📥 Commits

Reviewing files that changed from the base of the PR and between 864886f and bd66374.

📒 Files selected for processing (7)
  • examples/explore-macro/dag.html
  • examples/explore-macro/macro.html
  • examples/explore-seed/dag.html
  • examples/explore/dag.html
  • src/domain/project_def.rs
  • src/domain/vars.rs
  • templates/explore-lineage.js

📝 Walkthrough

Walkthrough

Three redundant-traversal hotspots are addressed. filterByModels in templates/explore-lineage.js and four HTML examples now resolves each Cytoscape node once into a {id, node} cache before the batch dim/trace/sel update. In src/domain/project_def.rs, provenance_for_fqn is rewritten as a single-pass BTreeMap accumulator, removing keys_along_fqn. In src/domain/vars.rs, project_var_inventory precomputes per-model packages and passes them into a refactored inventory_entry.

Changes

filterByModels Cytoscape Lookup Dedup

Layer / File(s) Summary
filterByModels template source refactor
templates/explore-lineage.js
Builds a present array of {id, node} once per id, uses those cached objects inside cy.batch to clear/dim/trace/sel, and derives activeFilter.matched from present.map(p => p.id).
Same refactor in four HTML example copies
examples/explore/dag.html, examples/explore-seed/dag.html, examples/explore-macro/dag.html, examples/explore-macro/macro.html
Identical filterByModels change propagated to all four compiled/example HTML pages, mirroring the template source.

Rust Domain Single-Pass and Precompute Refactors

Layer / File(s) Summary
provenance_for_fqn single-pass BTreeMap rewrite
src/domain/project_def.rs
Replaces the collect-keys-then-re-resolve loop with a single fqn descent that accumulates deepest-match winners in a BTreeMap, then maps them to ConfigProvenance; the keys_along_fqn helper is removed.
project_var_inventory package precompute
src/domain/vars.rs
project_var_inventory materializes a scanned_pkgs vector of (id, scan, package) triples before calling inventory_entry; inventory_entry's signature is updated to accept and iterate over that slice instead of resolving packages internally.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • breezy-bays-labs/cute-dbt#218: Originally introduced the Cytoscape-based explore lineage engine in templates/explore-lineage.js, establishing the filterByModels dim/trace/sel batch logic that this PR refactors.

Poem

🐇 One lookup, not three — the rabbit hops free,
No redundant trails through the Cytoscape tree.
A BTreeMap of winners, one pass down the FQN,
Packages precomputed before we begin.
Less work, same truth — efficiency's the key! 🌿

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf-433-traversal-lookup-dedup

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ast-grep (0.43.0)
examples/explore-macro/dag.html
examples/explore-macro/macro.html
examples/explore-seed/dag.html
  • 1 others

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 15, 2026 •

Copy link
Copy Markdown
Contributor

📄 Rendered report preview

All golden examples regenerated cleanly.

🟡 Golden examples

Committed to examples/ and byte-identity gated — the canonical reports contributors and consumers browse. Stable across PRs.

Report View Download
macro-heavy-report.html ▶ Open ↗ ⬇ Download
jaffle-shop-report.html ▶ Open ↗ ⬇ Download
seed-showcase-report.html ▶ Open ↗ ⬇ Download
prdiff-minidag-report.html ▶ Open ↗ ⬇ Download
diff-showcase-report.html ▶ Open ↗ ⬇ Download
playground-report.html ▶ Open ↗ ⬇ Download

🐶 Live dogfood preview

This PR doesn't touch dbt-project/, so there's no live dogfood preview.

🧭 Explore preview

The two-page cute-dbt explore explorer — dag.html (model lineage) + tests.html (unit-test viewer). Same golden/live split as the report.

🟡 Golden explore

The committed examples/explore/ playground golden (the full synthetic playground manifest). Byte-identity gated in Example report check. Stable across PRs.

Page View Download
explore/dag.html ▶ Open ↗ ⬇ Download
explore/tests.html ▶ Open ↗ ⬇ Download

🐶 Live explore

This PR doesn't touch dbt-project/, so there's no live explore preview.

▶ Open ↗ opens the report or explorer in your browser in one
click — published to this repo's GitHub Pages under
/pr-441/.
⬇ Download fetches the same self-contained HTML as a workflow
artifact (auth-gated; works fully offline). Either way the report
makes zero external resource requests.

The Pages preview may take ~1 min to update after this comment
posts. On PRs from forks the Open link is unavailable (read-only
token) — use Download.

Alternative: GitHub CLI
# gh CLI >= 2.63 extracts into ./report-preview-playground/.
gh run download 27558780877 -R breezy-bays-labs/cute-dbt -n report-preview-playground
open report-preview-playground/playground-report.html

Posted by report-preview.yml for bd663745e71f0ddaec34115464d153d033d73e3d. Affordance only — never blocks merge.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces performance optimizations by caching and precomputing lookups. In the HTML and JS visualization files, Cytoscape node resolutions are cached to avoid redundant lookups. In the Rust backend, FQN config resolution is optimized to a single descent, and model package resolutions are precomputed to avoid repeated manifest lookups. The review feedback suggests further optimizing the Rust package resolution by using a flat vector instead of a BTreeMap, which avoids O(log M) map lookups in the inner loop and reduces the overall complexity to O(V * M).

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/domain/vars.rs Outdated
Comment thread src/domain/vars.rs Outdated
…a map lookup

gemini-code-assist flagged the var-inventory precompute on PR #441: pairing each
scanned model with its resolved package in a flat Vec lets inventory_entry
iterate the slice directly, dropping the per-iteration O(log M) BTreeMap lookup
(O(V×M log M) -> O(V×M)) and collapsing two params into one. Behavior-neutral
(same iteration order / tallies / var_models); the explore project-pane goldens
are byte-identical (re-rendered + diffed, verified).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cmbays
cmbays merged commit 0437201 into main Jun 15, 2026
44 checks passed
@cmbays
cmbays deleted the perf-433-traversal-lookup-dedup branch June 15, 2026 16:01
@cmbays

cmbays commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

Merged — debrief

Shipped (squash chore: → no version bump; v0.1.0 hold intact): three correctness-neutral traversal/lookup dedups on the #262 project-config surfaces — provenance_for_fqn single-pass (O(K·L)→O(L), keys_along_fqn removed), project_var_inventory package precompute, filterByModels resolve-once.

Deviation (review-driven, intentional): the var-inventory precompute first used a BTreeMap<&str, Option<&str>> lookup (f30d293). gemini flagged the per-iteration O(log M) map lookup; I adopted its flat Vec<(&id, &scan, package)> form (bd66374) — iterate the slice directly, no lookups (O(V×M log M)→O(V×M)), two params collapsed to one. Strictly better and the issue's compute-once thesis; behavior-neutral.

Bot disposition: gemini — 2 medium threads, both real (probed before disposition) → adopted + replied + resolved. CodeRabbit — Review completed, no findings.

Gates: all green locally (fmt/clippy --locked/doc/deny=0, nextest 2367/2367, BDD 242/242) and on CI for the final commit (crap4rs ✓, byte-identity ✓, headless zero-egress ✓, 3 OS test matrices ✓). Explore project-pane goldens verified byte-identical after the gemini refactor (behavior-neutral).

AC note: the "no measured baseline" AC was honored via complexity-reduction + the provenance_for_fqn maintainability win rather than criterion infra (zero-compute tool, no perf budget).

Risks / follow-ups: none — behavior-neutral, no output change, no new surface.

github-actions Bot added a commit that referenced this pull request Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore: post-v0.1 perf pass — provenance_for_fqn single-pass, var-inventory package precompute, filterByModels lookup dedup

1 participant