Skip to content

#200 — design-2 data contracts: manifest_nodes lookup, TestPayload.overrides (Value passthrough), ModelPayload.description - #216

Merged
cmbays merged 7 commits into
mainfrom
adapters-200-design2-data-contracts
Jun 11, 2026
Merged

cmbays merged 7 commits into
mainfrom
adapters-200-design2-data-contracts

Conversation

@cmbays

@cmbays cmbays commented Jun 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Implements the three genuinely-new Phase-2 data contracts (design-return-2 integration audit D12–D14), Rust + fixtures only — the consuming shelf/hover-card JS lands in #201/#202 and the report degrades gracefully until then.

Everything is skip_serializing_if (all-empty manifest_nodes entries are skipped too), so findings-free + manifest_nodes-free payloads stay byte-stable, and emission stays on the #194 payload_json_for_html_script path. That escaper gains = in its tag-opener set: the new description content puts SQL-ish prose like encounter_start_at <= current_timestamp on the wire and tl-based extractors mis-scan a raw <= — the same scanner-hostile class #170 fixed for <letter/<?; output still JSON-decodes to the original text.

Fusion-first verification (pinned 9977b6cbb1b761065536300037560d8e3c037011)

  • overrides: UnitTestOverrides (crates/dbt-schemas/src/schemas/properties/unit_test_properties.rs:32) is {env_vars, macros, vars}: Option<BTreeMap<String, YmlValue>> under #[skip_serializing_none] — three sibling open maps with arbitrary scalar values, unset = key absent. ManifestUnitTest.overrides: Option<UnitTestOverrides> (manifest_nodes.rs:268). env_vars IS modeled by fusion (the Discovery question — parity confirmed at the type level).
  • description/tags: ManifestMaterializableCommonAttributes.{description: Option<String>, tags: Vec<String>} (manifest_nodes.rs) — top-level on the serialized node. The ""-emitting custom serializer (serialize_dbt_column_desc, dbt_column.rs) is column-specific; the model-level description is plain skip-none.
  • REAL committed fixture evidence (the second leg — synthetic JSON misses the unset shapes):
    • tests/fixtures/playground-current.json carries the diff: surface unit-test overrides (env_vars/macros/vars) changes in the YAML text-diff fallback #125 overrides splice: {"env_vars": {}, "macros": {"is_incremental": true}, "vars": {}} — a native JSON bool plus empty-map unset channels, and five tests with explicit "overrides": null. The grouped ingestion drops null AND {} channels individually (asserted on the real fixture in tests/manifest_ingestion.rs).
    • The same fixture proves the tags decision: mart_dq_summary top-level tags = ["analytics","data_quality","marts"] while config.tags = ["analytics","data_quality","marts","analytics","data_quality"] — the nested list carries project+model merge duplicates, so cute-dbt reads only the authoritative deduplicated top-level list.
    • jaffle-shop-current.json: authored customers description ingests verbatim; stg_customers's "" description is dropped.
  • The demo payload in the design package's reference HTML (the wire-shape truth) was decoded and matched: 7 manifest_nodes entries with {description, materialized, tags, columns[{name, description, type, tests}], model_tests[{name, detail}]}, and three tests carrying native-scalar overrides (true, 7, 0.05, "ci").

Tests

  • Domain: round-trips for Node.description/tags and the grouped overrides (native bool/int/float/string types pinned; deterministic BTreeMap group order; pre-feature: design-2 integration PR-3 — data contracts: report manifest_nodes lookup + TestPayload overrides + ModelPayload description #200 JSON without the new keys still deserializes).
  • Adapter wire matrix over both engine dialects: absent key / explicit null blob / {} blob / null channels / empty channels / per-channel drops / native scalar retention; model description ""-drop + null/absent tags.
  • Render units: a manifest_nodes entry carries description/materialized/tags/columns/model_tests; ref()-ed upstream included + unrelated model excluded (+ this/source()/unresolvable-ref arms); all-empty entries skipped and the key omitted (byte-stability); overrides group+scalar round-trip at the serialized-JSON level; description threads + omitted-when-None; <= escaper round-trip.
  • Real-fixture integration test (above) + the updated chrome snapshot (jaffle now shows its manifest_nodes entry).
  • BDD: one payload-fact scenario in report_generation.feature (no new .feature file — feature-count stays 14, no mirror bump) through the real subprocess: manifest_nodes for the in-scope model and its NOT-in-scope ref()-ed upstream, ModelPayload.description, and native-scalar overrides (type probes reject "7"/"true").

Dogfood + goldens

Gates (run directly — lefthook skips in fresh worktrees)

fmt ✓ · clippy --all-targets --locked -D warnings ✓ (exit 0) · nextest 1111 ✓ · BDD 114 scenarios ✓ · headless ignored pair (zero-egress 2 + toggle 38) ✓ · cargo doc -D warnings (incl. --document-private-items) ✓ · cargo deny ✓ · crap4rs PASS (0 above threshold 25, worst 23.0)

Closes #200

🤖 Generated with Claude Code


Open in Stage

Summary by CodeRabbit

Release Notes

  • New Features

    • Reports now display model descriptions and tags for enhanced context visibility
    • Unit test overrides are captured and included in generated report payloads
    • Added model context lookup in reports showing descriptions, tags, materialization, and associated test details
    • Native JSON data types preserved for override values in reports
  • Tests

    • Enhanced test coverage for report generation with model metadata and unit test overrides
    • Improved integration testing for manifest ingestion

github-actions Bot and others added 6 commits June 10, 2026 23:34
…ob (#200)

Node gains the authored model description (empty-string unset shapes
dropped at the adapter, the #165 precedent) and the resolved top-level
tags (the deduplicated wire list — never the duplicate-carrying
config.tags), via a with_model_metadata builder. UnitTest retains the
FULL overrides blob as the grouped UnitTestOverrides map (group -> name
-> serde_json::Value): native scalars preserved end-to-end per the #197
founder decision — the BTreeMap<String, BTreeMap<String, String>>
sketch was rejected as lossy. is_incremental_mode (#145) keeps its
lifted field; with_overrides follows the same no-constructor-churn
builder pattern. Domain stays std+serde only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ive-scalar overrides (#200)

WireNode reads the node's top-level description (fusion omits unset —
skip_serializing_none on ManifestMaterializableCommonAttributes,
dbt-schemas manifest_nodes.rs @ 9977b6cbb1b761065536300037560d8e3c037011;
dbt-core emits "" — dropped) and top-level tags (Option for null-fill
tolerance; config.tags deliberately unread — the committed playground
fixture shows it carrying project+model merge duplicates).
WireUnitTestOverrides widens to all three fusion channels
({env_vars, macros, vars}: Option<BTreeMap<String, YmlValue>> per
UnitTestOverrides in dbt-schemas unit_test_properties.rs @ 9977b6cb…)
and folds them into the domain's grouped map, dropping null AND
empty-map channels individually (both unset shapes verified on real
committed fixtures: jaffle "overrides": null; playground
"vars": {}, "env_vars": {} beside a populated macros). The #145
is_incremental lift is unchanged.

Real-fixture coverage: jaffle (authored description verbatim, ""
dropped, null overrides -> None) + playground (deduplicated top-level
tags, the #125 splice's macros group with a native JSON bool).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…+ ModelPayload.description (#200)

ReportPayload.manifest_nodes: BTreeMap<String, ManifestNodePayload>
keyed by bare model name — the in-scope models + every model referenced
by a rendered test's given.input ref(), never the whole project graph
(source() inputs and unresolvable refs contribute nothing; absent entry
= graceful no-hover). Each entry carries description / materialized
(#145 re-plumb) / tags / declared columns (name, description, type from
the ingested Node.columns data_type map, and the SHIPPED
ColumnTestPayload §2.2 tests — reused, no parallel type) / model_tests
(the NEW grouping: attached_node = model AND column_name = None, mapped
through column_test_payload to {name, detail}). TestPayload.overrides
serializes the grouped map with serde Value values (native bool/number/
string — the #197 founder decision), skip-when-None.
ModelPayload.description rides skip-when-None. All-empty manifest_nodes
entries are skipped and every new field is skip_serializing_if, so
findings-free + manifest_nodes-free payloads stay byte-stable; emission
stays on the #194 payload_json_for_html_script path.

That escaper gains '=' in the tag-opener set: authored descriptions put
SQL-ish prose like 'encounter_start_at <= current_timestamp' on the
wire (the committed playground fixture), and tl-based extractors
mis-scan a raw '<=' — same class as the #170 '<letter'/'<?' fix;
round-trips to the original text through JSON.parse.

Chrome snapshot: the real jaffle fixture now carries its
manifest_nodes entry (materialized + the customer_id column tests;
the "" description correctly absent).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One scenario in report_generation.feature (no new .feature file — the
feature-count mirror stays 14): a synthetic current/baseline pair built
from the domain builders, serialized natively (description/tags are
top-level node fields and the grouped overrides matches the wire shape,
so serialize_to_tmp round-trips through the REAL manifest reader — no
wire injection needed), run through the real subprocess. Asserts
ModelPayload.description, the manifest_nodes entries for the in-scope
model AND its NOT-in-scope ref()-ed upstream (proving the ref() arm),
and the overrides groups carrying a native JSON number + bool (type
probes would fail on "7"/"true" strings).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…yload

order_metrics gains config.tags ['marts', 'dogfood'] — the live
dbt-project dogfood previously carried zero model tags, so the new
manifest_nodes tags row would have been invisible on the PR-preview
report (the dogfood-pairing rule). Its #125 overrides block (vars +
env_vars) and the model/column descriptions already light up the other
new fields.

Golden regen (all three, the ci.yml recipe). Byte audit: every changed
byte is one of the new contract fields — manifest_nodes on all three
(jaffle: stg_customers, materialized + column tests, "" description
correctly absent; playground: 7 in-scope models + 4 ref()-ed upstreams;
diff-showcase: 3 + 4), ModelPayload.description on every described
in-scope model, overrides {macros: {is_incremental: true}} (native
bool, empty channels dropped) on the playground incremental test, and
three \u003c= escapes inside NEW description content. Zero
pre-existing payload bytes changed (old payloads carried no raw <=).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lefthook pre-push docs hook runs --document-private-items, where a
domain doc link to the adapters' ModelPayload is unresolvable (and
names an outward item the domain must not reference anyway).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@coderabbitai

coderabbitai Bot commented Jun 11, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

Pull request was closed or merged during review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 62d8f33d-3371-4277-ab5d-19d6b7c99ca6

📥 Commits

Reviewing files that changed from the base of the PR and between 71fba89 and 4054129.

⛔ Files ignored due to path filters (1)
  • tests/snapshots/render_integration__rendered_chrome_jaffle_shop.snap is excluded by !**/*.snap
📒 Files selected for processing (13)
  • dbt-project/models/marts/_marts__models.yml
  • examples/diff-showcase-report.html
  • examples/jaffle-shop-report.html
  • examples/playground-report.html
  • features/report_generation.feature
  • src/adapters/manifest.rs
  • src/adapters/render.rs
  • src/domain/manifest.rs
  • src/domain/mod.rs
  • src/domain/unit_test.rs
  • tests/manifest_ingestion.rs
  • tests/steps/report_generation.rs
  • tests/steps/world.rs

📝 Walkthrough

Walkthrough

This PR implements the Phase-2 manifest context feature by adding model descriptions, tags, and unit-test overrides as rich payloads. Domain models are extended with optional metadata fields; the manifest adapter translates dbt v12 data into grouped overrides while preserving native JSON types; the render layer builds manifest_nodes lookups with scoped model context for hover cards; and Cucumber BDD tests validate the full end-to-end data contracts.

Changes

Phase 2 Manifest Context and Overrides

Layer / File(s) Summary
Domain model contracts for description, tags, and overrides
src/domain/unit_test.rs, src/domain/manifest.rs, src/domain/mod.rs
UnitTestOverrides type alias (nested BTreeMap preserving native JSON scalar types) and UnitTest field added; Node extends with optional description and tags fields; serde defaults for backward compatibility; builder and getter methods for safe access.
Wire-to-domain translation for manifest v12 metadata and overrides
src/adapters/manifest.rs
WireNode deserializes top-level description and tags; WireUnitTestOverrides captures all three channels (macros, vars, env_vars); into_grouped folds channels and filters empty ones; WireUnitTest::into_domain and WireManifest::into_domain chain metadata setters without changing incremental-mode logic; comprehensive test coverage validates scalar passthrough, overrides() collapsing to None for unset shapes, and per-channel dropping.
Report payload structures and manifest_nodes assembly
src/adapters/render.rs
ModelPayload adds optional description; TestPayload adds optional overrides; ReportPayload adds manifest_nodes map; new ManifestNodePayload, ManifestColumnPayload, ModelTestPayload types for rich context; build_manifest_nodes helper scopes referenced models (in-scope + unit-test ref() upstreams); HTML JSON escaping broadened for tag-opener edge cases; payload assembly wires fields from domain.
Integration and BDD test coverage for data contracts
features/report_generation.feature, tests/steps/world.rs, tests/steps/report_generation.rs, tests/manifest_ingestion.rs
New ContextPlan struct holds per-model metadata and override state; Cucumber steps define context models (with optional tags), attach unit tests with native scalar overrides, and assert model descriptions, manifest_nodes entries, and override JSON types in rendered payload; manifest ingestion test validates real-fixture round-trip behavior.
Example and fixture updates to reflect new metadata
dbt-project/models/marts/_marts__models.yml, examples/jaffle-shop-report.html
dbt model configuration adds explicit tags for manifest consumption; HTML example payload refreshed with new stg_customers model metadata.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

  • breezy-bays-labs/cute-dbt#200: Main issue implements the exact Phase-2 data contracts (ReportPayload.manifest_nodes, TestPayload.overrides, ModelPayload.description) and corresponding manifest/domain ingestion in the same Rust modules.
  • breezy-bays-labs/cute-dbt#201: UI/layout work that depends on manifest_nodes and overrides payload surfaces this PR supplies.
  • breezy-bays-labs/cute-dbt#197: Related render/manifest/overrides contract changes (ModelPayload.description, ReportPayload.manifest_nodes, TestPayload.overrides).

Possibly related PRs

Poem

🐰 A rabbit hops through manifests,
gathering tags and descriptions in nested tests,
building context lookup maps so bright,
with native JSON types held just right—
Phase 2 blooms, the payload takes flight! 🌿

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the three main components of the PR: manifest_nodes lookup, TestPayload.overrides with Value passthrough, and ModelPayload.description.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch adapters-200-design2-data-contracts

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ast-grep (0.43.0)
examples/jaffle-shop-report.html

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ghost

ghost commented Jun 11, 2026 •

Copy link
Copy Markdown

Ready to review this PR? Stage has broken it down into 6 individual chapters for you:

Title
1 Define domain types for model metadata and overrides
2 Ingest model metadata and overrides from manifest
3 Build report payload with manifest_nodes lookup
4 Escape SQL comparisons in JSON output
5 Verify data contracts with integration tests
6 Other changes
Open in Stage

Chapters generated by Stage for commit 4054129 on Jun 11, 2026 3:44am UTC.

…howcase) + golden regen

Conflict resolution by REGENERATION, never manual picking: all three
examples re-rendered from the merged tree with the ci.yml recipe.
Non-payload bytes are byte-identical to origin/main's goldens (the
#213 template/CSS delta carries through untouched); the payload delta
vs origin/main is exclusively the #200 contract fields — manifest_nodes,
ModelPayload.description, the native-scalar overrides, and the three
\u003c= escapes inside new description content. The #214
external-fixture given rides through unchanged (no existing payload
key altered). Double-render byte-identity verified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📄 Rendered report preview

All golden examples regenerated cleanly.

🟡 Golden examples

Committed to examples/ and byte-identity gated — the canonical reports contributors and consumers browse. Stable across PRs.

Report View Download
jaffle-shop-report.html ▶ Open ↗ ⬇ Download
diff-showcase-report.html ▶ Open ↗ ⬇ Download
playground-report.html ▶ Open ↗ ⬇ Download

🐶 Live dogfood preview

This PR's own dbt-project/ diff, freshly compiled by fusion into an ephemeral manifest (never committed) and rendered with --pr-diff. Regenerated every PR — the live self-dogfood, not a committed example.

Report View Download
dbt-project-report.html ▶ Open ↗ ⬇ Download

▶ Open ↗ opens the report in your browser in one click —
published to this repo's GitHub Pages under /pr-216/.
⬇ Download fetches the same self-contained HTML as a workflow
artifact (auth-gated; works fully offline). Either way the report
makes zero external resource requests.

The Pages preview may take ~1 min to update after this comment
posts. On PRs from forks the Open link is unavailable (read-only
token) — use Download.

Alternative: GitHub CLI
# gh CLI >= 2.63 extracts into ./report-preview-playground/.
gh run download 27322326748 -R breezy-bays-labs/cute-dbt -n report-preview-playground
open report-preview-playground/playground-report.html

Posted by report-preview.yml for 4054129fe615d1aa816345357fc527d65164ab43. Affordance only — never blocks merge.

@cmbays
cmbays merged commit 353caff into main Jun 11, 2026
31 of 32 checks passed
@cmbays
cmbays deleted the adapters-200-design2-data-contracts branch June 11, 2026 03:48
github-actions Bot added a commit that referenced this pull request Jun 11, 2026
cmbays pushed a commit that referenced this pull request Jun 11, 2026
- tests/steps/world.rs: kept BOTH accumulators — main's ContextPlan
  (#216) and the explore fields/structs (#100).
- tests/steps/report_generation.rs: the NEW #216 context-report
  invocation (landed on main pre-verb) gains the report verb.
- examples: report goldens taken from main and verified byte-identical
  over the merged tree under the report verb; explore goldens
  REGENERATED over the merged tree (stale twice: #214 changed the
  playground manifest, #216 added payload fields tests.html embeds) +
  double-render byte-identity + leak-grep clean.
- ci.yml/lefthook.yml auto-merged: verb-aware gate + tripwire +
  feature-count 16 intact (no upstream bump — no renumber).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: design-2 integration PR-3 — data contracts: report manifest_nodes lookup + TestPayload overrides + ModelPayload description

1 participant