Skip to content

feat(v3): add ValueBody list substrate and std.unicode bootstrap - #920

Merged
briansrls merged 20 commits into
mainfrom
session/neat-dove-411
Apr 26, 2026
Merged

briansrls merged 20 commits into
mainfrom
session/neat-dove-411

Conversation

@briansrls

@briansrls briansrls commented Apr 26, 2026 •

Copy link
Copy Markdown
Contributor

Brief

Assigned brief: docs/briefs/t-substrate-valuebody-list-worker.md.

Acceptance Receipt

  • Added ValueBody::List(Vec<FieldValue>) for top-level list-shaped data bodies.
  • Lowered SurfaceExpr::List in lower_data_item through existing FieldValue structural lowering; element shape remains Vec<FieldValue>.
  • rust_pilot_primitives now lowers from Dag::new() as ValueBody::List with 10 FieldValue::Variant elements; integration test checks count and first/last constructor labels.
  • Added std.unicode to the bootstrap std fixture path; integration test verifies CharClass resolves from fresh Dag::new().
  • Regenerated bootstrap_std_generated.rs, bootstrap_generated.rs, and bootstrap_generated_without_parse_surface.rs; regen_bootstrap --verify passes.
  • Added the four-pattern coproduct dissolution receipt on the new ValueBody::List variant.
  • Updated exhaustive ValueBody consumers, including snapshot rendering, lens/test-runner conversion, refinement checks, and structural-only test helpers. No wildcard swallowing.
  • resolve_data_path handles ValueBody::List deliberately as None: DB-10 scalar/record field path reads do not expose list terminal reads yet.
  • Kept ValueBody::Map / kernel_algebra_profile out of scope.

Feedback Addressed

  • Regenerated bootstrap snapshots so runtime Dag::new() observes rust_pilot_primitives.value_body = ValueBody::List(_).
  • Fixed exhaustive test/helper matches for ValueBody::List.
  • Kept the R14 unparsed-data diagnostic generic across failed structural fallback cases instead of claiming every Unparsed data body is a map literal.
  • Refreshed parse_corpus_manifest.txt after the tokenize.dag comment update.
  • Merged current origin/main; the effective diff no longer contains prior neat-dove-411 #882 docs lineage.

Verification

Local:

  • cargo fmt --all --check
  • cargo test -p v3-compiler --test integration extdeps_rust_primitives_loader_test -- --nocapture
  • cargo test -p v3-compiler --test integration parse_stage4_prep::handwritten_parse_snapshot_matches_manifest -- --nocapture
  • cargo run -p v3-compiler --features bootstrap-regen-fresh --bin regen_bootstrap -- --verify
  • cargo clippy -p v3-compiler --all-targets -- -D warnings
  • RUSTC_BOOTSTRAP=1 cargo test -p v3-compiler -- -Z unstable-options --report-time reproduced the stale manifest failure before the latest fix; targeted manifest test passes after refresh.

Pending/CI:

  • New head 40740a6b should rerun GitHub checks with the refreshed parser manifest.
  • DB-8/self-host ratchet must still be read from CI for final merge readiness.

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 934a8f45 · Trigger: schedule
  • Thinking: 92s wall

⚠️ Review could not be completed because the required source-of-truth command gh pr diff 920 --repo gunb-ai/gunbc failed to connect to api.github.com.

@briansrls
briansrls marked this pull request as ready for review April 26, 2026 21:23

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 934a8f453a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +2747 to +2749
Some(list_expr @ SurfaceExpr::List { .. }) => {
lower_list_to_structural(name, list_expr, ty_decl_id, symbols, dag)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Regenerate bootstrap snapshots for top-level list lowering

lower_data_item now lowers SurfaceExpr::List into ValueBody::List, but Dag::new() does not re-lower sources; it clones the committed snapshot from bootstrap_generated::bootstrapped_fixture_dag in dag.rs. Because this commit does not regenerate bootstrap_generated*.rs, the shipped bootstrap graph still contains rust_pilot_primitives as ValueBody::Unparsed (see src/v3/compiler/src/bootstrap_generated.rs), so runtime Dag::new() consumers will not observe the new behavior yet. Please run regen_bootstrap and commit updated generated snapshots with this change.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified against current PR head 717026a: this is addressed. The bootstrap snapshots were regenerated and now contain structural list bodies; src/v3/compiler/src/bootstrap_generated.rs has rust_pilot_primitives as value_body: Some(ValueBody::List(...)), and the no-parse-surface snapshot matches. I also reran the focused coverage: cargo test -p v3-compiler --test integration top_level_list_data_body -- --nocapture (2 passed) and cargo test -p v3-compiler --test integration extdeps_rust_primitives_loader_test -- --nocapture (3 passed).

Comment thread src/v3/compiler/src/lower.rs Outdated
Diagnostic::ResolveError {
name: format!(
"data `{name}` has an opaque body — M1(2.8) user code cannot yet use record / list / map literals inside data bodies (see DOWNSTREAM_REQUIREMENTS.md class-5 gap #3)"
"data `{name}` has an opaque body — M1(2.8) user code cannot yet use map literals inside data bodies (see DOWNSTREAM_REQUIREMENTS.md class-5 gap #3)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep unparsed-data diagnostic accurate across fallback cases

This message now claims unparsed data bodies are specifically due to unsupported map literals, but reject_user_unparsed_scaffolds is reached for any ValueBody::Unparsed, including record/list bodies that failed structural/type checks and fell back to Unparsed in lower_data_item. In those cases users get a misleading extra error about maps, which obscures the real failure path; the text should remain generic or be emitted only when the original body was actually a map.

Useful? React with 👍 / 👎.

@briansrls briansrls changed the title neat-dove-411 feat(v3): add ValueBody list substrate and std.unicode bootstrap Apr 26, 2026
@briansrls

Copy link
Copy Markdown
Contributor Author

Review metadata

  • Provider / model: openai-pro / gpt-5-5-pro
  • Commit: 40740a6b · Trigger: manual
  • Conversation: View conversation

1. Story of the diff

This PR turns top-level .dag list-valued data declarations from opaque bootstrap facts into first-class substrate data. The central move is adding ValueBody::List(Vec<FieldValue>) in src/v3/compiler/src/dag.rs:320, wiring lowering so data xs: List<T> = [...] becomes that structural body at src/v3/compiler/src/lower.rs:2747, and teaching downstream readers/renderers to handle it in lens_apply, test_runner, and bootstrap emission. That unlocks rust_pilot_primitives as an actually walkable 10-element list rather than an Unparsed body, and the regenerated bootstrap snapshots show the same shape for newly bootstrapped std.unicode list data.

The other half of the diff brings dsl/std/unicode.dag into the std bootstrap set: bootstrap_regen_fresh.rs:31 includes the file and bootstrap_regen_fresh.rs:125 adds it to std_fixtures(), while the generated snapshots now include CharClass, Unicode block data, and list-backed Unicode tables. Tests were refreshed to assert the new structured list for Rust primitives and the presence of std.unicode::CharClass in Dag::new().

2. Invariant categories

  1. LAYER MODEL (substrate vs implementation).

Compliant — this is substrate-touching, and the new substrate carrier is modeled explicitly rather than hidden in an implementation mirror: src/v3/compiler/src/dag.rs:320 adds List(Vec<FieldValue>), while src/v3/compiler/src/lower.rs:2788 checks that the declared type is actually List<_> before constructing it. That keeps the value fact on the Dag and avoids a parallel Rust-only enumeration.

  1. INVARIANTS.md + modeling-discipline.md.

Compliant — fail-closed/API-level enforcement is handled in the new lowering path: src/v3/compiler/src/lower.rs:2788 branches on list_element_type(...), src/v3/compiler/src/lower.rs:2793 emits the typed diagnostic message when the declared type is not List<_>, and src/v3/compiler/src/lower.rs:2799 returns None instead of fabricating a plausible body. Coproduct discipline is also explicitly documented for the new enum variant at src/v3/compiler/src/dag.rs:304 through src/v3/compiler/src/dag.rs:319, including the four-pattern check and terminal verdict.

  1. CODING.md.

Compliant — the new logic stays in data + free-function style. The substrate remains a data enum at src/v3/compiler/src/dag.rs:320, lowering is a standalone helper at src/v3/compiler/src/lower.rs:2781, and rendering/adapter code is handled by local pure matches such as src/v3/compiler/src/regen_bootstrap_emit.rs:286 and src/v3/compiler/src/test_runner.rs:2481. No new builder object, hidden global state, or method-heavy API surface is introduced.

  1. TESTING.md.

Finding (NON-BLOCKING) — the tests pin the generated/bootstrap outcome, but not the new lowering behavior directly. src/v3/compiler/tests/integration/extdeps_rust_primitives_loader_test.rs:116 asserts that the committed bootstrap declaration is now ValueBody::List, and src/v3/compiler/tests/integration/extdeps_rust_primitives_loader_test.rs:142 asserts std.unicode::CharClass loads. What is not directly covered is the new runtime branch at src/v3/compiler/src/lower.rs:2747 or its fail-closed non-List<_> diagnostic at src/v3/compiler/src/lower.rs:2793. A focused compile/lower test for data xs: List<Int> = [...] and a negative data xs: Int = [...] case would make the behavior test match the interface that changed. I do not consider this blocking because the bootstrap feature this PR needs is covered.

  1. LOCKED DESIGN DECISIONS.

N/A — I did not see this diff altering a locked design decision. It moves an already-named R2 substrate lane forward by adding ValueBody::List and bootstrapping std.unicode, rather than changing external realization, target emission authority, or another locked seam.

  1. TRACKED vs UNTRACKED DEBT.

Finding (NON-BLOCKING) — two updated comments appear to reference a not-yet-landed ValueBody::Map lane as though it exists: src/v3/compiler/src/bootstrap.rs:51 says map-shaped bootstrap data remains on the sibling ValueBody::Map lane, and src/v3/compiler/src/dag.rs:2893 repeats that wording. The diff only adds ValueBody::List at src/v3/compiler/src/dag.rs:320; map bodies still appear to be unimplemented/opaque in this PR. If “ValueBody::Map lane” is meant as future debt, the comment should name the documentation row, bounds, and dissolution trigger; if it is just shorthand for a future sibling, I would reword it to avoid implying a live enum variant.

3. Verdict

APPROVE_WITH_COMMENTS

The substrate shape is sound: list bodies are represented once on ValueBody, lowering fails closed on non-list declarations, and bootstrap/tests now consume the new structured fact. The only issues I found are non-blocking: direct lower-path test coverage would strengthen the regression net, and the ValueBody::Map wording should be clarified so it does not look like a live substrate lane.

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 40740a6b · Trigger: schedule
  • Thinking: 80s wall

⚠️ Review is blocked because the required source-of-truth command gh pr diff 920 --repo gunb-ai/gunbc failed to connect to api.github.com.

@briansrls

Copy link
Copy Markdown
Contributor Author

Addressed the review comments in head 8d2575cca.

  • Added direct lower-path tests for data xs: List<Int> = [1, 2, 3] lowering to ValueBody::List, and data xs: Int = [1, 2, 3] producing the fail-closed non-List<_> diagnostic.
  • Clarified the map wording so it no longer reads as a live ValueBody::Map enum/implementation; it now states that map-shaped bootstrap data is future T-Substrate debt and still lowers to ValueBody::Unparsed today.
  • The scheduled Codex item was an API connectivity failure, not actionable code feedback.

Local verification before push: cargo fmt --all --check, cargo test -p v3-compiler --test integration top_level_list_data_body -- --nocapture, and cargo test -p v3-compiler --test integration extdeps_rust_primitives_loader_test -- --nocapture.

# Conflicts:
#	src/v3/compiler/src/bootstrap_generated.rs
#	src/v3/compiler/src/bootstrap_generated_without_parse_surface.rs
@briansrls

Copy link
Copy Markdown
Contributor Author

Queued feedback disposition after recheck:

  • Director review at 2026-04-26T23:12:48Z is approval-only and requested merge; feat(v3): add ValueBody list substrate and std.unicode bootstrap #920 is already merged.
  • R2 Modeling consumer review is approval-only/consumer readiness; the pending ratchet it mentioned finished green before merge.
  • Earlier bootstrap-regen and generic unparsed-data diagnostic findings were already addressed before merge; current review data shows the bootstrap thread answered and the diagnostic thread outdated.
  • The Cursor scheduled review's test-style finding was valid: the negative list-body test used a substring match on the diagnostic text. I pushed follow-up head 3002c04 on session/neat-dove-411 tightening that assertion to an exact expected diagnostic string, making the wording lock explicit instead of a loose contains match.
  • The exploratory lower.rs unreachable note is optional and I left it unchanged: the helper is private, called only after the SurfaceExpr::List dispatch, and the current unreachable documents that internal invariant.

Verification: git diff --check passed. I could not rerun the Rust integration test in this restored shell because cargo/rustup/nix are not installed on PATH in the execution environment.

@briansrls

Copy link
Copy Markdown
Contributor Author

Review metadata

  • Provider / model: claude / claude-opus-4-7
  • Commit: 50a082a4 · Trigger: schedule
  • Thinking: 41s wall

Verdict: APPROVE — Diff is narrowly scoped: adds ValueBody::List substrate variant, structural lowering for top-level data xs: List<T> = [...], and pulls std.unicode into the bootstrap fixture load set. The new variant reuses the existing FieldValue::List (single source for nested + top-level list values — no parallel representation), carries a complete 4-pattern receipt at dag.rs:302-319, and every match site (lens_apply.rs, lower.rs, regen_bootstrap_emit.rs, test_runner.rs, three integration tests) is updated exhaustively. Fail-closed path is in place: lower_list_to_structural (lower.rs:2783-2820) emits a ResolveError when the list body's declared type isn't List<_>, with a regression test (m1_substrate_test.rs:2010-2024). Comments in bootstrap.rs, dag.rs, and tokenize.dag are updated to reflect that the list-shape gap closed while the map-shape gap (kernel_algebra_profile) remains tracked debt with a named dissolution lane. Bulk changes in bootstrap_*_generated.rs are mechanical regen artifacts (DeclarationId renumbering for the inserted std.unicode decls).

No findings.

@briansrls

Copy link
Copy Markdown
Contributor Author

Review metadata

  • Provider / model: codex / codex-default
  • Commit: 50a082a4 · Trigger: schedule
  • Thinking: 84s wall

Verdict: APPROVE

The diff cleanly extends the substrate with ValueBody::List, adds the required coproduct/dissolution receipt, routes list bodies through existing structural validation, and adds focused fail-closed tests for the new path. I didn’t find a concrete violation of the pinned modeling, coding, or testing rules in the changed lines.

I wasn’t able to run the targeted cargo tests because the local exec tool started rejecting new processes, so this is a static review.

briansrls added a commit that referenced this pull request Apr 27, 2026
Single control surface for cleanup-lane harvest of #900/#901/#920/#897/#824/#825
per dispatch from tidy-dove-734 (#941). No ctrl#263 in repo; this docs/audit
artifact is the agreed fallback. Rows: source PR, gap, file/invariant, owner
lane, dissolution trigger, acceptance check, tracking authority, disposition.
briansrls added a commit that referenced this pull request Apr 27, 2026
This was referenced Apr 27, 2026
briansrls added a commit that referenced this pull request Apr 27, 2026
Codex-connector inline on #954 flagged that the original "additions
fall to Narrow by default" wording was misleading: char_display_width
checks wide_blocks / zero_width_blocks first, so newly assigned code
points inside an already-enumerated range inherit that block's class,
not Narrow. Only code points outside every enumerated range fall to
Narrow. Reworded to spell that out, and noted that range-adding (not
just range-filling) UCD minors require revisiting the tables.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
briansrls added a commit that referenced this pull request Apr 27, 2026
* docs(audit): W-C1 follow-up harvest table

Single control surface for cleanup-lane harvest of #900/#901/#920/#897/#824/#825
per dispatch from tidy-dove-734 (#941). No ctrl#263 in repo; this docs/audit
artifact is the agreed fallback. Rows: source PR, gap, file/invariant, owner
lane, dissolution trigger, acceptance check, tracking authority, disposition.

* docs(audit): close #897/#824/#825 rows per bright-wolf-465 audit

Per bright-wolf-465 (inbox #945), no missed BLOCKING findings on these PRs;
flip rows 6/7/8 to closed with audit citation.

* docs(audit): reconcile note with row dispositions for #897/#824/#825

* WIP: calm-ant-861

* chore: apply cargo fmt

* WIP: calm-ant-861

* fix(test): fold B5 Loop closure receipt into m1_substrate_test

SG-0 census ratchet forbids new hand-authored .rs files. Move the
every_loop_node_originates_from_recursive_function_lowering test (and the
LoopNode/LoopBound import + two helper fns) into the existing
m1_substrate_test.rs and drop the standalone file + its integration.rs
registration. Behavior unchanged.

* docs: correct B5 receipt file path in synthesis doc (m1_substrate_test, not standalone file)

* docs: mark ROADMAP loop-emission row resolved + correct synthesis cite

Codex BLOCKING on be4a6ab: ROADMAP.md still listed the loop-emission
semantic invariant as open debt while the synthesis doc declared it
resolved — tracker-authority mismatch.

- ROADMAP.md:436: rewrite the row as RESOLVED 2026-04-27 with PR cite,
  audit summary (two production sites in lower.rs), receipt location
  (m1_substrate_test.rs), and marker-retired note.
- synthesis-doc §3 line 155: fix the receipt path
  (loop_construction_closure_test.rs → m1_substrate_test.rs after the
  SG-0 fold).

* fix(test): split global vs fixture-scoped Loop closure assertions

Codex BLOCKING on db7b773: previous test mixed a global all-Dag closure
check with fixture-only variant claims, so bootstrap loops could mask the
fixture-coverage claim.

Split into two phases:
- Global: every Behavior::Loop in the Dag (fixture + bootstrap) satisfies
  the recursive-function-lowering signature (loop.output is a Bind value
  port + Descent cluster id resolves). This is the closure invariant.
- Fixture-scoped: filter loops by span.file == fixture file before claiming
  both LoopBound variants are produced by THIS fixture. Bootstrap loops are
  excluded so the variant claim is mechanically about the fixture's
  recursive functions.

* post-merge: drop superseded artifacts; correct ROADMAP B5 receipt cite

The parallel B5 lane landed `r2_b5_loop_construction_closure_test.rs` on
main with a more rigorous receipt (substring push-site ratchet + per-fixture
DAG walk + Origin::Accumulated provenance). The harvest table in this PR is
also being landed via aggregate #949.

Drop:
- `docs/audit/w-c1-followup-harvest-2026-04-27.md` (canonical surface is
  #949).
- The m1_substrate_test additions (auto-reverted via merge --theirs;
  superseded by the standalone r2_b5_loop_construction_closure_test.rs on
  main).

Keep:
- `ROADMAP.md` row marking loop-emission RESOLVED, with citation rewritten
  to point at the actual landed receipt file (r2_b5_…) instead of the
  retired m1_substrate_test cite.
briansrls added a commit that referenced this pull request Apr 27, 2026
* docs(std.unicode): cite UCD 15.x / UAX-11 authority

Closes the #920 post-merge citation gap. Header now states the file
is sourced from UCD 15.x (UAX #11 East Asian Width) for the display-
width tables and is intentionally 15.x compatible rather than pinned
to a specific minor. UAX #9 is explicitly not consulted (no bidi).
No behavior changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: tighten P5 per-PR dissolution gate (b) for ROADMAP-cited deferrals

Require exactly one checkable receipt: delete path, SG-0 census
before/after counts, or lane plus concrete ROADMAP row/link. Call out
vague deferrals as insufficient. Align INVARIANTS §P5 (b) with the
template without duplicating the checklist.

Made-with: Cursor

* docs(audit): W-C1 follow-up harvest table

Single control surface for cleanup-lane harvest of #900/#901/#920/#897/#824/#825
per dispatch from tidy-dove-734 (#941). No ctrl#263 in repo; this docs/audit
artifact is the agreed fallback. Rows: source PR, gap, file/invariant, owner
lane, dissolution trigger, acceptance check, tracking authority, disposition.

* docs(audit): mark #920 citation follow-up closed

* docs(audit): close #897 #824 #825 harvest rows

* docs(audit): cite closure evidence for harvest rows

* WIP: Cleanup

* docs(std.unicode): clarify UAX 11 coverage

* docs: link cleanup harvest from roadmap

* docs(std.unicode): refresh bootstrap spans

* docs(std.unicode): refresh full bootstrap spans

* docs(std.unicode): refresh no-parse bootstrap spans

* docs(audit): stabilize harvest code references

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
briansrls added a commit that referenced this pull request Apr 29, 2026
Codex review: spot-check claimed no landed slice vs HEAD; #920 merged
ValueBody::List + std.unicode bootstrap. Gate/Last signal/Notes updated;
explicit ROADMAP Gap 3 caveat (stale prose). P1 live-state discipline.

Made-with: Cursor
briansrls added a commit that referenced this pull request Apr 29, 2026
* docs(r2): closure-ledger — Substrate/PB rows + R1 path-a signal

- T-Substrate-Lens-Primitive: in-flight, #1186 (carrier); instance gates pending
- ValueBody list/sum: spot-check note (ROADMAP gap unchanged this pass)
- PB patch-lower-helpers: note #1014 + #1192 ratchet scope
- R3 bridge retirements: in-flight #1171 #1183 #1192; Verification gate explicit
- Incoming surface: path-(a) closure — no residuals absorbed; named R1C-B deferral

Cross-program drift sweep per PM detection + Director endorsement.

Made-with: Cursor

* docs(r2): ledger — ValueBody row reflects #920 list + unicode slice

Codex review: spot-check claimed no landed slice vs HEAD; #920 merged
ValueBody::List + std.unicode bootstrap. Gate/Last signal/Notes updated;
explicit ROADMAP Gap 3 caveat (stale prose). P1 live-state discipline.

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant