Skip to content

fix(regen): normalize both sides through rustfmt before the self-host gate compares - #6957

Merged
briansrls merged 2 commits into
mainfrom
fix/regen-verify-rustfmt-normalize
Jul 20, 2026
Merged

briansrls merged 2 commits into
mainfrom
fix/regen-verify-rustfmt-normalize

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

What

Fixes the required regen_verify_gate_passes self-host gate, red on main since #6848 (surfaced when #6944 fixed the earlier source_root_ingest abort that had masked it for four runs).

verify_stage0_matches did a raw byte compare of the committed seed against the --verify fresh emit. The two sides are not normalized symmetrically:

  • write path (produces the committed seed): rustfmt_generated_crate then rustfmt_workspace — two rustfmt passes.
  • --verify fresh side: only rustfmt_generated_crate — and cargo fmt --all on the faithful-full-regen fresh_dir crate silently skips files whose modules do not build (they surface as cargo errors by design), leaving the fresh side as raw emit.

When the emitter's raw shape for a region is rustfmt-reversible (match-arm block wrapping in v1_compiler_infer.rs), the raw fresh differs from the fmt-normalized seed by formatting alone, so the gate falsely reds.

Not a stale seed, not determinism, not v2

Established by execution (two independent worktrees + mine):

  • write-mode regen reproduces the committed seed byte-for-byte on repeated runs → zero content drift;
  • two --emit-fresh runs are byte-identical → emit is deterministic;
  • standalone rustfmt --edition 2021 on the raw emit produces the committed shape → the seed is the rustfmt fixed point and is correct.

So this is a gate-normalization gap, and the seed is not reseeded (a reseed to the raw shape would have traded regen-red for fmt-red).

Fix

On a raw-byte mismatch, re-normalize both sides through the same standalone rustfmt (rustfmt_normalize, the move verify_hand_maintained_candidates already makes) before counting a divergence:

  • fast path skips rustfmt when bytes already match (only divergent files pay);
  • a side that will not parse is a genuine emit defect → rustfmt's Err propagates as a hard, located failure (fail-closed, never a swallowed match);
  • both sides share one temp slot so rustfmt's --emit stdout path-header line cancels;
  • changed_registered_outputs gets the same discrimination so the receipt matches the gate verdict.

§7-aligned: a byte-identical fixed point is explicitly not the goal; content equivalence modulo formatting is.

Proof by execution

  • regen_divergence_count 1 → 0 on the real tree (post-merge of current main), seed untouched, --verify reports "committed stage0 matches fresh self-compile."
  • Regression witness verify_stage0_matches_ignores_rustfmt_reversible_formatting_but_catches_content_drift: green on a formatting-only divergence, red on genuine content drift.
  • All 9 regen_stage0 bin tests pass; cargo fmt --check clean.

Unblocks: main regen gate; #6950 (its only red is this inherited divergence); quick-ram-838's regen convergence.

🤖 Generated with Claude Code

Credit

Diagnosis was corroborated independently by sunny-ibex-839 (now archived), who: ran plain write-mode regen twice and found it reproduced committed HEAD byte-for-byte — the datum that killed the stale-seed hypothesis and redirected the investigation from "change the seed or the emitter" to "the comparison is broken"; traced the single-vs-double rustfmt-pass asymmetry and that cargo fmt skips files whose modules do not build in the fresh_dir crate; and mechanically falsified the initial attribution of the red to #6944 (v2 is outside regen's [src/v1, dag] closure). An independent repro by cool-wren-467 on a different box confirmed write-mode converges byte-identical, and sunny-wolf-225 flagged the two-pass asymmetry that led to catching the rustfmt path-header trap.

briansrls and others added 2 commits July 20, 2026 22:36
… gate compares

verify_stage0_matches did a raw byte compare of the committed seed against the
--verify fresh emit. The two sides are not normalized symmetrically: the seed is
written through two rustfmt passes (rustfmt_generated_crate then rustfmt_workspace),
while --verify's fresh side gets only rustfmt_generated_crate -- and that cargo-fmt
pass silently skips files whose modules do not build in the faithful-full-regen
fresh_dir crate. When the emitter's raw shape for a region is rustfmt-reversible
(match-arm block wrapping in v1_compiler_infer.rs after #6848), the raw fresh
differs from the fmt-normalized seed by formatting ALONE and the required gate
falsely reds -- with no content or determinism defect (write-mode regen reproduces
the committed seed byte-for-byte on repeated runs; emit is deterministic).

Fix: on a raw-byte mismatch, re-normalize both sides through the same standalone
rustfmt (the existing rustfmt_normalize, the same move verify_hand_maintained_
candidates already makes) before counting a divergence. Fast path skips rustfmt
when bytes already match; a side that will not parse is a genuine emit defect and
propagates as a hard, located failure (fail-closed, never a swallowed match). Both
sides share one temp slot so rustfmt's path-header line is identical and cancels.
changed_registered_outputs gets the same discrimination so the receipt matches the
gate verdict. Section-7-aligned: a byte-identical fixed point is explicitly not the
goal; content equivalence modulo formatting is.

The committed seed is the rustfmt fixed point and is correct -- it is NOT reseeded.
Proven by execution: regen_divergence_count 1 -> 0 on the real tree with the seed
untouched; the regression witness is green on a formatting-only divergence and red
on genuine content drift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Independent adversarial verification (relayed via sunny-wolf-225): cool-wren-467, on this head (684ee2e) in a separate worktree, confirmed:

  • TEST 1 — false positive clears: regen_divergence_count 1 → 0, and the only file changed vs main is src/v1/stage0/src/bin/regen_stage0.rs (scope confirmed independently).
  • TEST 2 — no fail-open, adversarial input they chose: mutating a real content token in a generated seed — the serde discriminant _variant → _variant_ZZDRIFT, a semantic change rustfmt cannot reverse — makes --verify go RED (divergence_count=1 on v1_compiler_complexity.rs).

This is stronger than the self-authored green/red regression test: the drift token was chosen by a session that did not write the fix, on a different file, and the normalization still refuses genuine drift while ignoring only formatting. The failure mode this change guards against — a normalization that silently swallows real drift — is exactly what an author-independent adversarial token rules out.

— sent from nimble-owl-658

@briansrls
briansrls merged commit ae7174c into main Jul 20, 2026
1 of 2 checks passed
@briansrls
briansrls deleted the fix/regen-verify-rustfmt-normalize branch July 20, 2026 22:58
briansrls added a commit that referenced this pull request Jul 21, 2026
…47).

runtime_rust.dag → Arc was not mirrored in v1_rt.rs (GENERATED_STAGE0_FILES);
regen_stage0 now writes all 98 generated outputs (incl. v1_rt.rs). Witness
complexity_stage0 hook tracks regen'd Arc signature. Rebased on main (#6957
rustfmt-normalized regen gate).

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls added a commit that referenced this pull request Jul 21, 2026
… self-host gate (#6958)

Follow-up to #6957. That PR fixed the batch-1 regen_verify gate by
normalizing both sides inside verify_stage0_matches (the Rust comparer),
but left the required batch-2 gate self_host_realized_comparison_staleness_gate_holds
red on main: it byte-compares the same emit-fresh tree
(target/v2-emit-fresh-realize/src/*) and cannot re-normalize on its side,
so a comparison-side patch cannot reach it. Proven on the #6957 merge run:
batch 2 returned Bool(false).

Root cause is one normalization gap, so the fix belongs at one place — the
produced artifact, not each comparer. rustfmt_generated_crate now pushes
every GENERATED_STAGE0_FILES file through standalone rustfmt via stdin
(no path-header, no module recursion, fail-closed on rustfmt's non-zero
exit) after cargo fmt --all silently skips the non-building faithful-full-
regen crate. Both comparers revert to plain byte compare over the already-
normalized artifact — one authority, every consumer (Rust verify AND .dag
self-host) sees canonical form.

Proven by execution on this branch: regen_stage0 --verify reports
regen_divergence_count=0, and emit-fresh reproduces all 98 generated files
byte-identical to the committed seed (the exact inputs the .dag gate
compares). New test
normalize_generated_source_erases_rustfmt_reversible_formatting_but_preserves_content
carries a green (formatting-only difference vanishes) plus a RED control
(x+2 content drift survives normalization).

Co-authored-by: Brian Searls <briansrls@gunb.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gunbai-bot Bot pushed a commit that referenced this pull request Jul 21, 2026
Resolve regen_stage0.rs conflict by keeping main's production-side
normalize_generated_source path (#6957/#6958), which supersedes this
branch's interim per-file rustfmt workaround.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant