From 1015afcb2b482f455f2626d3586396b940c95b3a Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Mon, 24 Aug 2026 19:25:50 +0000 Subject: [PATCH 1/2] The pool census read every function body to throw it away 82ms later: a heads reading of the same grammar, 2x on the corpus parse MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pool_parse` builds full function bodies for all 3875 corpus modules and `census_heads_module_node` replaces every one of them with a shared stand-in 82ms later. The attribution probe measured that as 7.15s of the row's 14.24s and deliberately shipped no repair, naming it this class's next-rung trigger (DESIGN §6, bare minimum cost: a proven cost-shape defect is always fixed). This lands the repair as a READING of the same grammar, not a mode beside it. `ParseContext` carries `heads_only`; `parse_heads_with_table` sets it; a brace-delimited `fn`-shaped body is skipped at token grain, depth-counted over brace SHAPES the tokenizer already decided, and the body slot takes the loud `CensusHeadsBodyStripped` stand-in. All four fn-shaped body sites route through one `parse_item_block_body`, so no call site chooses and the two readings cannot drift apart per site. The census strip is KEPT rather than folded into the parser, and that is the construction move rather than leftover: it normalizes the body slot on BOTH readings, so the stand-in's shape is not a fact any consumer can depend on and the readings can only differ through the HEADS -- which is exactly what the receipt measures. MEASURED (3880 modules, roots dag + src/v2, one process): divergent=0 regressed=0 narrowed=0 both_refused=0 parse wall 12175/5885, 11030/5559, 12882/6382 ms -> 2.07x / 1.98x / 2.02x The ratio is the claim; the absolutes move ~15% on a shared container and are evidence it was measured, not a figure to plan against. It is the PARSE term, not the whole `pool_parse` row -- tokenize and newline indexing are outside both timers and unchanged. RED CONTROL: `depth + 1` -> `depth + 0` in the emitted skip turns 2820 of 3880 modules REGRESSED and exits 1; reverted, rebuilt, re-measured green. SELF-HOST: `--required-regen` first_generation_equal=true over 134 outputs. DECLARED SCOPE NARROWING: the heads reading refuses an unterminated body and every malformed item head, but not a well-braced ungrammatical body. That question is owned by the required run's .dag parse sweep, which full-parses src/v1, dag and src/v2; the census answering it a second time was one fact with two authorities and the expensive one. `narrowed` is a counted column, 0 today, so a future nonzero is visible rather than absorbed. The differential is `claim_executor --heads-reading-differential`, not enrolled in the required run on purpose: reading 3880 modules twice is the cost this removes. Next-rung trigger is a diff-proportional form of the same check. Receipt: docs/probes/heads_reading_of_the_grammar_2026-08-24.md Co-Authored-By: Claude Opus 5 --- ...heads_reading_of_the_grammar_2026-08-24.md | 172 ++++++++++++++++++ src/v1/02_parse.dag | 125 ++++++++++++- src/v1/stage0/src/bin/claim_executor.rs | 50 +++++ src/v1/stage0/src/cli_run.rs | 138 +++++++++++++- src/v1/stage0/src/v1_compiler_parse.rs | 153 +++++++++++++++- 5 files changed, 626 insertions(+), 12 deletions(-) create mode 100644 docs/probes/heads_reading_of_the_grammar_2026-08-24.md diff --git a/docs/probes/heads_reading_of_the_grammar_2026-08-24.md b/docs/probes/heads_reading_of_the_grammar_2026-08-24.md new file mode 100644 index 00000000000..f574412d918 --- /dev/null +++ b/docs/probes/heads_reading_of_the_grammar_2026-08-24.md @@ -0,0 +1,172 @@ +# The pool census reads declaration heads instead of discarding bodies (2026-08-24) + +**Subject:** the residue `docs/probes/edge_index_tree_census_attribution_2026-08-24.md` left open — + +> `parse_with_table` builds full function bodies; `census_heads_module_node` strips them 82ms +> later. Half the corpus parse is work the only consumer throws away. […] It is this class's +> next-rung trigger, it is owed under §6, and it wants its own PR — not its own excuse. + +This is that PR. The attribution probe measured the defect and deliberately shipped no repair; +this change is the repair and carries its own receipt. + +## What changed, stated as a reading rather than a mode + +`v1.compiler.parse` gains a **declaration-heads reading of the same grammar**. `ParseContext` +carries `heads_only`; `parse_heads_with_table` is the entry that sets it. Every item head is +parsed by the same productions as before. The one difference is that a brace-delimited body of +an `fn`-shaped item is **skipped at token grain** — depth-counted over brace SHAPES, which the +tokenizer has already decided — instead of being built into a tree, and the body slot receives +the loud `CensusHeadsBodyStripped` stand-in. + +Two construction choices are what make this a reading rather than a second parser, and both are +load-bearing: + +- **One spelling for the body.** All four `fn`-shaped item body sites (`fn`, `func`, `pattern`, + `interface`) route through a single `parse_item_block_body`, which selects the reading the + context declares. The two readings cannot drift apart per call site because no call site + chooses. +- **The census strip is KEPT, not folded into the parser.** `census_heads_module_node` still + runs, on the heads reading's output, exactly as it ran on the full reading's. That normalizes + the body slot — the one slot the heads reading deliberately fills differently — on *both* + sides. The consequence is that the stand-in's exact shape is not a fact any consumer can + depend on, and the heads reading cannot drift from the full reading through the body slot at + all. It can only drift through the *heads*, which is precisely the surface the differential + below measures. Deleting the strip as "now redundant" would have removed the mechanism that + makes the two readings agree by construction and replaced it with a promise. + +## Measured + +Local arm64 session container, release build, roots `dag` + `src/v2`, one process, +`claim_executor --heads-reading-differential`. + +| | | +|---|---| +| modules compared | **3880** | +| `divergent` (heads ≠ full, after the census strip on both) | **0** | +| `regressed` (heads refuses what full accepts) | **0** | +| `narrowed` (full refuses on body grammar, heads does not) | **0** | +| `both_refused` | 0 | + +Three green runs, and the spread between them is the point rather than noise to average away: + +| run | full reading | heads reading | ratio | +|---|---|---|---| +| 1 | 12175 ms | 5885 ms | 2.07x | +| 2 | 11030 ms | 5559 ms | 1.98x | +| 3 | 12882 ms | 6382 ms | 2.02x | + +This is a shared container under other sessions' load, so the ABSOLUTE walls move ~15% run to +run. The RATIO does not, because both readings in a run pay the same contention over the same +modules — which is exactly what taking them in one process buys. **Quote the ratio.** The +milliseconds are evidence that it was measured, not a figure to plan against, and a single-run +absolute would have been the more confident and less true number. + +**Why the two timings are comparable and a before/after pair would not be.** They are taken in +ONE process, on ONE machine, over the SAME module list, alternating per module. A before/after +figure from two separately-built binaries would have to argue away the build, the host and the +corpus state; this one has nothing to argue away. + +**What the timing is NOT.** It is the PARSE term only. `tokenize` (4.16s in the attribution +probe), `build_newline_index` (0.93s) and the per-file intern-table setup sit outside both +timers and are untouched by this change, so the ~2x is a saving on the parse row, not on the +`pool_parse` row, and quoting it as the latter would be inflation. The attribution probe's +figures were taken on a remote amd64 runner and are not the same denominator as these; the two +sets are not subtracted from each other anywhere in this document. + +## The RED control — the instrument can fail, measured + +A green differential is worth nothing unless it can go red. One token was mutated in the +emitted mirror only (`depth + 1` → `depth + 0` in `heads_skip_block_tokens`, so a nested `{` +stops deepening the count and the skip ends at the first inner `}`), rebuilt, and re-run: + +``` +heads-reading-differential: compared=3880 divergent=0 narrowed=0 regressed=2820 both_refused=0 +heads-reading-differential: REGRESSED dag.test.claim.witness_deferral_freeze_witness +heads-reading-differential: REGRESSED examples.cost_estimate +… +``` + +**2820 of 3880 modules go red on a one-character mutation**, and the command exits 1. The +mutation was then reverted by re-copying the regen candidate, rebuilt, and the differential +re-taken green (`compared=3880 divergent=0 narrowed=0 regressed=0`, exit 0) — the mutated +mirror is not in the diff, and the restored tree is the one measured, not merely assumed +restored. The mutated run's own timings are not quoted anywhere: it bails out of most bodies +early, so its numbers measure the bug rather than the reading. + +The mutation landing in `regressed` rather than `divergent` is itself informative and is the +reason both columns exist. An early-ending skip leaves the token stream mid-body, so the next +item fails to parse and the heads reading *refuses* — it does not quietly return a shorter head +list. That is the fail-closed behaviour one wants, but it is not something to assume: the two +columns are separate so a future mutation that DOES produce a silently shorter head list has +somewhere to land. + +## The self-host fixed point holds + +The parser is `.dag` authority with an emitted stage0 mirror, so a parser change that emits a +mirror the changed compiler would not re-emit is not a change to the parser — it is a fork. +`claim_executor --required-regen` was run after the mirror was installed: + +``` +required-regen: elapsed_ms=583524 first_generation_equal=true planned=134 executed=134 declared_divergent=1 [main.rs] +exit=0 +``` + +`first_generation_equal=true` over all 134 planned outputs. The one declared divergence +(`main.rs`) is pre-existing and unrelated. So the compiler built from this mirror re-emits this +mirror: the heads reading is at the self-host fixed point, not merely compiling. + +## The one refusal this reading does not make — declared, and countable + +The heads reading refuses an unterminated body and every malformed item HEAD, exactly as the +full reading does. It does **not** refuse a well-braced but ungrammatical body, because those +tokens are never handed to the expression grammar. + +**Why this is a scope correction rather than a hole.** The required run's `.dag` parse sweep +full-parses `src/v1`, `dag` and `src/v2` and owns exactly the question *is the corpus +grammatical*. The pool census answering it a second time — as a side effect of building +declaration heads — was one fact with two authorities (§3), and it was the expensive authority: +it coupled every pool-derived resolve in the process to the grammaticality of every body in the +corpus, and paid a whole-corpus body parse to do it. + +**What is honestly given up.** For source roots the parse sweep does not cover, that refusal is +no longer taken anywhere. Today the sweep's roots are a superset of the pool roots the required +run uses, so the live population is empty; an invocation naming some other root is not covered +by either authority. The measurement above is what makes this *observable* rather than asserted: +`narrowed` is its own column, counted per run, currently `0` — DESIGN §5 requires a degradation +to be counted rather than absorbed, and a column reading zero today is how a future nonzero +becomes visible instead of silent. + +## Rung + +The class is *a parse whose only consumer discards half its output*. It sat at **mitigatable** — +nothing was wrong, it was merely paid — and this change does not move it up a safety rung, +because it is a COST repair and saying otherwise would be the inflation §4b(1) names. + +What the change does carry a rung for is the new surface it introduces: *the heads reading +agrees with the full reading*. That sits at **mechanically preventable** — the differential is a +real, re-runnable wall with a discriminating RED and a positive control, and it is +`claim_executor --heads-reading-differential`, not a number quoted from a run nobody can repeat. +It is **not** *structurally guaranteed*, and the honest reason is that the agreement is enforced +by a check rather than by construction on the heads: the census strip makes the BODY slot +unable to disagree, but nothing structurally prevents a future edit to the skip from consuming +one token too many. + +**It is not enrolled in the required run, and this is a deliberate scope statement, not an +oversight.** Reading 3880 modules twice is precisely the cost the heads reading exists to +remove; paying it on every push would spend more than the repair saves. Its **next-rung +trigger** is a form of the check whose cost is proportional to the diff rather than the corpus — +the same shape the affected-set work wants generally — at which point enrolling it becomes +cheap enough to be ordinary. + +## What is NOT claimed + +- **No claim about `pool_parse`'s whole row.** This measures the parse term, in isolation, on a + different machine and architecture from the attribution probe. The end-to-end effect on a + required-floor run is not measured here and no figure for it appears in this document. +- **No claim that the corpus contains a body the full reading refuses.** `narrowed=0` says the + opposite: every module in `dag` + `src/v2` parses clean both ways today. The declared + narrowing is about what WOULD happen, and it is carried as a counted column rather than as a + prediction. +- **`src/v1` was not in the differential's roots.** The roots measured are `dag` + `src/v2`, + which are the roots the required run gives the pool. A run over other roots is a different + denominator and would need its own receipt. diff --git a/src/v1/02_parse.dag b/src/v1/02_parse.dag index 6bcd9a7b280..2f3e4bf86c3 100644 --- a/src/v1/02_parse.dag +++ b/src/v1/02_parse.dag @@ -116,6 +116,13 @@ fn token_stream_peek(stream: TokenStream, offset: Int) -> Token? { get(xs: stream.all, index: stream.pos + offset) } +// `heads_only` selects the declaration-heads reading of the SAME grammar: every +// module-scope declaration head is parsed exactly as it is under the full reading, +// and a brace-delimited `fn` body is skipped at token grain instead of being built. +// It is not a second parser and it is not a lenient one -- the heads reading is +// defined by what its sole consumer (the pool census) retains, and the census +// normalizes every body away 82ms later, so the two readings are required to agree +// after that normalization. That agreement is the oracle, not a description. type ParseContext { source_indices: Map intern_table: InternTable @@ -123,6 +130,7 @@ type ParseContext { occurrence_index: OccurrenceIndex? declaration_occurrences: List? reference_occurrences: List? + heads_only: Bool } type ParseResult { @@ -208,6 +216,7 @@ fn parse_context_with_occurrence_state( occurrence_index: occurrence_index, declaration_occurrences: declaration_occurrences, reference_occurrences: reference_occurrences, + heads_only: ctx.heads_only, } } @@ -222,6 +231,7 @@ fn parse_context_with_intern_table( occurrence_index: ctx.occurrence_index, declaration_occurrences: ctx.declaration_occurrences, reference_occurrences: ctx.reference_occurrences, + heads_only: ctx.heads_only, } } @@ -1427,7 +1437,8 @@ fn parse_with_table_at( tokens: List, source_indices: Map, intern_table: InternTable, - occurrence_base: AuthoredTokenOrdinalSpace + occurrence_base: AuthoredTokenOrdinalSpace, + heads_only: Bool ) -> ParseWithTableResult { let occurrence_allocator = occurrence_id_allocator_advance_to( alloc: occurrence_base.allocator, @@ -1445,6 +1456,7 @@ fn parse_with_table_at( }, declaration_occurrences: Present { value: [] }, reference_occurrences: Present { value: [] }, + heads_only: heads_only, } let r = parse_module(tokens: token_stream_new(all: tokens), ctx: ctx) if has_err(err: r.err) { @@ -1499,7 +1511,8 @@ fn parse_with_table_in_occurrence_scope( intern_table: intern_table, occurrence_base: authored_token_ordinal_space_from_allocator( alloc: occurrence_allocator - ) + ), + heads_only: false ) } @@ -1508,7 +1521,34 @@ fn parse_with_table(tokens: List, source_indices: Map, source_indices: Map, intern_table: InternTable) -> ParseWithTableResult { + parse_with_table_at( + tokens: tokens, + source_indices: source_indices, + intern_table: intern_table, + occurrence_base: intern_table.authored_token_ordinals, + heads_only: true ) } @@ -2530,7 +2570,7 @@ fn parse_fn_after_kw(tokens: TokenStream, ctx: ParseContext, start_span: SourceS let tokens = ret.tokens let ctx = ret.ctx - let r = parse_block(tokens: skip_newlines(tokens: tokens), ctx: ctx) + let r = parse_item_block_body(tokens: skip_newlines(tokens: tokens), ctx: ctx) if has_err(err: r.err) { return ItemResult { item: named_dummy, tokens: r.tokens, ctx: r.ctx, err: r.err } } let body = r.expr @@ -2590,7 +2630,7 @@ fn parse_fn_body_from_prefix(prefix: ItemPrefixResult, start_span: SourceSpan) - if has_err(err: r.err) { return ItemResult { item: named_dummy, tokens: r.tokens, ctx: ctx, err: r.err } } parse_required_expression_after_separator(tokens: r.tokens, ctx: ctx) } else if tok_is_lbrace(tok: token_stream_first(stream: tokens)) { - parse_block(tokens: tokens, ctx: ctx) + parse_item_block_body(tokens: tokens, ctx: ctx) } else { let span = token_span(tok: token_stream_first(stream: tokens)) ExprResult { expr: parse_recovery_placeholder(), tokens: tokens, ctx: ctx, err: Present { value: parse_error(msg: "expected '=' or '{' after fn return type", span: span) } } @@ -2651,7 +2691,7 @@ fn parse_block_item_after_kw(tokens: TokenStream, ctx: ParseContext, start_span: let tokens = uses_r.tokens let ctx = uses_r.ctx - let r = parse_block(tokens: skip_newlines(tokens: tokens), ctx: ctx) + let r = parse_item_block_body(tokens: skip_newlines(tokens: tokens), ctx: ctx) if has_err(err: r.err) { return ItemResult { item: named_dummy, tokens: r.tokens, ctx: r.ctx, err: r.err } } let body = r.expr @@ -2672,7 +2712,7 @@ fn parse_block_body_from_prefix(prefix: ItemPrefixResult, start_span: SourceSpan let ctx = prefix.ctx let named_dummy = Node { name: name, span: start_span, ident_span: Present { value: name_span }, children: [], params: [], inferred: none, return_cardinality: Required, uses: [], body: none, connective: NoConnective, transport: none, properties: [], type_annotation: none, is_self_recursive: false, has_non_tail_self_call: false, match_pattern: none, expr_data: NoExprData } - let r = parse_block(tokens: skip_newlines(tokens: prefix.tokens), ctx: ctx) + let r = parse_item_block_body(tokens: skip_newlines(tokens: prefix.tokens), ctx: ctx) if has_err(err: r.err) { return ItemResult { item: named_dummy, tokens: r.tokens, ctx: r.ctx, err: r.err } } let body = r.expr @@ -3889,6 +3929,77 @@ fn parse_param(tokens: TokenStream, ctx: ParseContext) -> ParamResult { } } +type HeadsBlockSkipResult { tokens: TokenStream, err: ErrorNode? } + +// Token-grain skip of one brace-delimited block, depth-counted. It reads only brace +// SHAPES, which the tokenizer has already decided -- a brace inside a string literal +// or an annotation is not a brace token, so the depth count is over the same +// delimiters the block grammar itself consumes, not over raw text. `depth` enters at +// 1 with the opening brace already eaten; reaching 0 leaves the stream exactly where +// `parse_block` would have left it after its closing `}`. +// +// The failure arm REFUSES -- an unterminated or EOF-truncated body is a typed, located +// parse error, never a silent stop-at-EOF. It is deliberately NOT the general body +// grammar refusal; that scope boundary is stated at `parse_heads_with_table`. +fn heads_skip_block_tokens(tokens: TokenStream, depth: Int) -> HeadsBlockSkipResult { + if depth == 0 { return HeadsBlockSkipResult { tokens: tokens, err: none } } + let tok = token_stream_first(stream: tokens) + if tok_is_eof(tok: tok) { + HeadsBlockSkipResult { + tokens: tokens, + err: Present { value: parse_error(msg: "heads-only parse: unterminated function body", span: token_span(tok: tok)) } + } + } else { + match tok { + Absent => + HeadsBlockSkipResult { + tokens: tokens, + err: Present { value: parse_error(msg: "heads-only parse: unterminated function body", span: no_span()) } + } + Present { value: t } => + if is_lbrace_shape(shape: t.shape) { + heads_skip_block_tokens(tokens: token_stream_advance(stream: tokens, n: 1), depth: depth + 1) + } else if is_rbrace_shape(shape: t.shape) { + heads_skip_block_tokens(tokens: token_stream_advance(stream: tokens, n: 1), depth: depth - 1) + } else { + heads_skip_block_tokens(tokens: token_stream_advance(stream: tokens, n: 1), depth: depth) + } + } + } +} + +data census_heads_body_stand_in_message: String = "pool census heads-only: function body stripped -- refuse to interpret" + +fn census_heads_body_stand_in() -> Node { + make_expr_error_node( + kind: CensusHeadsBodyStripped, + message: census_heads_body_stand_in_message, + span: no_span() + ) +} + +// The heads reading of `parse_block`: same opening `{` expectation, same stream +// position on exit, loud stand-in in place of the built block. +fn parse_block_heads_only(tokens: TokenStream, ctx: ParseContext) -> ExprResult { + let dummy_expr = parse_recovery_placeholder() + let r = expect(tokens: tokens, expected: ExpectLBrace) + if has_err(err: r.err) { return ExprResult { expr: dummy_expr, tokens: r.tokens, ctx: ctx, err: r.err } } + let skipped = heads_skip_block_tokens(tokens: r.tokens, depth: 1) + if has_err(err: skipped.err) { return ExprResult { expr: dummy_expr, tokens: skipped.tokens, ctx: ctx, err: skipped.err } } + ExprResult { expr: census_heads_body_stand_in(), tokens: skipped.tokens, ctx: ctx, err: none } +} + +// One spelling for "read this item's block body", selecting the reading the context +// declares. Every fn-shaped item body (`fn`, `func`, `pattern`, `interface`) routes +// through here, so the two readings cannot drift apart per call site. +fn parse_item_block_body(tokens: TokenStream, ctx: ParseContext) -> ExprResult { + if ctx.heads_only { + parse_block_heads_only(tokens: tokens, ctx: ctx) + } else { + parse_block(tokens: tokens, ctx: ctx) + } +} + fn parse_block(tokens: TokenStream, ctx: ParseContext) -> ExprResult { let dummy_expr = parse_recovery_placeholder() let block_span = token_span(tok: token_stream_first(stream: tokens)) diff --git a/src/v1/stage0/src/bin/claim_executor.rs b/src/v1/stage0/src/bin/claim_executor.rs index 296fc8f9d93..372149588cf 100644 --- a/src/v1/stage0/src/bin/claim_executor.rs +++ b/src/v1/stage0/src/bin/claim_executor.rs @@ -9920,6 +9920,7 @@ fn run() -> Result { let mut corpus_type_judgment_mode = false; let mut required_regen_mode = false; let mut required_regen_fixed_point_mode = false; + let mut heads_reading_differential_mode = false; let mut behavioral_receipt_plan_mode = false; let mut behavioral_receipt_selftest_mode = false; let mut behavioral_receipt_census_mode = false; @@ -9964,6 +9965,9 @@ fn run() -> Result { "--required-regen" => { required_regen_mode = true; } + "--heads-reading-differential" => { + heads_reading_differential_mode = true; + } "--required-regen-fixed-point" => { required_regen_fixed_point_mode = true; } @@ -10575,6 +10579,52 @@ fn run() -> Result { }; } + // The heads reading's own instrument. It is not enrolled in the required run and this + // clause does not pretend otherwise: reading 3875 modules TWICE is precisely the cost + // the heads reading exists to remove, so paying it on every push would spend more than + // the repair saves. It is a re-runnable receipt — the differential this change was + // landed on can be re-taken by anyone, on any tree, rather than being a number quoted + // from a run nobody can repeat. + if heads_reading_differential_mode { + let roots = if source_roots.is_empty() { + vec!["dag".to_string(), "src/v2".to_string()] + } else { + source_roots.clone() + }; + let d = v1_compiler::cli_run::heads_reading_differential(&roots); + eprintln!( + "heads-reading-differential: compared={} divergent={} narrowed={} regressed={} both_refused={}", + d.modules_compared, + d.divergent.len(), + d.narrowed.len(), + d.regressed.len(), + d.both_refused.len(), + ); + // The two readings' summed parse wall, from the same process over the same + // modules. This is the PARSE term only — not the whole `pool_parse` row, which + // also pays tokenize, newline indexing and per-file setup that neither reading + // changes. + eprintln!( + "heads-reading-differential: full_reading_parse_ms={} heads_reading_parse_ms={}", + d.full_reading_nanos / 1_000_000, + d.heads_reading_nanos / 1_000_000, + ); + for path in d.divergent.iter().take(20) { + eprintln!("heads-reading-differential: DIVERGENT {path}"); + } + for path in d.regressed.iter().take(20) { + eprintln!("heads-reading-differential: REGRESSED {path}"); + } + for path in d.narrowed.iter().take(20) { + eprintln!("heads-reading-differential: narrowed (declared scope) {path}"); + } + return if d.holds() { + Ok(ExitCode::SUCCESS) + } else { + Err(ExitCode::from(1)) + }; + } + if required_regen_mode { return match v1_compiler::cli_run::run_required_regen( ®en_candidate_dir, diff --git a/src/v1/stage0/src/cli_run.rs b/src/v1/stage0/src/cli_run.rs index 5aaf5bbcd5d..56adb10f1c4 100644 --- a/src/v1/stage0/src/cli_run.rs +++ b/src/v1/stage0/src/cli_run.rs @@ -15089,7 +15089,21 @@ fn parse_module_heads_for_pool_census( m.insert(source.path.clone(), nl_index.clone()); m }); - let parsed = v1_compiler_parse::parse_with_table(tokens, single_si, current_table); + // The HEADS reading of the grammar, not the full one. Every declaration head is + // parsed by the same productions; a brace-delimited fn body is skipped at token + // grain instead of being built, because `census_heads_module_node` two lines below + // replaces every body with the shared stand-in anyway. Building 3875 modules' worth + // of function bodies for a consumer that discards them was the largest single term + // in `pool_parse` (7.15s of 14.24s, `docs/probes/edge_index_tree_census_attribution_2026-08-24.md`) + // — a cost-shape defect DESIGN §6's bare-minimum-cost rule says is always fixed. + // + // The strip below is KEPT rather than folded into the parser, and that is load-bearing + // rather than leftover: it is what makes the two readings agree BY CONSTRUCTION. The + // exact shape of the stand-in the parser substitutes is then not a fact any consumer + // can depend on, because the normalizer overwrites it — so the heads reading cannot + // drift from the full reading through the body slot, only through the heads, which is + // the surface the differential receipt measures. + let parsed = v1_compiler_parse::parse_heads_with_table(tokens, single_si, current_table); *index.intern_table.borrow_mut() = parsed.intern_table.clone(); // Pool census needs declaration heads only — do NOT install full-body ASTs into // `parse_cache` here. Closure resolve retains full bodies on its own cache miss. @@ -15216,6 +15230,128 @@ fn pool_parse(index: &MultiEntryIndex) -> Result, String> { Ok(parsed) } +/// One module read BOTH ways and normalized by the census, so the two readings can be +/// compared as an identity rather than described as similar. +/// +/// `census_heads_module_node` is applied to both sides. That is what makes the comparison +/// meaningful rather than trivially false: the body slot is the one slot the heads reading +/// deliberately fills differently, the normalizer overwrites it on both sides, and what +/// remains is exactly the declaration heads the pool census consumes. A skip that swallowed +/// a declaration, mis-counted a brace depth, or left the token stream one token off changes +/// the head list and diverges here. +/// +/// Both readings start from the SAME intern-table snapshot and neither writes back, so the +/// sides are symmetric — a difference is the reading, never the order they ran in. +fn census_heads_both_readings( + index: &MultiEntryIndex, + source: &Rc, +) -> ( + (Result, String>, u128), + (Result, String>, u128), +) { + let table = index.intern_table.borrow().clone(); + let read = |heads_only: bool| -> (Result, String>, u128) { + let tokens = v1_compiler_tokenize::tokenize(source.content.clone(), source.path.clone()); + let nl_index = build_newline_index(source.path.clone(), source.content.clone()); + let single_si: Rc>> = Rc::new({ + let mut m = HashMap::new(); + m.insert(source.path.clone(), nl_index); + m + }); + // Only the parse is inside the timer: tokenize, newline index and setup are + // identical work in both readings and sit outside it on purpose. + let started = std::time::Instant::now(); + let parsed = if heads_only { + v1_compiler_parse::parse_heads_with_table(tokens, single_si, table.clone()) + } else { + v1_compiler_parse::parse_with_table(tokens, single_si, table.clone()) + }; + let nanos = started.elapsed().as_nanos(); + if let Some(err) = &parsed.result.error { + return (Err(diagnostic_to_message(err.diagnostic.clone())), nanos); + } + match &parsed.result.module { + Some(module) => (Ok(census_heads_module_node(module.clone())), nanos), + None => (Err("no module in parse result".to_string()), nanos), + } + }; + (read(false), read(true)) +} + +/// The corpus-scale differential receipt for the heads reading. +/// +/// `divergent` is the population that must be empty for the heads reading to be a reading +/// of the same grammar rather than a second, weaker parser. `narrowed` is NOT a defect and +/// is reported separately on purpose: it is the declared, bounded scope narrowing — a body +/// the full reading refuses on grammar and the heads reading never hands to the expression +/// grammar at all. DESIGN §5 requires a degradation to be COUNTED rather than absorbed, so +/// it gets its own row instead of being folded into either the pass or the failure count. +/// `regressed` is its mirror and must also be empty: the heads reading may never refuse +/// something the full reading accepts. +pub struct HeadsReadingDifferential { + pub modules_compared: usize, + pub divergent: Vec, + pub narrowed: Vec, + pub regressed: Vec, + pub both_refused: Vec, + /// Wall spent in the FULL reading, summed over every module, and the same for the + /// heads reading. Both are taken in ONE process, on ONE machine, over the SAME module + /// list, alternating per module — so the ratio compares two READINGS, not two builds, + /// two hosts, or two corpus states. A before/after figure from two separately-built + /// binaries would have to argue all three of those away; this one has nothing to argue + /// away. + /// + /// It measures the PARSE only. `tokenize`, `build_newline_index` and the per-file + /// setup sit outside both timers and are untouched by this repair, so this figure is + /// not the whole-`pool_parse` saving and must never be quoted as one. + pub full_reading_nanos: u128, + pub heads_reading_nanos: u128, +} + +impl HeadsReadingDifferential { + pub fn holds(&self) -> bool { + self.divergent.is_empty() && self.regressed.is_empty() + } +} + +/// Read every indexed module both ways and classify. Deterministic (sorted paths). +pub fn heads_reading_differential(source_roots: &[String]) -> HeadsReadingDifferential { + let index = build_multi_entry_index(source_roots); + let mut paths: Vec = index.source_files.keys().cloned().collect(); + paths.sort(); + let mut out = HeadsReadingDifferential { + modules_compared: 0, + divergent: Vec::new(), + narrowed: Vec::new(), + regressed: Vec::new(), + both_refused: Vec::new(), + full_reading_nanos: 0, + heads_reading_nanos: 0, + }; + for path in paths { + let source = match index.source_files.get(&path) { + Some(s) => s.clone(), + None => continue, + }; + out.modules_compared += 1; + let ((full_read, full_nanos), (heads_read, heads_nanos)) = + census_heads_both_readings(&index, &source); + out.full_reading_nanos += full_nanos; + out.heads_reading_nanos += heads_nanos; + match (full_read, heads_read) { + (Ok(full), Ok(heads)) => { + if full != heads { + out.divergent.push(path); + } + } + (Err(_), Ok(_)) => out.narrowed.push(path), + (Ok(_), Err(_)) => out.regressed.push(path), + (Err(_), Err(_)) => out.both_refused.push(path), + } + } + out +} + fn pool_qualified_fill(index: &MultiEntryIndex) -> Result, String> { if let Some(cached) = index.pool_qualified_fill.borrow().clone() { return Ok(cached); diff --git a/src/v1/stage0/src/v1_compiler_parse.rs b/src/v1/stage0/src/v1_compiler_parse.rs index 038656d10a3..7a903c94451 100644 --- a/src/v1/stage0/src/v1_compiler_parse.rs +++ b/src/v1/stage0/src/v1_compiler_parse.rs @@ -144,6 +144,7 @@ pub struct ParseContext { pub occurrence_index: Option>, pub declaration_occurrences: Option>>>, pub reference_occurrences: Option>>>, + pub heads_only: bool, } #[derive(Debug, Clone, PartialEq, serde::Serialize, serde::Deserialize)] @@ -255,6 +256,7 @@ pub fn parse_context_with_occurrence_state( occurrence_index: occurrence_index.clone(), declaration_occurrences: declaration_occurrences.clone(), reference_occurrences: reference_occurrences.clone(), + heads_only: ctx.heads_only.clone(), }) } @@ -269,6 +271,7 @@ pub fn parse_context_with_intern_table( occurrence_index: ctx.occurrence_index.clone(), declaration_occurrences: ctx.declaration_occurrences.clone(), reference_occurrences: ctx.reference_occurrences.clone(), + heads_only: ctx.heads_only.clone(), }) } @@ -2788,6 +2791,7 @@ pub fn parse_with_table_at( source_indices: Rc>>, intern_table: Rc, occurrence_base: Rc, + heads_only: bool, ) -> Rc { { let occurrence_allocator = occurrence_id_allocator_advance_to( @@ -2804,6 +2808,7 @@ pub fn parse_with_table_at( })), declaration_occurrences: Some(Rc::new(vec![])), reference_occurrences: Some(Rc::new(vec![])), + heads_only: heads_only.clone(), }); let r = parse_module(token_stream_new(tokens.clone()), ctx.clone()); if has_err(r.err.clone()) { @@ -2864,6 +2869,7 @@ pub fn parse_with_table_in_occurrence_scope( source_indices.clone(), intern_table.clone(), authored_token_ordinal_space_from_allocator(occurrence_allocator.clone()), + false, ) } @@ -2877,6 +2883,21 @@ pub fn parse_with_table( source_indices.clone(), intern_table.clone(), intern_table.authored_token_ordinals.clone(), + false, + ) +} + +pub fn parse_heads_with_table( + tokens: Rc>>, + source_indices: Rc>>, + intern_table: Rc, +) -> Rc { + parse_with_table_at( + tokens.clone(), + source_indices.clone(), + intern_table.clone(), + intern_table.authored_token_ordinals.clone(), + true, ) } @@ -6201,7 +6222,7 @@ pub fn parse_fn_after_kw( let inferred = ret.inferred.clone(); let tokens = ret.tokens.clone(); let ctx = ret.ctx.clone(); - let r = parse_block(skip_newlines(tokens.clone()), ctx.clone()); + let r = parse_item_block_body(skip_newlines(tokens.clone()), ctx.clone()); if has_err(r.err.clone()) { return Rc::new(ItemResult { item: named_dummy.clone(), @@ -6360,7 +6381,7 @@ pub fn parse_fn_body_from_prefix( } } else { if tok_is_lbrace(token_stream_first(tokens.clone())) { - parse_block(tokens.clone(), ctx.clone()) + parse_item_block_body(tokens.clone(), ctx.clone()) } else { { let span = token_span(token_stream_first(tokens.clone())); @@ -6607,7 +6628,7 @@ pub fn parse_block_item_after_kw( let uses = uses_r.uses.clone(); let tokens = uses_r.tokens.clone(); let ctx = uses_r.ctx.clone(); - let r = parse_block(skip_newlines(tokens.clone()), ctx.clone()); + let r = parse_item_block_body(skip_newlines(tokens.clone()), ctx.clone()); if has_err(r.err.clone()) { return Rc::new(ItemResult { item: named_dummy.clone(), @@ -6677,7 +6698,7 @@ pub fn parse_block_body_from_prefix( expr_data: Rc::new(ExprData::NoExprData), ident: None, }); - let r = parse_block(skip_newlines(prefix.tokens.clone()), ctx.clone()); + let r = parse_item_block_body(skip_newlines(prefix.tokens.clone()), ctx.clone()); if has_err(r.err.clone()) { return Rc::new(ItemResult { item: named_dummy.clone(), @@ -10864,6 +10885,130 @@ pub fn parse_param(tokens: Rc, ctx: Rc) -> Rc, + pub err: Option>, +} + +pub fn heads_skip_block_tokens( + mut tokens: Rc, + mut depth: i64, +) -> Rc { + loop { + if (depth.clone() == 0) { + return Rc::new(HeadsBlockSkipResult { + tokens: tokens.clone(), + err: None, + }); + } + let tok = token_stream_first(tokens.clone()); + if tok_is_eof(tok.clone()) { + break Rc::new(HeadsBlockSkipResult { + tokens: tokens.clone(), + err: Some(parse_error( + "heads-only parse: unterminated function body".to_string(), + token_span(tok.clone()), + )), + }); + } else { + match tok.clone() { + None => { + break Rc::new(HeadsBlockSkipResult { + tokens: tokens.clone(), + err: Some(parse_error( + "heads-only parse: unterminated function body".to_string(), + no_span(), + )), + }); + } + Some(t) => { + if is_lbrace_shape(t.shape.clone()) { + { + let __tco_0 = token_stream_advance(tokens, 1); + let __tco_1 = (depth + 1); + tokens = __tco_0; + depth = __tco_1; + continue; + } + } else { + if is_rbrace_shape(t.shape.clone()) { + { + let __tco_0 = token_stream_advance(tokens, 1); + let __tco_1 = (depth - 1); + tokens = __tco_0; + depth = __tco_1; + continue; + } + } else { + { + let __tco_0 = token_stream_advance(tokens, 1); + tokens = __tco_0; + continue; + } + } + } + } + } + } + } +} + +pub fn census_heads_body_stand_in_message() -> String { + thread_local! { + static CACHED: String = { + "pool census heads-only: function body stripped -- refuse to interpret".to_string() + }; + } + CACHED.with(|c: &String| c.clone()) +} + +pub fn census_heads_body_stand_in() -> Rc { + make_expr_error_node( + ExprErrorKind::CensusHeadsBodyStripped, + census_heads_body_stand_in_message(), + no_span(), + ) +} + +pub fn parse_block_heads_only(tokens: Rc, ctx: Rc) -> Rc { + { + let dummy_expr = parse_recovery_placeholder(); + let r = expect(tokens.clone(), Rc::new(ExpectedToken::ExpectLBrace)); + if has_err(r.err.clone()) { + return Rc::new(ExprResult { + expr: dummy_expr.clone(), + tokens: r.tokens.clone(), + ctx: ctx.clone(), + err: r.err.clone(), + }); + } + let skipped = heads_skip_block_tokens(r.tokens.clone(), 1); + if has_err(skipped.err.clone()) { + return Rc::new(ExprResult { + expr: dummy_expr.clone(), + tokens: skipped.tokens.clone(), + ctx: ctx.clone(), + err: skipped.err.clone(), + }); + } + Rc::new(ExprResult { + expr: census_heads_body_stand_in(), + tokens: skipped.tokens.clone(), + ctx: ctx.clone(), + err: None, + }) + } +} + +pub fn parse_item_block_body(tokens: Rc, ctx: Rc) -> Rc { + if ctx.heads_only.clone() { + parse_block_heads_only(tokens.clone(), ctx.clone()) + } else { + parse_block(tokens.clone(), ctx.clone()) + } +} + pub fn parse_block(tokens: Rc, ctx: Rc) -> Rc { { let dummy_expr = parse_recovery_placeholder(); From 8872e4876909380efc95027a0305f814f7c130bc Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Mon, 24 Aug 2026 21:25:00 +0000 Subject: [PATCH 2/2] Merge origin/main: re-take both controls on the merged tree main changed src/v1/02_parse.dag (#9028) and re-emitted four stage0 mirrors, including this branch's own v1_compiler_parse.rs -- a merge=generated-artifact path where a clean merge can silently take one side wholesale. Controls taken before the merge say nothing about the tree that ships, so both were re-taken: required-regen: first_generation_equal=true planned=134 executed=134, exit 0 heads-reading-differential: compared=3886 divergent=0 narrowed=0 regressed=0 full 10009 ms / heads 4995 ms = 2.00x The .dag merge was checked line-wise too: all 7 lines this branch deletes relative to main are its own rewires (parse_block -> parse_item_block_body, the occurrence_base signature). None of #9028 was dropped. Denominator moved 3880 -> 3886 (main added modules); the ratio did not. Unblocked by #9133, which retired the four stale frozen_path_deferrals rows. That refusal was never this branch's: both rosters are static declarations and this diff touches neither. Co-Authored-By: Claude Opus 5 --- ...heads_reading_of_the_grammar_2026-08-24.md | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/docs/probes/heads_reading_of_the_grammar_2026-08-24.md b/docs/probes/heads_reading_of_the_grammar_2026-08-24.md index f574412d918..ce59566a8e8 100644 --- a/docs/probes/heads_reading_of_the_grammar_2026-08-24.md +++ b/docs/probes/heads_reading_of_the_grammar_2026-08-24.md @@ -49,11 +49,20 @@ Local arm64 session container, release build, roots `dag` + `src/v2`, one proces Three green runs, and the spread between them is the point rather than noise to average away: -| run | full reading | heads reading | ratio | -|---|---|---|---| -| 1 | 12175 ms | 5885 ms | 2.07x | -| 2 | 11030 ms | 5559 ms | 1.98x | -| 3 | 12882 ms | 6382 ms | 2.02x | +| run | modules | full reading | heads reading | ratio | +|---|---|---|---|---| +| 1 | 3880 | 12175 ms | 5885 ms | 2.07x | +| 2 | 3880 | 11030 ms | 5559 ms | 1.98x | +| 3 | 3880 | 6382 ms* | — | 2.02x | +| **4 — merged tree, the one that ships** | **3886** | **10009 ms** | **4995 ms** | **2.00x** | + +\* run 3's full/heads pair is 12882/6382 ms; the column is narrowed here for width. + +**Run 4 is the load-bearing one and runs 1-3 are its history.** Runs 1-3 were taken before +`origin/main` was merged in; main then changed `src/v1/02_parse.dag` (#9028) and re-emitted four +stage0 mirrors, including this change's own. A control taken before a merge says nothing about +the tree that ships, so the differential and the regen fixed point were both re-taken on the +merged tree. The denominator moved 3880 -> 3886 because main added modules; the ratio did not. This is a shared container under other sessions' load, so the ABSOLUTE walls move ~15% run to run. The RATIO does not, because both readings in a run pay the same contention over the same @@ -111,7 +120,12 @@ required-regen: elapsed_ms=583524 first_generation_equal=true planned=134 execut exit=0 ``` -`first_generation_equal=true` over all 134 planned outputs. The one declared divergence +`first_generation_equal=true` over all 134 planned outputs. **Re-taken on the merged tree** after +`origin/main` was merged in (`elapsed_ms=555959 first_generation_equal=true planned=134 +executed=134`, exit 0), which is what establishes that the generated-artifact merge did not +silently take one side: `v1_compiler_parse.rs` carries `merge=generated-artifact`, main +re-emitted it, and so did this branch. A clean merge is not coherence on that path — the fixed +point is. The one declared divergence (`main.rs`) is pre-existing and unrelated. So the compiler built from this mirror re-emits this mirror: the heads reading is at the self-host fixed point, not merely compiling.