diff --git a/docs/probes/heads_reading_of_the_grammar_2026-08-24.md b/docs/probes/heads_reading_of_the_grammar_2026-08-24.md new file mode 100644 index 00000000000..ce59566a8e8 --- /dev/null +++ b/docs/probes/heads_reading_of_the_grammar_2026-08-24.md @@ -0,0 +1,186 @@ +# The pool census reads declaration heads instead of discarding bodies (2026-08-24) + +**Subject:** the residue `docs/probes/edge_index_tree_census_attribution_2026-08-24.md` left open — + +> `parse_with_table` builds full function bodies; `census_heads_module_node` strips them 82ms +> later. Half the corpus parse is work the only consumer throws away. […] It is this class's +> next-rung trigger, it is owed under §6, and it wants its own PR — not its own excuse. + +This is that PR. The attribution probe measured the defect and deliberately shipped no repair; +this change is the repair and carries its own receipt. + +## What changed, stated as a reading rather than a mode + +`v1.compiler.parse` gains a **declaration-heads reading of the same grammar**. `ParseContext` +carries `heads_only`; `parse_heads_with_table` is the entry that sets it. Every item head is +parsed by the same productions as before. The one difference is that a brace-delimited body of +an `fn`-shaped item is **skipped at token grain** — depth-counted over brace SHAPES, which the +tokenizer has already decided — instead of being built into a tree, and the body slot receives +the loud `CensusHeadsBodyStripped` stand-in. + +Two construction choices are what make this a reading rather than a second parser, and both are +load-bearing: + +- **One spelling for the body.** All four `fn`-shaped item body sites (`fn`, `func`, `pattern`, + `interface`) route through a single `parse_item_block_body`, which selects the reading the + context declares. The two readings cannot drift apart per call site because no call site + chooses. +- **The census strip is KEPT, not folded into the parser.** `census_heads_module_node` still + runs, on the heads reading's output, exactly as it ran on the full reading's. That normalizes + the body slot — the one slot the heads reading deliberately fills differently — on *both* + sides. The consequence is that the stand-in's exact shape is not a fact any consumer can + depend on, and the heads reading cannot drift from the full reading through the body slot at + all. It can only drift through the *heads*, which is precisely the surface the differential + below measures. Deleting the strip as "now redundant" would have removed the mechanism that + makes the two readings agree by construction and replaced it with a promise. + +## Measured + +Local arm64 session container, release build, roots `dag` + `src/v2`, one process, +`claim_executor --heads-reading-differential`. + +| | | +|---|---| +| modules compared | **3880** | +| `divergent` (heads ≠ full, after the census strip on both) | **0** | +| `regressed` (heads refuses what full accepts) | **0** | +| `narrowed` (full refuses on body grammar, heads does not) | **0** | +| `both_refused` | 0 | + +Three green runs, and the spread between them is the point rather than noise to average away: + +| run | modules | full reading | heads reading | ratio | +|---|---|---|---|---| +| 1 | 3880 | 12175 ms | 5885 ms | 2.07x | +| 2 | 3880 | 11030 ms | 5559 ms | 1.98x | +| 3 | 3880 | 6382 ms* | — | 2.02x | +| **4 — merged tree, the one that ships** | **3886** | **10009 ms** | **4995 ms** | **2.00x** | + +\* run 3's full/heads pair is 12882/6382 ms; the column is narrowed here for width. + +**Run 4 is the load-bearing one and runs 1-3 are its history.** Runs 1-3 were taken before +`origin/main` was merged in; main then changed `src/v1/02_parse.dag` (#9028) and re-emitted four +stage0 mirrors, including this change's own. A control taken before a merge says nothing about +the tree that ships, so the differential and the regen fixed point were both re-taken on the +merged tree. The denominator moved 3880 -> 3886 because main added modules; the ratio did not. + +This is a shared container under other sessions' load, so the ABSOLUTE walls move ~15% run to +run. The RATIO does not, because both readings in a run pay the same contention over the same +modules — which is exactly what taking them in one process buys. **Quote the ratio.** The +milliseconds are evidence that it was measured, not a figure to plan against, and a single-run +absolute would have been the more confident and less true number. + +**Why the two timings are comparable and a before/after pair would not be.** They are taken in +ONE process, on ONE machine, over the SAME module list, alternating per module. A before/after +figure from two separately-built binaries would have to argue away the build, the host and the +corpus state; this one has nothing to argue away. + +**What the timing is NOT.** It is the PARSE term only. `tokenize` (4.16s in the attribution +probe), `build_newline_index` (0.93s) and the per-file intern-table setup sit outside both +timers and are untouched by this change, so the ~2x is a saving on the parse row, not on the +`pool_parse` row, and quoting it as the latter would be inflation. The attribution probe's +figures were taken on a remote amd64 runner and are not the same denominator as these; the two +sets are not subtracted from each other anywhere in this document. + +## The RED control — the instrument can fail, measured + +A green differential is worth nothing unless it can go red. One token was mutated in the +emitted mirror only (`depth + 1` → `depth + 0` in `heads_skip_block_tokens`, so a nested `{` +stops deepening the count and the skip ends at the first inner `}`), rebuilt, and re-run: + +``` +heads-reading-differential: compared=3880 divergent=0 narrowed=0 regressed=2820 both_refused=0 +heads-reading-differential: REGRESSED dag.test.claim.witness_deferral_freeze_witness +heads-reading-differential: REGRESSED examples.cost_estimate +… +``` + +**2820 of 3880 modules go red on a one-character mutation**, and the command exits 1. The +mutation was then reverted by re-copying the regen candidate, rebuilt, and the differential +re-taken green (`compared=3880 divergent=0 narrowed=0 regressed=0`, exit 0) — the mutated +mirror is not in the diff, and the restored tree is the one measured, not merely assumed +restored. The mutated run's own timings are not quoted anywhere: it bails out of most bodies +early, so its numbers measure the bug rather than the reading. + +The mutation landing in `regressed` rather than `divergent` is itself informative and is the +reason both columns exist. An early-ending skip leaves the token stream mid-body, so the next +item fails to parse and the heads reading *refuses* — it does not quietly return a shorter head +list. That is the fail-closed behaviour one wants, but it is not something to assume: the two +columns are separate so a future mutation that DOES produce a silently shorter head list has +somewhere to land. + +## The self-host fixed point holds + +The parser is `.dag` authority with an emitted stage0 mirror, so a parser change that emits a +mirror the changed compiler would not re-emit is not a change to the parser — it is a fork. +`claim_executor --required-regen` was run after the mirror was installed: + +``` +required-regen: elapsed_ms=583524 first_generation_equal=true planned=134 executed=134 declared_divergent=1 [main.rs] +exit=0 +``` + +`first_generation_equal=true` over all 134 planned outputs. **Re-taken on the merged tree** after +`origin/main` was merged in (`elapsed_ms=555959 first_generation_equal=true planned=134 +executed=134`, exit 0), which is what establishes that the generated-artifact merge did not +silently take one side: `v1_compiler_parse.rs` carries `merge=generated-artifact`, main +re-emitted it, and so did this branch. A clean merge is not coherence on that path — the fixed +point is. The one declared divergence +(`main.rs`) is pre-existing and unrelated. So the compiler built from this mirror re-emits this +mirror: the heads reading is at the self-host fixed point, not merely compiling. + +## The one refusal this reading does not make — declared, and countable + +The heads reading refuses an unterminated body and every malformed item HEAD, exactly as the +full reading does. It does **not** refuse a well-braced but ungrammatical body, because those +tokens are never handed to the expression grammar. + +**Why this is a scope correction rather than a hole.** The required run's `.dag` parse sweep +full-parses `src/v1`, `dag` and `src/v2` and owns exactly the question *is the corpus +grammatical*. The pool census answering it a second time — as a side effect of building +declaration heads — was one fact with two authorities (§3), and it was the expensive authority: +it coupled every pool-derived resolve in the process to the grammaticality of every body in the +corpus, and paid a whole-corpus body parse to do it. + +**What is honestly given up.** For source roots the parse sweep does not cover, that refusal is +no longer taken anywhere. Today the sweep's roots are a superset of the pool roots the required +run uses, so the live population is empty; an invocation naming some other root is not covered +by either authority. The measurement above is what makes this *observable* rather than asserted: +`narrowed` is its own column, counted per run, currently `0` — DESIGN §5 requires a degradation +to be counted rather than absorbed, and a column reading zero today is how a future nonzero +becomes visible instead of silent. + +## Rung + +The class is *a parse whose only consumer discards half its output*. It sat at **mitigatable** — +nothing was wrong, it was merely paid — and this change does not move it up a safety rung, +because it is a COST repair and saying otherwise would be the inflation §4b(1) names. + +What the change does carry a rung for is the new surface it introduces: *the heads reading +agrees with the full reading*. That sits at **mechanically preventable** — the differential is a +real, re-runnable wall with a discriminating RED and a positive control, and it is +`claim_executor --heads-reading-differential`, not a number quoted from a run nobody can repeat. +It is **not** *structurally guaranteed*, and the honest reason is that the agreement is enforced +by a check rather than by construction on the heads: the census strip makes the BODY slot +unable to disagree, but nothing structurally prevents a future edit to the skip from consuming +one token too many. + +**It is not enrolled in the required run, and this is a deliberate scope statement, not an +oversight.** Reading 3880 modules twice is precisely the cost the heads reading exists to +remove; paying it on every push would spend more than the repair saves. Its **next-rung +trigger** is a form of the check whose cost is proportional to the diff rather than the corpus — +the same shape the affected-set work wants generally — at which point enrolling it becomes +cheap enough to be ordinary. + +## What is NOT claimed + +- **No claim about `pool_parse`'s whole row.** This measures the parse term, in isolation, on a + different machine and architecture from the attribution probe. The end-to-end effect on a + required-floor run is not measured here and no figure for it appears in this document. +- **No claim that the corpus contains a body the full reading refuses.** `narrowed=0` says the + opposite: every module in `dag` + `src/v2` parses clean both ways today. The declared + narrowing is about what WOULD happen, and it is carried as a counted column rather than as a + prediction. +- **`src/v1` was not in the differential's roots.** The roots measured are `dag` + `src/v2`, + which are the roots the required run gives the pool. A run over other roots is a different + denominator and would need its own receipt. diff --git a/src/v1/02_parse.dag b/src/v1/02_parse.dag index eb1dba9dfc7..eac8920e2cd 100644 --- a/src/v1/02_parse.dag +++ b/src/v1/02_parse.dag @@ -116,6 +116,13 @@ fn token_stream_peek(stream: TokenStream, offset: Int) -> Token? { get(xs: stream.all, index: stream.pos + offset) } +// `heads_only` selects the declaration-heads reading of the SAME grammar: every +// module-scope declaration head is parsed exactly as it is under the full reading, +// and a brace-delimited `fn` body is skipped at token grain instead of being built. +// It is not a second parser and it is not a lenient one -- the heads reading is +// defined by what its sole consumer (the pool census) retains, and the census +// normalizes every body away 82ms later, so the two readings are required to agree +// after that normalization. That agreement is the oracle, not a description. type ParseContext { source_indices: Map intern_table: InternTable @@ -123,6 +130,7 @@ type ParseContext { occurrence_index: OccurrenceIndex? declaration_occurrences: List? reference_occurrences: List? + heads_only: Bool } type ParseResult { @@ -208,6 +216,7 @@ fn parse_context_with_occurrence_state( occurrence_index: occurrence_index, declaration_occurrences: declaration_occurrences, reference_occurrences: reference_occurrences, + heads_only: ctx.heads_only, } } @@ -222,6 +231,7 @@ fn parse_context_with_intern_table( occurrence_index: ctx.occurrence_index, declaration_occurrences: ctx.declaration_occurrences, reference_occurrences: ctx.reference_occurrences, + heads_only: ctx.heads_only, } } @@ -1427,7 +1437,8 @@ fn parse_with_table_at( tokens: List, source_indices: Map, intern_table: InternTable, - occurrence_base: AuthoredTokenOrdinalSpace + occurrence_base: AuthoredTokenOrdinalSpace, + heads_only: Bool ) -> ParseWithTableResult { let occurrence_allocator = occurrence_id_allocator_advance_to( alloc: occurrence_base.allocator, @@ -1445,6 +1456,7 @@ fn parse_with_table_at( }, declaration_occurrences: Present { value: [] }, reference_occurrences: Present { value: [] }, + heads_only: heads_only, } let r = parse_module(tokens: token_stream_new(all: tokens), ctx: ctx) if has_err(err: r.err) { @@ -1499,7 +1511,8 @@ fn parse_with_table_in_occurrence_scope( intern_table: intern_table, occurrence_base: authored_token_ordinal_space_from_allocator( alloc: occurrence_allocator - ) + ), + heads_only: false ) } @@ -1508,7 +1521,34 @@ fn parse_with_table(tokens: List, source_indices: Map, source_indices: Map, intern_table: InternTable) -> ParseWithTableResult { + parse_with_table_at( + tokens: tokens, + source_indices: source_indices, + intern_table: intern_table, + occurrence_base: intern_table.authored_token_ordinals, + heads_only: true ) } @@ -2530,7 +2570,7 @@ fn parse_fn_after_kw(tokens: TokenStream, ctx: ParseContext, start_span: SourceS let tokens = ret.tokens let ctx = ret.ctx - let r = parse_block(tokens: skip_newlines(tokens: tokens), ctx: ctx) + let r = parse_item_block_body(tokens: skip_newlines(tokens: tokens), ctx: ctx) if has_err(err: r.err) { return ItemResult { item: named_dummy, tokens: r.tokens, ctx: r.ctx, err: r.err } } let body = r.expr @@ -2590,7 +2630,7 @@ fn parse_fn_body_from_prefix(prefix: ItemPrefixResult, start_span: SourceSpan) - if has_err(err: r.err) { return ItemResult { item: named_dummy, tokens: r.tokens, ctx: ctx, err: r.err } } parse_required_expression_after_separator(tokens: r.tokens, ctx: ctx) } else if tok_is_lbrace(tok: token_stream_first(stream: tokens)) { - parse_block(tokens: tokens, ctx: ctx) + parse_item_block_body(tokens: tokens, ctx: ctx) } else { let span = token_span(tok: token_stream_first(stream: tokens)) ExprResult { expr: parse_recovery_placeholder(), tokens: tokens, ctx: ctx, err: Present { value: parse_error(msg: "expected '=' or '{' after fn return type", span: span) } } @@ -2651,7 +2691,7 @@ fn parse_block_item_after_kw(tokens: TokenStream, ctx: ParseContext, start_span: let tokens = uses_r.tokens let ctx = uses_r.ctx - let r = parse_block(tokens: skip_newlines(tokens: tokens), ctx: ctx) + let r = parse_item_block_body(tokens: skip_newlines(tokens: tokens), ctx: ctx) if has_err(err: r.err) { return ItemResult { item: named_dummy, tokens: r.tokens, ctx: r.ctx, err: r.err } } let body = r.expr @@ -2672,7 +2712,7 @@ fn parse_block_body_from_prefix(prefix: ItemPrefixResult, start_span: SourceSpan let ctx = prefix.ctx let named_dummy = Node { name: name, span: start_span, ident_span: Present { value: name_span }, children: [], params: [], inferred: none, return_cardinality: Required, uses: [], body: none, connective: NoConnective, transport: none, properties: [], type_annotation: none, is_self_recursive: false, has_non_tail_self_call: false, match_pattern: none, expr_data: NoExprData } - let r = parse_block(tokens: skip_newlines(tokens: prefix.tokens), ctx: ctx) + let r = parse_item_block_body(tokens: skip_newlines(tokens: prefix.tokens), ctx: ctx) if has_err(err: r.err) { return ItemResult { item: named_dummy, tokens: r.tokens, ctx: r.ctx, err: r.err } } let body = r.expr @@ -3887,6 +3927,77 @@ fn parse_param(tokens: TokenStream, ctx: ParseContext) -> ParamResult { } } +type HeadsBlockSkipResult { tokens: TokenStream, err: ErrorNode? } + +// Token-grain skip of one brace-delimited block, depth-counted. It reads only brace +// SHAPES, which the tokenizer has already decided -- a brace inside a string literal +// or an annotation is not a brace token, so the depth count is over the same +// delimiters the block grammar itself consumes, not over raw text. `depth` enters at +// 1 with the opening brace already eaten; reaching 0 leaves the stream exactly where +// `parse_block` would have left it after its closing `}`. +// +// The failure arm REFUSES -- an unterminated or EOF-truncated body is a typed, located +// parse error, never a silent stop-at-EOF. It is deliberately NOT the general body +// grammar refusal; that scope boundary is stated at `parse_heads_with_table`. +fn heads_skip_block_tokens(tokens: TokenStream, depth: Int) -> HeadsBlockSkipResult { + if depth == 0 { return HeadsBlockSkipResult { tokens: tokens, err: none } } + let tok = token_stream_first(stream: tokens) + if tok_is_eof(tok: tok) { + HeadsBlockSkipResult { + tokens: tokens, + err: Present { value: parse_error(msg: "heads-only parse: unterminated function body", span: token_span(tok: tok)) } + } + } else { + match tok { + Absent => + HeadsBlockSkipResult { + tokens: tokens, + err: Present { value: parse_error(msg: "heads-only parse: unterminated function body", span: no_span()) } + } + Present { value: t } => + if is_lbrace_shape(shape: t.shape) { + heads_skip_block_tokens(tokens: token_stream_advance(stream: tokens, n: 1), depth: depth + 1) + } else if is_rbrace_shape(shape: t.shape) { + heads_skip_block_tokens(tokens: token_stream_advance(stream: tokens, n: 1), depth: depth - 1) + } else { + heads_skip_block_tokens(tokens: token_stream_advance(stream: tokens, n: 1), depth: depth) + } + } + } +} + +data census_heads_body_stand_in_message: String = "pool census heads-only: function body stripped -- refuse to interpret" + +fn census_heads_body_stand_in() -> Node { + make_expr_error_node( + kind: CensusHeadsBodyStripped, + message: census_heads_body_stand_in_message, + span: no_span() + ) +} + +// The heads reading of `parse_block`: same opening `{` expectation, same stream +// position on exit, loud stand-in in place of the built block. +fn parse_block_heads_only(tokens: TokenStream, ctx: ParseContext) -> ExprResult { + let dummy_expr = parse_recovery_placeholder() + let r = expect(tokens: tokens, expected: ExpectLBrace) + if has_err(err: r.err) { return ExprResult { expr: dummy_expr, tokens: r.tokens, ctx: ctx, err: r.err } } + let skipped = heads_skip_block_tokens(tokens: r.tokens, depth: 1) + if has_err(err: skipped.err) { return ExprResult { expr: dummy_expr, tokens: skipped.tokens, ctx: ctx, err: skipped.err } } + ExprResult { expr: census_heads_body_stand_in(), tokens: skipped.tokens, ctx: ctx, err: none } +} + +// One spelling for "read this item's block body", selecting the reading the context +// declares. Every fn-shaped item body (`fn`, `func`, `pattern`, `interface`) routes +// through here, so the two readings cannot drift apart per call site. +fn parse_item_block_body(tokens: TokenStream, ctx: ParseContext) -> ExprResult { + if ctx.heads_only { + parse_block_heads_only(tokens: tokens, ctx: ctx) + } else { + parse_block(tokens: tokens, ctx: ctx) + } +} + fn parse_block(tokens: TokenStream, ctx: ParseContext) -> ExprResult { let dummy_expr = parse_recovery_placeholder() let block_span = token_span(tok: token_stream_first(stream: tokens)) diff --git a/src/v1/stage0/src/bin/claim_executor.rs b/src/v1/stage0/src/bin/claim_executor.rs index 559068c1c7b..31fc5e3044b 100644 --- a/src/v1/stage0/src/bin/claim_executor.rs +++ b/src/v1/stage0/src/bin/claim_executor.rs @@ -9952,6 +9952,7 @@ fn run() -> Result { let mut corpus_type_judgment_mode = false; let mut required_regen_mode = false; let mut required_regen_fixed_point_mode = false; + let mut heads_reading_differential_mode = false; let mut behavioral_receipt_plan_mode = false; let mut behavioral_receipt_selftest_mode = false; let mut behavioral_receipt_census_mode = false; @@ -9996,6 +9997,9 @@ fn run() -> Result { "--required-regen" => { required_regen_mode = true; } + "--heads-reading-differential" => { + heads_reading_differential_mode = true; + } "--required-regen-fixed-point" => { required_regen_fixed_point_mode = true; } @@ -10607,6 +10611,52 @@ fn run() -> Result { }; } + // The heads reading's own instrument. It is not enrolled in the required run and this + // clause does not pretend otherwise: reading 3875 modules TWICE is precisely the cost + // the heads reading exists to remove, so paying it on every push would spend more than + // the repair saves. It is a re-runnable receipt — the differential this change was + // landed on can be re-taken by anyone, on any tree, rather than being a number quoted + // from a run nobody can repeat. + if heads_reading_differential_mode { + let roots = if source_roots.is_empty() { + vec!["dag".to_string(), "src/v2".to_string()] + } else { + source_roots.clone() + }; + let d = v1_compiler::cli_run::heads_reading_differential(&roots); + eprintln!( + "heads-reading-differential: compared={} divergent={} narrowed={} regressed={} both_refused={}", + d.modules_compared, + d.divergent.len(), + d.narrowed.len(), + d.regressed.len(), + d.both_refused.len(), + ); + // The two readings' summed parse wall, from the same process over the same + // modules. This is the PARSE term only — not the whole `pool_parse` row, which + // also pays tokenize, newline indexing and per-file setup that neither reading + // changes. + eprintln!( + "heads-reading-differential: full_reading_parse_ms={} heads_reading_parse_ms={}", + d.full_reading_nanos / 1_000_000, + d.heads_reading_nanos / 1_000_000, + ); + for path in d.divergent.iter().take(20) { + eprintln!("heads-reading-differential: DIVERGENT {path}"); + } + for path in d.regressed.iter().take(20) { + eprintln!("heads-reading-differential: REGRESSED {path}"); + } + for path in d.narrowed.iter().take(20) { + eprintln!("heads-reading-differential: narrowed (declared scope) {path}"); + } + return if d.holds() { + Ok(ExitCode::SUCCESS) + } else { + Err(ExitCode::from(1)) + }; + } + if required_regen_mode { return match v1_compiler::cli_run::run_required_regen( ®en_candidate_dir, diff --git a/src/v1/stage0/src/cli_run.rs b/src/v1/stage0/src/cli_run.rs index 5b6fb7bc9c5..347aaa84b9a 100644 --- a/src/v1/stage0/src/cli_run.rs +++ b/src/v1/stage0/src/cli_run.rs @@ -15503,7 +15503,21 @@ fn parse_module_heads_for_pool_census( m.insert(source.path.clone(), nl_index.clone()); m }); - let parsed = v1_compiler_parse::parse_with_table(tokens, single_si, current_table); + // The HEADS reading of the grammar, not the full one. Every declaration head is + // parsed by the same productions; a brace-delimited fn body is skipped at token + // grain instead of being built, because `census_heads_module_node` two lines below + // replaces every body with the shared stand-in anyway. Building 3875 modules' worth + // of function bodies for a consumer that discards them was the largest single term + // in `pool_parse` (7.15s of 14.24s, `docs/probes/edge_index_tree_census_attribution_2026-08-24.md`) + // — a cost-shape defect DESIGN §6's bare-minimum-cost rule says is always fixed. + // + // The strip below is KEPT rather than folded into the parser, and that is load-bearing + // rather than leftover: it is what makes the two readings agree BY CONSTRUCTION. The + // exact shape of the stand-in the parser substitutes is then not a fact any consumer + // can depend on, because the normalizer overwrites it — so the heads reading cannot + // drift from the full reading through the body slot, only through the heads, which is + // the surface the differential receipt measures. + let parsed = v1_compiler_parse::parse_heads_with_table(tokens, single_si, current_table); *index.intern_table.borrow_mut() = parsed.intern_table.clone(); // Pool census needs declaration heads only — do NOT install full-body ASTs into // `parse_cache` here. Closure resolve retains full bodies on its own cache miss. @@ -15630,6 +15644,128 @@ fn pool_parse(index: &MultiEntryIndex) -> Result, String> { Ok(parsed) } +/// One module read BOTH ways and normalized by the census, so the two readings can be +/// compared as an identity rather than described as similar. +/// +/// `census_heads_module_node` is applied to both sides. That is what makes the comparison +/// meaningful rather than trivially false: the body slot is the one slot the heads reading +/// deliberately fills differently, the normalizer overwrites it on both sides, and what +/// remains is exactly the declaration heads the pool census consumes. A skip that swallowed +/// a declaration, mis-counted a brace depth, or left the token stream one token off changes +/// the head list and diverges here. +/// +/// Both readings start from the SAME intern-table snapshot and neither writes back, so the +/// sides are symmetric — a difference is the reading, never the order they ran in. +fn census_heads_both_readings( + index: &MultiEntryIndex, + source: &Rc, +) -> ( + (Result, String>, u128), + (Result, String>, u128), +) { + let table = index.intern_table.borrow().clone(); + let read = |heads_only: bool| -> (Result, String>, u128) { + let tokens = v1_compiler_tokenize::tokenize(source.content.clone(), source.path.clone()); + let nl_index = build_newline_index(source.path.clone(), source.content.clone()); + let single_si: Rc>> = Rc::new({ + let mut m = HashMap::new(); + m.insert(source.path.clone(), nl_index); + m + }); + // Only the parse is inside the timer: tokenize, newline index and setup are + // identical work in both readings and sit outside it on purpose. + let started = std::time::Instant::now(); + let parsed = if heads_only { + v1_compiler_parse::parse_heads_with_table(tokens, single_si, table.clone()) + } else { + v1_compiler_parse::parse_with_table(tokens, single_si, table.clone()) + }; + let nanos = started.elapsed().as_nanos(); + if let Some(err) = &parsed.result.error { + return (Err(diagnostic_to_message(err.diagnostic.clone())), nanos); + } + match &parsed.result.module { + Some(module) => (Ok(census_heads_module_node(module.clone())), nanos), + None => (Err("no module in parse result".to_string()), nanos), + } + }; + (read(false), read(true)) +} + +/// The corpus-scale differential receipt for the heads reading. +/// +/// `divergent` is the population that must be empty for the heads reading to be a reading +/// of the same grammar rather than a second, weaker parser. `narrowed` is NOT a defect and +/// is reported separately on purpose: it is the declared, bounded scope narrowing — a body +/// the full reading refuses on grammar and the heads reading never hands to the expression +/// grammar at all. DESIGN §5 requires a degradation to be COUNTED rather than absorbed, so +/// it gets its own row instead of being folded into either the pass or the failure count. +/// `regressed` is its mirror and must also be empty: the heads reading may never refuse +/// something the full reading accepts. +pub struct HeadsReadingDifferential { + pub modules_compared: usize, + pub divergent: Vec, + pub narrowed: Vec, + pub regressed: Vec, + pub both_refused: Vec, + /// Wall spent in the FULL reading, summed over every module, and the same for the + /// heads reading. Both are taken in ONE process, on ONE machine, over the SAME module + /// list, alternating per module — so the ratio compares two READINGS, not two builds, + /// two hosts, or two corpus states. A before/after figure from two separately-built + /// binaries would have to argue all three of those away; this one has nothing to argue + /// away. + /// + /// It measures the PARSE only. `tokenize`, `build_newline_index` and the per-file + /// setup sit outside both timers and are untouched by this repair, so this figure is + /// not the whole-`pool_parse` saving and must never be quoted as one. + pub full_reading_nanos: u128, + pub heads_reading_nanos: u128, +} + +impl HeadsReadingDifferential { + pub fn holds(&self) -> bool { + self.divergent.is_empty() && self.regressed.is_empty() + } +} + +/// Read every indexed module both ways and classify. Deterministic (sorted paths). +pub fn heads_reading_differential(source_roots: &[String]) -> HeadsReadingDifferential { + let index = build_multi_entry_index(source_roots); + let mut paths: Vec = index.source_files.keys().cloned().collect(); + paths.sort(); + let mut out = HeadsReadingDifferential { + modules_compared: 0, + divergent: Vec::new(), + narrowed: Vec::new(), + regressed: Vec::new(), + both_refused: Vec::new(), + full_reading_nanos: 0, + heads_reading_nanos: 0, + }; + for path in paths { + let source = match index.source_files.get(&path) { + Some(s) => s.clone(), + None => continue, + }; + out.modules_compared += 1; + let ((full_read, full_nanos), (heads_read, heads_nanos)) = + census_heads_both_readings(&index, &source); + out.full_reading_nanos += full_nanos; + out.heads_reading_nanos += heads_nanos; + match (full_read, heads_read) { + (Ok(full), Ok(heads)) => { + if full != heads { + out.divergent.push(path); + } + } + (Err(_), Ok(_)) => out.narrowed.push(path), + (Ok(_), Err(_)) => out.regressed.push(path), + (Err(_), Err(_)) => out.both_refused.push(path), + } + } + out +} + fn pool_qualified_fill(index: &MultiEntryIndex) -> Result, String> { if let Some(cached) = index.pool_qualified_fill.borrow().clone() { return Ok(cached); diff --git a/src/v1/stage0/src/v1_compiler_parse.rs b/src/v1/stage0/src/v1_compiler_parse.rs index f3d7525aeee..3cd2e81d0c4 100644 --- a/src/v1/stage0/src/v1_compiler_parse.rs +++ b/src/v1/stage0/src/v1_compiler_parse.rs @@ -144,6 +144,7 @@ pub struct ParseContext { pub occurrence_index: Option>, pub declaration_occurrences: Option>>>, pub reference_occurrences: Option>>>, + pub heads_only: bool, } #[derive(Debug, Clone, PartialEq, serde::Serialize, serde::Deserialize)] @@ -255,6 +256,7 @@ pub fn parse_context_with_occurrence_state( occurrence_index: occurrence_index.clone(), declaration_occurrences: declaration_occurrences.clone(), reference_occurrences: reference_occurrences.clone(), + heads_only: ctx.heads_only.clone(), }) } @@ -269,6 +271,7 @@ pub fn parse_context_with_intern_table( occurrence_index: ctx.occurrence_index.clone(), declaration_occurrences: ctx.declaration_occurrences.clone(), reference_occurrences: ctx.reference_occurrences.clone(), + heads_only: ctx.heads_only.clone(), }) } @@ -2787,6 +2790,7 @@ pub fn parse_with_table_at( source_indices: Rc>>, intern_table: Rc, occurrence_base: Rc, + heads_only: bool, ) -> Rc { { let occurrence_allocator = occurrence_id_allocator_advance_to( @@ -2803,6 +2807,7 @@ pub fn parse_with_table_at( })), declaration_occurrences: Some(Rc::new(vec![])), reference_occurrences: Some(Rc::new(vec![])), + heads_only: heads_only.clone(), }); let r = parse_module(token_stream_new(tokens.clone()), ctx.clone()); if has_err(r.err.clone()) { @@ -2863,6 +2868,7 @@ pub fn parse_with_table_in_occurrence_scope( source_indices.clone(), intern_table.clone(), authored_token_ordinal_space_from_allocator(occurrence_allocator.clone()), + false, ) } @@ -2876,6 +2882,21 @@ pub fn parse_with_table( source_indices.clone(), intern_table.clone(), intern_table.authored_token_ordinals.clone(), + false, + ) +} + +pub fn parse_heads_with_table( + tokens: Rc>>, + source_indices: Rc>>, + intern_table: Rc, +) -> Rc { + parse_with_table_at( + tokens.clone(), + source_indices.clone(), + intern_table.clone(), + intern_table.authored_token_ordinals.clone(), + true, ) } @@ -6200,7 +6221,7 @@ pub fn parse_fn_after_kw( let inferred = ret.inferred.clone(); let tokens = ret.tokens.clone(); let ctx = ret.ctx.clone(); - let r = parse_block(skip_newlines(tokens.clone()), ctx.clone()); + let r = parse_item_block_body(skip_newlines(tokens.clone()), ctx.clone()); if has_err(r.err.clone()) { return Rc::new(ItemResult { item: named_dummy.clone(), @@ -6359,7 +6380,7 @@ pub fn parse_fn_body_from_prefix( } } else { if tok_is_lbrace(token_stream_first(tokens.clone())) { - parse_block(tokens.clone(), ctx.clone()) + parse_item_block_body(tokens.clone(), ctx.clone()) } else { { let span = token_span(token_stream_first(tokens.clone())); @@ -6606,7 +6627,7 @@ pub fn parse_block_item_after_kw( let uses = uses_r.uses.clone(); let tokens = uses_r.tokens.clone(); let ctx = uses_r.ctx.clone(); - let r = parse_block(skip_newlines(tokens.clone()), ctx.clone()); + let r = parse_item_block_body(skip_newlines(tokens.clone()), ctx.clone()); if has_err(r.err.clone()) { return Rc::new(ItemResult { item: named_dummy.clone(), @@ -6676,7 +6697,7 @@ pub fn parse_block_body_from_prefix( expr_data: Rc::new(ExprData::NoExprData), ident: None, }); - let r = parse_block(skip_newlines(prefix.tokens.clone()), ctx.clone()); + let r = parse_item_block_body(skip_newlines(prefix.tokens.clone()), ctx.clone()); if has_err(r.err.clone()) { return Rc::new(ItemResult { item: named_dummy.clone(), @@ -10845,6 +10866,130 @@ pub fn parse_param(tokens: Rc, ctx: Rc) -> Rc, + pub err: Option>, +} + +pub fn heads_skip_block_tokens( + mut tokens: Rc, + mut depth: i64, +) -> Rc { + loop { + if (depth.clone() == 0) { + return Rc::new(HeadsBlockSkipResult { + tokens: tokens.clone(), + err: None, + }); + } + let tok = token_stream_first(tokens.clone()); + if tok_is_eof(tok.clone()) { + break Rc::new(HeadsBlockSkipResult { + tokens: tokens.clone(), + err: Some(parse_error( + "heads-only parse: unterminated function body".to_string(), + token_span(tok.clone()), + )), + }); + } else { + match tok.clone() { + None => { + break Rc::new(HeadsBlockSkipResult { + tokens: tokens.clone(), + err: Some(parse_error( + "heads-only parse: unterminated function body".to_string(), + no_span(), + )), + }); + } + Some(t) => { + if is_lbrace_shape(t.shape.clone()) { + { + let __tco_0 = token_stream_advance(tokens, 1); + let __tco_1 = (depth + 1); + tokens = __tco_0; + depth = __tco_1; + continue; + } + } else { + if is_rbrace_shape(t.shape.clone()) { + { + let __tco_0 = token_stream_advance(tokens, 1); + let __tco_1 = (depth - 1); + tokens = __tco_0; + depth = __tco_1; + continue; + } + } else { + { + let __tco_0 = token_stream_advance(tokens, 1); + tokens = __tco_0; + continue; + } + } + } + } + } + } + } +} + +pub fn census_heads_body_stand_in_message() -> String { + thread_local! { + static CACHED: String = { + "pool census heads-only: function body stripped -- refuse to interpret".to_string() + }; + } + CACHED.with(|c: &String| c.clone()) +} + +pub fn census_heads_body_stand_in() -> Rc { + make_expr_error_node( + ExprErrorKind::CensusHeadsBodyStripped, + census_heads_body_stand_in_message(), + no_span(), + ) +} + +pub fn parse_block_heads_only(tokens: Rc, ctx: Rc) -> Rc { + { + let dummy_expr = parse_recovery_placeholder(); + let r = expect(tokens.clone(), Rc::new(ExpectedToken::ExpectLBrace)); + if has_err(r.err.clone()) { + return Rc::new(ExprResult { + expr: dummy_expr.clone(), + tokens: r.tokens.clone(), + ctx: ctx.clone(), + err: r.err.clone(), + }); + } + let skipped = heads_skip_block_tokens(r.tokens.clone(), 1); + if has_err(skipped.err.clone()) { + return Rc::new(ExprResult { + expr: dummy_expr.clone(), + tokens: skipped.tokens.clone(), + ctx: ctx.clone(), + err: skipped.err.clone(), + }); + } + Rc::new(ExprResult { + expr: census_heads_body_stand_in(), + tokens: skipped.tokens.clone(), + ctx: ctx.clone(), + err: None, + }) + } +} + +pub fn parse_item_block_body(tokens: Rc, ctx: Rc) -> Rc { + if ctx.heads_only.clone() { + parse_block_heads_only(tokens.clone(), ctx.clone()) + } else { + parse_block(tokens.clone(), ctx.clone()) + } +} + pub fn parse_block(tokens: Rc, ctx: Rc) -> Rc { { let dummy_expr = parse_recovery_placeholder();