Skip to content

Building the module-path index was full-parsing the whole corpus as a side effect: one fact, two authorities, 21.8s to 13.4s - #9667

Merged
briansrls merged 1 commit into
mainfrom
session/sleek-cat-449
Aug 29, 2026
Merged

briansrls merged 1 commit into
mainfrom
session/sleek-cat-449

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What this is

build_module_path_index_uncached walks every .dag file under the source roots and full-parses each one, to project two facts that are both declaration HEADS: the module's name and that name's span. On the seed closure (--entry src/v1/compile.dag, 3031 .dag files in tree) that pass costs 21.8s — and it runs before compile.frontend emits its first phase mark, so it is invisible in the phase table.

The bodies were never the subject. What the full parse bought over the heads reading is a whole-tree "every authored .dag body is grammatical" refusal — and that question already has an authority: required-CI phase 1, cli_run::run_dag_parse_sweep over DAG_PARSE_SWEEP_ROOTS (src/v1, dag, src/v2), which full-parses exactly these roots and exists to answer it.

Two producers answering one question is the §3 fork. This is the same cut already taken once for the pool census — the scope statement at v1.compiler.parse.parse_heads_with_table records it, in the same words: answering it there "coupled every pool-derived resolve in the process to the grammaticality of every body in the corpus."

Consolidation, not a cache: no memo, no shared state, no second path. One producer stops answering a question that was never its own.

Scope of the narrowing

Still refuses, unchanged: an unterminated body, and every malformed item HEAD.

Narrows in exactly one arm: a well-braced but ungrammatical fn body in a pool file no longer refuses here. That is not an unrefused state — the sweep above refuses it, and any file this invocation actually COMPILES is full-parsed again by the front end. A pool file outside the closure is never compiled by this invocation, so its body grammar was never a fact this walk needed.

Measurement

Same instrument vintage in both arms (nested trace_mark plus an accumulating timer compiled into the mirrors, localized by sampled backtrace; throwaway, not in this diff):

pass before after
build_module_path_index_uncached 21.84s 13.38s −39%
reference_resolution_facts 24.56s 24.88s unchanged (control)
extend_sources_to_both_closure_fixpoint 29.70s 29.26s unchanged (control)

The two untouched whole-tree passes moving by under a second is the control that says the delta is this walk and not the machine. The residual 13.4s is corpus acquisition — read, tokenize, newline index — not parsing.

Behaviour, by execution on the real 3031-file tree

  • Emitted 108-file crate byte-identical (diff -r).
  • All 8723 diagnostic lines identical, including the resolved 102 sources … 2928 indexed modules receipt.

Discriminating REDs — executed, not argued

  • A pool file with a malformed item head still refuses at the index, typed and located: module index refused: 1 unparseable .dag source(s) with path and span.
  • A pool file with a well-braced ungrammatical body does NOT refuse at the index (the declared narrowing), and v1_src_dag_parse refuses it — exit 1 with file and span, against a control run of exit 0 with the probe removed. The fact kept its authority; it lost its second one.

Checks

cargo fmt --all --check clean. cargo clippy --lib -- -D warnings clean. The --all-targets clippy reds are pre-existing on this merge base — verified by compiling each named failing target on the unmodified tree and getting identical errors.

Not in this PR

Three more whole-tree sweeps remain (reference_resolution_facts 24.9s, extend_sources_to_both_closure_fixpoint 29.3s, parse_census_fill_sources 21s). Parsing is now a small share of each; the bulk is three independent read+tokenize acquisitions of the same 3031 files. Consolidating those onto one acquisition is the larger remaining prize, but the three walks use different collectors (collect_dag_files vs collect_dag_files_tolerant, different orderings), different path keys, and different refusal policies — unifying them is a behaviour decision, not a mechanical cleanup, so it is deliberately left out.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135

… side effect: one fact, two authorities, 21.8s to 13.4s

`build_module_path_index_uncached` walks every `.dag` file under the source roots and
FULL-PARSES each one, to project two facts that are both declaration HEADS: the module's
name, and that name's span. On the seed closure (`--entry src/v1/compile.dag`, 3031 `.dag`
files in tree) that pass costs 21.8s of a 451s run, and it runs before `compile.frontend`
emits its first phase mark, so it does not appear in the phase table at all.

The bodies were never the subject. What the full parse bought over the heads reading is a
whole-tree "every authored `.dag` body is grammatical" refusal -- and that question already
has an authority: required-CI phase 1, `cli_run::run_dag_parse_sweep` over
`DAG_PARSE_SWEEP_ROOTS` (`src/v1`, `dag`, `src/v2`), which full-parses exactly these roots
and exists to answer it. Two producers answering one question is the §3 fork, and this is
the same cut already taken once for the pool census -- the scope statement at
`v1.compiler.parse.parse_heads_with_table` records it, in the same words: answering it here
"coupled every pool-derived resolve in the process to the grammaticality of every body in
the corpus".

So the index walk drops to the heads reading. This is a consolidation, not a cache: no
memo, no shared state, no second path -- one producer stops answering a question that was
never its own.

WHAT STILL REFUSES, unchanged: an unterminated body, and every malformed item HEAD. What
narrows is exactly one arm -- a well-braced but UNGRAMMATICAL `fn` BODY in a pool file no
longer refuses HERE. It is not an unrefused state: the sweep above refuses it, and any file
this invocation actually COMPILES is full-parsed again by the front end, which refuses there.
A pool file outside the closure is never compiled by this invocation, so its body grammar
was never a fact this walk needed.

Measured, same instrument vintage both arms (nested `trace_mark` plus an accumulating timer
compiled into the mirrors, localized by sampled backtrace; throwaway, not in this diff):

  build_module_path_index_uncached   21.84s -> 13.38s   (-39%)
  reference_resolution_facts         24.56s -> 24.88s   unchanged (control)
  extend_sources_to_both_closure     29.70s -> 29.26s   unchanged (control)

The two untouched whole-tree passes moving by less than a second is the control that says
the delta is this walk and not the machine. The residual 13.4s is corpus acquisition --
read, tokenize, newline index -- not parsing.

BEHAVIOUR, by execution on the real 3031-file tree: the emitted 108-file crate is
BYTE-IDENTICAL (`diff -r`), and the run's 8723 diagnostic lines including the
`resolved 102 sources ... 2928 indexed modules` receipt are identical.

Discriminating REDs, both arms executed rather than argued:
  - a pool file with a malformed item head still refuses at the index, typed and located
    ("module index refused: 1 unparseable .dag source(s)", with path and span);
  - a pool file with a well-braced ungrammatical body does NOT refuse at the index (the
    declared narrowing), and `v1_src_dag_parse` refuses it -- exit 1 with the file and span,
    against a control run of exit 0 with the probe removed. The fact kept its authority; it
    lost its second one.

`cargo fmt --all --check` clean; `cargo clippy --lib -- -D warnings` clean. The
`--all-targets` clippy reds are pre-existing on this merge base -- verified by compiling
each named target on the unmodified tree and getting the identical errors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135
@gunbai-bot

gunbai-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

CI red on this head is inherited from main, not from this diff

The aggregating job says so itself — attribution=unestablished, with the instruction to read the lane log before assuming a defect in the diff. I did.

required-witnesses-build: pass. required-witnesses-floor: fail, and the witnesses job (3s) is just the aggregator reflecting it.

The floor failure set on this head is byte-identical to the same lane on dda42363 — this PR's exact merge base, and the tip of main:

FAIL test.claim.ci_floor_on_success_materialization_receipt_hand_rust_witness.scaffold_names_bounded_deferral_receipt
FAIL test.claim.discovery_census_witness.w_counts_partition_the_offered_population
FAIL test.claim.discovery_census_witness.w_derived_aggregate_is_satisfiable_against_standings
FAIL test.claim.discovery_census_witness.w_required_aggregate_dependencies_are_exactly_the_planned_subset
FAIL test.claim.observation_emit_census_witness_test.w_every_rostered_marker_still_exists_in_the_seed
FAIL test.claim.observation_emit_census_witness_test.w_post_snapshot_live_tags_are_rostered_bidirectional
FAIL test.claim.observation_lockstep_witness_test.w_heartbeat_source_still_refuses_to_fabricate
verdict=FloorRefused unexpected_failures=7

Same seven, same verdict, same count. Main has been red on every witnesses run since c3f534401 (six consecutive runs, ~7h), so this predates the branch.

Names alone would not settle it, so the partition line as well — identical on every discovery and outcome field:

offered=13963 routed=13148 declined_long=541 declined_fixture=0
planned=13148 executed=13148 not_attempted=0 terminal=13148
failed=7 known_red_held=65 stale_quarantine=3 route_gap_held=203

Where the two runs differ, the change is an improvement

main dda42363 this head cf78da95
passed 12853 12856 +3
interrupted_before_verdict 16 13 −3
over_cost_line_diagnostic 709 616 −93

Three witnesses that were previously budget-refused — undecided, not failed — now reach a verdict and pass, and 93 fewer rows breach the 100ms cost line. That is the expected downstream effect of a cheaper module-path index: witnesses that resolve .dag get cheaper, so fewer hit the deadline before producing a verdict. failed is unchanged at 7.

So this diff introduces zero new failures and converts three undecided rows into decided passes.

No fix pushed

There is nothing here to fix. Pushing a speculative repair for main's breakage into a perf PR would be scope creep, and it would reset this head's approval for no gain. The seven failures are observation/discovery-census witnesses, unrelated to the module-path index — they want their own PR and, given they have stood for ~7h across six runs, probably their own owner.

Happy to take that on as a separate change if nobody else holds it — say the word.

— sent from sleek-cat-449

@briansrls
briansrls merged commit b305e4e into main Aug 29, 2026
1 of 3 checks passed
@briansrls
briansrls deleted the session/sleek-cat-449 branch August 29, 2026 08:11
@briansrls
briansrls restored the session/sleek-cat-449 branch August 29, 2026 09:40
gunbai-bot Bot pushed a commit that referenced this pull request Aug 29, 2026
…s consumer reads as heads: 21s to 13s, identical census

`parse_census_fill_sources` parses every `.dag` file OUTSIDE the compile closure -- ~2,900
of the tree's 3,031 -- so qualified references can resolve against modules the closure does
not compile. It full-parsed them. Its consumer never reads a body.

`build_symbol_index_qualified_fill` reads module items, their names, single-occurrence `Disj`
variants, and fn SIGNATURES. The one place a body appears is `local_binding_for_item`, which
tests `item.body != none` to tell a fn from a type and an alias -- and then builds its binding
with `body: none`, discarding it. Body PRESENCE is exactly what the heads reading preserves,
via the `CensusHeadsBodyStripped` stand-in; body CONTENT is what it drops, and no consumer
asks for it.

So the fill takes the heads reading. This is the same cut as #9667 (the module-path index),
which makes it the second instance of one shape: an INDEX built by full-parsing the corpus,
where the index's own consumer wants heads. The scope statement at `parse_heads_with_table`
governs both.

`parse_heads_in_occurrence_scope` is added beside `parse_with_table_in_occurrence_scope`,
completing at occurrence-scope grain the pair the file already carried at table grain
(`parse_with_table` / `parse_heads_with_table`). It is the existing `parse_with_table_at` with
`heads_only: true` -- no new reading, the missing combination.

MEASURED, same `trace_mark` and same placement both arms:

  census fill parse   21s -> 13s
  compile.reconcile   71s -> 60s   (clean runs, no instrument)

EQUIVALENCE, established on the census itself rather than inferred from a downstream green.
Dumping the fill's `SymbolIndex` under both readings over the real 2,928-module fill:

  modules  2928 = 2928
  entries  67558 = 67558
  keyhash  efe9a7ea7f3e6b54 = efe9a7ea7f3e6b54   (the census NAME SET is identical)

32,235 entries differ by value, and normalizing `OccurrenceId { value: N }` out of both dumps
makes the two files COMPARE EQUAL BYTE FOR BYTE. The whole difference is occurrence NUMBERING:
the allocator no longer advances over skipped body tokens, so ids are smaller. Names, spans,
ident_spans, cardinality, type annotations and shapes are unchanged. Those ids come from the
fill's own allocator (`occurrence_id_allocator_initial`), a private per-parse counter that was
never comparable with the closure's, and no consumer reads occurrence identity off a
symbol-index binding.

WHOLE-CORPUS BEHAVIOUR: the emitted tree differs from the pre-change baseline in exactly two
files -- `v1_compiler_compile.rs` and `v1_compiler_parse.rs`, the mirrors of the two sources
edited here -- in exactly the 3 and 1 hunks those edits imply. The other 106 emitted files are
byte-identical. All 2,180 diagnostics are identity-identical with positions normalized, and the
only positional shifts are inside the two edited files.

WHAT I COULD NOT ESTABLISH, stated rather than left to be assumed. I tried to make the
diagnostics evidence discriminating by starving the census -- passing `[]` as
`census_fill_modules` -- and the run produced the IDENTICAL 2,180 diagnostics and the identical
`resolved 102 sources ... 2928 indexed modules` receipt. So on this entry the census fill
contributes nothing to the outcome, and no downstream green here can distinguish a working fill
from no fill at all. That is why the equivalence above is measured on the index itself. It also
means the receipt's "2928 indexed modules" counts fill SOURCES, not indexed names.

`cargo fmt --all --check` clean; `cargo clippy --lib -- -D warnings` clean. Mirrors regenerated
via `claim_executor --required-regen`; regen named exactly these two files as drifted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135
gunbai-bot Bot pushed a commit that referenced this pull request Aug 29, 2026
… consumers, and the second one reads bodies

Reverts c98094e. The change was measured (census fill 21s -> 13s) and its census-equivalence
evidence was real, but its PREMISE was wrong, and the premise is what the change rested on.

WHAT I CLAIMED: "its consumer never reads a body" -- `build_symbol_index_qualified_fill` reads
module items, names, `Disj` variants and fn signatures, and `local_binding_for_item` discards the
body after testing that one exists. All of that is true, and it is still true.

WHAT I MISSED: it is A consumer, not THE consumer. `cli_run::run_dag_parse_sweep` also calls
`parse_census_fill_sources` -- deliberately, and its own comment says so in as many words: "ONE
FILE THROUGH THE FRONTEND'S OWN CENSUS-FILL PARSE, not `parse` alone", because source-annotation
GRAIN (DESIGN §4c) must not be decided differently by two frontends. The sweep builds the
declaration index, and the declaration index derives CITATIONS -- which are authored inside fn
BODIES. Heads-only deleted them.

MEASURED, my branch against `origin/main` content, same binary, same roots, one variable:

  citations     1877 -> 1711   (-166)
  in_fixtures    205 ->   87   (-118)
  CITATION-DEBT-ROW-STALE   0 -> 31

The 31 findings are the visible end of it: a citation that is no longer INDEXED cannot refuse,
so its debt row reads as spent and the required check demands its deletion. Deleting those rows
to green the check would have been the repair that consumes its own evidence -- the rows are
correct and the index had gone blind.

WHY MY VERIFICATION DID NOT CATCH IT. I checked the census itself -- dumping the fill's
`SymbolIndex` under both readings and getting an identical key set over 67,558 entries -- and the
compile route's emitted tree and diagnostics. Every one of those measurements was sound. They
were all measurements of ONE consumer. The question I did not ask is whether the failing path
would have gone through the thing I measured, and for the sweep it would not: the sweep runs in
the required-CI parse phase, not in `gunbc compile`, so no amount of compile-route evidence could
have gone red. A shared function needs its consumers ENUMERATED, not sampled.

The saving is still real and still available, but it belongs to a change that gives the sweep a
full reading and the census a heads reading, since those are two different questions about one
source -- names versus citations. That is a larger change than this one claimed to be, and it
needs the §4c grain argument answered rather than stepped around, so it is not being improvised
here. #9667 (the module-path index, already on main) is unaffected: it has no second consumer,
and `run_dag_parse_sweep` does not route through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135
briansrls pushed a commit that referenced this pull request Aug 29, 2026
…hority, 12,121 tokenizations to 6,061 (#9671)

* Building the module-path index was full-parsing the whole corpus as a side effect: one fact, two authorities, 21.8s to 13.4s

`build_module_path_index_uncached` walks every `.dag` file under the source roots and
FULL-PARSES each one, to project two facts that are both declaration HEADS: the module's
name, and that name's span. On the seed closure (`--entry src/v1/compile.dag`, 3031 `.dag`
files in tree) that pass costs 21.8s of a 451s run, and it runs before `compile.frontend`
emits its first phase mark, so it does not appear in the phase table at all.

The bodies were never the subject. What the full parse bought over the heads reading is a
whole-tree "every authored `.dag` body is grammatical" refusal -- and that question already
has an authority: required-CI phase 1, `cli_run::run_dag_parse_sweep` over
`DAG_PARSE_SWEEP_ROOTS` (`src/v1`, `dag`, `src/v2`), which full-parses exactly these roots
and exists to answer it. Two producers answering one question is the §3 fork, and this is
the same cut already taken once for the pool census -- the scope statement at
`v1.compiler.parse.parse_heads_with_table` records it, in the same words: answering it here
"coupled every pool-derived resolve in the process to the grammaticality of every body in
the corpus".

So the index walk drops to the heads reading. This is a consolidation, not a cache: no
memo, no shared state, no second path -- one producer stops answering a question that was
never its own.

WHAT STILL REFUSES, unchanged: an unterminated body, and every malformed item HEAD. What
narrows is exactly one arm -- a well-braced but UNGRAMMATICAL `fn` BODY in a pool file no
longer refuses HERE. It is not an unrefused state: the sweep above refuses it, and any file
this invocation actually COMPILES is full-parsed again by the front end, which refuses there.
A pool file outside the closure is never compiled by this invocation, so its body grammar
was never a fact this walk needed.

Measured, same instrument vintage both arms (nested `trace_mark` plus an accumulating timer
compiled into the mirrors, localized by sampled backtrace; throwaway, not in this diff):

  build_module_path_index_uncached   21.84s -> 13.38s   (-39%)
  reference_resolution_facts         24.56s -> 24.88s   unchanged (control)
  extend_sources_to_both_closure     29.70s -> 29.26s   unchanged (control)

The two untouched whole-tree passes moving by less than a second is the control that says
the delta is this walk and not the machine. The residual 13.4s is corpus acquisition --
read, tokenize, newline index -- not parsing.

BEHAVIOUR, by execution on the real 3031-file tree: the emitted 108-file crate is
BYTE-IDENTICAL (`diff -r`), and the run's 8723 diagnostic lines including the
`resolved 102 sources ... 2928 indexed modules` receipt are identical.

Discriminating REDs, both arms executed rather than argued:
  - a pool file with a malformed item head still refuses at the index, typed and located
    ("module index refused: 1 unparseable .dag source(s)", with path and span);
  - a pool file with a well-braced ungrammatical body does NOT refuse at the index (the
    declared narrowing), and `v1_src_dag_parse` refuses it -- exit 1 with the file and span,
    against a control run of exit 0 with the probe removed. The fact kept its authority; it
    lost its second one.

`cargo fmt --all --check` clean; `cargo clippy --lib -- -D warnings` clean. The
`--all-targets` clippy reds are pre-existing on this merge base -- verified by compiling
each named target on the unmodified tree and getting the identical errors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135

* The census-only fill full-parsed 2,900 files to build a name index its consumer reads as heads: 21s to 13s, identical census

`parse_census_fill_sources` parses every `.dag` file OUTSIDE the compile closure -- ~2,900
of the tree's 3,031 -- so qualified references can resolve against modules the closure does
not compile. It full-parsed them. Its consumer never reads a body.

`build_symbol_index_qualified_fill` reads module items, their names, single-occurrence `Disj`
variants, and fn SIGNATURES. The one place a body appears is `local_binding_for_item`, which
tests `item.body != none` to tell a fn from a type and an alias -- and then builds its binding
with `body: none`, discarding it. Body PRESENCE is exactly what the heads reading preserves,
via the `CensusHeadsBodyStripped` stand-in; body CONTENT is what it drops, and no consumer
asks for it.

So the fill takes the heads reading. This is the same cut as #9667 (the module-path index),
which makes it the second instance of one shape: an INDEX built by full-parsing the corpus,
where the index's own consumer wants heads. The scope statement at `parse_heads_with_table`
governs both.

`parse_heads_in_occurrence_scope` is added beside `parse_with_table_in_occurrence_scope`,
completing at occurrence-scope grain the pair the file already carried at table grain
(`parse_with_table` / `parse_heads_with_table`). It is the existing `parse_with_table_at` with
`heads_only: true` -- no new reading, the missing combination.

MEASURED, same `trace_mark` and same placement both arms:

  census fill parse   21s -> 13s
  compile.reconcile   71s -> 60s   (clean runs, no instrument)

EQUIVALENCE, established on the census itself rather than inferred from a downstream green.
Dumping the fill's `SymbolIndex` under both readings over the real 2,928-module fill:

  modules  2928 = 2928
  entries  67558 = 67558
  keyhash  efe9a7ea7f3e6b54 = efe9a7ea7f3e6b54   (the census NAME SET is identical)

32,235 entries differ by value, and normalizing `OccurrenceId { value: N }` out of both dumps
makes the two files COMPARE EQUAL BYTE FOR BYTE. The whole difference is occurrence NUMBERING:
the allocator no longer advances over skipped body tokens, so ids are smaller. Names, spans,
ident_spans, cardinality, type annotations and shapes are unchanged. Those ids come from the
fill's own allocator (`occurrence_id_allocator_initial`), a private per-parse counter that was
never comparable with the closure's, and no consumer reads occurrence identity off a
symbol-index binding.

WHOLE-CORPUS BEHAVIOUR: the emitted tree differs from the pre-change baseline in exactly two
files -- `v1_compiler_compile.rs` and `v1_compiler_parse.rs`, the mirrors of the two sources
edited here -- in exactly the 3 and 1 hunks those edits imply. The other 106 emitted files are
byte-identical. All 2,180 diagnostics are identity-identical with positions normalized, and the
only positional shifts are inside the two edited files.

WHAT I COULD NOT ESTABLISH, stated rather than left to be assumed. I tried to make the
diagnostics evidence discriminating by starving the census -- passing `[]` as
`census_fill_modules` -- and the run produced the IDENTICAL 2,180 diagnostics and the identical
`resolved 102 sources ... 2928 indexed modules` receipt. So on this entry the census fill
contributes nothing to the outcome, and no downstream green here can distinguish a working fill
from no fill at all. That is why the equivalence above is measured on the index itself. It also
means the receipt's "2928 indexed modules" counts fill SOURCES, not indexed names.

`cargo fmt --all --check` clean; `cargo clippy --lib -- -D warnings` clean. Mirrors regenerated
via `claim_executor --required-regen`; regen named exactly these two files as drifted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135

* The tree was lexed four times to compile it once: one acquisition authority, 12,121 tokenizations to 6,061

Four whole-tree walks stand between `gunbc compile` and a parsed closure, and each one
independently read, tokenized and newline-indexed the same files. Measured on the seed closure:
`tokenize_artifact` ran 12,121 times over a 3,031-file tree, and the multiplicity was exact --
`distinct_file_spellings=3031`, of which 3,030 were tokenized exactly 4x. 25.9s of lexing plus
7.4s of newline indexing, to lex a corpus that has one right answer per file.

Lexing a file is a PURE FUNCTION of its bytes and the spelling those bytes are reported under.
One right answer means one authority (DESIGN §2/§3), so `cli_run::pool_acquire` computes it and
the walks ask, instead of each recomputing.

WHAT THIS DELIBERATELY DOES NOT DO, because it is the part that is not a cleanup. It does not
merge the walks, unify their collectors (`collect_dag_files` sorts per directory,
`collect_dag_files_tolerant` sorts globally), unify their path keys, or touch their refusal
policies. Those genuinely differ, and unifying them would change which file wins a duplicate
module path -- a behaviour decision. Every walk keeps its own collector, order, key and policy
and merely stops re-lexing what another walk already lexed. That the walks already AGREE on the
spelling per file is measured, not assumed: 3,031 files, 3,031 distinct spellings. Had they
disagreed this change would have bought nothing, and I would have needed that decision first.

THE KEY IS THE INPUT, NOT A PATH. A memo keyed on a path asserts the file has not changed, which
is a claim about the world this process cannot make -- the `cache_impurity` failure mode. The key
here is (spelling, content length, content hash), and a hit compares the stored bytes before it
is served, so a hash collision returns a miss rather than another file's tokens. A file rewritten
mid-run hashes differently and is re-lexed. The spelling is in the key because `tokenize` bakes
it into every span, so two spellings of one file are two different answers.

MEASURED, same instrument and placement as the baselines:

  tokenize_artifact   25.86s x12121  ->  13.16s x6061   (exactly halved, as the 4x/2x predicts)
  PASS2 reference_resolution_facts   24.88s -> 15.54s
  PASS3 extend_sources_to_both_closure_fixpoint   29.26s -> 22.76s
  PASS1 build_module_path_index_uncached   13.41s -> 13.41s (first walk; it does the lexing the
                                                             other two now reuse, so it is the
                                                             control that should NOT move)

Prologue total 67.6s -> 51.7s. The remaining 6,061 tokenizations are the `.dag` frontend and
census-fill path, which cannot reach a hand-Rust authority; routing them is a separate change
against `v1.compiler.compile` and is not attempted here.

BEHAVIOUR: the emitted 108-file crate is BYTE-IDENTICAL to the pre-change baseline (`diff -r`),
and all 8,723 diagnostic lines are byte-identical -- not merely identity-identical: this change
edits no `.dag`, so nothing shifts position and the strict comparison is the right one. The whole
3,031-file corpus flows through the new authority, so unlike a change confined to one walk this
control exercises every acquisition the run performs.

`cargo fmt --all --check` clean; `cargo clippy --lib -- -D warnings` clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135

* Revert the census-fill heads cut: `parse_census_fill_sources` has TWO consumers, and the second one reads bodies

Reverts c98094e. The change was measured (census fill 21s -> 13s) and its census-equivalence
evidence was real, but its PREMISE was wrong, and the premise is what the change rested on.

WHAT I CLAIMED: "its consumer never reads a body" -- `build_symbol_index_qualified_fill` reads
module items, names, `Disj` variants and fn signatures, and `local_binding_for_item` discards the
body after testing that one exists. All of that is true, and it is still true.

WHAT I MISSED: it is A consumer, not THE consumer. `cli_run::run_dag_parse_sweep` also calls
`parse_census_fill_sources` -- deliberately, and its own comment says so in as many words: "ONE
FILE THROUGH THE FRONTEND'S OWN CENSUS-FILL PARSE, not `parse` alone", because source-annotation
GRAIN (DESIGN §4c) must not be decided differently by two frontends. The sweep builds the
declaration index, and the declaration index derives CITATIONS -- which are authored inside fn
BODIES. Heads-only deleted them.

MEASURED, my branch against `origin/main` content, same binary, same roots, one variable:

  citations     1877 -> 1711   (-166)
  in_fixtures    205 ->   87   (-118)
  CITATION-DEBT-ROW-STALE   0 -> 31

The 31 findings are the visible end of it: a citation that is no longer INDEXED cannot refuse,
so its debt row reads as spent and the required check demands its deletion. Deleting those rows
to green the check would have been the repair that consumes its own evidence -- the rows are
correct and the index had gone blind.

WHY MY VERIFICATION DID NOT CATCH IT. I checked the census itself -- dumping the fill's
`SymbolIndex` under both readings and getting an identical key set over 67,558 entries -- and the
compile route's emitted tree and diagnostics. Every one of those measurements was sound. They
were all measurements of ONE consumer. The question I did not ask is whether the failing path
would have gone through the thing I measured, and for the sweep it would not: the sweep runs in
the required-CI parse phase, not in `gunbc compile`, so no amount of compile-route evidence could
have gone red. A shared function needs its consumers ENUMERATED, not sampled.

The saving is still real and still available, but it belongs to a change that gives the sweep a
full reading and the census a heads reading, since those are two different questions about one
source -- names versus citations. That is a larger change than this one claimed to be, and it
needs the §4c grain argument answered rather than stepped around, so it is not being improvised
here. #9667 (the module-path index, already on main) is unaffected: it has no second consumer,
and `run_dag_parse_sweep` does not route through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135

---------

Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant