Skip to content

The tree was lexed four times to compile it once: one acquisition authority, 12,121 tokenizations to 6,061 - #9671

Merged
briansrls merged 5 commits into
mainfrom
session/sleek-cat-449
Aug 29, 2026
Merged

The tree was lexed four times to compile it once: one acquisition authority, 12,121 tokenizations to 6,061#9671
briansrls merged 5 commits into
mainfrom
session/sleek-cat-449

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Aug 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this is

Four whole-tree walks stand between gunbc compile and a parsed closure, and each independently read, tokenized and newline-indexed the same files. Measured on the seed closure: tokenize_artifact ran 12,121 times over a 3,031-file tree — 25.9s of lexing plus 7.4s of newline indexing, to lex a corpus that has one right answer per file.

Lexing a file is a pure function of its bytes and the spelling those bytes are reported under. One right answer means one authority (DESIGN §2/§3), so cli_run::pool_acquire computes it and the walks ask instead of each recomputing.

What it deliberately does not do

It does not merge the walks, unify their collectors (collect_dag_files sorts per directory, collect_dag_files_tolerant sorts globally), unify their path keys, or touch their refusal policies. Those genuinely differ, and unifying them would change which file wins a duplicate module path — a behaviour decision, not a cleanup.

Every walk keeps its own collector, order, key and policy and merely stops re-lexing what another already lexed. That the walks already agree on the spelling per file is measured, not assumed: distinct_file_spellings=3031 over 3,031 files, of which 3,030 were tokenized exactly 4×. Had they disagreed this would have bought nothing, and the decision above would have come first.

The key is the input, not a path

A memo keyed on a path asserts the file has not changed — a claim about the world this process cannot make, and the cache_impurity failure mode. The key here is (spelling, content length, content hash), and a hit compares the stored bytes before it is served, so a hash collision returns a miss rather than another file's tokens. A file rewritten mid-run hashes differently and is re-lexed. The spelling is in the key because tokenize bakes it into every span, so two spellings of one file are two different answers.

Measured — same instrument and placement both arms

before after
tokenize_artifact 25.86s ×12121 13.16s ×6061
reference_resolution_facts 24.88s 15.54s
extend_sources_to_both_closure_fixpoint 29.26s 22.76s
build_module_path_index_uncached 13.41s 13.41s

Prologue 67.6s → 51.7s. The last row is the control that should not move: it is the first walk, so it does the lexing the other two now reuse.

The remaining 6,061 tokenizations are the .dag frontend and census-fill path, which cannot reach a hand-Rust authority; routing them is a separate change against v1.compiler.compile.

Behaviour

The emitted 108-file crate is byte-identical to the pre-change baseline (diff -r), and all 8,723 diagnostic lines are byte-identical — not merely identity-identical: this change edits no .dag, so nothing shifts position and the strict comparison is the right one. The whole 3,031-file corpus flows through the new authority, so unlike a change confined to one walk, this control exercises every acquisition the run performs.

Required-lane declarations phase matches origin/main content field for field on the same binary: modules=4314 declared=83607 import_members=88930 citations=1877 debt=41 in_fixtures=205 outside_index=304, 0 stale rows.

cargo fmt --all --check clean; cargo clippy --lib -- -D warnings clean.

History on this branch

Two other commits appear in the log and net out to nothing:

  • Building the module-path index was full-parsing the whole corpus as a side effect: one fact, two authorities, 21.8s to 13.4s #9667 (module-path index heads cut) already landed on main as b305e4eb.
  • A census-fill heads cut was pushed and then reverted by me. It was measured (21s→13s) and its census-equivalence evidence was real, but its premise was wrong: I verified build_symbol_index_qualified_fill reads only heads, and that is a consumer, not the consumer. run_dag_parse_sweep also calls parse_census_fill_sources, deliberately, and builds the declaration index from it — whose citations live inside fn bodies. Heads-only deleted 166 of them (in_fixtures 205→87) and turned 31 correct citation-debt rows stale. Reverted; the revert restores main's counts exactly. That saving is still available, but it belongs to a change giving the sweep a full reading and the census a heads reading, which needs the §4c grain argument answered rather than stepped around.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135

gunbc-ci-auto-heal and others added 5 commits August 29, 2026 07:07
gunbc-ci-auto-heal @claude
Building the module-path index was full-parsing the whole corpus as a…
cf78da9 
… side effect: one fact, two authorities, 21.8s to 13.4s

`build_module_path_index_uncached` walks every `.dag` file under the source roots and
FULL-PARSES each one, to project two facts that are both declaration HEADS: the module's
name, and that name's span. On the seed closure (`--entry src/v1/compile.dag`, 3031 `.dag`
files in tree) that pass costs 21.8s of a 451s run, and it runs before `compile.frontend`
emits its first phase mark, so it does not appear in the phase table at all.

The bodies were never the subject. What the full parse bought over the heads reading is a
whole-tree "every authored `.dag` body is grammatical" refusal -- and that question already
has an authority: required-CI phase 1, `cli_run::run_dag_parse_sweep` over
`DAG_PARSE_SWEEP_ROOTS` (`src/v1`, `dag`, `src/v2`), which full-parses exactly these roots
and exists to answer it. Two producers answering one question is the §3 fork, and this is
the same cut already taken once for the pool census -- the scope statement at
`v1.compiler.parse.parse_heads_with_table` records it, in the same words: answering it here
"coupled every pool-derived resolve in the process to the grammaticality of every body in
the corpus".

So the index walk drops to the heads reading. This is a consolidation, not a cache: no
memo, no shared state, no second path -- one producer stops answering a question that was
never its own.

WHAT STILL REFUSES, unchanged: an unterminated body, and every malformed item HEAD. What
narrows is exactly one arm -- a well-braced but UNGRAMMATICAL `fn` BODY in a pool file no
longer refuses HERE. It is not an unrefused state: the sweep above refuses it, and any file
this invocation actually COMPILES is full-parsed again by the front end, which refuses there.
A pool file outside the closure is never compiled by this invocation, so its body grammar
was never a fact this walk needed.

Measured, same instrument vintage both arms (nested `trace_mark` plus an accumulating timer
compiled into the mirrors, localized by sampled backtrace; throwaway, not in this diff):

  build_module_path_index_uncached   21.84s -> 13.38s   (-39%)
  reference_resolution_facts         24.56s -> 24.88s   unchanged (control)
  extend_sources_to_both_closure     29.70s -> 29.26s   unchanged (control)

The two untouched whole-tree passes moving by less than a second is the control that says
the delta is this walk and not the machine. The residual 13.4s is corpus acquisition --
read, tokenize, newline index -- not parsing.

BEHAVIOUR, by execution on the real 3031-file tree: the emitted 108-file crate is
BYTE-IDENTICAL (`diff -r`), and the run's 8723 diagnostic lines including the
`resolved 102 sources ... 2928 indexed modules` receipt are identical.

Discriminating REDs, both arms executed rather than argued:
  - a pool file with a malformed item head still refuses at the index, typed and located
    ("module index refused: 1 unparseable .dag source(s)", with path and span);
  - a pool file with a well-braced ungrammatical body does NOT refuse at the index (the
    declared narrowing), and `v1_src_dag_parse` refuses it -- exit 1 with the file and span,
    against a control run of exit 0 with the probe removed. The fact kept its authority; it
    lost its second one.

`cargo fmt --all --check` clean; `cargo clippy --lib -- -D warnings` clean. The
`--all-targets` clippy reds are pre-existing on this merge base -- verified by compiling
each named target on the unmodified tree and getting the identical errors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135
gunbc-ci-auto-heal @claude
The census-only fill full-parsed 2,900 files to build a name index it…
c98094e 
…s consumer reads as heads: 21s to 13s, identical census

`parse_census_fill_sources` parses every `.dag` file OUTSIDE the compile closure -- ~2,900
of the tree's 3,031 -- so qualified references can resolve against modules the closure does
not compile. It full-parsed them. Its consumer never reads a body.

`build_symbol_index_qualified_fill` reads module items, their names, single-occurrence `Disj`
variants, and fn SIGNATURES. The one place a body appears is `local_binding_for_item`, which
tests `item.body != none` to tell a fn from a type and an alias -- and then builds its binding
with `body: none`, discarding it. Body PRESENCE is exactly what the heads reading preserves,
via the `CensusHeadsBodyStripped` stand-in; body CONTENT is what it drops, and no consumer
asks for it.

So the fill takes the heads reading. This is the same cut as #9667 (the module-path index),
which makes it the second instance of one shape: an INDEX built by full-parsing the corpus,
where the index's own consumer wants heads. The scope statement at `parse_heads_with_table`
governs both.

`parse_heads_in_occurrence_scope` is added beside `parse_with_table_in_occurrence_scope`,
completing at occurrence-scope grain the pair the file already carried at table grain
(`parse_with_table` / `parse_heads_with_table`). It is the existing `parse_with_table_at` with
`heads_only: true` -- no new reading, the missing combination.

MEASURED, same `trace_mark` and same placement both arms:

  census fill parse   21s -> 13s
  compile.reconcile   71s -> 60s   (clean runs, no instrument)

EQUIVALENCE, established on the census itself rather than inferred from a downstream green.
Dumping the fill's `SymbolIndex` under both readings over the real 2,928-module fill:

  modules  2928 = 2928
  entries  67558 = 67558
  keyhash  efe9a7ea7f3e6b54 = efe9a7ea7f3e6b54   (the census NAME SET is identical)

32,235 entries differ by value, and normalizing `OccurrenceId { value: N }` out of both dumps
makes the two files COMPARE EQUAL BYTE FOR BYTE. The whole difference is occurrence NUMBERING:
the allocator no longer advances over skipped body tokens, so ids are smaller. Names, spans,
ident_spans, cardinality, type annotations and shapes are unchanged. Those ids come from the
fill's own allocator (`occurrence_id_allocator_initial`), a private per-parse counter that was
never comparable with the closure's, and no consumer reads occurrence identity off a
symbol-index binding.

WHOLE-CORPUS BEHAVIOUR: the emitted tree differs from the pre-change baseline in exactly two
files -- `v1_compiler_compile.rs` and `v1_compiler_parse.rs`, the mirrors of the two sources
edited here -- in exactly the 3 and 1 hunks those edits imply. The other 106 emitted files are
byte-identical. All 2,180 diagnostics are identity-identical with positions normalized, and the
only positional shifts are inside the two edited files.

WHAT I COULD NOT ESTABLISH, stated rather than left to be assumed. I tried to make the
diagnostics evidence discriminating by starving the census -- passing `[]` as
`census_fill_modules` -- and the run produced the IDENTICAL 2,180 diagnostics and the identical
`resolved 102 sources ... 2928 indexed modules` receipt. So on this entry the census fill
contributes nothing to the outcome, and no downstream green here can distinguish a working fill
from no fill at all. That is why the equivalence above is measured on the index itself. It also
means the receipt's "2928 indexed modules" counts fill SOURCES, not indexed names.

`cargo fmt --all --check` clean; `cargo clippy --lib -- -D warnings` clean. Mirrors regenerated
via `claim_executor --required-regen`; regen named exactly these two files as drifted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135
gunbc-ci-auto-heal @claude
The tree was lexed four times to compile it once: one acquisition aut…
f033f6e 
…hority, 12,121 tokenizations to 6,061

Four whole-tree walks stand between `gunbc compile` and a parsed closure, and each one
independently read, tokenized and newline-indexed the same files. Measured on the seed closure:
`tokenize_artifact` ran 12,121 times over a 3,031-file tree, and the multiplicity was exact --
`distinct_file_spellings=3031`, of which 3,030 were tokenized exactly 4x. 25.9s of lexing plus
7.4s of newline indexing, to lex a corpus that has one right answer per file.

Lexing a file is a PURE FUNCTION of its bytes and the spelling those bytes are reported under.
One right answer means one authority (DESIGN §2/§3), so `cli_run::pool_acquire` computes it and
the walks ask, instead of each recomputing.

WHAT THIS DELIBERATELY DOES NOT DO, because it is the part that is not a cleanup. It does not
merge the walks, unify their collectors (`collect_dag_files` sorts per directory,
`collect_dag_files_tolerant` sorts globally), unify their path keys, or touch their refusal
policies. Those genuinely differ, and unifying them would change which file wins a duplicate
module path -- a behaviour decision. Every walk keeps its own collector, order, key and policy
and merely stops re-lexing what another walk already lexed. That the walks already AGREE on the
spelling per file is measured, not assumed: 3,031 files, 3,031 distinct spellings. Had they
disagreed this change would have bought nothing, and I would have needed that decision first.

THE KEY IS THE INPUT, NOT A PATH. A memo keyed on a path asserts the file has not changed, which
is a claim about the world this process cannot make -- the `cache_impurity` failure mode. The key
here is (spelling, content length, content hash), and a hit compares the stored bytes before it
is served, so a hash collision returns a miss rather than another file's tokens. A file rewritten
mid-run hashes differently and is re-lexed. The spelling is in the key because `tokenize` bakes
it into every span, so two spellings of one file are two different answers.

MEASURED, same instrument and placement as the baselines:

  tokenize_artifact   25.86s x12121  ->  13.16s x6061   (exactly halved, as the 4x/2x predicts)
  PASS2 reference_resolution_facts   24.88s -> 15.54s
  PASS3 extend_sources_to_both_closure_fixpoint   29.26s -> 22.76s
  PASS1 build_module_path_index_uncached   13.41s -> 13.41s (first walk; it does the lexing the
                                                             other two now reuse, so it is the
                                                             control that should NOT move)

Prologue total 67.6s -> 51.7s. The remaining 6,061 tokenizations are the `.dag` frontend and
census-fill path, which cannot reach a hand-Rust authority; routing them is a separate change
against `v1.compiler.compile` and is not attempted here.

BEHAVIOUR: the emitted 108-file crate is BYTE-IDENTICAL to the pre-change baseline (`diff -r`),
and all 8,723 diagnostic lines are byte-identical -- not merely identity-identical: this change
edits no `.dag`, so nothing shifts position and the strict comparison is the right one. The whole
3,031-file corpus flows through the new authority, so unlike a change confined to one walk this
control exercises every acquisition the run performs.

`cargo fmt --all --check` clean; `cargo clippy --lib -- -D warnings` clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135
gunbc-ci-auto-heal
Merge remote-tracking branch 'origin/main' into session/sleek-cat-449
ca3a4b5 
# Conflicts:
#	src/v1/stage0/src/module_path_index/index.rs
gunbc-ci-auto-heal @claude
Revert the census-fill heads cut: parse_census_fill_sources has TWO…
584bb99 
… consumers, and the second one reads bodies

Reverts c98094e. The change was measured (census fill 21s -> 13s) and its census-equivalence
evidence was real, but its PREMISE was wrong, and the premise is what the change rested on.

WHAT I CLAIMED: "its consumer never reads a body" -- `build_symbol_index_qualified_fill` reads
module items, names, `Disj` variants and fn signatures, and `local_binding_for_item` discards the
body after testing that one exists. All of that is true, and it is still true.

WHAT I MISSED: it is A consumer, not THE consumer. `cli_run::run_dag_parse_sweep` also calls
`parse_census_fill_sources` -- deliberately, and its own comment says so in as many words: "ONE
FILE THROUGH THE FRONTEND'S OWN CENSUS-FILL PARSE, not `parse` alone", because source-annotation
GRAIN (DESIGN §4c) must not be decided differently by two frontends. The sweep builds the
declaration index, and the declaration index derives CITATIONS -- which are authored inside fn
BODIES. Heads-only deleted them.

MEASURED, my branch against `origin/main` content, same binary, same roots, one variable:

  citations     1877 -> 1711   (-166)
  in_fixtures    205 ->   87   (-118)
  CITATION-DEBT-ROW-STALE   0 -> 31

The 31 findings are the visible end of it: a citation that is no longer INDEXED cannot refuse,
so its debt row reads as spent and the required check demands its deletion. Deleting those rows
to green the check would have been the repair that consumes its own evidence -- the rows are
correct and the index had gone blind.

WHY MY VERIFICATION DID NOT CATCH IT. I checked the census itself -- dumping the fill's
`SymbolIndex` under both readings and getting an identical key set over 67,558 entries -- and the
compile route's emitted tree and diagnostics. Every one of those measurements was sound. They
were all measurements of ONE consumer. The question I did not ask is whether the failing path
would have gone through the thing I measured, and for the sweep it would not: the sweep runs in
the required-CI parse phase, not in `gunbc compile`, so no amount of compile-route evidence could
have gone red. A shared function needs its consumers ENUMERATED, not sampled.

The saving is still real and still available, but it belongs to a change that gives the sweep a
full reading and the census a heads reading, since those are two different questions about one
source -- names versus citations. That is a larger change than this one claimed to be, and it
needs the §4c grain argument answered rather than stepped around, so it is not being improvised
here. #9667 (the module-path index, already on main) is unaffected: it has no second consumer,
and `run_dag_parse_sweep` does not route through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XJLHPutzoQ4FCJ9fofP135
@gunbai-bot gunbai-bot Bot changed the title v1 frontend performance The tree was lexed four times to compile it once: one acquisition authority, 12,121 tokenizations to 6,061 Aug 29, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review August 29, 2026 11:52
@briansrls
briansrls merged commit f1f9dd8 into main Aug 29, 2026
4 checks passed
@briansrls
briansrls deleted the session/sleek-cat-449 branch August 29, 2026 12:00
@gunbai-bot gunbai-bot Bot mentioned this pull request Aug 31, 2026
Merged
gunbai-bot Bot pushed a commit that referenced this pull request Sep 2, 2026
gunbc-ci-auto-heal @claude
Merge main (9bd0a76); record the admission as an EXACT authorized exc…
5fb520e 
…eption, not a category

INTEGRATION: roster auto-merged (both one_refusal_two_destinations and #9956's
unlanded_citation_indistinguishable_at_the_citing_end present and declared);
both projections regenerated, superset clean on both sides; lib.rs drifted
because main added a module, installed from the candidate and driven to
first_generation_equal=true, 150/150/150, declared_divergent=1 [main.rs].
Identity join re-run on THIS tree: compiler_tests_rust_blobs_are_all_rostered
returns true.

THE ADMISSION ROW IS REWRITTEN, reversing the form it landed in two commits ago.
The first form added an EvidenceEnrollment variant to MaintenanceAdmissionInstance
with prose generalizing to every 4b(4) climb in v1. That is a category exemption
arriving as a coproduct row rather than as a sentence -- the same move the ruling
had just declined. Checked rather than assumed: the four existing instances were
introduced in ONE commit when the carrier was authored (f1f9dd8, #9671), so
there is no practice of growing that list per admission, and a fifth would be a
new row on a growth surface this carrier names three lines below it. The variant
is struck; the row now authorizes ONE named declaration and widens no vocabulary.

TWO CORRECTIONS OF FACT in the same row. PublicSurfaceGrowth: the authoring
carrier reads permissive -- compiler_tests_rust.dag spells all 45 declarations as
bare fn with zero pub fn -- but the class is a diff over the EMITTED seed, and
stage0 emits . The emitter adds the visibility, so the class fires and
reading the .dag would say it does not. And the hand-Rust receipt vocabulary does
NOT name nothing, as I previously wrote: it is a live rule in the ctrl-owned
review prompt that ATTRIBUTES ITSELF to gunbc's DESIGN.md, where it does not
exist. So the class is authority substitution in a review prompt, its repair
belongs to ctrl, and reviews 58359 and 58380 are two executions of ONE prompt
rather than two independent sightings -- one producer corroborates nothing about
itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FdxzwWekWhHR2FCTTf8a1b
gunbai-bot Bot pushed a commit that referenced this pull request Sep 2, 2026
gunbc-ci-auto-heal @claude
Merge main (9bd0a76); record the admission as an EXACT authorized exc…
8b423c1 
…eption, not a category

INTEGRATION: roster auto-merged (both one_refusal_two_destinations and #9956's
unlanded_citation_indistinguishable_at_the_citing_end present and declared);
both projections regenerated, superset clean on both sides; lib.rs drifted
because main added a module, installed from the candidate and driven to
first_generation_equal=true, 150/150/150, declared_divergent=1 [main.rs].
Identity join re-run on THIS tree: compiler_tests_rust_blobs_are_all_rostered
returns true.

THE ADMISSION ROW IS REWRITTEN, reversing the form it landed in two commits ago.
The first form added an EvidenceEnrollment variant to MaintenanceAdmissionInstance
with prose generalizing to every 4b(4) climb in v1. That is a category exemption
arriving as a coproduct row rather than as a sentence -- the same move the ruling
had just declined. Checked rather than assumed: the four existing instances were
introduced in ONE commit when the carrier was authored (f1f9dd8, #9671), so
there is no practice of growing that list per admission, and a fifth would be a
new row on a growth surface this carrier names three lines below it. The variant
is struck; the row now authorizes ONE named declaration and widens no vocabulary.

TWO CORRECTIONS OF FACT in the same row. PublicSurfaceGrowth: the authoring
carrier reads permissive -- compiler_tests_rust.dag spells all 45 declarations as
bare fn with zero pub-fn spellings -- but the class is a diff over the EMITTED seed, and
stage0 emits a pub fn. The emitter adds the visibility, so the class fires and
reading the .dag would say it does not. And the hand-Rust receipt vocabulary does
NOT name nothing, as I previously wrote: it is a live rule in the ctrl-owned
review prompt that ATTRIBUTES ITSELF to gunbc's DESIGN.md, where it does not
exist. So the class is authority substitution in a review prompt, its repair
belongs to ctrl, and reviews 58359 and 58380 are two executions of ONE prompt
rather than two independent sightings -- one producer corroborates nothing about
itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FdxzwWekWhHR2FCTTf8a1b
@gunbai-bot gunbai-bot Bot mentioned this pull request Sep 2, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@briansrls