Skip to content

Model census heads in the authority and drop non-closure bodies after parsing - #11093

Merged
briansrls merged 8 commits into
mainfrom
session/valiant-badger-803
Sep 12, 2026
Merged

briansrls merged 8 commits into
mainfrom
session/valiant-badger-803

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Entry-scoped compilation still retained parsed bodies for every non-closure census module after #11072 streamed their tokens. This change models the declaration-head projection in v1.compiler.compile, then retains that projection after full parsing and annotation admission. The seed delegates to the generated projection instead of maintaining a second strip implementation.

The projection preserves declaration identity, names, kinds, signatures, imports, and spans. Functions retain the parser’s refusing body marker for function-kind discrimination; executable bodies, uses, match patterns, and expression data are discarded. Both census accumulator references share the projected parse result. Full-body parse errors still reach the compile refusal.

Only compile_to_resolved_with_options selects RetainDeclarationHeads for its non-closure sources. The existing parse_census_fill_sources API retains full nodes for namespace-wave admission, singleton CLI admission, entry resolution, and the diagnostics-only compile-clean caller. This avoids changing occurrence/namespace consumers for no meaningful singleton retention saving. Closure modules continue through the normal frontend. Frontier (b), the whole-pool edge index, is separate.

Validation:

  • Native arm64 release build (-p v1-compiler --bin gunbc --bin claim_executor -j 2) and cargo fmt --all --check pass. Binaries were copied out of shared /cargo-target immediately after each build.
  • Model-first commit f6581910ab2 contains the authority projection and its generated mirror. Retention and seed delegation follow in 812c542d491.
  • claim_executor --required-regen --source-root dag --source-root src/v2 passes at 812c542d491: first_generation_equal=true, planned/executed/adjudicated 155/155/155, changed paths empty. Authority digest fnv1a64:1321de6fd99aff1b; committed/candidate generated digest fnv1a64:d70e079d37a101da.
  • Only the emitted v1_compiler_compile.rs candidate was installed, using the parent-approved interim route for the installer gap tracked by Include host-shell mirror ownership in regen installation #11094. No generated Rust was hand-edited.
  • Rebuilt-seed claim_executor --required-regen-fixed-point --source-root dag --source-root src/v2 passes: fixed_point_equal=true, referenced_first_generation_equal=true, both at 812c542d491. CI build (including cargo clippy --all-targets -- -D warnings) and generated-artifact regeneration pass. Dashboard review 64084 approves with no findings. The witness lane is infrastructure-refused; see below.

Isolated before/after, same worktree corpus and invocation (16 closure sources, 5,429 census modules):

Run VmHWM (KiB) GiB Wall (s) Exit Margin to 15 GiB
Streaming baseline, positive 9,063,388 8.644 177.530 0 6.356 GiB
Heads retained, positive 7,886,192 7.521 167.962 0 7.479 GiB
Streaming baseline, balanced-body red 9,063,308 8.643 161.038 1 6.357 GiB
Heads retained, balanced-body red 7,886,224 7.521 169.985 1 7.479 GiB

The measured positive saving is 1,177,196 KiB = 1.123 GiB (12.99%). This is smaller than bright-bear-236’s prior heads-retention probe (8,986,140 → 7,663,268 KiB, about 1.26 GiB); #11072’s streaming-only result was 8,986,500 KiB after 10,006,804 KiB. Those are predecessor measurements, not substituted for this session’s results. The admitted budget stays 16,106,127,360 bytes (15 GiB). No floor budget rows or parse witnesses changed, and no floor JOB verdict is claimed as proof.

Both positive runs report 0 blocking errors, 213 advisories, and one emitted file. The discriminating red appends fn body_probe(x: Int) -> Int { let = = = 9 } to non-closure dag/std/access.dag, then restores the file. Both runs refuse at emit with two hard diagnostics and the identical error[dag/std/access.dag]: expected name, found Eq (span 2646–2647). Balanced braces make this invisible to the heads index; the full-body census parse must catch it. Artifact byte equality is not used because registry ordering is nondeterministic on the baseline.

Reproduction (run the identical command against each snapshotted binary, polling /proc/<pid>/status for VmHWM and timing with a monotonic clock):

gunbc compile --source-root dag --source-root src/v2 \
  --entry src/v2/workflow/floor2_prepared_subject.dag \
  --output-dir /tmp/heads-measure-output --target dag

Baseline gunbc SHA-256: c5c07879131d35731fec3f90c1524d8acdf128e9f8b9c13fae75142c33e918b3.
Final gunbc SHA-256: d0a38657f28295aaacdcba875557617896627741211dd45b11a305bce8edb375.

CI disposition: witness job 103431616486 refused with MemoryStallRefusedPageThrash while resolving floor_terminal_ledger_wire and typechecking floor2_prepared_subject: 282,224 major faults/minute, 4% CPU share over 34,140 ms, admitted budget unchanged at 15 GiB. The terminal ledger could not be rendered, so this is explicitly not a green floor. The run classified itself class=infra; it is a draw, and the isolated before/after measurements above remain the evidence for this change. GitHub is blocked pending disposition of that check.

gunbc-ci-auto-heal and others added 6 commits September 11, 2026 17:23
`gunbc compile --source-root dag --source-root src/v2 --entry
src/v2/workflow/floor2_prepared_subject.dag` peaks at 9.54 GiB RSS for an entry
whose reference-derived closure is 16 files. Reproduced at c560821:
VmHWM 10,006,804 KiB, wall ~230s (sleek-swift-789 measured 9,964,140 KiB).

Half of that peak is one term. `parse_census_fill_sources` is handed the whole
indexed pool minus the compile closure -- 5,399 modules the entry never
references -- and built `prepared`, a list holding EVERY one of those modules'
token streams, before parsing any of them. The fold then consumes each entry
exactly once, in order, so module N's tokens stayed resident through the parse
of modules N+1..5399 with no consumer. Measured by emptying the census fill
behind a throwaway probe: 10,006,804 -> 5,027,748 KiB, so the fill accounts for
4.75 GiB -- ~79 bytes of resident compiler state per byte of unreferenced source.

Fusing the tokenize into the fold frees each module's tokens before the next is
read. This is DESIGN §2's "minimise the demand before materialising" applied to
a RETENTION shape rather than to redundant work: same work, same order, same
results, held only for the interval it is used. It is explicitly NOT a cache --
nothing is memoised, and the pool is not scoped to the closure.

The pre-intern pass dissolves with `prepared` as a consequence, not a separate
decision: it existed only because the token lists were already all in hand.
Interning is idempotent and `parse_with_table_in_occurrence_scope` already
threads an updated table through the fold, so pre-interning module N's tokens
immediately before parsing module N reaches that parse with the same table
contents. Only id assignment ORDER changes, which no consumer reads -- names
resolve positionally via `authored_name_at`, never by intern id.

Every module is still parsed with FULL BODIES and every parse diagnostic still
surfaces. That is load-bearing and executed, not a formality: a body-level parse
break in a non-closure module is invisible to the heads-only module index and is
caught here and nowhere else.

Evidence, same invocation:
  before  VmHWM 10,006,804 KiB (9.54 GiB)  wall ~230s
  after   VmHWM  8,986,500 KiB (8.57 GiB)  wall  163s
Both: `resolved 16 sources ... 5399 indexed modules in the name census only`,
213 advisory diagnostics, 1 file emitted, exit 0.

Discriminating RED (the wall this change risks deleting), planted in
dag/std/access.dag, which is outside the entry's closure:
  fn body_probe(x: Int) -> Int { let = = = 9 }
Balanced braces, so the heads-only module index does NOT catch it. Before and
after, identically: exit 1, `refused at emit`, 2 hard diagnostics,
`error[dag/std/access.dag]: expected name, found Eq`. A brace-unbalanced probe
was tried first and rejected as a control -- it fires, but the module index
refuses it before the census fill ever runs, so it does not discriminate.

Positive control: `--source-root src/v1 --entry src/v1/compile.dag` produces
2487 hard diagnostics before and after -- the edit adds none.

NOT claimed: the emitted dag-artifact.json is run-to-run nondeterministic on
main (baseline binary, two identical runs, differing output), so artifact
equality was not usable as an equivalence oracle and is not cited as one.

Named, not done:
- Retention of full-body ASTs for the same 5,399 modules. `fill.modules` reaches
  only `build_symbol_index_qualified_fill`, which reads declaration heads. A
  probe retaining `census_heads_module_node(node)` measured a further
  8,986,140 -> 7,663,268 KiB (1.26 GiB). Deferred because the post-parse heads
  strip exists only in the seed (`cli_run::census_heads`), so landing it in the
  authority needs that projection declared in .dag first -- `02_parse.dag`
  already carries `census_heads_body_stand_in`, so the vocabulary is half there.
- `build_both_closure_edge_index` computes reference and bare pull edges for all
  ~5.4k pool files, while an --entry BFS queries only the closure's. That work is
  not merely duplicated, it is never demanded. It sits inside the residual
  4.79 GiB and is the brief's originally-named term.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdEbLY7W5T2jBtqXQLBgVE
Review 63932: the annotation added to parse_census_fill_sources carried two
transcribed number pairs (VmHWM before/after, wall before/after). DESIGN §6
forbids exactly that -- a copied number is unreachable from the producer that
owns it, so it rots without anyone touching either end. Naming the entry point
was the right half and is kept; the digits are replaced by the invocation that
re-derives them, plus why this entry is the discriminating one.

The figures stay in the PR and commit message, which are bound to a revision
and do not claim to stay true.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdEbLY7W5T2jBtqXQLBgVE
CI caught what the file's own comment denies: `v1_compiler_compile.rs` is NOT
hand-synced, it is regenerated from `compile.dag` and gated.

  required-ci: generated-artifact population=stage0-mirrors
    FAIL generated surface drift: v1_compiler_compile.rs

The drift was exactly, and only, the 8-line comment block I hand-added inside
`parse_census_fill_sources`. Diffing the committed file against
`target/stage0-regen-candidate/src/v1_compiler_compile.rs` produced 9 diff
lines, all of them that comment; the generated bytes are otherwise identical
to what I had written by hand, which is independent evidence that the manual
sync was semantically right and procedurally wrong.

The reasoning the comment carried already lives in `src/v1/compile.dag`, which
is the authority and the only correct home for it. A generated file cannot hold
commentary: the next regeneration deletes it, so it would rot by construction.

Produced by `claim_executor --regen-round-cost` (run twice -- the first pass
refuses because its own seed build replaces the running executable). The round
itself then refuses at its rebuild step with
`RebuildScopeRefused MirrorHasNoOwningPackage mirror=v1_compiler_compile.rs`
and restores its checkpoint, so the emitted candidate was installed for this
one path rather than via the round's install. Only that path was copied --
copying the candidate tree wholesale silently reverts hand-authored mirrors.

Re-verified after regeneration, not assumed from the earlier run: release build
clean, `cargo fmt --all --check` clean, and the discriminating RED still refuses
identically -- planted `fn body_probe(x: Int) -> Int { let = = = 9 }` in
dag/std/access.dag (outside the entry's closure) gives exit 1,
`error[dag/std/access.dag]: expected name, found Eq`, 2 hard diagnostics.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdEbLY7W5T2jBtqXQLBgVE
@gunbai-bot gunbai-bot Bot changed the title O(closure) frontier (a): model the post-parse heads strip in the authority, then retain heads only for non-closure modules (~1.26 GiB) Model census heads in the authority and drop non-closure bodies after parsing Sep 11, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review September 11, 2026 21:43
@gunbai-bot

gunbai-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Operator handoff for 812c542d491, under the parent’s refused-floor disposition:

  1. Build and generated artifacts are green. Required build, cargo clippy --all-targets -- -D warnings, and heal-generated-artifacts pass. Local first-generation equality covers all 155 mirrors; the rebuilt-seed fixed point also passes at this head.
  2. The floor is a draw, not green. Run 34649265163, witness job 103431616486 refused with MemoryStallRefusedPageThrash, explicitly floor_class=infra, during terminal-ledger grammar resolution. No rerun or budget increase is being used to manufacture a pass.
  3. The isolated retention saving is 1.123 GiB. The same entry invocation and 16-source/5,429-module population went from 9,063,388 to 7,886,192 KiB (8.644 → 7.521 GiB), with positive wall time 177.530 → 167.962 seconds. Margin to the unchanged 15 GiB budget is 7.479 GiB. The PR body carries commands and binary hashes.
  4. The discriminating parse red is preserved. The balanced-brace non-closure body probe refuses before and after with exit 1, two hard diagnostics, and error[dag/std/access.dag]: expected name, found Eq at the same 2646–2647 span. Bodies are fully parsed before projection; the probe was restored.
  5. Review is approved. Dashboard review 64084 (cursor) approves with no findings; its complete artifact was read and there are no outstanding review changes.

Implementation and evidence are complete. Per sunny-tern-813’s ruling, stopping here for the operator to land this PR; GitHub’s refused-floor check remains explicitly red.

— sent from valiant-badger-803

@briansrls
briansrls merged commit 88c67ac into main Sep 12, 2026
2 of 4 checks passed
@briansrls
briansrls deleted the session/valiant-badger-803 branch September 12, 2026 04:23
@briansrls
briansrls restored the session/valiant-badger-803 branch September 12, 2026 04:26
@gunbai-bot
gunbai-bot Bot deleted the session/valiant-badger-803 branch September 12, 2026 04:29
briansrls pushed a commit that referenced this pull request Sep 12, 2026
gunbai-bot Bot pushed a commit that referenced this pull request Sep 12, 2026
gunbc#11047 moved the separator into the fold -- `authored` is now
`join(r.receipts, " ")` -- and deleted the per-receipt trailing-space
convention this row was authored under. All eight of my receipts ended in a
space, so after the merge every one of the seven boundaries would have rendered
double-spaced. Stripped the trailing space from all eight; the diff is 8/8 and
each changed line is a receipt terminator, not prose.

The coupling is the RENDERER, not the import graph. This row imports only
RecurringFailureMode and NonEmptyStr, neither touched by #11047, so an
import-graph reading clears it and is wrong -- which is how #11100 was judged
safe.

Re-derived against the merged tree, with a pre-existing row on the same page as
the control:
  - regen EXIT=0; my rendered paragraph 7426 chars, double-space count 0;
    control `absorbing_fallback` paragraph 3602 chars, double-space count 0.
  - all seven receipt boundaries render with exactly one space
    ("...by no one. WHY IT IS WORSE", "...a climb. THE SPECIMEN", and so on).
  - second regen byte-identical to the first (cmp clean).

The mutation evidence is RE-DERIVED under the new fold rather than carried over
from the old one: one `receipts` element replaced by 42 gives regen EXIT=1, a
located resolve refusal at :10:5 "type mismatch: expected 'Primitive(String)',
got 'Primitive(Int)'", and no projection written. Restoring the row reproduces
the projection byte-for-byte.

This merge also brings in #11093 (census heads modelled, non-closure bodies
dropped after parsing), which is the cause of the MemoryStallRefusedPageThrash
that refused the floor on the previous head. A rerun would have replayed the
merge ref pinned before it; only a push re-measures the current memory shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuEqKq5Jxcy4Wj3fYEwYFL
gunbai-bot Bot pushed a commit that referenced this pull request Sep 12, 2026
The floor lane refused MemoryStallRefusedPageThrash on three attempts. All
three compiled a base predating the fix: the run was created 02:50:42Z and
gunbc#11093 -- which drops non-closure bodies from census heads, ~1.26 GiB
less per compile -- merged at 04:23:22Z. A re-run reuses the original merge
ref, so no amount of re-running could have picked it up.

This reverses my earlier refusal to re-integrate. That refusal was correct
when main carried no fix (merging more commits only enlarges the compile that
thrashed); it is wrong now, because the content in main is the REDUCTION.
Pushing also forces a synchronize so the merge ref is recomputed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013YcKjgpfRzhjNJrjaNZjJG
gunbai-bot Bot added a commit that referenced this pull request Sep 12, 2026
… adjudicate it (#11020)

* File the class: routing a subject to an adjudicator never observed to adjudicate it

One row under dag/gunbc/recurring_failure_mode/. The invalid state: a subject
whose adjudication is transferred to an authority never OBSERVED to adjudicate
it, with the transfer recorded as a routing rather than as a loss of coverage.
The origin gate goes green on a typed, honest decline; the destination's silence
is indistinguishable from coverage.

Specimen: the near-miss of 2026-09-09/10 on the four native_decl_selection
witnesses, the ruling that they be routed to required-v2-native, and
clever-dove-148's out-of-scope check of the DESTINATION, which found it refusing
at its own emit/resolve stage before adjudicating any witness. The row records
what actually caught it -- an acceptance condition demanding the destination
COMPLETE and NAME all four identities -- as the repair it points at.

Bounded explicitly against shared_precondition_re_derived_once_per_claim_frame
(cost, not destination), absorbing_fallback (widens, does not refuse), the inert
lens (never consulted; here the destination is consulted and loud), and the
escape hatch (proceeds as if no refusal; here the refusal fires and is located).

Rung found at mitigatable; ceiling argued at mechanically preventable -- a
decline arm required to cite a destination receipt naming the routed subject --
and explicitly NOT structural, since a receipt can be stale or name a run that
reached no verdict. Trigger names the capability: a cross-gate adjudication
receipt carrying destination lane, run, and the subject identities that run
actually reached a verdict on.

EVIDENCE, by execution against a release gunbc built from this tree, through
tools.docs_projection_gate regen (gunbc.design_ledgers
expected_design_failure_modes_md):

  - green: regen EXIT=0, writes docs/design-failure-modes.md 1563582 bytes, and
    the row's identity and prose render into it.
  - MUTATION, consumed field: one element of `receipts` replaced by `42`. regen
    EXIT=1, resolve refusal located at
    routing_a_subject_to_an_unverified_adjudicator.dag:10:5 "type mismatch:
    expected 'Primitive(String)', got 'Primitive(Int)'", and NO
    docs/design-failure-modes.md is written -- the fold reds and the text is gone.
  - CONTROL, unconsumed field: `evidence: 42` on the same row. regen EXIT=0 and
    the projection writes normally, reproducing exactly the green that #11015's
    author found and threw the resolve-only check out over. The pair is what
    makes the first result discriminating rather than incidental.
  - regen re-run after restore is byte-identical to the first generation (cmp
    clean), no drift; docs_projection_gate main EXIT=0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuEqKq5Jxcy4Wj3fYEwYFL

* Describe the sibling class by shape, not by an identity that resolves nowhere

review 63417 (cursor/auto, REQUEST_CHANGES) is correct: the BOUNDARIES receipt
cited `shared_precondition_re_derived_once_per_claim_frame` as a peer ledger
identity, and that row is on neither tree -- empty on origin/main and empty in
git ls-files at this head, so the only mention of the name in the corpus was my
own citation of it. The named sibling is filed but unlanded.

Repaired the way `unlanded_citation_indistinguishable_at_the_citing_end`
prescribes: describe the sibling by SHAPE -- "the class filed by clever-dove-148
over the same specimen, whose subject is a shared precondition re-derived once
per claim frame" -- and say in the same breath why the identity is withheld, so
a later reader is not left to wonder whether the name rotted or never existed.
The boundary the receipt draws is unchanged; only the handle is. Landing the
sibling in this closure was the other admissible repair and is not mine to do:
it belongs to that row's author and to its own PR.

Every remaining backticked ledger identity in the row resolves against the
directory: `absorbing_fallback` and
`unlanded_citation_indistinguishable_at_the_citing_end`.

Re-verified by execution: tools.docs_projection_gate regen EXIT=0, the reworded
receipt renders into docs/design-failure-modes.md, and a second regen is
byte-identical to the first (cmp clean).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuEqKq5Jxcy4Wj3fYEwYFL

* Migrate the row's receipts to the #11047 separator convention

gunbc#11047 moved the separator into the fold -- `authored` is now
`join(r.receipts, " ")` -- and deleted the per-receipt trailing-space
convention this row was authored under. All eight of my receipts ended in a
space, so after the merge every one of the seven boundaries would have rendered
double-spaced. Stripped the trailing space from all eight; the diff is 8/8 and
each changed line is a receipt terminator, not prose.

The coupling is the RENDERER, not the import graph. This row imports only
RecurringFailureMode and NonEmptyStr, neither touched by #11047, so an
import-graph reading clears it and is wrong -- which is how #11100 was judged
safe.

Re-derived against the merged tree, with a pre-existing row on the same page as
the control:
  - regen EXIT=0; my rendered paragraph 7426 chars, double-space count 0;
    control `absorbing_fallback` paragraph 3602 chars, double-space count 0.
  - all seven receipt boundaries render with exactly one space
    ("...by no one. WHY IT IS WORSE", "...a climb. THE SPECIMEN", and so on).
  - second regen byte-identical to the first (cmp clean).

The mutation evidence is RE-DERIVED under the new fold rather than carried over
from the old one: one `receipts` element replaced by 42 gives regen EXIT=1, a
located resolve refusal at :10:5 "type mismatch: expected 'Primitive(String)',
got 'Primitive(Int)'", and no projection written. Restoring the row reproduces
the projection byte-for-byte.

This merge also brings in #11093 (census heads modelled, non-closure bodies
dropped after parsing), which is the cause of the MemoryStallRefusedPageThrash
that refused the floor on the previous head. A rerun would have replayed the
merge ref pinned before it; only a push re-measures the current memory shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuEqKq5Jxcy4Wj3fYEwYFL

* Cite the sibling class by identity: it landed, so the withholding clause rotted

review 64357 (claude/opus[1m], REQUEST_CHANGES) is correct and I verified it
rather than taking it: shared_precondition_re_derived_once_per_claim_frame is
on this tree. It landed via #11015 (ec1571e), was appended by #11037
(ce76f34), and this PR's own merge of main brought it in -- so the file is
present in the worktree and the identity resolves.

The BOUNDARIES receipt was therefore printing a justification that was TRUE WHEN
WRITTEN AND IS NOW FALSE, and a false one in the worst direction: it told a
reader the name does not exist when it does, which inverts
unlanded_citation_indistinguishable_at_the_citing_end rather than applying it.
Restored the direct citation and deleted the withholding clause entirely; the
boundary the receipt draws is unchanged, only the handle.

Every backticked ledger identity in the row now resolves against the directory:
absorbing_fallback and shared_precondition_re_derived_once_per_claim_frame.
(namespace_graft is the specimen's error name, not a ledger identity.)

Re-derived: regen EXIT=0, rendered paragraph 7096 chars with 0 double spaces,
second regen byte-identical to the first (cmp clean), and the receipt
trailing-space count stays 0 under the #11047 join convention.

Worth naming, since this row is about exactly this: the withheld citation was a
correct judgement about a tree that then moved underneath it, and nothing in the
row re-read its own premise. That is the same shape as the class being filed --
a conclusion whose condition was checked once and never again -- which is why a
reviewer reading the current tree caught it and the author who wrote it did not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuEqKq5Jxcy4Wj3fYEwYFL

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant