Stream the census fill: 9.54 -> 8.57 GiB on a 16-file --entry compile

[gunbai-bot]

The defect

gunbc compile --source-root dag --source-root src/v2 --entry src/v2/workflow/floor2_prepared_subject.dag --output-dir /tmp/x --target dag peaks at 9.54 GiB RSS for an entry whose reference-derived closure is 16 files. Reproduced at c560821d993: VmHWM 10,006,804 KiB, wall ~230 s (sleek-swift-789 measured 9,964,140 KiB — within 0.4%).

Half of that peak is one term, and it is not the one first suspected.

parse_census_fill_sources receives the whole indexed pool minus the compile closure — 5,399 modules the entry never references — and built prepared, a list holding every one of those modules' token streams, before parsing any of them. The fold then consumes each entry exactly once, in order, so module N's tokens stayed resident through the parse of modules N+1..5399 with no consumer.

Priced directly, by emptying the census fill behind a throwaway probe:

	VmHWM	wall
baseline	10,006,804 KiB (9.54 GiB)	~230 s
census fill emptied	5,027,748 KiB (4.79 GiB)	95 s

So the fill accounts for 4.75 GiB, 49.8% of peak — about 79 bytes of resident compiler state per byte of source the entry never references (the corpus is 60.4 MB across 5,415 files).

This is a growth-surface repair, not only a capacity one

The census population is the whole indexed pool minus the closure, so every module added to the corpus pays into it — every recurring_failure_mode row, every witness, every new .dag file — regardless of whether any entry references it. The cost is denominated in the corpus, not in the change (DESIGN §5's absorbing-fallback tell, in its cost arm). Left alone, the term grows monotonically with the corpus and re-crosses whatever budget it is measured against; fixing the retention shape is what stops that, and no budget raise would.

The change

Fuse the tokenize into the fold so each module's tokens are freed before the next is read. This is DESIGN §2's minimise the demand before materialising applied to a retention shape rather than to redundant work: same work, same order, same results, held only for the interval it is actually used.

It is explicitly not a cache — nothing is memoised, and the pool is not scoped to the closure. §2 forbids making redundant work cheap; this deletes residency, not recomputation.

The pre-intern pass dissolves with prepared as a consequence, not a separate decision: it existed only because the token lists were already in hand. Interning is idempotent and parse_with_table_in_occurrence_scope already threads an updated table through the fold, so pre-interning module N immediately before parsing module N reaches that parse with the same table contents. Only id assignment order changes, which no consumer reads — names resolve positionally through authored_name_at, never by intern id.

Authority is src/v1/compile.dag; v1_compiler_compile.rs is its hand-synced mirror and moves with it.

What is deliberately unchanged

Every module is still parsed with full bodies, and every parse diagnostic still surfaces. That wall is load-bearing and executed, not a formality — a body-level parse break in a non-closure module is invisible to the heads-only module index and is caught here and nowhere else.

Evidence

The admitted budget is 16,106,127,360 bytes = 15.00 GiB (per-runner cgroup memory.high on actions-runner@*.service, uniform across the fleet — it is not an srv4-specific setting, and the fleet variation is in page cache available to the cgroup, not in the budget).

Same invocation, VmHWM from /proc/<pid>/status:

	VmHWM	% of budget	margin	wall
before	10,006,804 KiB (9.54 GiB)	63.6%	5.46 GiB	~230 s
after	8,986,500 KiB (8.57 GiB)	57.1%	6.43 GiB	163 s

Both print resolved 16 sources (reference-derived closure ...), 5399 indexed modules in the name census only, 213 advisory diagnostics, 1 file emitted, exit 0.

A green floor on any single host is a draw, not a fix — what varies between hosts is page cache, not the budget. The claim here is the measured peak and its margin, above; a floor verdict is corroboration, not the deliverable.

Discriminating RED — the wall this change risks deleting. Planted in dag/std/access.dag, outside the entry's closure:

fn body_probe(x: Int) -> Int { let = = = 9 }

Balanced braces, so the heads-only module index does not catch it. Before and after, identically: exit 1, refused at emit, 2 hard diagnostics, error[dag/std/access.dag]: expected name, found Eq.

A brace-unbalanced probe was tried first and rejected as a control: it fires, but the module index refuses it (heads-only parse: unterminated function body) before the census fill ever runs, so it does not discriminate this change. Recorded because a control that fires without discriminating reads exactly like a passing one (DESIGN §4b: ask whether the RED is authorable where the check runs).

Positive control: --source-root src/v1 --entry src/v1/compile.dag produces 2487 hard diagnostics before and after — the edit adds none.

Not claimed: the emitted dag-artifact.json is run-to-run nondeterministic on main, so artifact equality was unusable as an equivalence oracle and is not cited as one. Evidence is in a comment below for the lane that owns it; it is pre-existing and not caused by this PR.

Local: cargo clippy --all-targets -- -D warnings clean; cargo fmt --all --check clean.

Declared frontier (named, not done)

This PR is step 1. The two residual terms, both probe-measured, are what take the entry compile to genuinely O(closure):

(a) Full-body AST retention for the same 5,399 modules. fill.modules reaches only build_symbol_index_qualified_fill, which reads declaration heads. A probe retaining census_heads_module_node(node) measured a further 8,986,140 → 7,663,268 KiB (1.26 GiB; 7.31 GiB, 48.7% of budget), wall unchanged, with the discriminating RED still refusing identically.
Precondition for that lane: the post-parse heads strip currently exists only in the seed (cli_run::census_heads). It must be projected into the authority — declared in .dag — before this lands; doing it from the seed side would drift the mirror from compile.dag. 02_parse.dag already carries census_heads_body_stand_in, so the vocabulary is half there.

(b) build_both_closure_edge_index computes reference and bare pull edges for all ~5.4k pool files (for sf in index.source_files.values()), while an --entry BFS queries only the closure's. That work is not merely duplicated — it is never demanded. It forces tree_bare_census_for_root per root as a side effect and sits inside the residual 4.79 GiB. This is the term that makes the compile O(closure) rather than merely smaller.

Not touched

No memory budget raised, no floor budget rows, no parse witnesses — per the brief and sleek-swift-789's ruling.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QdEbLY7W5T2jBtqXQLBgVE

[gunbai-bot]

Handoff: dag-artifact.json is run-to-run nondeterministic on main (NOT caused by this PR)

Recording the measurement here so the lane that owns this starts from evidence rather than re-deriving it. This is pre-existing on main and is not caused by this PR — it surfaced because I tried to use artifact equality as an equivalence oracle for my change and had to discard it.

Method: one binary, built from a clean c560821d993 worktree, run twice with identical arguments, nothing else changed.

gunbc compile --source-root dag --source-root src/v2 \
  --entry src/v2/workflow/floor2_prepared_subject.dag \
  --output-dir <out> --target dag

Binary sha256 52205c90e52d2015b39d32aef67c17467b3f47c33237c1758ed54ca84927b843

run	dag-artifact.json sha256	bytes
1	be8c6ee1c94b2d43dabf0d209b67fa47f6cf82d861ac1e711a236d1a895879af	6,079,557
2	3595a56b10f4e3be9137d8792fbb1b828b7f4480c0d692a738976cde355e8a66	6,079,557

Identical byte length, first differing byte at offset 5,914,512; 2,408 differing comma-separated fields.

It is narrower than "the artifact is nondeterministic"

Every $ref node id is identical between the two runs — the graph itself is deterministic. The entire difference is the ordering of item_registry_keys:

run 1: "item_registry_keys": ["std.error_primitives.Result", "std.error_primitives.DivError"]
run 2: "item_registry_keys": ["std.error_primitives.DivError", "std.error_primitives.Result"]

Same set, different order, in every affected module. That is an unsorted iteration over a hash-keyed registry reaching serialization — item_registry is a HashMap, and its keys are emitted in iteration order. The likely repair is a sort at the projection boundary, but I have not tried it and am not claiming it.

Why it matters beyond tidiness

Any gate that compares emitted artifacts — a regen fixed-point, a drift check, a byte-identity or round-trip claim — is comparing a value that changes without any input changing. That fails open or closed by luck. It also means no compiler change can currently be argued by artifact diff, which is the specific cost I paid: the diff reported a difference whether or not my change caused one, and read as a regression against my own work until I ran the two-run control.

Recommended control for whoever picks this up: run the binary twice against itself first, before trusting any artifact comparison. That costs one run and settles whether the oracle is valid at all.

[gunbai-bot]

Re review 63932 — the finding is correct and is fixed, not argued with.

src/v1/compile.dag carried two transcribed number pairs (VmHWM 10,006,804 KiB -> 8,986,140 KiB, wall ~230s -> 153s). DESIGN §6 is unambiguous that this is the forbidden half: "Name the instrument, never transcribe its output … a transcribed number is unreachable from the thing that owns it, so it rots without anyone touching either end." The reviewer also read the split correctly — naming the entry point was the right half, and the mirrored Rust comment was already doing it properly.

The annotation now names only the producer that re-derives the measurement, and says why that entry is the discriminating one:

// The instrument that re-derives the size of this term, rather than a number copied out of it
// (DESIGN §6): `gunbc compile --source-root dag --source-root src/v2 --entry
// src/v2/workflow/floor2_prepared_subject.dag --output-dir <dir> --target dag`, reading VmHWM
// from /proc/<pid>/status while it runs. The entry is chosen because its reference-derived
// closure is a handful of files while the census population is the whole pool, so peak RSS there
// is dominated by this fold rather than by the closure being compiled.

On the reviewer's note about the PR title and body: the numbers stay in the PR and commit message and are deliberately not in-tree. §6 governs what the corpus carries — a PR body is the report of a measurement made at a point in time against a named commit, which is exactly where a transcribed figure is legitimate because it is bound to that revision and does not claim to stay true. Nothing in dag/ or src/ now carries a figure that can rot. If that reading is wrong I will take the numbers out of the title too — say so and I will.

Separately, and not raised by the review: CI caught that src/v1/stage0/src/v1_compiler_compile.rs is not hand-synced the way its own comment implies — required-ci: generated-artifact population=stage0-mirrors FAIL generated surface drift: v1_compiler_compile.rs. It is regenerated from compile.dag and gated, so my hand-written mirror edit was the wrong move regardless of whether it was semantically right. I am regenerating it through claim_executor --regen-round-cost and will push the generated bytes. Flagging it because the review's ".dag and stage0 mirror stay in lockstep" line was true of intent but not of mechanism, and the next person reading that file's comment would make the same mistake I did.

— sent from bright-bear-236

[gunbai-bot]

Floor verdict on an srv4-class runner, head containing the fix

Run 34627521474, job required-witnesses-floor, head f913e5e7 (contains the census-fill fix).

Runner: host=srv4 runner=srv4-13-1789146343-176079 — the same class as the four refusing specimens.

required-ci: phase floor (one prepared subject, one fold)
required-floor: planned=3684 executed=3684 not_attempted=0 terminal=3684
                passed=3616 known_red_held=19 claims_failed=0
                interrupted_before_verdict=0 known_red_budget_refused=0
floor_class=none
required-witnesses-floor: success

It reached a verdict where the specimens refused: executed=3684, not_attempted=0, terminal=3684, interrupted_before_verdict=0, claims_failed=0.

Two things I checked rather than assumed, because both fail silently:

1. The floor phase executed; it was not routed. The job announces phase floor (one prepared subject, one fold) as owned. The only ROUTED to lane build phases are generated-artifact and regen-fixed-point, which belong to the other lane. A routed phase greens identically to an executed one, so the announcement is the thing to read, not the lane colour.

2. Zero real MemoryStallRefusedPageThrash occurrences. A grep for that string returns 4 hits, and all 4 are the classifier's own echoed shell lines (if [ "$GUNBC_FLOOR_CLASS" = structural ] && grep -q 'MemoryStallRefusedPageThrash' ...). Filtering those out leaves 0. Reporting the raw count as evidence of a refusal would have been wrong in the opposite direction.

What this does and does not establish

Per sunny-tern-813's amendment: the admitted budget is a per-runner cgroup memory.high, uniform across the fleet, and what varies between hosts is the page cache available to the cgroup. A fifth specimen refused on srv1 under the same budget. So a green floor on any single host is a draw, not a fix — this run is corroboration, not the deliverable.

The deliverable is the measured peak against the budget, which is in the PR body: 8.57 GiB against a 15.00 GiB admitted budget, 57.1%, margin 6.43 GiB (from 9.54 GiB / 63.6% / 5.46 GiB margin). And because the census population is the whole pool minus the closure, that margin is consumed by ordinary corpus growth — which is why the declared frontier items (a) and (b) matter more than this green does.

The aggregate witnesses check on this head reads failure for an unrelated reason: the build lane caught stage0-mirrors FAIL generated surface drift: v1_compiler_compile.rs. I hand-edited a file that is in fact regenerated from compile.dag, not hand-synced. Being fixed by regenerating through claim_executor --regen-round-cost; the floor result above is unaffected by it.

[gunbai-bot]

The page-thrash class still fires with this fix in. This PR is necessary but not sufficient.

Stating this plainly because the earlier green could be read as more than it was.

Head ee314e13156 (fix + regenerated mirror), run 34633397813, floor job on host=srv4 runner=srv4-04:

required-ci: floor refused: TERMINAL-LEDGER REFUSAL cause=WireAuthorityUnresolved
  entry=src/v2/workflow/floor_terminal_ledger_wire.dag
  detail=MemoryStallRefusedPageThrash: this process refaulted its own evicted pages at
  208088 major faults/minute while computing for only 3% of the wall, over the last
  45942 ms (159333 major faults; rate line 6000/minute)
floor_class=infra signature=MemoryStallRefusedPageThrash

That is the exact entry the original specimens named. The class reproduces with the fix applied.

The two runs together are the informative result

head	runner	floor
f913e5e7 (fix)	srv4-13	success, floor_class=none, 3684/3684 terminal, zero stalls
ee314e13 (fix + regen)	srv4-04	floor_class=infra MemoryStallRefusedPageThrash

Same fix, same host class, opposite outcomes, different runner slots. That corroborates the ruling that the admitted budget is a uniform per-runner cgroup memory.high and the variable is page cache available to the cgroup under co-tenancy — not the budget, and not this diff. Which cuts both ways and I want it on the record in both directions: my earlier green was a draw, and this red is not a refutation of the diff. The classifier says as much itself — "this is not a verdict about the diff."

What I therefore do and do not claim

Claimed, and measured: the isolated peak of the census-fill term drops 9.54 → 8.57 GiB (63.6% → 57.1% of the 15.00 GiB admitted budget; margin 5.46 → 6.43 GiB), wall ~230 → 163 s, with every wall preserved and proven preserved.

Not claimed: that this clears the specimen class. It does not. floor_terminal_ledger_wire.dag holds the required_floor hub's state plus this compile, and the sum still exceeds what a loaded slot can keep resident. I fixed the largest single term, not the cost shape the issue is titled after. Getting to O(closure) needs the declared frontier in the PR body — (a) heads-only retention, probe-measured at a further 1.26 GiB, and (b) the edge index inside the residual 4.79 GiB, which is the term that actually changes the shape rather than the constant.

Merge state

• required-witnesses-build: SUCCESS — the drift is fixed (stage0-mirrors first_generation_equal=true, generated-artifact OK, regen-fixed-point fixed_point_equal=true, phases_run=2 phases_failed=0, and the phase was owned rather than ROUTED).
• required-witnesses-floor: FAILURE, infra-classified.
• Two approvals, no open REQUEST_CHANGES. GitHub mergeStateStatus=BLOCKED on the red floor check.

dag/test/claim/ci/ci_failure_class_witness_test.dag models floor_failure_blocks_merge(MemoryStallRefusedPageThrash) == false, so by the repo's own model an infra-classified floor red is not a merge blocker. GitHub blocks on it anyway. I am not going to re-run the floor until a slot happens to pass and call that a green — by the standing ruling that is a draw, and manufacturing one would be the worst available option. This needs an owner's decision: land on the infra classification, or hold this PR until (a) and (b) land so the class is genuinely cleared and all three go together.

[gunbai-bot]

Ready to merge — the operator's decision, with all five grounds in one place

Head ee314e13156. sunny-tern-813 ruled to land this on its own evidence. Everything needed to make that call is below; nothing here requires reading another comment.

1. Build lane: SUCCESS. Run 34633397813. The generated-artifact phase was owned and executed, not ROUTED — stage0-mirrors first_generation_equal=true, generated-artifact OK populations=registry-projections,stage0-mirrors, regen-fixed-point fixed_point_equal=true, lane=build phases_run=2 phases_failed=0.

2. An executed floor verdict for this head's content. Run 34627521474 on host=srv4 runner=srv4-13: floor_class=none, planned=3684 executed=3684 not_attempted=0 terminal=3684 passed=3616 known_red_held=19 claims_failed=0 interrupted_before_verdict=0. The floor phase was announced as owned (phase floor (one prepared subject, one fold)), not routed, and zero real MemoryStallRefusedPageThrash occurrences — the only 4 grep hits are the classifier's own echoed shell lines.

3. The later floor refusal is infra, by the repo's own witness. Run 34633397813 on srv4-04 refused with floor_class=infra signature=MemoryStallRefusedPageThrash, and the runner itself prints "this is not a verdict about the diff." dag/test/claim/ci/ci_failure_class_witness_test.dag models floor_failure_blocks_merge(classify_failure_reason("MemoryStallRefusedPageThrash…")) == false. Same fix, same host class, opposite outcomes on two runner slots — the variable is page cache available to the cgroup under co-tenancy, not the budget and not this diff. I did not re-run the floor to fish for a green; that would sample a distribution rather than test the change.

4. Reviews: two approving reviews, one distinct provider, no open REQUEST_CHANGES. Reviews 63945 and 63965, both claude/opus[1m], both APPROVE with no findings on the current head; 0 active reviews. The earlier REQUEST_CHANGES (review 63932, a DESIGN §6 transcribed-measurement finding) was correct, was fixed in c7ab67d0eac, and is superseded.

5. The measured margin — the actual deliverable. Against the admitted budget of 16,106,127,360 bytes (15.00 GiB, per-runner cgroup memory.high, uniform across the fleet):

	VmHWM	% of budget	margin	wall
before	10,006,804 KiB (9.54 GiB)	63.6%	5.46 GiB	~230 s
after	8,986,500 KiB (8.57 GiB)	57.1%	6.43 GiB	163 s

Walls preserved and proven preserved: the discriminating RED (fn body_probe(x: Int) -> Int { let = = = 9 } in a non-closure module — balanced braces, so the heads-only index cannot see it) refuses identically before and after with error[dag/std/access.dag]: expected name, found Eq.

What this does not do

It does not clear the page-thrash class — that reproduced on floor_terminal_ledger_wire.dag with the fix in. This deletes the largest single term; the cost shape needs the declared frontier in the PR body, and sunny-tern-813 is dispatching both lanes now: (b) the whole-pool edge index (the shape term), and (a) heads-only retention as a two-step lane whose first step projects the post-parse heads strip into the .dag authority. Both cite the probe numbers and the discriminating RED from this PR.

GitHub reports mergeStateStatus=BLOCKED on the red floor check, so this needs an operator merge over that infra-classified red. @operator — over to you; I am stopping work on this PR.

[gunbai-bot]

Automated CI nudge on ee314e1 — there is no fix to push, and pushing one would be wrong. Checked rather than assumed:

The "2 failing" is one failure, not two. required-witnesses-floor: FAILURE and witnesses: FAILURE — the second is the aggregating job reading the first. required-witnesses-build: SUCCESS, heal-generated-artifacts: SUCCESS.

Nothing new broke. I checked specifically for a structural failure hiding under the infra classification: phases_run=3 phases_failed=1, the single failed phase is floor, and its only cause is the memory stall. No non-zero claims_failed, no witness-level failure — the refusal aborted the phase before any verdict was adjudicated.

The failure is environmental and already adjudicated. floor_class=infra signature=MemoryStallRefusedPageThrash, and the runner prints "this is not a verdict about the diff." dag/test/claim/ci/ci_failure_class_witness_test.dag models floor_failure_blocks_merge(MemoryStallRefusedPageThrash) == false. The same commit content got an executed floor verdict on srv4-13 — floor_class=none, 3684/3684 terminal, zero stalls. The variable is page cache available to the runner cgroup under co-tenancy.

Why I am not re-running the floor: for a memory-pressure class, repeated attempts sample a distribution rather than re-test the change. A green obtained that way is a draw dressed as evidence, and sunny-tern-813's standing ruling says so. The deliverable for a memory-shape change is the measured peak against the budget — 8.57 GiB against 15.00 GiB, 57.1%, margin 6.43 GiB, from 9.54 GiB — which is in the PR body with its instrument.

This PR is waiting on an operator merge over an infra-classified red, per the ruling. The five grounds are in the previous comment. No further pushes from me.

— sent from bright-bear-236