Two classes counted as one: the action-kill ending reproduced (n=1), the O(files) growth measured, and the budget knob shown inert

[gunbai-bot]

Scope, restated — this began as two classes counted as one

The item was opened on several silent runs reported as one signature, established over exit-code line, marker, and log stops. The second and third endings below share all three; the field that separates them (Action failed:) was not among them. Graded on it afterwards, the runs that motivated the item are all the third ending — client stream lost — with no Action failed line, no signal or kill wording, and nothing about the runner at all.

So the two endings have opposite evidential standing and must not inherit each other's support:

• The O(files) accumulation and the inert budget stand on instrumented arms here and are independently corroborated by a second lane.
• The action kill is real and is n=1 — the arm taken for this PR.
• The client-stream-lost ending — n=2 confirmed — is what actually cost the reporting lanes their time, and is not explained here and not a fact about this runner or this compiler. (Earlier runs from that lane were described before the separating field existed and have not been re-graded, so they are unclassified candidates, not members. The counts are given exactly because an inflated one is what produced the merge.)

Standing consequence — actionable now, and it does not wait on an explanation

A log that ends without an exit-code line and without an Action failed: line is not evidence of a failed run. The observation ended before the subject did, so the runner-side outcome is unobserved, not bad. The opposite reading has been in live use — lanes treating these as runner failures and drawing conclusions about their own branches from them. Where such a conclusion has survived, check why before keeping it: a two-arm comparison in which both arms hit this ending is still sound, since a control does not need the phenomenon named to do its job — but that soundness is a property of the comparison's design, never of the log's readability.

The trap that produced the merge is recorded too, because this PR was caught by it: a supplied cause overwrites an accurate agnostic reading. "Control never returned and the stream was cut" was correctly agnostic; "a killed process cannot report its own death" was then accepted for runs where it had never been established, having been established for a different run entirely.

What was asked

Explain why .dag witness execution dies silently on a BuildBuddy runner — exit 255, log cut mid-stream, no returned status even with RUST_BACKTRACE demonstrably live — with a discriminating arm behind the explanation, or an honest "cannot be established from outside the runner" naming what was ruled out.

Answer

The signature is reproduced, and the kill happens at two grains — only one of which is loud.

Three dispatches of the same subject, each run to its own conclusion rather than stopped by a client cap:

ending	exit-code line	Action failed	trailer	unconditional echo
2 dispatches: guest OOM kills the child	present (0)	—	✓	✓ printed 137
1 dispatch: executor kills the action	absent	signal: killed	✓	never printed

The third is the signature this item was opened on: no status, log cut, wrapper 255, and RUST_BACKTRACE with nothing to print — because control never returned to a shell that no longer existed. The guest's OOM killer selects a child and that ending is fully reported; the executor kills the action and that ending can report nothing from inside, because there is no inside left. The severing run thrashed sustainedly first: 8,822,792 major faults, 659s of system time.

So the silent cut is memory-driven — the in-guest OOM killer is simply not what delivers it.

Two corrections this PR makes against its own earlier commits

Both are kept in the receipt rather than quietly replaced, because each was plausible and each came from generalising over the runs that existed at the time.

1. "Exhaustion here is loud, so it is not what cuts the log." Wrong — generalised from the first two dispatches before the third existed.
2. The trailer detector is withdrawn — and there are three endings, not two. A peer lane's independent n=6 showed two silent runs with no trailer, raising whether I had retired a working detector on a case that wasn't a killed action. Settled against the raw capture: my severing log is the direct client redirect (never a fetched server view), and its trailer follows Action failed: signal: killed by five lines — the client outlived the action. So the withdrawal stands, and their data adds the missing ending:

ending	exit-code line	Action failed	trailer
command reached an ending	present	—	✓
executor killed the action	absent	signal: killed	✓
the client stream was lost	absent	absent	✗

The third is real and I produced it accidentally, via a caller-side timeout on the reading end — which is exactly what made me mistake trailer-absence for action-severance. So a missing trailer is a fact about the observer, not the action; a present trailer proves nothing about whether a command ran.

The wrapper's status is transport-shaped in both directions and is never the answer: 0 over an action whose payload died at 137 (correct shell semantics — a trailing echo succeeded), 255 over the killed action.

Draw counts, stated because they are small

The loud ending is n=2; the silent ending is n=1. That establishes the silent ending exists and identifies its grain — which is what this PR claims — and does not establish which conditions select between the two endings. The sustained-thrash difference is written as the distinguishing feature of one specimen, not as a rule. Every reversal above came from treating a measured result as a replicated one.

What is still open

Not what happens but why the executor kills the action — a health check, a ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot tell them apart. The deciding instrument is the executor's server-side record of a severed action, reachable today by the invocation URL every dispatch prints. Acceptance half: the record must be for an invocation whose client log carries Action failed: signal: killed and no exit-code line — and the severed invocation is already identified (94352bd4-e91b-440a-9a43-f7daa96ee1dd), so the trigger does not require reproducing the phenomenon again. What is missing is only access: the bb client here exposes no invocation-fetch subcommand, so the record must be read through the executor's API by someone holding it. That is why this stays a trigger, not a task. An in-guest supervisor is refused by construction — this PR shows that limit from both sides, since the sampler survived a child SIGKILL well enough to establish the loud ending and reported nothing at all about the action's own death.

The budget knob is inert

Where the memory actually goes

Read off the [floor-phase] lines rather than inferred from the ending. Preparation is bounded and invariant under both the roster flag and the budget:

run	warm rss_growth	source_files
single --function (completed, rc=0)	819,593,216 B	4475
--roster-from-discovery @ 2 GiB (died)	820,158,464 B	4475
--roster-from-discovery @ 1 GiB (died)	776,818,688 B	4475

Every later preparation phase matches across all three (modules=3155, declarers=17, declarer-closure sources=18, closure-strict-resolve sources=18). The unbounded growth is after preparation and across witness files: --roster-from-discovery walks every discovered witness file in the corpus, and the resident set rises file after file without returning.

So the entry was never the variable — the same entry at the same budget with a single --function completes and returns zero. This also reconciles the two lanes: snappy-koi-879's observation that the edge-index warm runs before any roster narrowing is correct, but the warm completes at ~0.8 GiB and is not the killer; their HostBudgetUnreadable was a different failure that fires in the first phase regardless of roster, so their narrowing was never reached.

Two controls, each converting an absence into evidence

The guest OOM killer is loud. A child deliberately allocating past the cap prints Killed, the shell survives, an unconditional status echo reports 137, the end marker prints, and the action's trailer prints. A child SIGKILL does not sever the action — so a log that simply stops is not an in-guest OOM. The action itself is being destroyed.

A completed action emits a protocol terminator; a severed one does not. BuildBuddy prints the command's exit code line followed by Remote run completed at <ts>. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control, and absent on a severed run. This is the detector to use, and it needs nothing from the payload.

One correction to the "the wrapper fails toward green" framing: a wrapper exit of 0 is sometimes correct shell semantics, not a bug — in the OOM control the payload died at 137 and the script still exited 0 because the trailing echo succeeded. The defect is specifically the severed case, where no trailer exists and the status is arbitrary. The earlier framing would have sent someone hunting a bug in the wrong layer.

The stall names its instrument

"Not establishable from outside the runner" is a legitimate terminal answer and an untracked stall if it stops there. The deciding instrument is the executor's own server-side record of a severed action — its terminal disposition and its worker's — set beside the client's truncated view of the same run. It is reachable today and simply was not taken: every dispatch prints its invocation URL, and the receipts already in this module cite invocations by UUID, so a severed run is addressable by identity. That comparison is what separates the executor destroyed the action from the client lost the stream — two causes with different owners that the client's view cannot tell apart.

Acceptance half, so no lookalike retires it: the record must be for an invocation whose client log lacks the trailer, and must report that action's own terminal status.

Two neighbours are named only to be refused. An in-guest supervisor cannot answer it by construction — anything watching from inside dies with the guest; this PR shows that limit from the other side, since the in-guest heartbeat survives a child SIGKILL well enough to establish the loudness finding and would report nothing if the action were destroyed. A fast reproducer is blocked, not unbuilt: severance did not occur in any dispatch taken here, so there is no observed handle to shrink.

Where the specimen lands against the module's own verdict

A third dispatch sampled /proc/PID/stat. The run traverses both arms of memory_stall_verdict in a single execution:

window	major-fault rate (threshold 6000/min)	user-CPU share (floor 2000 bp)	verdict
early	0/min	9473 bp	ProgressUnderMemoryAdmissible
terminal (86s at the wall)	272,895/min	340 bp	StallRefusedPageThrash

Both conjuncts hold terminally, so the module's arm fires. That is stronger evidence for the conjunction than either arm alone, and it vindicates measuring user CPU specifically: at the wall the process is busy, but the time is system time servicing its own faults — a user-plus-system share would have read as healthy throughput and admitted the thrash.

I reported this the other way round mid-investigation, from the early window alone, and was wrong; the full curve inverts it.

The change

One receipt row on gunbc.memory_stall_refusal, the module that already owns this class — as the parent asked, the specimen goes to that module rather than opening a class beside it. It records the inert-knob measurement, the localization, both controls, and the reading rule, and states plainly that setting the override to clear a HostBudgetUnreadable refusal trades a loud typed answer for an unbounded run — the escape hatch DESIGN §5 forbids.

Verification

Every number above is from an executed dispatch on tree 4e6a597de0 with the runner echoing DIRTY=0; no figure is transcribed from another session. The --roster-from-discovery arms were run to the wall rather than stopped by a client-side cap — an earlier reading of mine was truncated by my own tool timeout and is not used as evidence.

Local gunbc in this container is stale (it cannot parse // annotations and errors on files this branch never touches). The edit was checked against that: the baseline errors on the same line content, shifted by exactly the two inserted lines, so this change introduces no new refusal. CI is the real check.

Follow-up, not in this PR

transport_close_read_as_completion in gunbc.recurring_failure_mode deserves this specimen — with the refinement that here the protocol event exists and is simply not consulted, so the remedy is available rather than blocked. That carrier projects into docs/design-ledgers.md and needs a regen round, so it belongs in its own PR. The reading rule is recorded in this receipt meanwhile, where a lane debugging this will actually hit it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6

[gunbai-bot]

The floor red is not this PR — it reproduces on main

required-witnesses-floor failed, and the aggregate witnesses job is downstream of it. There is no fix to push here, and I would rather say that than invent a change.

The floor refused with zero failures. From this run's own footer:

required-floor: planned=3489 executed=3489 terminal=3489 passed=3413 known_red_held=26
                failed=0 interrupted_before_verdict=1 interrupted_cpu_deadline=1
required-floor: INTERRUPTED-BEFORE-VERDICT
  test.claim.self_host_compile_phase_live_gate_witness.an_unreached_measurement_yields_no_observation_at_all
  raised_by=cpu_deadline cpu_at_least=502ms/500ms wall_at_least=504ms/8000ms
required-floor: verdict=FloorRefused unexpected_failures=0

One witness crossed the 500ms CPU line by 2ms (0.4%) and went undecided. Nothing failed.

Baseline check — the same failure occurs on main, without this diff:

run	branch	witness	cpu
33659826911	main	an_unreached_measurement_yields_no_observation_at_all	501ms/500ms
33656168449	main	same family, 6 interruptions	~501ms/500ms
33661155355	this PR	an_unreached_measurement_yields_no_observation_at_all	502ms/500ms

Each shows failed=0 with verdict=FloorRefused from CPU-deadline interruptions alone.

Why it cannot be this PR. The whole diff is dag/gunbc/host/memory_stall_refusal.dag | 2 ++ — two lines of prose in one receipt string, no logic, no types, no substrate. It cannot change the CPU time of self_host_compile_phase_live_gate_witness.

The underlying condition, visible in the same log: that witness family runs at 455–465ms against a 500ms line — roughly 91% of budget — with its siblings already flagged [over-cost]. A family sitting that close to the line crosses it on ordinary runner variance, which is why the interruption count moves between runs (1 here, 6 on one main run) while the population does not. That is a cost-line issue owned by the witness family, not a correctness signal, and it reds any PR that happens to run while it is marginal.

Rerunning the failed job once the in-progress lane finishes; the red is variance, not content.

— sent from proud-bear-215