From a6c48be838075a296bb1bba5fa74c43cb27224f9 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Wed, 2 Sep 2026 16:34:41 +0000 Subject: [PATCH 1/8] The budget knob is not a knob: a third specimen that retires the variable instead of narrowing its range gunbc.memory_stall_refusal carried two receipts bracketing GUNBC_MEMORY_BUDGET_BYTES from opposite sides -- one truthful declaration that thrashed, one overstated one that was killed -- and concluded that no good NUMBER exists on a 7 GiB runner. This adds the specimen that varies the number as its only input, and finds the knob is not connected to the quantity that ends the run at all: 2147483648 declared produced 6419416 kB resident, 1073741824 declared produced 6416112 kB. Five hundredths of one percent apart, at three and six times the declaration. That is stronger than an empty range, and it changes what a blocked lane should do. The override answers gunbc.host_budget_source's SOURCE question -- may a run begin -- and constrains nothing allocated afterwards, so setting it to clear a HostBudgetUnreadable refusal trades a loud, typed, immediate answer for an unbounded run: the escape hatch DESIGN section 5 forbids. The receipt also localizes the growth, off the floor-phase lines rather than off the ending. Preparation is bounded and invariant under both the roster flag and the budget -- the bare-reference edge-index warm completes at source_files=4475 for ~0.8 GiB in a completing run and in both dying arms alike, and every later preparation phase matches across all three. The unbounded growth is after preparation and across witness FILES, so the entry was never the variable: the same entry at the same budget with a single --function completes and returns zero. Two controls are recorded because each converts an absence into evidence. The guest OOM killer is LOUD here -- a child allocating past the cap prints Killed, the shell survives, an unconditional status echo reports 137, and the action's trailer prints -- so a log that simply stops is not an in-guest OOM. And a BuildBuddy action that completed emits a protocol terminator that a severed one does not, measured present on a genuine 255, a genuine 0 and the OOM control, and absent on a severed run; the wrapper's own status is transport-shaped and answers neither direction. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6 --- dag/gunbc/host/memory_stall_refusal.dag | 2 ++ 1 file changed, 2 insertions(+) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index 002b3ebf13a..8deea0fbec7 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,6 +24,8 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked." + data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." data memory_stall_admitted_under_pressure_receipt_note: String = "THE MEASURED CONTROL THAT FORCED THE CPU CONJUNCT, found by this refusal's own first execution rather than by reasoning. On the first CI run carrying a rate-only form of this verdict (gunbc PR #9755, run 33319823294, required-witnesses-floor, srv3 actions-runner slice, budget 15 GiB from cgroup memory.high), the arm fired at 12431 major faults/minute sustained over a 788-second window -- one 13-minute typecheck of std.constructors under the slice's memory.high reclaim throttle. That configuration is not a treadmill: the same floor completes green on main on the same runners day after day. A resolve can sustain hundreds of refaults per second under a reclaim throttle and still be spending most of its wall COMPUTING -- which is exactly the slow-but-progressing state this module is forbidden to refuse. Rate alone cannot separate the two; what separates them is where the wall goes. A progressing run under pressure still burns CPU between faults; the treadmill's defining property is that its CPU share of wall is near zero, because the process sits in disk wait re-reading its own text. So the refusal is a CONJUNCTION: the fault rate above its line AND the process's own CPU share of the window below its floor. Both halves are the process observing itself; neither is host-shared; neither is a timeout." From 243432ec09bec2bae662d9de61e446311918914e Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Wed, 2 Sep 2026 16:59:47 +0000 Subject: [PATCH 2/8] The ending refutes the hypothesis: exhausting this runner is LOUD, so it is not what cuts the log Both roster arms were run to their own conclusion rather than stopped by a client-side cap, and both ended the same way: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and the action's completion trailer printed. On both budgets, matching the synthetic control exactly. That refutes the explanation this investigation started from -- that the process tree is SIGKILLed and a killed process cannot report its own death. Exhausting this runner with the real workload returns a status and a complete log every time. So memory exhaustion is real on this path and is NOT the mechanism of a run whose log simply stops; the two failures are different, and a silent cut must stop being read as evidence of an out-of-memory condition. What destroys the action remains unestablished from outside the runner. The receipt now says that plainly rather than letting the exhaustion finding stand in for an answer it does not give. A third dispatch sampled /proc/PID/stat and places the specimen against this module's own verdict, which it traverses in both directions in a single run. Early: major faults flat at zero, user-CPU share 9473 bp -- correctly admitted as progress. Terminally, over an 86-second window: 391150 major faults for 272895/min against the 6000 threshold, user-CPU share collapsed to 340 bp against the 2000 floor -- both conjuncts hold, StallRefusedPageThrash. That is stronger evidence for the conjunction than either arm alone, and it vindicates the decision to measure USER cpu specifically: at the wall the process is busy, but the time is system time servicing its own faults, so a user-plus-system share would have read as healthy throughput and admitted the thrash. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6 --- dag/gunbc/host/memory_stall_refusal.dag | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index 8deea0fbec7..c5c44ab8104 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,7 +24,7 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." -data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING REFUTES THE HYPOTHESIS THAT SENT US HERE, WHICH IS THE MOST USEFUL THING THIS RECEIPT CARRIES. Both arms were run to their own conclusion rather than stopped by a client-side cap, and both ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and the action's completion trailer printed. The leading explanation for the silent deaths this receipt was opened to explain was that the process tree is SIGKILLed and a killed process cannot report its own death. THAT IS NOT WHAT HAPPENS HERE. Exhausting this runner with the real workload yields a returned status and a complete log every time, on both budgets, and matches the synthetic control exactly. So memory exhaustion is REAL on this path and is NOT the mechanism of a run whose log simply stops -- the two failures are different, and a lane must stop reading a silent cut as evidence of an out-of-memory condition. What destroys the ACTION remains unestablished from outside the runner; this receipt narrows the question rather than closing it, and records the exhaustion finding on its own terms.\n\nWHERE THE SPECIMEN LANDS AGAINST THIS MODULE'S OWN VERDICT, measured rather than assumed, and it traverses BOTH arms in one run. A third dispatch sampled the same subject's /proc/PID/stat every five seconds. EARLY, the run is genuinely computing and this module correctly declines to refuse it: major faults flat at ZERO with a user-CPU share of 9473 basis points. TERMINALLY, over an 86-second window at the wall, it becomes exactly the treadmill this module exists to stop: 391150 major faults for a rate of 272895 per minute against the 6000 threshold, while the user-CPU share collapses to 340 basis points against the 2000 floor. Both conjuncts hold, so memory_stall_verdict returns StallRefusedPageThrash. THE CONJUNCTION DISCRIMINATES CORRECTLY IN BOTH DIRECTIONS ON ONE SPECIMEN, which is stronger evidence for it than either arm alone, and the choice to measure USER cpu specifically is what makes it work: at the wall the process is busy, but the time is SYSTEM time servicing its own faults, so a share computed over user-plus-system would have read as healthy throughput and admitted the thrash. The sampler's own interval stretching from five seconds to about twelve under load is the same phenomenon seen from outside." data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." From 368695da29db89a45cbdbece2d0392b152727bce Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Wed, 2 Sep 2026 17:02:27 +0000 Subject: [PATCH 3/8] Name the instrument that would decide the severance, and refuse the two that cannot The receipt stopped at "not establishable from outside the runner", which is a legitimate terminal answer and an untracked stall if it names no next instrument (DESIGN 4b(2)). It now names one, and it is reachable today rather than aspirational: the executor's own server-side record of a SEVERED action, read for that action's terminal disposition and its worker's, set beside the client's truncated view of the same run. Every dispatch prints its invocation URL and the receipts above already cite invocations by UUID, so a severed run is addressable by identity -- nobody has simply fetched the server's side of one. That comparison is what separates the executor destroying the action from the client losing the stream: two causes, different owners, indistinguishable from the client alone. The acceptance half is stated so no lookalike retires it: the record must be for an invocation whose CLIENT log lacks the completion trailer, and must report that action's own terminal status. A record for a run that completed proves nothing about severance. Two neighbouring instruments are named only to be refused, so they are not proposed again. An in-guest supervisor cannot answer it by construction -- anything watching from inside dies with the guest, and this receipt demonstrates that limit from the other side, since the in-guest heartbeat survives a child SIGKILL well enough to establish the loudness finding and would report nothing if the action itself were destroyed. And a fast reproducer is BLOCKED rather than unbuilt: severance did not occur in any dispatch taken here, so there is no observed handle to shrink, and naming it as the next step would name work that cannot start. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6 --- dag/gunbc/host/memory_stall_refusal.dag | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index c5c44ab8104..7fe75bad6d8 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,7 +24,7 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." -data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING REFUTES THE HYPOTHESIS THAT SENT US HERE, WHICH IS THE MOST USEFUL THING THIS RECEIPT CARRIES. Both arms were run to their own conclusion rather than stopped by a client-side cap, and both ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and the action's completion trailer printed. The leading explanation for the silent deaths this receipt was opened to explain was that the process tree is SIGKILLed and a killed process cannot report its own death. THAT IS NOT WHAT HAPPENS HERE. Exhausting this runner with the real workload yields a returned status and a complete log every time, on both budgets, and matches the synthetic control exactly. So memory exhaustion is REAL on this path and is NOT the mechanism of a run whose log simply stops -- the two failures are different, and a lane must stop reading a silent cut as evidence of an out-of-memory condition. What destroys the ACTION remains unestablished from outside the runner; this receipt narrows the question rather than closing it, and records the exhaustion finding on its own terms.\n\nWHERE THE SPECIMEN LANDS AGAINST THIS MODULE'S OWN VERDICT, measured rather than assumed, and it traverses BOTH arms in one run. A third dispatch sampled the same subject's /proc/PID/stat every five seconds. EARLY, the run is genuinely computing and this module correctly declines to refuse it: major faults flat at ZERO with a user-CPU share of 9473 basis points. TERMINALLY, over an 86-second window at the wall, it becomes exactly the treadmill this module exists to stop: 391150 major faults for a rate of 272895 per minute against the 6000 threshold, while the user-CPU share collapses to 340 basis points against the 2000 floor. Both conjuncts hold, so memory_stall_verdict returns StallRefusedPageThrash. THE CONJUNCTION DISCRIMINATES CORRECTLY IN BOTH DIRECTIONS ON ONE SPECIMEN, which is stronger evidence for it than either arm alone, and the choice to measure USER cpu specifically is what makes it work: at the wall the process is busy, but the time is SYSTEM time servicing its own faults, so a share computed over user-plus-system would have read as healthy throughput and admitted the thrash. The sampler's own interval stretching from five seconds to about twelve under load is the same phenomenon seen from outside." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING REFUTES THE HYPOTHESIS THAT SENT US HERE, WHICH IS THE MOST USEFUL THING THIS RECEIPT CARRIES. Both arms were run to their own conclusion rather than stopped by a client-side cap, and both ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and the action's completion trailer printed. The leading explanation for the silent deaths this receipt was opened to explain was that the process tree is SIGKILLed and a killed process cannot report its own death. THAT IS NOT WHAT HAPPENS HERE. Exhausting this runner with the real workload yields a returned status and a complete log every time, on both budgets, and matches the synthetic control exactly. So memory exhaustion is REAL on this path and is NOT the mechanism of a run whose log simply stops -- the two failures are different, and a lane must stop reading a silent cut as evidence of an out-of-memory condition. What destroys the ACTION remains unestablished from outside the runner; this receipt narrows the question rather than closing it, and records the exhaustion finding on its own terms.\n\nAND THE STALL NAMES ITS INSTRUMENT RATHER THAN STOPPING AT NOT-ESTABLISHABLE, because a class that halts at the limit of the instruments someone happened to hold becomes permanent by default. THE DECIDING INSTRUMENT IS THE EXECUTOR'S OWN SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, and compared against the client's view of the same run. It is reachable today and simply was not taken: every dispatch prints its own invocation URL on the client, and the receipts above already cite invocations by UUID, so a severed run is addressable by identity. What is missing is that nobody has fetched the SERVER's record for one and set it beside the truncated client stream -- which is exactly the comparison that separates the executor destroying the action from the client losing the stream, two causes with different owners that the client's view cannot tell apart. THE ACCEPTANCE HALF, so no lookalike retires this: the record must be for an invocation whose CLIENT log lacks the completion trailer, and it must report that action's own terminal status. A record for a run that completed proves nothing about severance.\n\nTWO NEIGHBOURING INSTRUMENTS ARE NAMED ONLY TO BE REFUSED, so they are not proposed again. AN IN-GUEST SUPERVISOR CANNOT ANSWER IT BY CONSTRUCTION: anything watching from inside dies with the guest, and this receipt already demonstrates the limit of that approach from the other side -- the in-guest heartbeat survives a child SIGKILL perfectly well, which is how the loudness above was established, and would report nothing at all if the action itself were destroyed. A FAST REPRODUCER IS BLOCKED RATHER THAN UNBUILT: severance did not occur in any dispatch taken for this receipt, all of which ended loudly, so there is no observed handle to shrink. A reproducer cannot be built for a phenomenon that did not reproduce, and claiming one as the next step would name work that cannot start.\n\nWHERE THE SPECIMEN LANDS AGAINST THIS MODULE'S OWN VERDICT, measured rather than assumed, and it traverses BOTH arms in one run. A third dispatch sampled the same subject's /proc/PID/stat every five seconds. EARLY, the run is genuinely computing and this module correctly declines to refuse it: major faults flat at ZERO with a user-CPU share of 9473 basis points. TERMINALLY, over an 86-second window at the wall, it becomes exactly the treadmill this module exists to stop: 391150 major faults for a rate of 272895 per minute against the 6000 threshold, while the user-CPU share collapses to 340 basis points against the 2000 floor. Both conjuncts hold, so memory_stall_verdict returns StallRefusedPageThrash. THE CONJUNCTION DISCRIMINATES CORRECTLY IN BOTH DIRECTIONS ON ONE SPECIMEN, which is stronger evidence for it than either arm alone, and the choice to measure USER cpu specifically is what makes it work: at the wall the process is busy, but the time is SYSTEM time servicing its own faults, so a share computed over user-plus-system would have read as healthy throughput and admitted the thrash. The sampler's own interval stretching from five seconds to about twelve under load is the same phenomenon seen from outside." data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." From deea5a1ae8b27d43a6b4e792160ddd628136b183 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Wed, 2 Sep 2026 17:15:07 +0000 Subject: [PATCH 4/8] The third dispatch severs: the signature IS reproduced, and this receipt's own detector was wrong A third run of the same subject ended a third way, and it overturns two claims this receipt made one commit ago. Both corrections are recorded rather than quietly replaced, because each was plausible and each was reached by generalising from the runs that existed at the time. FIRST: the silent signature reproduced. Two dispatches ended loudly -- guest OOM killer, Killed naming the binary, unconditional echo reporting 137, an ordinary exit-code line -- and the third ended with no exit-code line at all, "Action failed: signal: killed" in its place, the unconditional echo NEVER printing, and the wrapper returning 255. That is the signature this receipt was opened to explain. So the kill happens at two grains and only one is loud: the guest OOM killer selects a CHILD and that ending is fully reported; the executor kills the ACTION and that ending can report nothing from inside, because there is no inside left. The previous claim -- exhaustion here is loud and therefore is not what cuts a log -- generalised from the first two dispatches before the third existed. The silent cut IS memory-driven; the in-guest OOM killer is simply not what delivers it. The severing run thrashed sustainedly first: 8822792 major faults and 659 seconds of system time. SECOND: the trailer detector was wrong and is withdrawn. The completion trailer is emitted in EVERY case measured -- genuine nonzero exit, clean exit, child OOM kill, killed action alike -- so its presence proves nothing, and offering its absence as the severance signal was a mistake. The discriminator is the EXIT-CODE LINE: present and numeric when a command reached an ending, replaced by "Action failed: signal: killed" when the action was killed. A protocol-looking line can still be transport-shaped, and the way to find out is to build the failure on purpose and watch which lines survive it. The open question narrows accordingly: not what happens, but why the executor kills the action, which the guest cannot see. The server-side record instrument stands, with its acceptance half re-pinned to the corrected signature. The fast-reproducer refusal is withdrawn -- there is one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6 --- dag/gunbc/host/memory_stall_refusal.dag | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index 7fe75bad6d8..56905dcc38d 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,7 +24,7 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." -data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING REFUTES THE HYPOTHESIS THAT SENT US HERE, WHICH IS THE MOST USEFUL THING THIS RECEIPT CARRIES. Both arms were run to their own conclusion rather than stopped by a client-side cap, and both ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and the action's completion trailer printed. The leading explanation for the silent deaths this receipt was opened to explain was that the process tree is SIGKILLed and a killed process cannot report its own death. THAT IS NOT WHAT HAPPENS HERE. Exhausting this runner with the real workload yields a returned status and a complete log every time, on both budgets, and matches the synthetic control exactly. So memory exhaustion is REAL on this path and is NOT the mechanism of a run whose log simply stops -- the two failures are different, and a lane must stop reading a silent cut as evidence of an out-of-memory condition. What destroys the ACTION remains unestablished from outside the runner; this receipt narrows the question rather than closing it, and records the exhaustion finding on its own terms.\n\nAND THE STALL NAMES ITS INSTRUMENT RATHER THAN STOPPING AT NOT-ESTABLISHABLE, because a class that halts at the limit of the instruments someone happened to hold becomes permanent by default. THE DECIDING INSTRUMENT IS THE EXECUTOR'S OWN SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, and compared against the client's view of the same run. It is reachable today and simply was not taken: every dispatch prints its own invocation URL on the client, and the receipts above already cite invocations by UUID, so a severed run is addressable by identity. What is missing is that nobody has fetched the SERVER's record for one and set it beside the truncated client stream -- which is exactly the comparison that separates the executor destroying the action from the client losing the stream, two causes with different owners that the client's view cannot tell apart. THE ACCEPTANCE HALF, so no lookalike retires this: the record must be for an invocation whose CLIENT log lacks the completion trailer, and it must report that action's own terminal status. A record for a run that completed proves nothing about severance.\n\nTWO NEIGHBOURING INSTRUMENTS ARE NAMED ONLY TO BE REFUSED, so they are not proposed again. AN IN-GUEST SUPERVISOR CANNOT ANSWER IT BY CONSTRUCTION: anything watching from inside dies with the guest, and this receipt already demonstrates the limit of that approach from the other side -- the in-guest heartbeat survives a child SIGKILL perfectly well, which is how the loudness above was established, and would report nothing at all if the action itself were destroyed. A FAST REPRODUCER IS BLOCKED RATHER THAN UNBUILT: severance did not occur in any dispatch taken for this receipt, all of which ended loudly, so there is no observed handle to shrink. A reproducer cannot be built for a phenomenon that did not reproduce, and claiming one as the next step would name work that cannot start.\n\nWHERE THE SPECIMEN LANDS AGAINST THIS MODULE'S OWN VERDICT, measured rather than assumed, and it traverses BOTH arms in one run. A third dispatch sampled the same subject's /proc/PID/stat every five seconds. EARLY, the run is genuinely computing and this module correctly declines to refuse it: major faults flat at ZERO with a user-CPU share of 9473 basis points. TERMINALLY, over an 86-second window at the wall, it becomes exactly the treadmill this module exists to stop: 391150 major faults for a rate of 272895 per minute against the 6000 threshold, while the user-CPU share collapses to 340 basis points against the 2000 floor. Both conjuncts hold, so memory_stall_verdict returns StallRefusedPageThrash. THE CONJUNCTION DISCRIMINATES CORRECTLY IN BOTH DIRECTIONS ON ONE SPECIMEN, which is stronger evidence for it than either arm alone, and the choice to measure USER cpu specifically is what makes it work: at the wall the process is busy, but the time is SYSTEM time servicing its own faults, so a share computed over user-plus-system would have read as healthy throughput and admitted the thrash. The sampler's own interval stretching from five seconds to about twelve under load is the same phenomenon seen from outside." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." From 17ee884788e8600d07ddb7aaca60be7f38b0081b Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Wed, 2 Sep 2026 17:18:07 +0000 Subject: [PATCH 5/8] State the draw counts, and hand the trigger a severed invocation it can be fetched against Two additions, both aimed at the next reader rather than at the finding. The observation counts are now stated in the receipt because they are small: the loud ending is n=2 and the silent ending is n=1. That is enough to establish that the silent ending exists and to identify its grain, which is what the receipt claims. It is not enough to establish which conditions select between the two endings, so the sustained-thrash difference is written as the distinguishing feature of one specimen rather than as a rule. Every reversal recorded in this receipt came from treating a measured result as a replicated one, and the counts are the cheapest guard against the next reader repeating that. And the next-instrument trigger no longer requires reproducing the phenomenon before it can be used. The severed invocation is identified -- 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries "Action failed: signal: killed" and no exit-code line, which is exactly the acceptance half. What is missing is only access: the bb client here exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access. That is why this stays a trigger and not a task. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6 --- dag/gunbc/host/memory_stall_refusal.dag | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index 56905dcc38d..2de7f9578b3 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,7 +24,7 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." -data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest. AND THE OBSERVATION COUNTS ARE STATED BECAUSE THEY ARE SMALL: the loud ending is n=2 and THE SILENT ENDING IS n=1. One dispatch is one draw. That is enough to establish that the silent ending EXISTS and to identify its grain, which is what this receipt claims; it is NOT enough to establish which conditions select between the two endings, and the sustained-thrash difference named above is the distinguishing feature of a single specimen rather than a rule. A reader deciding anything about frequency needs more draws than this receipt has, and every reversal recorded in it came from treating a measured result as a replicated one.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE SEVERED INVOCATION IS ALREADY IDENTIFIED so the trigger does not require reproducing the phenomenon again: 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries Action failed: signal: killed and no exit-code line. WHAT IS MISSING IS ONLY ACCESS: the bb client in this environment exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access, which is why this is a trigger and not a task. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." From 161863058b5592b8634783c2351802b14c9c0588 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Wed, 2 Sep 2026 17:20:25 +0000 Subject: [PATCH 6/8] Three endings, not two: the trailer's absence is a fact about the observer, not the action A peer lane's independent n=6 shows two silent runs with NO trailer, which is the opposite of what the severing run here measured, and raised the question of whether the withdrawn trailer detector had been retired on a case that was not a killed action. Settled against the raw capture: the severing arm's log is the direct client redirect, never a fetched server view, and its trailer follows the "Action failed: signal: killed" line by five lines. The client outlived the action it was running, so the trailer genuinely prints for a killed action and the withdrawal stands. What their data does establish is that the taxonomy was one ending short. There are three: command reached an ending exit-code line PRESENT, trailer PRESENT executor killed the action no exit-code line, "Action failed: ", trailer PRESENT client stream itself lost none of the three The third is real and was produced accidentally here by a caller-side timeout on the reading end, which is exactly what made an earlier draft mistake trailer-absence for action-severance. So a missing trailer means THE CLIENT did not finish -- a fact about the observer, not about the action -- while a present trailer proves nothing about whether any command ran. Only the exit-code line answers that, and it answers in both directions, which is why it is the discriminator and the trailer is not. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6 --- dag/gunbc/host/memory_stall_refusal.dag | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index 2de7f9578b3..4d8055b4884 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,7 +24,7 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." -data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest. AND THE OBSERVATION COUNTS ARE STATED BECAUSE THEY ARE SMALL: the loud ending is n=2 and THE SILENT ENDING IS n=1. One dispatch is one draw. That is enough to establish that the silent ending EXISTS and to identify its grain, which is what this receipt claims; it is NOT enough to establish which conditions select between the two endings, and the sustained-thrash difference named above is the distinguishing feature of a single specimen rather than a rule. A reader deciding anything about frequency needs more draws than this receipt has, and every reversal recorded in it came from treating a measured result as a replicated one.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE SEVERED INVOCATION IS ALREADY IDENTIFIED so the trigger does not require reproducing the phenomenon again: 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries Action failed: signal: killed and no exit-code line. WHAT IS MISSING IS ONLY ACCESS: the bb client in this environment exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access, which is why this is a trigger and not a task. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest. AND THE OBSERVATION COUNTS ARE STATED BECAUSE THEY ARE SMALL: the loud ending is n=2 and THE SILENT ENDING IS n=1. One dispatch is one draw. That is enough to establish that the silent ending EXISTS and to identify its grain, which is what this receipt claims; it is NOT enough to establish which conditions select between the two endings, and the sustained-thrash difference named above is the distinguishing feature of a single specimen rather than a rule. A reader deciding anything about frequency needs more draws than this receipt has, and every reversal recorded in it came from treating a measured result as a replicated one.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. AND THE TRAILER'S ABSENCE MEANS A THIRD THING, WHICH IS WHY IT IS NOT THE DISCRIMINATOR IN EITHER DIRECTION. There are three endings, not two. A command that reached an ending emits an exit-code line AND a trailer. AN ACTION KILLED BY THE EXECUTOR EMITS NO EXIT-CODE LINE, an Action failed line naming the signal, AND STILL A TRAILER -- measured in the raw client capture of this receipt's severing run, where the trailer follows the Action failed line by five lines, so the client outlived the action it was running. A CLIENT STREAM THAT WAS ITSELF LOST emits none of the three. That last ending is real and was produced accidentally here by a caller-side timeout on the reading end, which is precisely what made an earlier draft mistake trailer-absence for action-severance. So: trailer ABSENT means THE CLIENT did not finish, which is a fact about the observer and not about the action; trailer PRESENT proves nothing about whether any command ran. Only the exit-code line answers that, and it answers it in both directions. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE SEVERED INVOCATION IS ALREADY IDENTIFIED so the trigger does not require reproducing the phenomenon again: 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries Action failed: signal: killed and no exit-code line. WHAT IS MISSING IS ONLY ACCESS: the bb client in this environment exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access, which is why this is a trigger and not a task. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." From b9e407fc1c30585dfb7deb40ba7b426b4bb60715 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Wed, 2 Sep 2026 17:23:40 +0000 Subject: [PATCH 7/8] The class this began as was two classes: graded on the separating field, the reported runs are all client-stream-lost The separating field decided it. The runs that motivated this investigation carry no "Action failed:" line, no signal or kill wording, and nothing about the runner at all -- they are the third ending, CLIENT STREAM LOST. The action-kill ending has exactly one confirmed instance and it is the arm taken here. That matters because the original signature was established over exit-code-line, marker and log-stops, and the second and third endings SHARE all three. The field that separates them was not among them, so two classes were counted as one and the merged class became the subject. The receipt now says where the merge was rather than inheriting the framing. The consequence is that the two endings must not inherit each other's support. The O(files) accumulation and the inert budget stand on instrumented arms and are independently corroborated. The action kill is real and n=1. And the ending that actually cost the reporting lanes their time is the client-stream-lost one, which this receipt does not explain and which is not a fact about this runner or this compiler at all: such a log establishes nothing about what happened on the far side, because the observation ended before the subject did. A lane holding one has measured a failed measurement, not a failed run, and the runner-side outcome is UNOBSERVED rather than bad. Recorded with it is the trap that produced the merge, because this receipt was caught by it too: a supplied cause overwrites an accurate agnostic reading. "Control never returned and the stream was cut" was correctly agnostic between the endings; "a killed process cannot report its own death" was then accepted for runs where it had never been established, having been established for a different run entirely. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6 --- dag/gunbc/host/memory_stall_refusal.dag | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index 4d8055b4884..8dedfd87c7c 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,7 +24,7 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." -data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest. AND THE OBSERVATION COUNTS ARE STATED BECAUSE THEY ARE SMALL: the loud ending is n=2 and THE SILENT ENDING IS n=1. One dispatch is one draw. That is enough to establish that the silent ending EXISTS and to identify its grain, which is what this receipt claims; it is NOT enough to establish which conditions select between the two endings, and the sustained-thrash difference named above is the distinguishing feature of a single specimen rather than a rule. A reader deciding anything about frequency needs more draws than this receipt has, and every reversal recorded in it came from treating a measured result as a replicated one.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. AND THE TRAILER'S ABSENCE MEANS A THIRD THING, WHICH IS WHY IT IS NOT THE DISCRIMINATOR IN EITHER DIRECTION. There are three endings, not two. A command that reached an ending emits an exit-code line AND a trailer. AN ACTION KILLED BY THE EXECUTOR EMITS NO EXIT-CODE LINE, an Action failed line naming the signal, AND STILL A TRAILER -- measured in the raw client capture of this receipt's severing run, where the trailer follows the Action failed line by five lines, so the client outlived the action it was running. A CLIENT STREAM THAT WAS ITSELF LOST emits none of the three. That last ending is real and was produced accidentally here by a caller-side timeout on the reading end, which is precisely what made an earlier draft mistake trailer-absence for action-severance. So: trailer ABSENT means THE CLIENT did not finish, which is a fact about the observer and not about the action; trailer PRESENT proves nothing about whether any command ran. Only the exit-code line answers that, and it answers it in both directions. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE SEVERED INVOCATION IS ALREADY IDENTIFIED so the trigger does not require reproducing the phenomenon again: 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries Action failed: signal: killed and no exit-code line. WHAT IS MISSING IS ONLY ACCESS: the bb client in this environment exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access, which is why this is a trigger and not a task. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest. AND THE OBSERVATION COUNTS ARE STATED BECAUSE THEY ARE SMALL: the loud ending is n=2 and THE SILENT ENDING IS n=1. One dispatch is one draw. That is enough to establish that the silent ending EXISTS and to identify its grain, which is what this receipt claims; it is NOT enough to establish which conditions select between the two endings, and the sustained-thrash difference named above is the distinguishing feature of a single specimen rather than a rule. A reader deciding anything about frequency needs more draws than this receipt has, and every reversal recorded in it came from treating a measured result as a replicated one.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. AND THE TRAILER'S ABSENCE MEANS A THIRD THING, WHICH IS WHY IT IS NOT THE DISCRIMINATOR IN EITHER DIRECTION. There are three endings, not two. A command that reached an ending emits an exit-code line AND a trailer. AN ACTION KILLED BY THE EXECUTOR EMITS NO EXIT-CODE LINE, an Action failed line naming the signal, AND STILL A TRAILER -- measured in the raw client capture of this receipt's severing run, where the trailer follows the Action failed line by five lines, so the client outlived the action it was running. A CLIENT STREAM THAT WAS ITSELF LOST emits none of the three. That last ending is real and was produced accidentally here by a caller-side timeout on the reading end, which is precisely what made an earlier draft mistake trailer-absence for action-severance. So: trailer ABSENT means THE CLIENT did not finish, which is a fact about the observer and not about the action; trailer PRESENT proves nothing about whether any command ran. Only the exit-code line answers that, and it answers it in both directions. AND THE THREE-WAY SPLIT DISSOLVED THE CLASS THIS RECEIPT WAS OPENED ON, WHICH IS THE MOST IMPORTANT THING IT RECORDS. The investigation began from a report of several silent runs described as one signature, established over the fields exit-code-line, marker and log-stops -- all three of which the second and third endings SHARE. The field that separates them, the Action failed line, was not among them, so two classes were counted as one and the merged class was handed on as the subject. Graded afterwards on that field, the reported occurrences are ALL the third ending, CLIENT STREAM LOST: no Action failed line, no signal or kill wording, nothing about the runner at all. THE ACTION-KILL ENDING HAS EXACTLY ONE CONFIRMED INSTANCE AND IT IS THE ARM TAKEN FOR THIS RECEIPT.\n\nSO THE TWO ENDINGS HAVE OPPOSITE EVIDENTIAL STANDING AND MUST NOT INHERIT EACH OTHER'S SUPPORT. The O(files) accumulation and the inert budget stand on instrumented arms here and are independently corroborated. The action kill is real and is n=1. And the ending that actually cost the reporting lanes their time is the CLIENT-STREAM-LOST one, which this receipt does not explain at all and which is not a fact about this runner or this compiler: such a log establishes nothing whatever about what happened on the far side, because the observation ended before the subject did. A lane holding one has not measured a failed run; it has measured a failed measurement, and the runner-side outcome is UNOBSERVED rather than bad. THAT IS THE STANDING TRAP HERE AND IT IS THE ONE THIS RECEIPT WAS ITSELF CAUGHT BY: a supplied cause overwrites an accurate agnostic reading. The original report said control never returned and the stream was cut, which was correctly agnostic between the endings; the explanation that a killed process cannot report its own death was then accepted for runs where it had never been established, having been established for a different run entirely. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE SEVERED INVOCATION IS ALREADY IDENTIFIED so the trigger does not require reproducing the phenomenon again: 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries Action failed: signal: killed and no exit-code line. WHAT IS MISSING IS ONLY ACCESS: the bb client in this environment exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access, which is why this is a trigger and not a task. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." From d34ceb96da6236f2f19c59875fac91c7feb6a108 Mon Sep 17 00:00:00 2001 From: Brian Searls Date: Wed, 2 Sep 2026 17:26:58 +0000 Subject: [PATCH 8/8] Give the counts exactly, and promote the unobserved-not-bad reading to a standing consequence Two corrections to the correction, both aimed at not repeating the defect that produced the merged class in the first place. The count is now exact. Client-stream-lost is n=2 CONFIRMED -- two runs actually graded on the separating field. Earlier runs from the same lane were described before that field existed and have not been re-examined against it, so they are UNCLASSIFIED CANDIDATES, not members, and joining them would mean re-grading each. The previous wording said the reported occurrences were all the third ending, which asserted a grading that had only been performed on two of them. An inflated count is what produced the merge; a correction carrying the error's own defect is not a correction. And the operative sentence is promoted from background to a standing consequence, because it is actionable now and does not wait on any explanation of why a client stream is lost -- an explanation that may never arrive. A lane holding such a log has not measured a failed run; it has measured a failed measurement, and the runner-side outcome is UNOBSERVED rather than bad. The opposite reading was in live use, with lanes treating these as runner failures and drawing conclusions about their own branches from them. Where such a conclusion has survived, the receipt now says to check WHY before keeping it: a two-arm comparison in which both arms hit this ending is still sound, since a control does not need the phenomenon named, but that soundness is a property of the comparison's design and never of the log's readability. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01So6KE5WgyvEYENzwMtexS6 --- dag/gunbc/host/memory_stall_refusal.dag | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index 8dedfd87c7c..29de661db09 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,7 +24,7 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." -data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest. AND THE OBSERVATION COUNTS ARE STATED BECAUSE THEY ARE SMALL: the loud ending is n=2 and THE SILENT ENDING IS n=1. One dispatch is one draw. That is enough to establish that the silent ending EXISTS and to identify its grain, which is what this receipt claims; it is NOT enough to establish which conditions select between the two endings, and the sustained-thrash difference named above is the distinguishing feature of a single specimen rather than a rule. A reader deciding anything about frequency needs more draws than this receipt has, and every reversal recorded in it came from treating a measured result as a replicated one.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. AND THE TRAILER'S ABSENCE MEANS A THIRD THING, WHICH IS WHY IT IS NOT THE DISCRIMINATOR IN EITHER DIRECTION. There are three endings, not two. A command that reached an ending emits an exit-code line AND a trailer. AN ACTION KILLED BY THE EXECUTOR EMITS NO EXIT-CODE LINE, an Action failed line naming the signal, AND STILL A TRAILER -- measured in the raw client capture of this receipt's severing run, where the trailer follows the Action failed line by five lines, so the client outlived the action it was running. A CLIENT STREAM THAT WAS ITSELF LOST emits none of the three. That last ending is real and was produced accidentally here by a caller-side timeout on the reading end, which is precisely what made an earlier draft mistake trailer-absence for action-severance. So: trailer ABSENT means THE CLIENT did not finish, which is a fact about the observer and not about the action; trailer PRESENT proves nothing about whether any command ran. Only the exit-code line answers that, and it answers it in both directions. AND THE THREE-WAY SPLIT DISSOLVED THE CLASS THIS RECEIPT WAS OPENED ON, WHICH IS THE MOST IMPORTANT THING IT RECORDS. The investigation began from a report of several silent runs described as one signature, established over the fields exit-code-line, marker and log-stops -- all three of which the second and third endings SHARE. The field that separates them, the Action failed line, was not among them, so two classes were counted as one and the merged class was handed on as the subject. Graded afterwards on that field, the reported occurrences are ALL the third ending, CLIENT STREAM LOST: no Action failed line, no signal or kill wording, nothing about the runner at all. THE ACTION-KILL ENDING HAS EXACTLY ONE CONFIRMED INSTANCE AND IT IS THE ARM TAKEN FOR THIS RECEIPT.\n\nSO THE TWO ENDINGS HAVE OPPOSITE EVIDENTIAL STANDING AND MUST NOT INHERIT EACH OTHER'S SUPPORT. The O(files) accumulation and the inert budget stand on instrumented arms here and are independently corroborated. The action kill is real and is n=1. And the ending that actually cost the reporting lanes their time is the CLIENT-STREAM-LOST one, which this receipt does not explain at all and which is not a fact about this runner or this compiler: such a log establishes nothing whatever about what happened on the far side, because the observation ended before the subject did. A lane holding one has not measured a failed run; it has measured a failed measurement, and the runner-side outcome is UNOBSERVED rather than bad. THAT IS THE STANDING TRAP HERE AND IT IS THE ONE THIS RECEIPT WAS ITSELF CAUGHT BY: a supplied cause overwrites an accurate agnostic reading. The original report said control never returned and the stream was cut, which was correctly agnostic between the endings; the explanation that a killed process cannot report its own death was then accepted for runs where it had never been established, having been established for a different run entirely. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE SEVERED INVOCATION IS ALREADY IDENTIFIED so the trigger does not require reproducing the phenomenon again: 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries Action failed: signal: killed and no exit-code line. WHAT IS MISSING IS ONLY ACCESS: the bb client in this environment exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access, which is why this is a trigger and not a task. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest. AND THE OBSERVATION COUNTS ARE STATED BECAUSE THEY ARE SMALL: the loud ending is n=2 and THE SILENT ENDING IS n=1. One dispatch is one draw. That is enough to establish that the silent ending EXISTS and to identify its grain, which is what this receipt claims; it is NOT enough to establish which conditions select between the two endings, and the sustained-thrash difference named above is the distinguishing feature of a single specimen rather than a rule. A reader deciding anything about frequency needs more draws than this receipt has, and every reversal recorded in it came from treating a measured result as a replicated one.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. AND THE TRAILER'S ABSENCE MEANS A THIRD THING, WHICH IS WHY IT IS NOT THE DISCRIMINATOR IN EITHER DIRECTION. There are three endings, not two. A command that reached an ending emits an exit-code line AND a trailer. AN ACTION KILLED BY THE EXECUTOR EMITS NO EXIT-CODE LINE, an Action failed line naming the signal, AND STILL A TRAILER -- measured in the raw client capture of this receipt's severing run, where the trailer follows the Action failed line by five lines, so the client outlived the action it was running. A CLIENT STREAM THAT WAS ITSELF LOST emits none of the three. That last ending is real and was produced accidentally here by a caller-side timeout on the reading end, which is precisely what made an earlier draft mistake trailer-absence for action-severance. So: trailer ABSENT means THE CLIENT did not finish, which is a fact about the observer and not about the action; trailer PRESENT proves nothing about whether any command ran. Only the exit-code line answers that, and it answers it in both directions. AND THE THREE-WAY SPLIT DISSOLVED THE CLASS THIS RECEIPT WAS OPENED ON, WHICH IS THE MOST IMPORTANT THING IT RECORDS. The investigation began from a report of several silent runs described as one signature, established over the fields exit-code-line, marker and log-stops -- all three of which the second and third endings SHARE. The field that separates them, the Action failed line, was not among them, so two classes were counted as one and the merged class was handed on as the subject. Graded afterwards on that field, TWO of the reported runs are the third ending, CLIENT STREAM LOST: no Action failed line, no signal or kill wording, nothing about the runner at all. THE COUNTS ARE GIVEN EXACTLY BECAUSE AN INFLATED ONE IS WHAT PRODUCED THE MERGE, and a correction carrying the error's own defect is not a correction. Client-stream-lost is n=2 CONFIRMED; earlier runs from the same lane were described before the separating field existed and have NOT been re-examined against it, so they are UNCLASSIFIED CANDIDATES rather than members, and joining them would require re-grading each. The action-kill ending has exactly ONE confirmed instance and it is the arm taken for this receipt.\n\nSO THE TWO ENDINGS HAVE OPPOSITE EVIDENTIAL STANDING AND MUST NOT INHERIT EACH OTHER'S SUPPORT. The O(files) accumulation and the inert budget stand on instrumented arms here and are independently corroborated. The action kill is real and is n=1. And the ending that actually cost the reporting lanes their time is the CLIENT-STREAM-LOST one, which this receipt does not explain at all and which is not a fact about this runner or this compiler: such a log establishes nothing whatever about what happened on the far side, because the observation ended before the subject did. A LANE HOLDING ONE HAS NOT MEASURED A FAILED RUN; IT HAS MEASURED A FAILED MEASUREMENT, AND THE RUNNER-SIDE OUTCOME IS UNOBSERVED RATHER THAN BAD. THAT SENTENCE IS A STANDING CONSEQUENCE OF THE TAXONOMY AND NOT A REMARK ABOUT IT: it is actionable now, it does not wait on any explanation of why a client stream is lost, and the explanation may never arrive. It is stated here because the opposite reading was in live use -- lanes treating such logs as runner failures and drawing conclusions about their own branches from them. Where such a conclusion has survived, check WHY before keeping it: a two-arm comparison in which BOTH arms hit this ending is still sound, because a control does not need the phenomenon named to do its job, but that soundness is a property of the comparison's design and never of the log's readability. THAT IS THE STANDING TRAP HERE AND IT IS THE ONE THIS RECEIPT WAS ITSELF CAUGHT BY: a supplied cause overwrites an accurate agnostic reading. The original report said control never returned and the stream was cut, which was correctly agnostic between the endings; the explanation that a killed process cannot report its own death was then accepted for runs where it had never been established, having been established for a different run entirely. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE SEVERED INVOCATION IS ALREADY IDENTIFIED so the trigger does not require reproducing the phenomenon again: 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries Action failed: signal: killed and no exit-code line. WHAT IS MISSING IS ONLY ACCESS: the bb client in this environment exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access, which is why this is a trigger and not a task. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site."