diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index 002b3ebf13a..29de661db09 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -24,6 +24,8 @@ data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODUL data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." +data memory_stall_inert_budget_knob_receipt_note: String = "THE THIRD SPECIMEN IN THE BUDGET SERIES, AND IT RETIRES THE VARIABLE RATHER THAN NARROWING ITS RANGE. The two receipts above bracket a declared budget from opposite sides -- one truthful and thrashing, one overstated and killed -- and together they conclude that no good NUMBER exists on a machine of that size. This one varies the number as its only input and finds the knob is not connected to the quantity that kills the run, which is a stronger statement and a cheaper one to act on: the search for a better value is not merely unpromising, it is ill-posed.\n\n2026-09-02, session proud-bear-215, default BuildBuddy remote-execution runner (MemTotal 7451388 kB, no cgroup memory.max or memory.high at any depth), tree 4e6a597de0 with the runner echoing DIRTY=0. Subject src/v2/workflow/floor_terminal_ledger_test.dag driven through the claim_batch binary with --roster-from-discovery --hermetic. TWO DISPATCHES DIFFERING IN NOTHING BUT GUNBC_MEMORY_BUDGET_BYTES, with arrival of the variable confirmed in both (ctrl-build printed its forwarding-env line naming RUST_BACKTRACE and GUNBC_MEMORY_BUDGET_BYTES, and the runner echoed the value back before starting): declared 2147483648 produced a claim_batch resident set of 6419416 kB; declared 1073741824 produced 6416112 kB. Halving the declaration moved resident memory by five hundredths of one percent, to three times and six times the declared figure respectively, and both arms then climbed monotonically to 7002676 kB and 7171452 kB with MemAvailable reaching zero. THE DECLARATION DOES NOT BOUND THE PROCESS IN EITHER DIRECTION.\n\nWHAT THE OVERRIDE ACTUALLY BUYS, stated plainly because a lane was handed it as a remedy this same day. It answers gunbc.host_budget_source's SOURCE question -- may a run begin at all -- and constrains nothing the process allocates afterwards. So setting it to clear a HostBudgetUnreadable refusal trades a loud, immediate, typed answer for an unbounded run, which is the escape hatch DESIGN section 5 forbids: a toggle whose only effect is to proceed as if the refusal had not fired. The receipt above records the same trade from the overstated end; this one shows the trade is not improved by understating, because understating changes nothing at all.\n\nWHERE THE MEMORY ACTUALLY GOES, read off the floor-phase lines rather than inferred from the ending. Preparation is BOUNDED AND INVARIANT under both the roster flag and the budget: across a completing single-function run and both dying arms, bare-reference-edge-index-warm completed at source_files=4475 for rss_growth of 819593216, 820158464 and 776818688 bytes, and module-index (modules=3155), declarer-discovery (declarers=17), whole-tree-graph-facts, declarer-closure and closure-strict-resolve (sources=18) completed identically in all three. The unbounded growth is entirely AFTER preparation and is ACROSS WITNESS FILES: --roster-from-discovery walks every discovered witness file in the corpus rather than the entry's own, and the resident set rises file after file without returning. THE ENTRY IS THEREFORE NOT THE VARIABLE AND NEVER WAS -- the same entry at the same budget with a single --function completes and returns zero. Narrowing the dispatch to one witness file is the working shape, and it works for a reason other than the one it appears to have: narrowing does not shrink preparation, which costs about 0.8 GiB either way.\n\nWHAT THIS IS NOT, established by a POSITIVE CONTROL on the same runner class the same day rather than by the absence of a line. The guest OOM killer is LOUD here: a child deliberately allocating past the cap prints Killed, the shell SURVIVES it, an unconditional exit-status echo reports 137, the script's end marker prints, and the action's own completion trailer prints. A child SIGKILL does not sever the action. So a run whose log simply STOPS, with no kill line and no returned status, is not being OOM-killed in the guest -- the ACTION itself is being destroyed, which is a different mechanism with a different owner. Building that control first is what converts no kill line appeared from an absence into evidence.\n\nAND THE READING RULE FOR ANYONE MEASURING THIS FROM OUTSIDE, because the wrapper's exit status cannot answer it. A BuildBuddy action that ran to completion emits its own protocol terminator -- a line reporting the command's exit code, followed by a Remote run completed line carrying a timestamp -- and a severed action emits neither. Measured present on a genuine exit 255, present on a genuine exit 0, present on the OOM control above, and absent on a severed run. The wrapper's status is transport-shaped and is not a substitute in either direction: it reported zero over an action whose payload had died at 137, because the script's trailing echo succeeded, which is correct shell semantics and not a wrapper defect. THE ABSENCE OF THE TRAILER IS THE SEVERANCE SIGNAL, whatever the status says. This is gunbc.recurring_failure_mode transport_close_read_as_completion read at its own recognition rule, and with the useful refinement that here the protocol event EXISTS and is simply not consulted -- so the remedy is available rather than blocked.\n\nAND THE ENDING SPLITS INTO TWO REACHABLE OUTCOMES ON ONE WORKLOAD, WHICH IS THE FINDING AND ALSO THE CORRECTION OF THIS RECEIPT'S OWN FIRST DRAFT. THREE dispatches of the same subject were run to their own conclusion rather than stopped by a client cap, and they did not end the same way. TWO ended LOUDLY: the guest OOM killer printed Killed naming the binary, the unconditional status echo reported 137, and an ordinary exit-code line was emitted -- identical to a synthetic control in which a child deliberately allocates past the cap, where the shell likewise SURVIVES its child's death. THE THIRD ENDED SILENTLY, and it reproduces the signature this receipt was opened to explain: no exit-code line at all, an Action failed: signal: killed line in its place, the unconditional echo NEVER PRINTING, and the client wrapper returning 255. Control never came back to the shell because the shell was killed with everything else.\n\nSO THE KILL HAPPENS AT TWO DIFFERENT GRAINS AND ONLY ONE OF THEM IS LOUD. The guest's own OOM killer selects a CHILD, and that ending is fully reported: a status, a diagnostic, a complete log. The executor kills the ACTION, and that ending can report nothing from inside, because there is no inside left. A reader must not generalise either ending to the other, and the earlier reading here -- that exhaustion on this path is loud and therefore cannot be what cuts a log -- WAS WRONG BECAUSE IT GENERALISED FROM THE FIRST TWO DISPATCHES BEFORE THE THIRD EXISTED. The silent cut IS memory-driven; it is simply not the in-guest OOM killer that delivers it. What distinguished the severing run was sustained thrashing rather than a fast climb: it reached 8822792 major faults with 659 seconds of SYSTEM time before dying, where the loudly-killed arms were selected by the guest first. THIS RECEIPT DOES NOT CLAIM THE EXECUTOR'S KILL RULE, which is the executor's own and is not observable from the guest. AND THE OBSERVATION COUNTS ARE STATED BECAUSE THEY ARE SMALL: the loud ending is n=2 and THE SILENT ENDING IS n=1. One dispatch is one draw. That is enough to establish that the silent ending EXISTS and to identify its grain, which is what this receipt claims; it is NOT enough to establish which conditions select between the two endings, and the sustained-thrash difference named above is the distinguishing feature of a single specimen rather than a rule. A reader deciding anything about frequency needs more draws than this receipt has, and every reversal recorded in it came from treating a measured result as a replicated one.\n\nTHE READING RULE, CORRECTED, AND THE FIRST VERSION OF IT IS RECORDED HERE BECAUSE IT WAS PLAUSIBLE AND WRONG. A completion trailer naming the run and a timestamp is emitted by the client in EVERY case measured -- a genuine nonzero exit, a clean exit, a child OOM kill, and the killed action alike -- so TRAILER PRESENCE PROVES NOTHING and an earlier draft of this receipt was mistaken to offer its absence as the severance signal. THE DISCRIMINATOR IS THE EXIT-CODE LINE: the client reports the command's own exit code when a command ran and returned one, and reports Action failed: signal: killed INSTEAD when the action was killed. Present-and-numeric means a process reached an ending and its status is trustworthy; absent-with-a-signal means nothing ran to an ending and the wrapper's own status is arbitrary. AND THE TRAILER'S ABSENCE MEANS A THIRD THING, WHICH IS WHY IT IS NOT THE DISCRIMINATOR IN EITHER DIRECTION. There are three endings, not two. A command that reached an ending emits an exit-code line AND a trailer. AN ACTION KILLED BY THE EXECUTOR EMITS NO EXIT-CODE LINE, an Action failed line naming the signal, AND STILL A TRAILER -- measured in the raw client capture of this receipt's severing run, where the trailer follows the Action failed line by five lines, so the client outlived the action it was running. A CLIENT STREAM THAT WAS ITSELF LOST emits none of the three. That last ending is real and was produced accidentally here by a caller-side timeout on the reading end, which is precisely what made an earlier draft mistake trailer-absence for action-severance. So: trailer ABSENT means THE CLIENT did not finish, which is a fact about the observer and not about the action; trailer PRESENT proves nothing about whether any command ran. Only the exit-code line answers that, and it answers it in both directions. AND THE THREE-WAY SPLIT DISSOLVED THE CLASS THIS RECEIPT WAS OPENED ON, WHICH IS THE MOST IMPORTANT THING IT RECORDS. The investigation began from a report of several silent runs described as one signature, established over the fields exit-code-line, marker and log-stops -- all three of which the second and third endings SHARE. The field that separates them, the Action failed line, was not among them, so two classes were counted as one and the merged class was handed on as the subject. Graded afterwards on that field, TWO of the reported runs are the third ending, CLIENT STREAM LOST: no Action failed line, no signal or kill wording, nothing about the runner at all. THE COUNTS ARE GIVEN EXACTLY BECAUSE AN INFLATED ONE IS WHAT PRODUCED THE MERGE, and a correction carrying the error's own defect is not a correction. Client-stream-lost is n=2 CONFIRMED; earlier runs from the same lane were described before the separating field existed and have NOT been re-examined against it, so they are UNCLASSIFIED CANDIDATES rather than members, and joining them would require re-grading each. The action-kill ending has exactly ONE confirmed instance and it is the arm taken for this receipt.\n\nSO THE TWO ENDINGS HAVE OPPOSITE EVIDENTIAL STANDING AND MUST NOT INHERIT EACH OTHER'S SUPPORT. The O(files) accumulation and the inert budget stand on instrumented arms here and are independently corroborated. The action kill is real and is n=1. And the ending that actually cost the reporting lanes their time is the CLIENT-STREAM-LOST one, which this receipt does not explain at all and which is not a fact about this runner or this compiler: such a log establishes nothing whatever about what happened on the far side, because the observation ended before the subject did. A LANE HOLDING ONE HAS NOT MEASURED A FAILED RUN; IT HAS MEASURED A FAILED MEASUREMENT, AND THE RUNNER-SIDE OUTCOME IS UNOBSERVED RATHER THAN BAD. THAT SENTENCE IS A STANDING CONSEQUENCE OF THE TAXONOMY AND NOT A REMARK ABOUT IT: it is actionable now, it does not wait on any explanation of why a client stream is lost, and the explanation may never arrive. It is stated here because the opposite reading was in live use -- lanes treating such logs as runner failures and drawing conclusions about their own branches from them. Where such a conclusion has survived, check WHY before keeping it: a two-arm comparison in which BOTH arms hit this ending is still sound, because a control does not need the phenomenon named to do its job, but that soundness is a property of the comparison's design and never of the log's readability. THAT IS THE STANDING TRAP HERE AND IT IS THE ONE THIS RECEIPT WAS ITSELF CAUGHT BY: a supplied cause overwrites an accurate agnostic reading. The original report said control never returned and the stream was cut, which was correctly agnostic between the endings; the explanation that a killed process cannot report its own death was then accepted for runs where it had never been established, having been established for a different run entirely. The wrapper status is transport-shaped in both directions and is never the answer: it returned 0 over an action whose payload died at 137, which is correct shell semantics because a trailing echo succeeded, and 255 over the killed action. This is gunbc.recurring_failure_mode transport_close_read_as_completion at its own recognition rule -- ask whether the completion predicate names a PROTOCOL event or a TRANSPORT one -- with the sharp lesson that a protocol-looking line can still be transport-shaped, and that the way to find out is to build the failure on purpose and watch which lines survive it.\n\nWHAT REMAINS UNESTABLISHED, NARROWED TO ONE QUESTION. The phenomenon is reproduced and its grain is identified, so the open question is no longer what happens but WHY THE EXECUTOR KILLS THE ACTION -- a health check, a memory ceiling above the guest's own, or a supervisor timeout are all consistent with what the guest can see, and the guest cannot distinguish them. THE DECIDING INSTRUMENT IS THE EXECUTOR'S SERVER-SIDE RECORD OF A SEVERED ACTION, read for that action's terminal disposition and its worker's, set beside the client's view of the same run; it is reachable today, since every dispatch prints its own invocation URL and the receipts above already cite invocations by UUID. THE SEVERED INVOCATION IS ALREADY IDENTIFIED so the trigger does not require reproducing the phenomenon again: 94352bd4-e91b-440a-9a43-f7daa96ee1dd, whose client log carries Action failed: signal: killed and no exit-code line. WHAT IS MISSING IS ONLY ACCESS: the bb client in this environment exposes no invocation-fetch subcommand, so the record must be read through the executor's API or web surface by someone holding that access, which is why this is a trigger and not a task. THE ACCEPTANCE HALF: the record must be for an invocation whose client log carries Action failed: signal: killed and NO exit-code line, and must report that action's own terminal status. AN IN-GUEST SUPERVISOR IS REFUSED AS AN ALTERNATIVE BY CONSTRUCTION -- anything watching from inside dies with the guest, and this receipt demonstrates exactly that limit from both sides: the in-guest sampler survived a child SIGKILL well enough to establish the loud ending, and reported nothing whatever about the action's own death." + data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." data memory_stall_admitted_under_pressure_receipt_note: String = "THE MEASURED CONTROL THAT FORCED THE CPU CONJUNCT, found by this refusal's own first execution rather than by reasoning. On the first CI run carrying a rate-only form of this verdict (gunbc PR #9755, run 33319823294, required-witnesses-floor, srv3 actions-runner slice, budget 15 GiB from cgroup memory.high), the arm fired at 12431 major faults/minute sustained over a 788-second window -- one 13-minute typecheck of std.constructors under the slice's memory.high reclaim throttle. That configuration is not a treadmill: the same floor completes green on main on the same runners day after day. A resolve can sustain hundreds of refaults per second under a reclaim throttle and still be spending most of its wall COMPUTING -- which is exactly the slow-but-progressing state this module is forbidden to refuse. Rate alone cannot separate the two; what separates them is where the wall goes. A progressing run under pressure still burns CPU between faults; the treadmill's defining property is that its CPU share of wall is near zero, because the process sits in disk wait re-reading its own text. So the refusal is a CONJUNCTION: the fault rate above its line AND the process's own CPU share of the window below its floor. Both halves are the process observing itself; neither is host-shared; neither is a timeout."