From 86abefb20df74aa18dce172134641ff6c61396b6 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Tue, 1 Sep 2026 09:31:55 +0000 Subject: [PATCH 1/2] Price the entry-subject reservation: a declared budget on a cgroup-less runner is an rc=137, not a measurement The scoped-compile note reserves entry subjects outside this module's admission arm and records the reservation's intent. It did not record what the reservation costs, so the gap was rediscovered twice in one night. Receipt: entry subject through the emission instrument on the default BuildBuddy runner, which exposes no cgroup limit. The typed refusal was bypassed with a declared 25 GB on a machine this module's own kill receipt puts near 4.9 GiB, taking the DeclaredUnverified arm. measure=137 after about twenty minutes, no diagnostic. The asymmetry priced here is why it belongs on this note: a whole-root subject maps DeclaredUnverified to a refusal, an entry subject admits, and the declaration is never compared against an observable because there is nothing on that runner to compare it against. --- dag/gunbc/whole_corpus_compile_admission.dag | 27 ++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/dag/gunbc/whole_corpus_compile_admission.dag b/dag/gunbc/whole_corpus_compile_admission.dag index 34ded3cd995..14328e950a4 100644 --- a/dag/gunbc/whole_corpus_compile_admission.dag +++ b/dag/gunbc/whole_corpus_compile_admission.dag @@ -96,6 +96,33 @@ data whole_corpus_compile_measured_demand_note: String = "WHERE THE THRESHOLD CO // An unasked question, not an all-clear: a scoped compile is not admitted by this module, it is // outside its subject. If scoped compiles later need their own bound it will be a different // measurement with a different demand figure, never this one reused at a discount. +// +// A RECEIPT FOR WHAT THE UNASKED QUESTION COSTS, so the reservation carries its price and not only +// its intent. 2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner, +// entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument +// measure. That runner exposes no cgroup memory.high or memory.max, so gunbc.host_budget_source +// answers Unreadable and the run refuses immediately and typed. The refusal was bypassed by +// declaring GUNBC_MEMORY_BUDGET_BYTES=25000000000 on a machine this module's own kill receipt +// records as roughly 4.9 GiB, which took the DeclaredUnverified arm. Two runs, invocations +// b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2: the second +// reported measure=137 after about twenty minutes, the first stopped at the same step. SIGKILL, +// no diagnostic, no measurement. +// +// THE ASYMMETRY THAT RECEIPT PRICES, which is the reason it belongs on THIS note rather than on the +// budget authority. A whole-root subject maps DeclaredUnverified to +// WholeCorpusCompileRefusedBudgetUnreadable and stops. An entry subject is outside this module's +// arm by the placement above, so it ADMITS, and the only behaviour left when the declaration is +// false is to be killed. The declaration is never compared against an observable, because on a +// runner with no cgroup there is nothing to compare it against. +// +// SO THE PRACTICAL CONSEQUENCE, recorded because it was rediscovered twice in one night by two +// sessions: on a runner exposing no cgroup limit there is no good value for that variable at a +// whole-closure entry. Too high is this SIGKILL; gunbc.host_budget_source memory_stall_refusal +// already records that a truthful 7000000000 produced four runs of two to four and a half hours +// without completing. An entry board figure needs a runner that exposes a real limit, at which +// point no declaration is involved and the refusal never fires. Declaring a budget to get past the +// refusal is the workaround DESIGN section 5 names, and the arm it defeats names this exact exit +// code in its own refusal text. // The demand figure: the highest available COMPLETING-run peak on the compile route (15871708 kB, // --target dag), rounded up to whole-gibibyte grain. This session's own three peaks are all lower From efa7aef0f330b790b258db0478f850375b6bc820 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Tue, 1 Sep 2026 09:36:56 +0000 Subject: [PATCH 2/2] Move the receipt to the module that owns the failure, and keep only the placement rationale here The receipt data -- invocations, exit code, timings, the declared literal -- now lives as a row in gunbc.memory_stall_refusal beside the truthful-declaration specimen it pairs with. That module owns what happens when the budget half was satisfied and the machine still cannot deliver it; whole_corpus_compile_admission owns only the placement that routes an entry subject past the refusal. The pair is the point: a truthful budget thrashes and an admitting budget is killed, so the honest range is empty for a whole-closure entry on that runner class rather than merely badly chosen. Neither reading establishes that alone. The annotation left behind carries the irreducible rationale and cites the receipt row by name. --- dag/gunbc/host/memory_stall_refusal.dag | 2 + dag/gunbc/whole_corpus_compile_admission.dag | 41 ++++++++------------ 2 files changed, 19 insertions(+), 24 deletions(-) diff --git a/dag/gunbc/host/memory_stall_refusal.dag b/dag/gunbc/host/memory_stall_refusal.dag index c3655af05a9..002b3ebf13a 100644 --- a/dag/gunbc/host/memory_stall_refusal.dag +++ b/dag/gunbc/host/memory_stall_refusal.dag @@ -22,6 +22,8 @@ data memory_stall_refusal_authority_disposition: Disposition = Terminal { data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODULE REFUSES, so the thresholds below face a real specimen rather than a thought experiment. 2026-08-30, default BuildBuddy remote-execution VM (7276 MiB total, 6 CPUs, swap 0, no procfs PSI, no cgroup memory controller files at any depth), GUNBC_MEMORY_BUDGET_BYTES=7000000000: four whole-closure gunbc runs (invocations 0cf5e9cb-f09e-47b6-a953-9bb4528ea9ec, e11bc9bf-61a8-432f-b596-e1d74efc5b55, ecd54e06-8b02-48c6-9f63-95e912c9edff, 33d93219-793c-40fe-a2c1-84f64d0fb743, measured by session loyal-raven-297) ran 2 to 4.5 hours without completing and without printing a further line, while the same census completes in about 10 minutes on a 20 GB runner. The budget was READABLE (env override) and ADMITTED; the machine simply cannot hold the working set beside its own page cache, and with swap disabled the kernel's only recourse is evicting file-backed pages -- including the binary's own text -- which the process immediately refaults. Wall time is consumed by disk reads the process never asked for; memory.current never crosses a kill line; no signal any existing arm reads ever fires. The run is not slow, it is a treadmill.\n\nWHY NO EXISTING ARM COVERS IT. gunbc.host_budget_source refuses an UNREADABLE budget -- this budget was readable. gunbc.whole_corpus_compile_admission refuses a whole-root subject below measured demand -- these were Entry subjects, deliberately outside that arm (whole_corpus_compile_measured_peak_demand's annotation reserves the gap). gunbc.typed_module_cache_capacity bounds the typed cache -- the thrash is at PAGE grain underneath the allocator, invisible to entry counts. And the deleted AIMD controller's PSI arm could not run here even if it survived: the VM synthesizes no /proc/pressure files at all (measured, same day)." +data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none." + data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site." data memory_stall_admitted_under_pressure_receipt_note: String = "THE MEASURED CONTROL THAT FORCED THE CPU CONJUNCT, found by this refusal's own first execution rather than by reasoning. On the first CI run carrying a rate-only form of this verdict (gunbc PR #9755, run 33319823294, required-witnesses-floor, srv3 actions-runner slice, budget 15 GiB from cgroup memory.high), the arm fired at 12431 major faults/minute sustained over a 788-second window -- one 13-minute typecheck of std.constructors under the slice's memory.high reclaim throttle. That configuration is not a treadmill: the same floor completes green on main on the same runners day after day. A resolve can sustain hundreds of refaults per second under a reclaim throttle and still be spending most of its wall COMPUTING -- which is exactly the slow-but-progressing state this module is forbidden to refuse. Rate alone cannot separate the two; what separates them is where the wall goes. A progressing run under pressure still burns CPU between faults; the treadmill's defining property is that its CPU share of wall is near zero, because the process sits in disk wait re-reading its own text. So the refusal is a CONJUNCTION: the fault rate above its line AND the process's own CPU share of the window below its floor. Both halves are the process observing itself; neither is host-shared; neither is a timeout." diff --git a/dag/gunbc/whole_corpus_compile_admission.dag b/dag/gunbc/whole_corpus_compile_admission.dag index 14328e950a4..e6c74e4b04e 100644 --- a/dag/gunbc/whole_corpus_compile_admission.dag +++ b/dag/gunbc/whole_corpus_compile_admission.dag @@ -97,32 +97,25 @@ data whole_corpus_compile_measured_demand_note: String = "WHERE THE THRESHOLD CO // outside its subject. If scoped compiles later need their own bound it will be a different // measurement with a different demand figure, never this one reused at a discount. // -// A RECEIPT FOR WHAT THE UNASKED QUESTION COSTS, so the reservation carries its price and not only -// its intent. 2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner, -// entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument -// measure. That runner exposes no cgroup memory.high or memory.max, so gunbc.host_budget_source -// answers Unreadable and the run refuses immediately and typed. The refusal was bypassed by -// declaring GUNBC_MEMORY_BUDGET_BYTES=25000000000 on a machine this module's own kill receipt -// records as roughly 4.9 GiB, which took the DeclaredUnverified arm. Two runs, invocations -// b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2: the second -// reported measure=137 after about twenty minutes, the first stopped at the same step. SIGKILL, -// no diagnostic, no measurement. +// WHAT THE UNASKED QUESTION COSTS ON THE ENTRY PATH, because a reservation that records only its +// intent gets rediscovered. A whole-root subject maps a declared-but-unverified budget to +// WholeCorpusCompileRefusedBudgetUnreadable and stops. An entry subject is outside this arm by the +// placement above, so it ADMITS -- and the declaration is never compared against an observable, +// because on a runner exposing no cgroup limit there is nothing to compare it against. The only +// behaviour left when the declaration is false is for the kernel to end the process. // -// THE ASYMMETRY THAT RECEIPT PRICES, which is the reason it belongs on THIS note rather than on the -// budget authority. A whole-root subject maps DeclaredUnverified to -// WholeCorpusCompileRefusedBudgetUnreadable and stops. An entry subject is outside this module's -// arm by the placement above, so it ADMITS, and the only behaviour left when the declaration is -// false is to be killed. The declaration is never compared against an observable, because on a -// runner with no cgroup there is nothing to compare it against. +// That is a real ending and not a hypothesis. Its receipt is gunbc.memory_stall_refusal +// memory_stall_declared_budget_kill_receipt_note, which is also where the truthful-declaration +// specimen lives; the two together are what establish that the honest range is EMPTY for a +// whole-closure entry on that runner class, rather than merely badly chosen. The receipts are +// carried there because that module owns what happens when the budget half was satisfied and the +// machine still cannot deliver it -- this module owns only the placement that routes an entry +// subject past the refusal. // -// SO THE PRACTICAL CONSEQUENCE, recorded because it was rediscovered twice in one night by two -// sessions: on a runner exposing no cgroup limit there is no good value for that variable at a -// whole-closure entry. Too high is this SIGKILL; gunbc.host_budget_source memory_stall_refusal -// already records that a truthful 7000000000 produced four runs of two to four and a half hours -// without completing. An entry board figure needs a runner that exposes a real limit, at which -// point no declaration is involved and the refusal never fires. Declaring a budget to get past the -// refusal is the workaround DESIGN section 5 names, and the arm it defeats names this exact exit -// code in its own refusal text. +// The consequence for anyone measuring here: declaring a budget to get past an Unreadable refusal +// is the workaround DESIGN section 5 names, and the arm it defeats cites the resulting exit code in +// its own refusal text. An entry measurement needs a runner that exposes a real limit, at which +// point no declaration is involved and the refusal never fires. // The demand figure: the highest available COMPLETING-run peak on the compile route (15871708 kB, // --target dag), rounded up to whole-gibibyte grain. This session's own three peaks are all lower