Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions dag/gunbc/host/memory_stall_refusal.dag
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ data memory_stall_refusal_authority_disposition: Disposition = Terminal {

data memory_stall_observed_receipt_note: String = "THE MEASURED STALL THIS MODULE REFUSES, so the thresholds below face a real specimen rather than a thought experiment. 2026-08-30, default BuildBuddy remote-execution VM (7276 MiB total, 6 CPUs, swap 0, no procfs PSI, no cgroup memory controller files at any depth), GUNBC_MEMORY_BUDGET_BYTES=7000000000: four whole-closure gunbc runs (invocations 0cf5e9cb-f09e-47b6-a953-9bb4528ea9ec, e11bc9bf-61a8-432f-b596-e1d74efc5b55, ecd54e06-8b02-48c6-9f63-95e912c9edff, 33d93219-793c-40fe-a2c1-84f64d0fb743, measured by session loyal-raven-297) ran 2 to 4.5 hours without completing and without printing a further line, while the same census completes in about 10 minutes on a 20 GB runner. The budget was READABLE (env override) and ADMITTED; the machine simply cannot hold the working set beside its own page cache, and with swap disabled the kernel's only recourse is evicting file-backed pages -- including the binary's own text -- which the process immediately refaults. Wall time is consumed by disk reads the process never asked for; memory.current never crosses a kill line; no signal any existing arm reads ever fires. The run is not slow, it is a treadmill.\n\nWHY NO EXISTING ARM COVERS IT. gunbc.host_budget_source refuses an UNREADABLE budget -- this budget was readable. gunbc.whole_corpus_compile_admission refuses a whole-root subject below measured demand -- these were Entry subjects, deliberately outside that arm (whole_corpus_compile_measured_peak_demand's annotation reserves the gap). gunbc.typed_module_cache_capacity bounds the typed cache -- the thrash is at PAGE grain underneath the allocator, invisible to entry counts. And the deleted AIMD controller's PSI arm could not run here even if it survived: the VM synthesizes no /proc/pressure files at all (measured, same day)."

data memory_stall_declared_budget_kill_receipt_note: String = "THE SIBLING SPECIMEN OF THE ONE ABOVE, ON THE SAME RUNNER CLASS AND THE SAME RESERVED GAP, FAILING THE OTHER WAY. The treadmill receipt records a TRUTHFUL declaration (7000000000 on a 7276 MiB VM) producing four runs that never completed. This one records an OVERSTATED declaration on the same class of machine producing the opposite ending, and the pair is what shows there is no good value rather than merely a badly chosen one.\n\n2026-09-01, session vivid-deer-102, default BuildBuddy remote-execution runner (no cgroup memory.high or memory.max at any depth, so gunbc.host_budget_source answers Unreadable and the run refuses immediately and typed). The refusal was bypassed with GUNBC_MEMORY_BUDGET_BYTES=25000000000 -- roughly five times what that runner class holds -- which takes the DeclaredUnverified arm. Entry subject src/v2/compiler/00_compile.dag driven through tools.emission_entry_instrument measure, invocations b6fb4af7-1268-468f-ba0a-5ea9ec42314f and e7da3bd4-a99d-4b03-9c95-a1ff729366f2. The second reported exit 137 after about twenty minutes; the first stopped at the same step, gunbc.WitnessBin.Run. SIGKILL, no diagnostic, no measurement. The cache cap printed by the degraded_budget_source line was 4000, the clamp ceiling, because it is derived from the declared bytes and the declaration was false.\n\nWHY THE PAIR MATTERS MORE THAN EITHER READING. A reader holding only the treadmill receipt concludes the operator declared too little; a reader holding only this one concludes they declared too much. Together they bound the honest range from both sides on a machine of that size, and the range is empty for a whole-closure entry subject: a truthful budget thrashes and an admitting budget is killed. So the remedy is a runner that exposes a real cgroup limit -- at which point no declaration is involved, this module is not reached, and the host_budget_source refusal never fires -- and never a better number for the variable.\n\nWHAT THIS IS NOT. It is not a defect against the DeclaredUnverified arm, which is typed, graded degraded, and announced on stderr, and which the whole-root path maps to a refusal. It is the measured cost of the ENTRY-subject reservation that gunbc.whole_corpus_compile_admission holds deliberately and that the receipt above already names: on that path the declaration is never compared against an observable, because on a runner with no cgroup there is nothing to compare it against, and the only remaining behaviour when it is false is to be killed.\n\nAND THE AUTHORING LESSON, which is the part that cost two sessions rather than one: the arm that was bypassed names this exact exit code in its own refusal text, citing the BuildBuddy receipt of 2026-08-30. Declaring a budget to get past it is the workaround DESIGN section 5 names -- the line-stop signal is noticing you are routing around a refusal you do not understand -- and it converted an immediate typed answer into a twenty-minute SIGKILL carrying none."

data memory_stall_signal_choice_note: String = "WHY THE SIGNAL IS THE PROCESS'S OWN MAJOR-FAULT COUNTER AND NOTHING HOST-SHARED. majflt in /proc/self/stat (proc.5 field 12) counts the times THIS process had to read one of its own pages back from disk: it is the kernel's record of eviction-and-readmission at the grain the kernel actually evicts, observed about this process alone. A warm, healthy process sustains approximately zero major faults after its inputs are first read, whatever its size and however slow its arithmetic -- so a slow-but-progressing resolve and a thrashing one, which no wall-clock bound can tell apart, are cleanly separated by this counter: the first accumulates wall with majflt flat, the second accumulates majflt in the thousands per second. That is also why the verdict below carries NO elapsed-time arm: time appears only as the denominator of a rate, never as a refusal condition, because a timeout labelled pressure-detection would refuse exactly the honest slow case this signal exists to admit.\n\nWHAT WAS REJECTED. /proc/meminfo MemAvailable is a fact about the MACHINE and is precisely the absorbing substitution gunbc.host_budget_source deleted (host_budget_source_seed_mirror_disposition's annotation); reusing it here as a trigger would re-open that arm one layer up. PSI (/proc/pressure/memory) is the kernel's own stall accounting and would be preferable where it exists, but the VM this class was measured on does not synthesize it (no /proc/pressure, no cgroup controllers -- measured 2026-08-30), so an arm keyed on PSI is permanently green exactly where the harm lives. The typed-module cache's eviction/readmission counters are the governor's own bookkeeping and ARE carried in the refusal so the operator can see whether the cache was implicated, but they do not gate: recompute-on-miss across entries is designed-in cost, and no measured stall has yet implicated it (the observed specimens thrash below the allocator, at page grain). If a measured stall ever shows the cache readmitting the same module unboundedly, the threshold for that axis is added HERE with its receipt, not improvised at a call site."

data memory_stall_admitted_under_pressure_receipt_note: String = "THE MEASURED CONTROL THAT FORCED THE CPU CONJUNCT, found by this refusal's own first execution rather than by reasoning. On the first CI run carrying a rate-only form of this verdict (gunbc PR #9755, run 33319823294, required-witnesses-floor, srv3 actions-runner slice, budget 15 GiB from cgroup memory.high), the arm fired at 12431 major faults/minute sustained over a 788-second window -- one 13-minute typecheck of std.constructors under the slice's memory.high reclaim throttle. That configuration is not a treadmill: the same floor completes green on main on the same runners day after day. A resolve can sustain hundreds of refaults per second under a reclaim throttle and still be spending most of its wall COMPUTING -- which is exactly the slow-but-progressing state this module is forbidden to refuse. Rate alone cannot separate the two; what separates them is where the wall goes. A progressing run under pressure still burns CPU between faults; the treadmill's defining property is that its CPU share of wall is near zero, because the process sits in disk wait re-reading its own text. So the refusal is a CONJUNCTION: the fault rate above its line AND the process's own CPU share of the window below its floor. Both halves are the process observing itself; neither is host-shared; neither is a timeout."
Expand Down
20 changes: 20 additions & 0 deletions dag/gunbc/whole_corpus_compile_admission.dag
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,26 @@ data whole_corpus_compile_measured_demand_note: String = "WHERE THE THRESHOLD CO
// An unasked question, not an all-clear: a scoped compile is not admitted by this module, it is
// outside its subject. If scoped compiles later need their own bound it will be a different
// measurement with a different demand figure, never this one reused at a discount.
//
// WHAT THE UNASKED QUESTION COSTS ON THE ENTRY PATH, because a reservation that records only its
// intent gets rediscovered. A whole-root subject maps a declared-but-unverified budget to
// WholeCorpusCompileRefusedBudgetUnreadable and stops. An entry subject is outside this arm by the
// placement above, so it ADMITS -- and the declaration is never compared against an observable,
// because on a runner exposing no cgroup limit there is nothing to compare it against. The only
// behaviour left when the declaration is false is for the kernel to end the process.
//
// That is a real ending and not a hypothesis. Its receipt is gunbc.memory_stall_refusal
// memory_stall_declared_budget_kill_receipt_note, which is also where the truthful-declaration
// specimen lives; the two together are what establish that the honest range is EMPTY for a
// whole-closure entry on that runner class, rather than merely badly chosen. The receipts are
// carried there because that module owns what happens when the budget half was satisfied and the
// machine still cannot deliver it -- this module owns only the placement that routes an entry
// subject past the refusal.
//
// The consequence for anyone measuring here: declaring a budget to get past an Unreadable refusal
// is the workaround DESIGN section 5 names, and the arm it defeats cites the resulting exit code in
// its own refusal text. An entry measurement needs a runner that exposes a real limit, at which
// point no declaration is involved and the refusal never fires.

// The demand figure: the highest available COMPLETING-run peak on the compile route (15871708 kB,
// --target dag), rounded up to whole-gibibyte grain. This session's own three peaks are all lower
Expand Down
Loading