Remove cached triton launcher by merrymercy · Pull Request #656 · sgl-project/sglang

merrymercy · 2024-07-19T06:25:17Z

No description provided.

…ed G design, and the acceptance program Compact successor brief in docs/dev/631/HANDOFF_656.md. Points at the corpse table in phase_flip_presence as the design document, and carries: - current HEAD and what each of the last eight commits contributes; - THE MEASURED TRANSPORT FACTS as a numbered list, because every design that ignored one of them has died: is_completed() never fires for a posted irecv OR an isend (the latter not even after the peer has fully consumed), only wait() progresses a transfer, the two-sided posted-vs-not-posted wire fact, and the one positive behaviour to build on -- the recv side's wait() drives the transfer; - defect G and the approved ARMED SERVICE LOOP with the send-counter, including the load-bearing ordering constraint (publish the counter strictly AFTER the isend is posted, so the only skew is counter-lags-send: a message may be consumed late, never phantom received), the both-inbound-channels requirement, why this is not the bounded-recv corpse, and the channels-empty-at-entry assert; - the acceptance program verbatim, with the honesty bar spelled out: a pass is a completed PP->TP->PP cycle under load in one unmanned log; - pointers to the three three-rank specimens in /spinning/evidence-631/ and to the capture harness; - the traps that already cost time, including the mislabeled collective; - the standing warnings: POLICY=auto must not boot until G lands, production stays manual and serving, watchdog stays decommissioned, port 30099 untouched. Documentation only; no behaviour change.

… J's root cause Supersedes the defect-G and acceptance-readiness sections of the original sgl-project#656 handoff. Records: G fixed and proven on metal; the first policy-driven flip in the feature's history committing at 01:37:12Z; the self-contradictory quiescence predicate that was the real unblocker; and defect J in three parts -- J.1 slot scope (proven, fixed), J.2 row extent (measured, deliberately not cut, with the live-spec-reserve measurement owed before it is), J.3 the cutover not carrying the resident decode set (root cause, and the reason a flip under load is currently impossible). Names the next build precisely -- resident-request carry across the stack/topology swap in build_production_flip_cutover and the scheduler topology snapshot, covering request objects, scheduler bookkeeping and mamba/GDN state AND locks, not just KV cells -- and flags the standing architecture context that design must compose with (sgl-project#635/sgl-project#636 PP dcp_size=1 to TP dcp_size=3 handover and its four silent preconditions; sgl-project#212 store routes truncating GDN state). Also records the two hypotheses of mine that died on the way, so they are not re-derived from a symptom they both fit perfectly, and the SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=0 demotion that unmasked J.3. Documentation only; no behaviour change. Suite unchanged at 379 passed.

The sgl-project#656 spec item 2 run: POLICY=auto, flips both directions, CUDA graphs in TP decode, strict purity in PP prefill, MTP speculation, the largest KV pool that satisfies the anti-wedge condition, 65 minutes unmanned, real agent traffic through router 30099 from two qwen lanes launched with no model override. Full judged extract in PROD_BRINGUP_BENCH section 2h and at /spinning/evidence-631/s25/acceptance/extract.txt. Recorded by scripts/s25_finish_acceptance.sh so the evidence lands whether or not a session is still watching.

The gate was `self.chunked_req is not None`, so only an already in-flight partial prefill got a predicted chunk size and every prefill's opening chunk took the static one. User ruling 2026-08-10: that is a defect, not a documented quirk. It skipped the feature exactly where it pays. A prompt short enough to finish in one chunk never becomes a chunked_req at all, so it was never sized dynamically -- the whole class of requests the predictor could serve end to end was excluded. For long prompts the first chunk is the one that sets the pipeline's opening bubble, and it was the one chunk taken blind. Nothing in the predictor needed the in-flight request: it takes a history_len, and for a prefill that has not started that value is 0 -- known, not assumed. The decision moves into Scheduler.dynamic_chunked_prefill_size() so it can be pinned directly rather than through the whole batch path, where a branch above can return early and make the assertion vacuous. The refusals are kept and tested, because "always dynamic" would pass the positive tests and break every boot with the feature off: enable_dynamic_chunking self-clears when profiling raises at init (scheduler.py:1542), and predict_next_chunk_size returns None until the predictor is ready -- that None must fall back to the static size, not be handed on as a chunk width. test_first_chunk_dynamic_chunking.py: 5 tests, red first (the positive one failed on the old gate). The 580 fake scheduler now binds the real method rather than reimplementing it, so it cannot drift. 19 passed across both files.

…as missing Spec item 8 says MEASURE draft graphs and leave them out if NEXTN gains nothing. They were captured unconditionally, so the rule could not be applied and the item stayed open through eleven successors. On this rig draft decode, extend and verify all capture, 0.12-0.30 GB per rank per kind. An A/B that can only be run in one direction is not an A/B. THIS ALSO SEQUENCES ITEM 6, which I had been treating as independent. resolve_spill_depth refuses spill rungs 2-3 with a real reason, not a placeholder: "the TP decode CUDA graphs bake addresses into" the draft weights, so restoring them into a fresh arena would corrupt the graphs. No draft graphs, no baked addresses, no blocker. The removal experiment is a PREREQUISITE for the spill route to >=600000, so item 8 comes before rung 2 rather than after it. TWO WAYS THIS COULD HAVE SHIPPED SILENTLY INERT, both caught before the boot rather than after: * base_spec_worker.py has no module-level logger. The disabled branch would have raised NameError the first time anyone used the flag. * EagleDraftWorkerBase's own class comment records that several draft variants -- StandaloneDraftWorker, FrozenKVMTPDraftWorker, MultiLayerEagleDraftWorker -- bypass EagleDraftWorker.__init__. A gate reading only self.server_args would therefore be inert on exactly the workers this rig runs. It now falls back to get_global_server_args(). Inert in the safe direction is still inert. Tests are 7, and two of them drive the real init_cuda_graphs and assert on the CALL rather than the predicate -- the pure helper being right is not the same as the flag working. Default unchanged: a boot that does not ask captures exactly as before. --disable-cuda-graph now also suppresses the draft capture, which it did not; a boot asking for no graphs should not still pay that per-rank memory, and it is the same corridor the seam is fighting for. Suite 738 passed / 1 failed (inherited red only). ruff clean on touched.

…this rig runs The flag shipped last shift parsed, propagated and did nothing. The boot of 19:06 carried it on the live process cmdline and captured the draft graphs anyway: zero occurrences of the disable message in the log, and idle VRAM moved -108 / -234 / +274 MiB across the three cards -- mixed sign, i.e. boot noise, not a freed capture. Cause: --speculative-algorithm NEXTN runs EAGLEWorkerV2, a wrapper that delegates to EagleDraftWorker, and EagleDraftWorker OVERRIDES init_cuda_graphs. The gate sat on EagleDraftWorkerBase.init_cuda_graphs, which that path never reaches. Same shape as the span-forwarding defect of HANDOFF_670 section 1d: a switch tested against an ancestor the production path does not execute. Two gates, and the second is not a duplicate. The phase flip re-arms a layout by calling _capture_cuda_graphs directly, without passing through init_cuda_graphs, so a gate on the entry point alone would hold at boot and let the graphs return at the first flip -- disabled for exactly as long as nobody looked. The disabled branch delegates to _capture_cuda_graphs rather than returning bare: that is the shadow-rank branch's own exit and it leaves both runner attributes None, which every later `is None` check needs. Tests: 11 passed in test_draft_cuda_graph_removal.py. Red-first -- the two new assertions failed against the old code for the right reasons (the override captured despite the flag; the re-capture path fell through the gate) and pass now. Default unchanged: a boot that does not ask still captures. Also adds scripts/s28_graph_ab_probe.py, the item-8 A/B instrument. Its content control had to be rewritten after measurement: at temperature 0.0 this instance does not reproduce its own output within a single boot, so text-hash equality is unattainable and cannot gate the comparison. Replaced by pinned token count plus a mandatory same-boot A-vs-A floor, measured at accept_len +-1.16%, throughput +-0.89%. ruff: clean on the touched files apart from a pre-existing F401 in eagle_worker_v2.py (capture_safe_tp_broadcast), which is present at HEAD and left alone.

…efuse INSIDE it Second defect on the same flag, found on metal. With the gate wired to the worker the boot got further and then died at 19:18 on all three ranks at the first draft extend: AttributeError: 'ModelRunner' object has no attribute 'eager_runner' ... _draft_extend_for_decode -> draft_runner.forward -> _forward_raw ModelRunner.init_cuda_graphs does not only capture graphs: it CONSTRUCTS the eager runner and aliases the prefill and decode runners onto it. The eager path is not the absence of the graph path, it is an object somebody has to build. Refusing around the call skipped the construction, and the failure surfaced three minutes after boot rather than at boot -- long after the log had already said the graphs were disabled. The refusal now lives inside init_cuda_graphs, next to the existing phase-flip PP carve, and both share _install_eager_only_runners: the terminal state is four coupled assignments and two copies of them drift invisibly until a forward pass reaches the one that was missed. Also fixes a decorator I displaced in the previous commit -- @time_startup_latency(cuda_graph_capture) had ended up on the new helper instead of init_cuda_graphs. Tests: 13 passed. Two new cases pin this regression directly, including a can-fail proof that a boot WITHOUT the flag still walks past the carve. Verified on the 19:15 boot that the gate itself now fires: 3 'Draft CUDA graphs DISABLED' lines (one per rank), 0 draft decode captures.

…oughput) Measured ON vs OFF against a same-boot A-vs-A floor of +-0.9% to +-4.1%: accept length -5.9%, wall decode -41.1%, mean request latency +111.3%. The user's rule removes draft graphs only if NEXTN gains nothing from them; NEXTN gains a great deal, so they stay. This retires the convergence route the last three handoffs were built on. Removing the graphs would have freed 730 MiB on the binding card -- fourteen times the 52 MiB corridor deficit -- and dissolved resolve_spill_depth's refusal of spill rungs 2-3. That memory is real and it is not purchasable at 41% of decode. Any route to >=600000 must now work with draft graphs resident. Trimming the capture set is not available either: it is already bs=[1,2,3,4] = max_running_requests. Also adds seam_scaling_reboot --from-replay: when the previous boot DIED there is no live process to read, and the documented workaround was hand transcription of the replay record's argv and env -- on the two inputs whose entire purpose is that they must not drift. Its first --dry-run caught a bug in its own parser: env values ending in ':' (PATH-like) were read as section headers and silently dropped every variable after them. Section names are matched exactly now.

…ter spill is worth its full payload The measurement that had to precede any spill work, and it came up favourably. 7196 samples at 100 ms, 82.1% occupancy, 195 flips, 0 abandons, split into PP and TP windows by the log's own re-dispatch lines: card 0 (3080) pp 896 tp 1292 -> PP binds, -128 vs floor card 1 (5090) pp 3345 tp 3047 -> TP binds, +2023 (irrelevant) card 2 (3080) pp 1210 tp 1384 -> PP binds, +186 Both binding cards are bound by the PP prefill phase, and under strict purity the drafter is idle for the whole of that phase. So a draft-weight spill is worth its full 1925 MiB on the binding card rather than the 0 MiB it would have been worth had TP bound. That was a coin flip and this chain has lost six of them by pricing a spill before checking. Adds the series dump to route_a_631_corridor.py: the aggregate minimum is a worst-TIME quantity over both phases and structurally cannot answer this. Cost me a repeated 16-minute run to discover, which is why the flag carries the reason. Also books the priced route: KvVmmArena as the VA-stable carrier (spill becomes a zero-copy decommit; torch_memory_saver is OFF on this rig and would need four moving parts and a needless D2H), the binding rank's Gen4 x4 link putting the restore at 310-340 ms (+8.5-9.3% of that rank's leg), and the one risk that must not be skipped -- the restore's bytes belong in the _staging_affordable verdict, or a rank that cannot afford them dies inside the no-return region instead of abandoning.

Third under-collection in this file's history, and its own header warns about the first two. Six tests were added to test_draft_cuda_graph_removal.py today and the family total did not move -- 738 before, 738 after -- because the list is explicit and nobody extended it. The file guards the spec item 8 flag and the eager-runner regression that took all three ranks down, so a hand-run is not enough. Family now 751 passed / 1 failed; the single red is the inherited pre-existing _staging_bytes over-reservation in test_phase_flip_mover_streaming_631, untouched today.

…t move HANDOFF_671 designed and priced this and stopped before building it. The blocker was never the bytes, it was the addresses: the TP decode CUDA graphs bake the drafter's parameter pointers at capture, and the dead code in phase_flip_spill restored into a FRESHLY allocated arena, which moves them. resolve_spill_depth refused rungs 2 and 3 to keep that bug unreachable. Spec item 8 closed off the obvious escape. Removing the draft graphs would have dissolved the constraint and costs 41% of decode throughput, so the route to >=600k has to work WITH the graphs resident. KvVmmArena splits what a normal allocation fuses. The virtual range is reserved once and freed only at close; the physical pages underneath are mapped and unmapped freely. So the spill is a decommit -- pages go back to the driver, NVML free rises -- while every address the graphs baked stands still. Parameters are bound onto the carrier exactly once, at boot, and never rebound. Proven on hardware before any model boot (s29_carrier_metal_probe.py): VA identical across spill and restore (0x502000000), three cycles each returning the full 192 MiB payload to the driver, span readable after restore. This boot does not set --enable-vram-dial, so the carrier is the first user of KvVmmArena in the serving process; that path was unexercised on this configuration until now. - boot packs the carrier strictly between build_flip_draft_worker and init_cuda_graphs, and a pin AFTER capture refuses the boot if any draft parameter escaped the reservation. That corruption has no runtime symptom, so it is an assertion or it is nothing. - hooks live in _cutover, not the pre-wave site: on the PP leg scheduler.draft_worker is already None, so the drafter is unreachable by construction, and the corridor gain is unchanged because the binding minimum is a PP-phase minimum, not a seam one. - _staging_bytes returns max(wave_peak, restore_bytes) on pp->tp, so the commit that could die in the no-return region is priced before the flip commits. max(), not sum(): the two peaks do not coexist and summing would abandon flips that fit. - depth>=2 refuses non-strict purity at boot -- between spill and restore the parameters address unbacked memory, which is sound only because strict purity forbids decode in PP. - rung 3 stays refused with the real reason: a captured graph cannot be refilled from a host image, only re-captured. The carrier's allocation is injectable so the payload classes queued behind the drafter by spec items 11-14 -- idle-slot GDN states, cold layout bytes, session KV -- are further rungs of this ladder rather than new machinery. Tests: 22 new, CPU-hermetic. The fake arena SCRIBBLES the span on decommit so a restore that forgets to refill cannot pass, and a control test drives the old fresh-arena restore to prove it DOES move the addresses -- without it the stability test could pass vacuously. Registered in run_631_flip_family.sh. Suite 774 passed / 1 failed; the red is the inherited _staging_bytes over-reservation, verified to reproduce on a pristine HEAD tree. No corridor or capacity claim is made here: the gain, the flip-duration cost and the pool raise all need the depth=draft boot.

…matched my own shell Two defects from the first depth=draft boot attempt. 1. SHARED STORAGE IS NOT SPILLABLE. The boot refused at 21:02:04Z: 'lm_head.weight' views 848035840 bytes of a 14137090816-byte storage; a partial view would smuggle unowned bytes into the arena (V1 scope) 14137090816 B is 13481 MiB -- the TARGET model's weights arena. The drafter VIEWS the target's lm_head; it does not own it. Packing that onto the carrier would have re-pointed a slice of the target's arena, and spilling it would have released pages the target still reads during the phase where the target is the ONLY thing running. plan_arena_layout's V1-scope check caught it and the boot died loudly, which is the behaviour worth keeping. The carrier now carries only parameters the drafter exclusively owns (storage size == the tensor's own bytes) and logs the excluded ones by name and size. Consequence for the price: HANDOFF_671 costed this rung at 1925 MiB/rank from the "Load weight end ... mem usage=" delta, and that delta INCLUDES the shared bytes. The spillable payload is smaller; read the installed figure from the log rather than quoting 1925. 2. pgrep -f MATCHED MY OWN SHELL, TWICE. The reboot script selected the server with pgrep -f "sglang.launch_server.*--port 30030", which matches any process whose command line CONTAINS that text. It captured a bash wrapper's 5-entry argv as if it were the server's 58-entry argv and relaunched a stub. Separately, a health-wait loop with `! pgrep -f ...` in its exit condition matched itself and could never terminate. Same family as the pkill -f self-kill the brief forbids -- eleven occurrences in this chain, now thirteen. Selection is structural now: argv[0] is a python interpreter, argv[2] is exactly "sglang.launch_server", and "--port <PORT>" are adjacent argv entries. A shell that merely quotes those strings satisfies none of that. The same script's VRAM wait polled the parent, which exits before the sglang::scheduler_PP* children release their memory; it now polls the driver, since free memory is what the next boot needs. Tests: 4 new pinning the exclusion -- a partial view is excluded rather than refused, the shared parameter is untouched by a full cycle, adding a borrowed parameter does NOT grow the claimed payload (that is how a spill gets credited for memory it never freed), and a drafter that owns nothing is refused. 26 pass in the carrier file.

…of its estimate The whole chain executes on the rig: carrier installed (12 params), carrier pin OK AFTER graph capture on all three ranks, server ready, SPILL returns 440.0/286.0/286.0 MiB of physical pages to the driver with every parameter address unchanged, RESTORE re-commits behind the SAME addresses and refills from the pinned host image with the checksum verified on device. 24 flips, 6 spills, 3 restores, 0 abandons, and a real request answered HTTP 200 in 4.17 s with coherent output -- the drafter produces valid tokens after its pages were released and re-committed, which is the functional evidence that the graphs address correct memory. THE CORRECTION. HANDOFF_671 priced this rung at 1925 MiB/rank from the boot's "Load weight end ... mem usage=" delta. That delta counts bytes the drafter does not own: model.embed_tokens.weight and lm_head.weight, 809 MiB each, are VIEWS into the TARGET model's weights arena. 1617.5 MiB per rank is shared and stays resident. The exclusively-owned payload is 439.1 MiB on the 5090 and 285.5 MiB on each 3080 -- and the BINDING cards are the 3080s, so the rung is worth 285 MiB where it matters. That closes the pool-500000 corridor breach (card 0's PP minimum was 896 against a 1024 floor) but does NOT fund 600000, which needed +1140 MiB in PP. The section 2o conclusion "A+B clears by 39 MiB" rested on the 1925 figure and does not survive. Seventh capacity headline in this chain that did not survive contact, and the first to die BEFORE being claimed: plan_arena_layout's V1-scope check refused the boot rather than letting a shared view be packed. Do not price a spill from a memory-usage delta; price it from the bytes the payload exclusively owns.

…s reusable carrier Records rung 2's mechanism as the reusable part (VA-stable KvVmmArena span, so a graph survives a physical release) and pins the measured caveat that cost this shift a boot: the drafter's embed_tokens/lm_head are VIEWS into the target's arena, so the spillable payload is what the drafter exclusively owns -- 439/285 MiB, not the 1925 MiB its load delta suggests. Names allocate_carrier_tensor as the injection point for the payload classes spec items 11-14 queue behind the drafter.

…binding phase moved to TP 7196 samples at 100 ms, 12 min, 4 streams, pool 500000, 174 flips, 0 abandons, 82% occupancy. Same load and sampler as HANDOFF_671 section 3. card PP min cache->draft delta TP min binding 0 896 -> 1496 +600 1216 PP -> TP 1 3345 -> 4149 +804 3029 TP 2 1210 -> 1766 +556 1414 PP -> TP per-card MINIMUM free 1196 / 2942 / 1396, floor 1024 CORRIDOR HELD: True The pool-500000 breach is closed: card 0 was 128 MiB below the floor in PP and is now 472 above it. 671's warning came true exactly -- the binding phase MOVED to TP on all three cards. Every card now binds where the drafter is resident by design, so this rung is spent and the next spill must be priced against the TP row. Two unpredicted readings, both recorded so they are not misread later: the PP gain is roughly DOUBLE the payload (+556..+804 against 285.5..439.1 MiB) because the carrier also replaces ~2 GB of scattered per-tensor storages with one arena block, so part of the delta is a one-off defragmentation and not the spill; and the worst reading of the whole run is now the SEAM (1196), not either phase. Also records spec item 15, which was added at 20:48Z and was not in my brief: static pool ladders are explicitly rejected in favour of a runtime pressure controller with spill-before-alloc AT the allocation, two watermarks, and kvso host-tier continuation. "Draft layers are not kept resident in PP prefill at all" is what rung 2 now does.

… watermarks Spec item 15 (added 2026-08-10 20:48Z) rejects static pool ladders and puts residency under a runtime pressure controller. Its first build requirement is the one existing machinery does not satisfy: "SPILL-VOR-ALLOC -- Pruefung an der Allokation (frei-X >= 1024 sonst erst synchron spillen), nicht reaktive Schwellen-Beobachtung" KvPressureRuntime (sgl-project#287) already drives a spill ladder and already takes a spill_fn, but it observes at a ROUND BOUNDARY and reacts. The corridor law is a CONTINUOUS minimum -- it is broken by the trough, not the average -- so an allocation between two boundaries can breach it and a 100 ms sampler will record a dip the controller never saw. And the allocation that most needs guarding does not happen on a round boundary at all: it is the seam's commit_range, inside the flip's no-return region, which has already killed this instance once. CorridorGuard is therefore a gate AT THE CALL SITE: - arms on `free - want < floor`, i.e. against the allocation that is about to happen, not against the current reading - frees to `floor + delta` (item 15b), because freeing exactly to the floor guarantees the next allocation spills again -- that is thrashing, and the spill/restore pair costs more than the allocation it enabled - spends providers CHEAPEST FIRST, which is the reclaim-ordering law - re-probes the DRIVER after every provider instead of trusting its return value: a provider that hands bytes to torch's cache has freed nothing NVML can see, and this chain has already credited such a release once - REFUSES when providers are exhausted. Allocating anyway would launder a corridor breach as a check that passed; at the seam the caller's refusal path already exists and abandons unanimously. Item 15c ("all resident is hot -> kvso keeps computing over the host tier, the price is tempo, never a corridor breach") is expressed as the most expensive provider rather than a special case, so the gate reaches it only after the cheap payloads are spent and still never breaches. draft_carrier_provider adapts rung 2's carrier as the natural first registration. Not yet wired into a production allocation site -- the wiring wants its own commit and its own metal supervision. Tests: 14, hermetic with an injected probe. They pin the allocation-relative check (a guard that only reads current free passes right before a breach), both watermarks and the no-respill consequence, the spend order, the refusal, and the liar-provider case. Suite 792 passed / 1 failed (the inherited _staging_bytes red). Also adds "trough" to .codespellrc -- correct English, already used in this codebase, mis-flagged as "through".

…the gate to the seam The corridor gate (item 15a/15b) was built but consulted by nothing. It is now in front of the seam's staging allocation, and it carries the user's item-16 order: cards fill EVENLY, host RAM is the last resort. WIRING. The gate runs in _execute, before _staging_affordable and before the wave loop. Not at commit_range, where the bytes actually move: that is inside the no-return region, there is no try/except on the path by design, and a raise there climbs into the event loop and kills the rank -- which is the 2026-08-09 death this gate exists to prevent. So the verdict travels as a string into too_small, which already rides the _collective_min that makes the abandon unanimous. A rank-local refusal would half-flip the group. Ordering is load-bearing: the gate's providers hand pages back to the DRIVER, so whatever it reclaims is money _staging_affordable then sees. Reversed, the cheaper check would refuse flips the gate could have funded. Pinned by test. ITEM 16. Relief is now tiered above cost: REBALANCE, then PARK, then HOST. Cost still orders within a tier -- item 15's law is untouched -- but it may not promote a host spill ahead of a rebalance because the spill is cheaper to execute. Those answer different questions. The host tier is gated on a FLEET predicate read over NVML: every card at the floor, or the tier stays shut. NVML and not a collective, deliberately -- the gate runs where ranks can disagree, and a collective there deadlocks. Without a fleet probe the host tier also stays shut: item 16 is a permission that must be proven, not assumed. "Even" is defined on FREE HEADROOM, not bytes held. The cards are 32/20/20 GiB, so equal bytes would mean permanently unequal pressure. free_spread_mib is the levelness metric and belongs in every corridor CSV. The drafter is registered as a REBALANCE though its image lives in host RAM: the tier names what an action does to the free column, not where bytes land, and evacuating a non-layer-bound payload from the binding card is the PP levelling move the user prescribes. Argued in full in the module docstring so a successor can reverse it with one argument if the user meant otherwise. Tests: 37 new (13 item-16 hermetic, 10 wiring, 14 pre-existing still green). The wiring tests exist because a gate that is built, registered and never called looks exactly like a gate that works -- so _execute's source is asserted to call it, to fold the refusal into too_small, and to do both before the affordability check. Suite: 815 passed, 1 failed (the inherited _staging_bytes over-reservation red, reproduced on pristine HEAD, untouched by this change).

… the gate is metal-proven ERROR FIRST. CorridorGuard judged its REFUSAL by the arming floor. Those are different questions and conflating them wedged the instance. With the arming floor raised for a proof run, the gate refused seams that the 1024 MiB law permitted comfortably (free 2306, want 726, i.e. 1580 left -- legal, refused anyway). On the pp->tp leg that is not a conservative error, it is a DEADLOCK: strict purity forbids decode in PP, so a persistently refused pp->tp starves decode outright, and nothing the PP phase holds can free the memory that would end the refusal. Measured on metal: 411 abandons, 0 requests completed in 6 minutes, /health 503 while all three ranks were alive and logging normally. So the guard now carries two numbers. floor_mib is a POLICY target -- where it starts working and how far it frees. law_floor_mib is the user's corridor law and the ONLY thing a refusal may be justified by. They coincide by default, so nothing changes unless a floor is deliberately raised. A repeated pp->tp refusal is also named now, at 1, 10 and every 100th, because the gate cannot fix that state and the operator needs the diagnosis rather than a wall of identical abandons. METAL PROOF, the point of the whole exercise. Arming floor 1600, law 1024, depth=draft, POLICY=auto, strict purity, MTP and graphs on: CORRIDOR-GUARD cleared: want 820 MiB, free 2394 -> 2680 MiB, reclaimed 286 MiB from [draft-weights] (tp_to_pp) An allocation that would have crossed the floor spilled FIRST, and NVML's own free column rose by the payload before the allocation went ahead -- re-probed, not taken from the provider's return value. 26 arms, 0 refusals, 0 abandons, health 200, corridor HELD. And the refusal path is proven too, from the run that found the bug: 197 refusals, 330 clean unanimous abandons, corridor HELD, no breach, every rank alive. The gate declines rather than dying inside the no-return region, which is what it was built for. ITEM 16 BOOKING. The sampler now records the per-card free-headroom SPREAD per sample, because holding the floor everywhere says nothing about whether one card carried the pressure alone. Measured over three runs: spread mean 2901 MiB, worst 4013 MiB -- the 5090 sits on 3768-4582 MiB while both 3080s bind near 1720-2300. The cards are NOT evenly filled, and that is now a number in every CSV rather than an impression. Not capacity evidence: peak occupancy 199453/500000 = 40%, below the bar. Suite: 823 passed, 1 failed (the inherited _staging_bytes red, on pristine HEAD too).

…he phase that cannot reach it The arena is sized max(pp, tp) because both layouts bind views into it and only one is live. Each layout occupies a PREFIX, so in the smaller-layout phase the span above it is committed and addressable by nothing. Measured on this boot: 318.7 / 220.1 / 1191.0 MiB, idle in TP -- the phase that binds on all three cards after rung 2 moved the binding there. That measurement confirms the 319/220/1191 record and refutes the 1773/0/1191 one; the two disagreed. Cheapest provider in the system: no host round trip either way. The tail holds nothing live in the phase it is released in, and arena_refill rewrites the whole arena on the way back regardless -- a refill that already runs on every flip. So spill is a decommit and restore is a commit, with no copy. ORDERING, load-bearing in both directions and silent corruption if reversed: release AFTER the PP->TP refill (its restore= arm rewrites the PP layout, which reaches into the tail) and commit BEFORE the TP->PP refill (which writes it). Asserted against the source, because a fault on unbacked memory is not a clean error. PRICED. The TP->PP commit is an allocation inside the no-return region, the shape that killed this instance on 2026-08-09. _arena_tail_bytes folds it into _staging_bytes the way _draft_restore_bytes does -- but on the OPPOSITE LEG, because PP is the larger layout here. Pricing both on one leg would leave one unpriced, which is the whole failure mode. Opt-in: DEPTH_ARENA_TAIL = 3, inserted BELOW draft+graphs (now 4) rather than renumbering "draft", so recorded integer evidence keeps its meaning. The default path still calls allocate_arena exactly as before. METAL, depth=arena, floor 1024, POLICY=auto, strict purity, MTP + graphs: released 1180 / 300 / 210 MiB at boot, addresses unchanged; 57 release/restore cycles over a 12-min loaded run, 0 abandons, health 200, corridor HELD. min free draft 1720/3768/1960 -> arena 2036/3856/2312 (+316/+88/+352) Occupancy matched to 2 slots (199453 vs 199455), so the rows compare. The gain is smaller than the bytes released because the minimum starts being set by the PP phase once the tail is gone -- price the next payload against the phase that binds AFTER this rung. Item 16 unmoved: spread 2895 vs 2901 MiB. Releasing on every card in the same phase levels nothing; that needs the rebalance tier. Two depth pins updated: rung 3 is the arena tail and IS wired, so the refused rung is now 4 (draft graphs, which must be re-captured, not refilled). Suite: 838 passed, 1 failed (the inherited _staging_bytes red).

…he host tier when refusing would deadlock The pp->tp seam had no provider at all, and item 16's host gate would have blocked the only cure. Both halves of that are now addressed on the gate side. THE TENSION, stated plainly. Item 16 withholds host RAM while any card still has headroom, because the bytes belong on that card. Item 15c says that when everything resident is hot, kvso keeps computing over the host tier and "the price is tempo, NEVER a corridor breach". Those two collide exactly when one card binds and the others do not -- which is this rig's normal state (measured spread 2895-4013 MiB across every run this shift). Resolved by asking what a refusal COSTS, per leg. Refusing tp->pp is survivable: the instance stays in TP, decode keeps running, prefill defers. Refusing pp->tp is not survivable at all -- strict purity forbids decode in PP, so a refused pp->tp starves decode outright and nothing the PP phase holds can free the memory that would end it. Measured 2026-08-10: 411 abandons, 0 requests in 6 minutes, /health 503, every rank alive and logging. So ensure_headroom takes refusal_is_fatal, the pp->tp leg passes it, and on that leg the host tier is admitted even on an unlevel fleet. Refusing forever does not protect the corridor -- the corridor is fine in that state -- it kills serving instead. The escape OPENS the tier; it does not reorder the ladder. Rebalance and park are still spent first, so it is only reached when nothing cheaper exists, and a rebalance that covers the ask means the escape is never counted. host_forced_count counts every time this happens, and the warning prints the whole free column and the spread. Each one is a levelling failure item 16 wanted avoided, so the counter is the honest price of the missing rebalance tier rather than a licence to page to RAM. Still to come: the provider itself. This commit makes the gate willing to spend a host-tier provider on the fatal leg; there is not one registered yet. Suite: 845 passed, 1 failed (the inherited _staging_bytes red).

… them lied A catalog-vs-code audit caught drift I introduced when rung 3 changed meaning, including an error message that actively misinforms. THE LIE. resolve_spill_depth's refusal printed "the deepest implemented rung is 3 ('draft': ... the draft model's weights)". After this shift rung 3 is the ARENA TAIL and 'draft' is rung 2, so the integer and the name in one sentence described two different rungs. An operator reading it would conclude the arena rung was unimplemented. Fixed by deriving the name: DEPTH_NAMES_BY_VALUE is built from DEPTH_NAMES rather than hand-written, because a hand-written name there is exactly how this happened. The module docstring was also stale in two ways: the flag list omitted `arena` and advertised "integers 0..3" when MAX_DEPTH is 4, and the rung table still showed 3 = draft+graphs. Its rung-2 figure (~1.86-2.01 GB/rank) was the memory-usage delta that HANDOFF_672 disproved -- replaced with the exclusively-owned 439/285/285 MiB and a note saying which mistake it was. Pinned so the next insertion cannot repeat it: every rung name must appear in the flag list, the documented integer range must equal MAX_DEPTH, every rung integer must have a table row, the name map must be derived, and the refusal must name DEPTH_NAMES_BY_VALUE[IMPLEMENTED_DEPTH]. CATALOG (feature-catalog duty: update on every merge). Added rung 3 `arena` with its measured 318.7/220.1/1191.0 MiB, the load-bearing ordering, and the opposite-leg pricing. Added a full entry for corridor_guard.py, which was absent entirely: the gate at the allocation, the two watermarks, tier-above- cost relief, the NVML fleet gate, the two floors, refusal_is_fatal and why, and the metal proof line. Recorded that idle mamba/GDN slots and kvso are DISPROVEN payloads so nobody rebuilds them, and named the KV bytes that can actually be returned. Also fixed pre-existing catalog drift not of my making: §3 still listed --enable-hierarchical-cache as mutually exclusive with kvso, which sgl-project#550 converted to an opt-in behind KVSO_ALLOW_HICACHE=1; §12 documented the resolution but the §3 bullet was never updated. Suite: 850 passed, 1 failed (the inherited _staging_bytes red).

…acity table, promote the KV-to-host rung, and correct its vehicle away from kvso

…rch's unused cache to the driver The gate ran BEFORE the staging path's cache reclaim, so it formed its verdict against a free column that understated the truth by the size of the allocator hoard -- 1028-1426 MiB per card at idle on this rig. In that state it could refuse a pp->tp flip, the leg that starves decode outright under strict purity, or force the host tier onto an unlevel fleet, while a gibibyte of memory no payload owns sat on the card. RELIEF_LOCAL sorts ahead of rebalance/park/host because returning unowned cache moves NVML's free column without moving any payload anywhere: it levels nothing, spills nothing, and costs only the re-cudaMalloc of whatever asks next. Tier still outranks cost, so an expensive local reclaim precedes a cheap rebalance, which is the point. The provider reports a MEASURED driver delta, never memory_reserved() minus memory_allocated(): that difference is the size of the hoard, not the size of the release, and crediting it would be the same class of error as crediting a free-list push -- bytes the corridor law cannot see, which this chain has now made three times. A fall between the two probes reports 0 rather than a negative, so the accounting cannot lie permissively. Tests: 21 pass in test_corridor_guard_631.py (8 new), 313 pass across the corridor/spill/flip unit set, 1 inherited red (_staging_bytes, reproduces on pristine HEAD). ruff clean. Also: s30_reboot_corridor_guard.sh took its own session name as a hardcoded heartbeat exemption, so the next shift's first reboot refused against its own liveness proof. It is now $SELF.

…ocator cap that makes it safe The KV pool already sits on a VA reservation and the primitive to unmap its pages already existed (runtime_set_backing_rows -> shrink -> cuMemUnmap + cuMemRelease). What did not exist is what makes using it under load safe. shrink() states its precondition plainly -- rows above the new span must be dead -- and nothing computed a safe shrink point from the live set. The only existing shrink path, the sgl-project#330 dial, sidesteps it by DESTROYING the live set (tree_cache.reset, req_to_token_pool.clear, allocator.resize), which is fine between runs and impossible under serving load. Without a cap the allocator goes on believing it may hand out every id up to size, and the next allocation above the watermark writes unmapped VA: cudaErrorIllegalAddress, a fault that kills every rank rather than an error someone catches. KvRowCap closes that non-destructively -- it withholds high ids from the FREE list and never touches a live allocation, so available_size() falls out correct and the scheduler simply admits less work. Three leaks are pinned as tests: eviction does not compact (a freed high id walks back onto the free list), clear() rebuilds arange(1, size+1) and re-admits everything, and a cap that bought no driver bytes is released again rather than carried. This funds the pp->tp leg, which was the deadlock class: under strict purity the drafter is already spilled for the whole PP phase, so the only REBALANCE provider returns exactly 0 when the fatal leg needs it. The scheduler's pool IS the PP layout's pool. Bytes are MEASURED as a driver delta, not taken from the pool's return value: under SGLANG_FLIP_SEAM_RETAIN_HANDLES the arena parks handles instead of releasing them, so its number is address space and NVML never moves. That is the same trap that caught the drafter estimate, the idle mamba slots and kvso. Recovery is wired on the tp->pp leg. It is not bookkeeping: a cap that is never lifted turns dynamic residency into a permanently smaller pool, which is the one fix the standing rule forbids. FOUND AND FIXED, a fault this rung would have made reachable: zero_kv_data_buffers zeroed the WHOLE VA-sized tensor, so a /flush_cache against a pool whose backing had shrunk would write into unmapped memory and kill every rank. It was unreachable while the only shrink path destroyed the live set and never ran under load. The corridor procedure calls /flush_cache before every idle reading, so the two would have met on the acceptance run. Zeroing now stops at safe_zero_rows. Also: RELIEF_LOCAL now carries two providers (allocator cache, then KV backing), spent before anything that moves a real payload. Tests: 873 pass in the 631 flip family (16 new), 1 inherited red (_staging_bytes, reproduces on pristine HEAD). ruff clean. The new file is now in run_631_flip_family.sh -- it was silently uncollected.

…ing it: three fixes from metal Booted at a raised arming floor as the can-fail instrument, and it fired within one boot. The gate's own detail line read: CORRIDOR-GUARD REFUSED: want 488 MiB, free 3040 -> 460 MiB, reclaimed 428 MiB from [allocator-cache, draft-weights] Free fell by 2.5 GiB DURING a reclaim. The consumer was the new provider, and it ended in cuMemCreate failed: CUDA_ERROR_OUT_OF_MEMORY. ROOT CAUSE: the production pool's arena has NO COMMIT CHUNK. Without one it holds one extent per buffer, and decommit_range releases only extents lying WHOLLY above the keep point -- so a shrink to any watermark inside that extent releases exactly zero while still lowering pool.size. The measured driver delta correctly reported 0, which is the ledger law working. What was wrong was the response to it. 1. A CHUNKLESS ARENA IS NOW A DISQUALIFIER, not a warning. The provider is not registered at all. It was logged and then used anyway. 2. THE FAILURE PATH MUST NEVER RE-COMMIT. It called recover(), and recovery GROWS the pool: finalize -> cuMemCreate. The cleanup allocated, inside a gate that armed because memory was short, on every arm. The cap now stays on instead -- it is free, and it is the invariant that nothing is handed out above the watermark. Recovery happens on the tp->pp leg, at an idle boundary where an allocation is affordable. 3. A ZERO-BYTE SHRINK EXHAUSTS THE PROVIDER. One failed attempt is evidence about the arena, not about this moment; repeating it was what turned a one-off into 2.5 GiB. Also: a failed recovery keeps the cap engaged. A capacity loss is survivable; handing out unbacked ids is not. This is the fourth 'frees nothing the driver can see' catch in this chain and the first that actively consumed memory. The pattern that caught it was the standing one: measure the driver delta, never believe the payload's own number -- and raise the arming floor until the gate fires on real metal. Tests: 20 pass in test_kv_backing_relief_631.py (4 new pinning each fix plus the chunkless disqualification). ruff clean. Also: s25_acceptance_evidence.py now books the item-16 spread time series and a relief-ladder section (per-provider spend counts, host-forced count, kv-backing shrink/recover counts).

…eeps covering the deficit At the real 1024 MiB floor the gate armed repeatedly and allocator-cache paid 604-824 MiB every time, so the KV rung proposed nothing. That is the tier law working -- free money before KV capacity -- and it is also why the rung may almost never fire in production. Recorded as next-step 0 with the failure mode to watch: a refusal that lands AFTER the rung declined on the strength of a cache estimate that then under-delivers.

…coped KV residency Fork deltas only, per the features-doc principle. Both entries lead with the property that makes them different from the obvious version: the gate is synchronous at the allocation site because the corridor law is a continuous minimum broken by the trough, and the KV residency change is safe across captured graphs because the VA reservation never moves.

… paid Found by self-review of this shift's own diff, not by a failure. Release is one commit chunk in EVERY buffer and the three PP stages hold 28 / 20 / 16 of them, so the ranks do not share a release granularity. Once the shrink target became collective, a rank could be handed a target shallower than its own smallest possible release. It returns zero -- correctly -- and the old rule read that as 'this arena cannot pay' and stopped it asking for the rest of the phase. That is a voice with real bytes silenced by a number it did not choose, and it is invisible: every log line still looks right. Exhaustion is now recorded only when the ask was at least one of this rank's own quanta, which is the only case where returning nothing says anything about the arena. 51 tests in the two rung files pass.

…, before any number This shift produced two logs and they are not interchangeable. The can-fail boot at a 4000 MiB arming floor carries spec item 12; the acceptance boot at the real 1024 MiB law carries every other axis. Quoting the acceptance log for item 12 would be the 'green on the axes it checks' error HANDOFF_675 4b warned about, and quoting the can-fail log for the corridor would be worse. Said before the numbers so a reader cannot pick the wrong pairing.

…ce list HANDOFF_675 1c said the next rung that removes slots inherits the named-posten obligation. An audit turned that warning into a table: of the five pools that share invariant_checker's six-term check, ONLY the full-attention pool passes 'withheld'. SWA passes five terms, mamba four, and req_to_token_pool uses a separate two-term invariant that never calls the shared function. Today's cap is safe because it touches only the full-attention allocator. A rung that withholds mamba slots -- which spec item 11 names explicitly as spill class -- reads as a leak and kills every rank at the first idle check, exactly as the first cap did.

…exist s25_acceptance_evidence.py printed 'no corridor.series.csv -- run the sampler with --series'. An audit of corridor_sample.sh found it takes exactly two positional arguments and has no --series flag at all; it always writes ts_ms,gpu0_free,gpu1_free,gpu2_free at 100 ms. So the instruction was unfollowable, and every acceptance run that read it left item 16 unjudged -- an axis of the user's spec silently absent because a script asked for something that was never implemented. The spread is max-min over the three free columns the CSV already carries, so it is computed from those instead. This run: spread mean 2645 MiB, median 2763, best 545, worst 3089 over 10078 samples. Item 16 is NOT satisfied and now says so out loud.

…there Settled by audit rather than assumed. is_fully_idle() (scheduler.py:6211) requires an empty running batch, an empty waiting queue and drained PP microbatches. The flip's quiescence predicate deliberately requires none of them -- phase_flip_runtime.py:303 says 'NOT sgl-project#297 fully-idle' outright, :307 says it does not require an empty waiting queue or running batch, and :392 rejects _pp_microbatches_drained because it would demand the resident decode set be empty. kv_reshard's _execute is gated on is_fully_idle(), so it cannot run at the seam, and under continuous load a fully idle round may never arrive. That rules out the obvious shape for the REBALANCE tier and says why in the handoff, before someone spends a shift on it.

…uessed An audit of _profile_available_bytes (model_runner_kv_cache_mixin.py:608, clamped at :4406) settles register C15 and names two levers this chain has not used: --mem-fraction-static, the fraction of free memory held back as non-static slack, and --rank-gpu-memory-mib, the absolute per-rank budget. It also confirms in the source what the acceptance boot measured: --max-total-tokens can only LOWER. Raising it above the profiled value has no effect and emits a warning at :4423-4427, which is precisely why 620000 came back as 512552. A successor can stop looking there.

…s, 0 refusals, 555 flips /spinning/evidence-631/s32/accept/EXTRACT.txt, 28692 corridor samples at 100 ms, one instance, one log, code a6a65be. corridor 0 breaches on all three cards; MIN free 1139/2404/1623 MiB second half headroom +115/+1380/+599 (s31: +948/+2992/+1160) flips 279 pp_to_tp + 276 tp_to_pp, 0 abandons, 0 tracebacks strict purity True -- 43995 prefill batches, ZERO carrying a graph decode graphs 99.0% (909/918) MTP accept length 2.695 (n=918) occupancy live slots max 238607 = 46.6% of a 512552-row pool real traffic 101 completions, 58 messages, 58 count_tokens, 1 chat relief ladder gate ARMED 89 times, 0 refused, 0 host-forced; allocator-cache paid 178 times item 16 spread mean 2640 MiB, worst 3237 -- NOT satisfied host RAM peak 112.1 GiB, oom_kill 9 cumulative and unchanged from s31's reading, so zero new kills this run THE DIFFERENCE FROM s31'S GREEN: the gate now ARMS at the real law floor. s31's run never armed once, so its ladder was unreached -- which proves a ladder does not break, not that it works. This one armed 89 times under real load and cleared every one, and gpu0 held within 115 MiB of the corridor law for 66 minutes without breaching: the "filled to the limit" regime item 15 asks for. WHAT THE GREEN DOES NOT COVER, recorded next to it rather than below it: item 12's KV rung did not fire here (0 shrinks -- every arm was satisfied by the cheaper tier, which is the tier law working); item 16 is still unmet at 2640 MiB spread; and YaRN long context is not in this run at all, so spec item 4 remains unmeasured.

…g arm has a trap in it Two audits that change what the next shift should do first. THE HOST HALF IS OPEN AFTER ALL. flip_blocking_guards tests scheduler.kv_session_offload, not the pool class, and MHATokenToKVPoolHost (mem_cache/pool_host/mha.py:88) takes NO scheduler reference -- only the device pool plus sizing. _kv_sess_attach_host_pool (model_runner_kv_cache_mixin.py:2655-2808) already constructs it independently and the kvso manager merely reads it later. So the pinned pool can be built as a destination, exactly as the order asks, without turning the flip off. Entry points recorded: backup_from_device_all_layer (mha.py:352) and load_to_device_per_layer (mha.py:235). DYNAMIC CHUNKING WOULD ACTIVATE HERE -- the gate is enable_dynamic_chunking and pp_size > 1 and chunked, and this instance is pp_size=3 -- and it moves both ways: down to base//4 (128 from the shipped 512) and up to ceil(base*1.25). But the only line reporting a predicted chunk size is DEBUG-level (scheduler_pp_mixin.py:1770-1773). Nothing at INFO or above fires when the chunk deviates from the configured value, so the order's "engagement proof -- chunk size provably moved at runtime" is unobtainable at the default log level. Raise that logger or add an INFO line BEFORE the run, or the arm yields a throughput number with no evidence the mechanism engaged.

…lip guard now asks what it is doing flip_blocking_guards refused flip arming whenever scheduler.kv_session_offload was merely NOT NONE, which made the host half of spec items 6/12/15c and the phase flip mutually exclusive: enabling the spill destination turned the flip off. A feature was the guard. The blanket refusal was too broad but it protected something real, and lifting it without a replacement re-opens a correctness hole rather than a capability: a kvso host image is LAYOUT-SPECIFIC (PP holds this stage's layers for every token, TP a token shard of every layer), so restoring a PP-captured image into a TP layout returns the wrong K/V without raising -- the shapes still line up. And a spilled session is not passive; it keeps generating through host ticks. kvso_flip_contract replaces presence with state: absent / idle / parked / busy, and only busy refuses (for one round, since it is transient). Parked means every image is stamped with the phase it was captured in, no copy is in flight, and no stamp names the phase the flip is about to enter. The stamp is evidence; the enforcement is two independent gates on the same hazard, so a bug in either alone is still caught: * pin_spills_to_phase suppresses the tick of any foreign-layout session, re-applied EVERY round because suppress_tick is a one-shot the picker clears -- a latch would release itself on the first tick it prevented; * restore_permitted refuses the H2D copy back into a foreign layout, and reads a MISSING phase as permitted, because a process without the flip has one layout for its whole life. Refusing there would have switched kvso's restore path off for every user who never enabled the flip. Also: the dynamic-chunking arm can now prove it engaged. The only line reporting a predicted chunk width was DEBUG-level, so at the default log level an A/B produced a throughput number with no evidence the mechanism ever moved. The new INFO line is edge-triggered on a change of width -- the per-iteration path it sits on runs thousands of times a minute and an unconditional log there would perturb what it measures. Tests: 26 new (21 contract + 5 engagement), both files registered in the family list. Mutation-checked: forcing flip_safety_state to report idle turns 6 of them red. Suite 937 passed, 1 failed (the inherited _staging_bytes red, untouched).

…layout; metal disagreed Found by changing --pp-stage-ratio to 15,9,8, which derives 32,16,16 layers over 64 and puts the MIDDLE rank's PP layout (6690 MiB) BELOW its TP layout (7924 MiB). Rung 3 then decommitted the arena tail down to the PP layout on tp->pp, and the next pp->tp refill copied the larger TP image straight into the released tail: weights_arena.py:386 arena_refill -> dst.copy_(payload) torch.AcceleratorError: CUDA error: invalid argument inside the flip's no-return region, taking all three ranks down at the first flip after a tp->pp. The assumption was written down -- '_arena_tail_bytes' said 'the arena tail is re-committed on tp->pp, because PP is the larger layout on every rank of this rig' -- and it was true for 14,10,8 only. The invariant is symmetric and belongs to the LAYOUT SIZES, not to the direction: a refill writes one layout and its restore= arm may rewrite the other, so the safe span is the MAXIMUM of the two on either leg. Both legs now commit that high-water before the copy and release the tail after it, and the affordability gate prices whichever leg has to grow. The tp->pp leg also never released afterwards, so on a TP-larger rank the tail stayed committed for the whole PP phase -- rung 3's purpose given away. Tests: 11 new in test_arena_high_water_631.py, including a fake carrier that FAULTS on a write past its committed prefix, so the pre-fix tree reproduces the metal message verbatim. Red-first verified: reverting the two hunks turns 3 of them red. test_phase_flip_arena_tail_631.py's ordering and pricing tests encoded the old assumption and are updated with the correction stated at the call site; the concern the old pricing test carried (no double counting onto the drafter's leg) is a max() and stays pinned by its own test. Also: _arena_tail_bytes now reads _census_scheduler defensively, because pricing both legs means it is reached on every staging estimate including those from runtimes built without one.

…apture, and the levelling measurements s33_yarn_bs1_leg.py drives spec items 3 and 4 as ONE leg, because they are one run: a bs1 session whose context passes 262144 is the long-context proof by construction and simultaneously the largest KV residency a single request can demand, which is the pressure that makes the relief ladder reach past its cheapest tier. Its prompt length is CALIBRATED against the server's own tokenizer, not estimated. The first version assumed ~1 token per word; measured, this synthetic vocabulary is 3.935 tokens per word, so the prompt would have been ~1.12M tokens against a 393216 context, been rejected, and reported spec item 4 as failing when only the client's arithmetic was wrong. s33_boot_from_capture.sh boots from the argv/env capture a previous reboot left on disk. s30's reboot script correctly refuses when nothing is live -- it cannot invent a configuration -- but a CRASHED instance leaves the same need, and route_a_631_prod_boot.sh would change checkpoint, per-rank MiB vector and pool at once. s33_extract.sh adds the axes no earlier extract carried: the YaRN leg quoted from the server's usage accounting, item 16 read at the BINDING INSTANT as well as on average (the mean conflates idle waste with transient depth, and only the first is an item-16 failure), the kvso contract's observed states, and the dynamic-chunk engagement count. PROD_BRINGUP_BENCH gains the measured levers: the PP stage-ratio lever is quantized at 4 layers / ~1.7 GiB against a ~1.3 GiB imbalance, so it cannot level this rig; the TP vector is near a no-op at rest because it moves ownership, which shows up under load; KV is 32 KiB/token exactly.

…ontract, and item 16's two closed ends

… what an allocator cache can hold The KV backing rung sits BELOW the allocator cache, and measured twice now (s32's acceptance and this one's first quarter hour) the cache covers the whole deficit every time the gate arms -- so item 12's rung proposes nothing and goes unexercised. That is the tier law working, and it is also why the rung can sit unproven at a fill level where free money is always enough. The seam's deficit is floor + delta + staging - free, and the STAGING term scales with live slots. So the lever is OCCUPANCY, not weakening the tier above it: several mid-sized concurrent sessions reach a higher resident row count than one maximal one, because max_running_requests is 4 and a single session is capped by the context. That is the same 'fill to the limit, then spill' regime spec item 15 describes, so the leg drives the run INTO the state the order asks for rather than around it. The corridor gate stays the backstop: if this exceeds what the ladder can fund, the correct outcome is a refused flip and an abandoned cutover, both recorded, neither a breach. A leg that could only run safely with the safety off would not be evidence.

…else; the bs1 leg found the gap

… twice, and one corridor breach reported as one NOT GREEN, on one axis. 12 samples on gpu0 at 1001 MiB -- 23 MiB under the law, 1.6 s, one card. The cause is located and it is not the flip seam: the gate armed two seconds AFTER the dip and cleared it (1186 -> 2150, reclaimed 964 MiB). What breached was an ungated prefill allocation. Item 15a's spill-before-alloc exists and works; it is called from exactly one site, and the 272k bs1 prefill binds at another. That is next shift's item 0 and it is one more caller, not a new mechanism. Everything else is the strongest recorded on this branch: 648 flips both directions, 0 abandons, 0 tracebacks strict purity True over 44226 prefill batches, ZERO with a graph decode graphs 98.8%, MTP accept 2.649 occupancy 332159 live slots = 64.8% of pool, against s32's 46.6% SPEC ITEM 4 PROVEN TWICE: 271237 prompt tokens per leg, above 262144, 48 tokens decoded each -- an axis no run in this chain had touched item 16 spread at the binding instant 717 MiB, from s32's 1265 gate 50 cleared, 0 refused; arena rung 3 324 releases, 0 faults real qwen agent traffic through router 30099 throughout Item 12's rung still did not fire (0 shrinks in 50 arms; the allocator cache covered every one). The occupancy leg written to force it was cancelled before starting rather than compound a breach that had just occurred. Extract /spinning/evidence-631/s33/accept/EXTRACT.txt, config CONFIG.txt, log serving-accept.log.

…emaining sites, and book C16/C17 An audit of the whole phase-flip surface for the 'PP is always the larger layout' assumption found the two CODE sites were the ones already fixed, and that the assumption survived as prose in phase_flip_spill's carrier docstring ('pp is the max on every rank') and at its commit_range call ('the boot packs the PP layout, which is the larger one'). A comment that states a contingent measurement as a structural property is what produced the crash in the first place, so both now say which stage ratio they were measured on, what falsifies them, and that the carrier is symmetric by construction. CONTRADICTIONS_REGISTER: C13 RESOLVED -- kvso no longer refuses flip arming by existing; the guard asks flip_safety_state() and refuses only 'busy'. The remaining blocker is host RAM (~12.9 GB per full-context region at ctx 393216), not configuration. C16 NEW -- the falsified 'PP is the larger layout' invariant, with the metal fault it caused. C17 NEW -- the corridor law is enforced at ONE allocation site. True but partial: a 272k bs1 prefill walks the binding card to 1001 MiB and only the next seam pulls it back. Suite 957 passed, 1 failed (inherited _staging_bytes red).

…make the KV rung say why it declines Register C17: `CorridorGuard.ensure_headroom` is item 15a's spill-before-alloc check, it works, and it had exactly ONE caller -- the flip seam. Successor 33's acceptance breached at a different site: 11:20:47Z gpu0 = 1001 MiB free, 12 samples (~1.6 s), 23 MiB under 11:20:49Z CORRIDOR-GUARD cleared on device 0: free 1186 -> 2150, reclaimed 964 MiB from [allocator-cache] The gate armed two seconds after the dip and fixed it in one call. A 272k bs1 prefill had walked the binding card under the floor at a site with no caller. This adds the caller. IT SPILLS; IT NEVER REFUSES. The guard is rank-local -- it reads this rank's NVML column -- while prefill admission must stay rank-uniform. A rank-local refusal would let one rank decline work its peers admitted, which is the capacity desync that previously left a scheduler not heartbeating with every rank alive. The user's wording for item 15a is "frei-X >= 1024 sonst erst synchron spillen", not "refuse", and that is what this does: the verdict is logged, counted, and not consulted for the admission decision. It prices ACTIVATION, not KV rows: in this fork the KV pool's pages are committed once at finalize() and `alloc_extend` hands out already-committed slots, so charging KV bytes would arm on every admission for memory that is never allocated. The slope is the metrics reporter's own per-token figure, a movement proxy biased small; when it cannot be read `want` is 0 and the gate degrades to enforcing the floor itself. Both regimes are pinned by tests -- including the honest one, that an underpriced slope cannot PREEMPT the crossing and instead bounds a 5-sample trough to a single dipped sample. Spec item 12's rung has declined on ~324 seam legs across two acceptance runs while emitting not one line, because its only logging sat inside the branch that already works. Its four deficit terms are now traced, edge-triggered on the sign so an acceptance run keeps the signal without an env var. The reconstruction that motivated it: the deficit was negative on 100% of s33's 93 arms, and dropping `cheap_relief_bytes` alone -- reserved-minus-allocated, 766 MiB median against a 239 MiB median gap -- flips every one of them positive. The cheap tier really was covering everything; nothing said so. C17 sibling sweep, each site booked rather than assumed: decode extend allocates from the same pre-committed pool and its transient is bounded by bs<=4; CUDA-graph capture is boot-time on this path (runtime recapture exists only in cpu_graph_runner); `KvBackingRelief.recover` is already bounded against the LAW floor; `vram_dial`'s grow does physically allocate but is disabled on this configuration and carries its own NVML floor model -- a second source of truth for one law, and a latent C17-family divergence worth a register entry rather than a change today. Tests: 15 new, mutation-checked (disabling the arm turns 7 red). Suite 972 passed, 1 failed -- the inherited `_staging_bytes` red, untouched. The new file is added to the canonical family list, which is why the count moves.

…ned for the dial's second floor model

…wo extract bugs a dry run caught The extract nests s33 -> s32, and both write $OUT/EXTRACT.txt. The outer block redirected into $EX.tmp -- the SAME path s33_extract.sh writes and then renames -- so the inner mv carried the outer script's open file descriptor away with it: the inherited sections vanished and the final mv failed. The outer temp is now its own path. And grep -c prints 0 on no match while exiting 1, so the '|| echo 0' idiom appended a second zero and every downstream arithmetic test died on '0\n0'. Both were found by dry-running the extract against a partial log while the window was still open, which is the only reason they are not being discovered at the end of a 65-minute run with nothing left to re-measure.

…ation, since a successful recovery logs nothing Spec item 12's shrink is only a spill if the rows come back. Failed and deferred recoveries log; a SUCCESSFUL one does not, so '0 failed recoveries' was being read as 'the pool is whole' when it is a weaker claim. The extract now reads the backed row count off the next proposal's own view and compares it to the boot pool.

…h its restore

…nd before they ran, and the instrument that found the fourth The first acceptance boot with the prefill gate logged NOTHING across 6066 admissions. "Installed but never needed to arm" and "inert because the guard lookup returned None" are opposite in value and were identical in the log -- which is the exact accusation this module's own docstring levels at the KV rung. So the gate now ANNOUNCES itself once per process: INFO with the floors, the slope and the providers when it is armed, WARNING when it is inert. Every other line it can emit is conditional on arming, so without this the mechanism is unfalsifiable. A review pass over the diff then found three more, and two of them were masking each other: D1, a crash. `dynamic_chunked_prefill_size()` is annotated `-> int` and returns None whenever chunked prefill is off (explicit -1, multimodal without chunked support, prefix_lm, --enable-mis). `int(None)` sat OUTSIDE the gate's try, in the scheduler event loop. Tokens are coerced once at the top now. D2, why the log was silent. `_qkv_act_bytes_per_token` and its FFN partner exist only under `--enable-metrics AND --enable-mfu-metrics`. This rig sets the first and not the second, so the slope is absent, `want` is 0, and the gate degrades to plain floor enforcement. That is the DEFAULT case, not the corner case the docstring called it, and the announcement now reports the resolved slope so it is read rather than inferred. D3, the one that would have hurt. When the slope IS readable it is summed over every layer -- each term in the reporter carries a `* num_layers`. For this model that is ~9.4 MiB/token, so a 512-token chunk would have been priced at ~4.7 GiB: larger than any card's free column, hence a guard target that can never be met, hence EVERY provider spent on EVERY admission -- including the rebalance provider that evacuates the drafter and takes speculative decoding with it. Transient activation is reused down the stack, so the resident figure is one layer's share. The slope is now divided by the layer count, refuses to price at all when the layer count is unknown, and is hard-capped at 256 MiB. The cap is the safety property: no error in a proxy may make the ladder's target unreachable. D4, the default path. `get_corridor_guard` had exactly one caller and it lived on the flip seam, so the guard, its provider ladder, its NVML fleet probe and its KvBackingRelief were built only on phase-flip boots. Calling it from the unconditional prefill path built all of that on EVERY boot of this fork. The gate is now off unless --enable-phase-flip is set. D5 is booked in the handoff rather than fixed: on the two paths where the deficit is never computed, the KV trace still prints "the cheaper tier covers the whole gap", which is the wrong reason. Tests: 21 in this file now (was 15), one per defect, plus the armed/inert announcement pair. Suite 978 passed, 1 failed -- the inherited _staging_bytes red, untouched. Run 1 of the acceptance is archived at evidence-631/s34/accept-run1: corridor HELD with 0 breaches over 26 minutes, 111/111 flips, 12 KV-rung shrinks with the pool returning to 512552, and YaRN at 271237 tokens. It is not the deliverable run, because the axis this shift exists to close could not be observed in it.

… two claims they falsified in this same document

…tivity table, and the layer-summed slope trap

…evaluate On the two paths where propose() skips the deficit computation entirely -- the arena is exhausted, or floor_rows >= current so there is no slack above the live set -- the new trace still printed 'the cheaper tier covers the whole gap'. That cause was never computed on those paths. A diagnostic that states a FALSE cause is worse than one that states none, because the next reader stops looking. This shift has already lost an acceptance run to a log that could not tell two opposite states apart; it should not also ship a log that confidently names the wrong one. A 'skipped' reason is now threaded out of propose and names the real cause. 3 tests, and they assert the ABSENCE of the false string as well as the presence of the true one. LOG STRING ONLY: the shrink arithmetic and the returned proposal tuple are unchanged. Landed while acceptance run 2 was still open, so it is NOT in that run's binary -- the modules were long since imported. CONFIG.txt says so explicitly rather than leaving 'code commit' to imply otherwise. Suite 981 passed, 1 failed (inherited _staging_bytes red).

…e fact it is, not as a pass With the arming floor at 1536 and the gate installed, 0 arms is ambiguous unless the extract says which ambiguity it is. It now states that the ARMED line proves installation, that 0 means the corridor was never under the floor AT AN ADMISSION INSTANT, and that the deep troughs on this boot are intra-forward transients and flip-seam moments -- neither reachable from a scheduling boundary. The gate closes s33's SUSTAINED under-floor state; it does not claim the transient, and the extract should not let a reader think it does.

… including the three a skim would miss

…lues filled at window close)

merrymercy added 2 commits July 18, 2024 23:23

remove cached triton launcher

408f56f

fix lint

1e5fe65

merrymercy merged commit e1792cc into main Jul 19, 2024

merrymercy deleted the profile-stream branch July 19, 2024 06:28

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Remove cached triton launcher (sgl-project#656)

c5d4e95

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] user priority: there is no max KV -- demote the cap…

7b3d4b4

…acity table, promote the KV-to-host rung, and correct its vehicle away from kvso

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] handoff 677 draft: the arena assumption, the kvso c…

df357d2

…ontract, and item 16's two closed ends

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] the corridor is gated at the flip seam and nowhere …

c715101

…else; the bs1 leg found the gap

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] register: C17 resolved by the prefill gate, C18 ope…

e520914

…ned for the dial's second floor model

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] HANDOFF_678 draft: errors first, item 12 proven wit…

4fce148

…h its restore

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] HANDOFF_678: the four gate defects, and correct the…

1c2b3be

… two claims they falsified in this same document

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] bench: the KV rung's deficit terms, the floor sensi…

f297b19

…tivity table, and the layer-summed slope trap

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] HANDOFF_678: count every failure signature by name,…

38407a7

… including the three a skim would miss

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 11, 2026

[sgl-project#656] HANDOFF_678: scaffold the acceptance axis table (va…

e24be1a

…lues filled at window close)

Navigation Menu

Uh oh!

Remove cached triton launcher - #656

Remove cached triton launcher#656
merrymercy merged 2 commits into
mainfrom
profile-stream

merrymercy commented Jul 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

merrymercy commented Jul 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant