Tune params - #696
Merged
Merged
Tune params#696
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
5 tasks
hhhh1252023
pushed a commit
to hhhh1252023/sglang_public
that referenced
this pull request
Jun 5, 2026
modify param
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… evaporates
THE DEFECT, MEASURED under lane load 2026-08-16: pp_to_tp abandoned on PP1
every ~3 s -- 18 abandons against 9 cutovers in 8 minutes, three ARM-UNFUNDED
in nine seconds -- while decode sat at 187.5 s against a 180 s budget. The
shortfall was identical every time, which is what gave it away:
staging 733 MiB needed but only 691 MiB is spendable (x8)
A runtime fluctuation does not repeat to the megabyte. That is a SIZING-time
constant.
WHERE IT CAME FROM. arming_draw_bytes excused the weights-arena tail from the
arming floor whenever the KV rung was RECORDED as able to pay it. On PP1: arena
814.9 MiB, rung_fund_bytes 953.8 MiB, so the draw fell 815 -> 138.9 and 815 MiB
stopped being permanently reserved. The pool was sized that much larger, and
the enlarged pool is what left 691 MiB spendable against a 733 MiB need.
WHY THE EXCUSE IS UNSOUND -- and it is not that the measurement was wrong.
rung_fund_bytes is a SINGLE-POINT reading. The rung's deliverable is
(current_rows - floor_rows) * bytes_per_row, and floor_rows tracks the LIVE
SET. At the abandon:
current=473088 rows, floor=471983, slack=1105 -> 12.3 MiB deliverable
954 MiB promised, 12.3 MiB available. The funding collapses precisely when
occupancy is high, which is precisely when a seam is hard to fund. An excuse
granted at low fill is spent at high fill.
THE TWO CANDIDATE REPAIRS ARE ONE REPAIR. "Price the excuse against the worst
case at max fill" and "drop the excuse" give the same answer, because at max
fill current approaches floor_rows and the guaranteed deliverable is ZERO. So
the predicate is written in the general form -- excuse only what the rung is
GUARANTEED to deliver, via a new rung_guaranteed_bytes -- and a record that has
never measured a guarantee excuses nothing. The mechanism survives for a future
measurement instead of being deleted, and sgl-project#685's own condition ("conditional on
the recorded funding, never an unconditional removal") finally holds, because
the condition is now a bound the rung cannot fall below rather than a number it
was once observed at.
ON THE "THIRD PAYMENT" OBJECTION, which is the real tension and is answered
rather than ignored. TheArenaTailIsNotChargedTwice quotes
arming_floor_subtrahend_bytes: "Reserving it here as well would be the third
payment for one requirement." That reasoning holds only if payments one and two
actually discharge the requirement at flip time. Measured, they do not: the
rung covers 12.3 MiB of an 815 MiB tail at the fill where the seam runs. The
third payment is therefore not redundant -- it is the only one that holds when
it matters.
CORROBORATION I DID NOT CONSTRUCT. The restored floors are
rank0 1728 (unchanged), rank1 1728 -> 1825, rank2 1728 -> 2467 MiB
which is exactly SHIP_PIN.basis_arming_floor_mib = (1728, 1825, 2467). The pin
already encodes the UN-EXCUSED floors, so the excuse had silently undercut the
pin its own regime was derived at; this returns the live floors to it.
COST, stated plainly: ~8.7k tokens of pool on rank1 and ~83k on rank2. rank0 is
untouched. That is the price of a seam that can arm, against a pool that is
larger and cannot.
TESTS (hermetic, CUDA_VISIBLE_DEVICES=""):
test_arming_draw_excuse_696.py 6 passed (new, red-first: pre-fix it fails
with "138.9 not >= 814.0", reproducing the
PP1 arithmetic that produced the 42 MiB
shortfall)
test_arming_floor_funding_662 66 passed -- the four sgl-project#685 fixtures now state
the condition as a GUARANTEE. Their intent is
the conditional-relief mechanism, so only the
field carrying "the rung can pay" changed; no
expected value was flipped.
managers + mem_cache 64 failed / 1908 passed, the same 64 as the
pre-change baseline.
NOT DEPLOYED. Rides the next window with 8fb86ef, e21e87f, c738ef5
and the W-probe.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… dry run, and the vouch The notes were written at 3c984ad and had gone stale in three ways. Chain: 18 commits at e21e87f -> 22 at 5939d0e. Added sgl-project#697 (8fb86ef), sgl-project#441 (5e0fa1e), sgl-project#524 (5939d0e) and the revision-1 docs commit. Runtime-touching is now 8 of 22, and sgl-project#441 is recorded as NOT runtime despite its ticket -- it adds a handover script and one test guard. Target: integration/r2 has moved a73a0d8 -> a157bf1, so the dry run was re-run rather than carried over. Still zero conflicts. Scope shrank with the target's advance: 115 -> 109 commits, 129 -> 124 files, +22731 -> +22196. Merged-tree tests: managers 2093 passed / 0 failed; planner plus the three new test files 2594 passed / 2 failed. Those 2 are PpWithSpecEvidenceTest in test_rejected_evidence_pins.py and were re-verified as pre-existing by aborting the merge and running that file on clean a157bf1 -- same 2 failed, 2 passed. The chain introduces no new failure. Because the target tip moved, this was re-checked instead of quoting revision 1's finding. Holds: sgl-project#690 (e21e87f) and sgl-project#697 (8fb86ef) are in F4-r4's bundle, so revision 1's objection to landing sgl-project#690 ahead of the sgl-project#694 soak is spent; both lift to group 1 on his boot commit. sgl-project#524 (5939d0e) joins the hold list -- desk-verified only, never on metal, A/B window-gated. sgl-project#685 (5301b94) stays held on the unmade R' decision. Vouch (new section 5): the four F4-r4 cherry-picks are byte-identical to the originals by git patch-id --stable -- f630947/c41645c8c9, 7c58aba/ce6035884d, f1f31d2/658ea3ac11, de92bb6/84b0171fa6 -- so preferring the originals is free and the only real risk is double attribution. 1073702 is excluded: patch-id 8595e66885ac matches nothing here, it is diagnostic scaffolding and not mine to vouch for. Section 6 records that 5af1531 (sgl-project#696) and c738ef5 (sgl-project#689) are F4-r4's line, not this branch, and are listed only because they lift holds; sgl-project#696 is what was DoSing the lanes. Docs only. No merge performed; the dry-run worktree was removed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…s standing in front of
54 MINUTES OF SILENT OUTAGE, health 200 throughout. From 16:23:10, 325
consecutive lines of:
BOTH BLOCKED: nothing can run in the pp layout and the target cannot admit
either (0 req resident, 10495392 tok pending)
Last real batch 16:23:11: full token usage 1.00, #running-req 0, mamba 0.17.
The whole KV pool was radix cache with ZERO resident requests -- every row
unlocked and evictable -- while 10.5M tokens queued and three GPUs sat at 0%.
ONE SWALLOWED EXCEPTION. _post_evict_rows asked tree_cache.evictable_size().
MambaRadixCache does not return a number from that method; it raises
NotImplementedError and says "use full_evictable_size() and
mamba_evictable_size() instead". The probe caught it and used 0, so on the
class this rig runs it returned `available` ALONE -- the exact error its own
docstring warns about, committed three lines below the warning.
At usage 1.00 that reads ~0, so every admissibility question answered no:
pp could not admit, tp had nothing resident to decode, and sgl-project#688's BOTH BLOCKED
branch declined. That branch returns BEFORE alloc_token_slots -- so the
allocator was never reached, eviction never ran, and the unlocked cache was
never freed. The receipt called it "an evict trigger" while no evict could
occur, which is the same counter-vs-actuator shape as sgl-project#681/sgl-project#694 in a third
place: a message naming an action nothing performs.
CONFIRMED BY ABSENCE in the specimen (WEDGE-2026-08-16T1623Z.txt): zero RADIX
SHAPE, zero "Out of memory", zero EVICTION UNDER-DELIVERED. The allocation path
was never entered. py-spy shows the ranks spinning the event loop building
nothing.
THE TRAP IS ALREADY DOCUMENTED IN THIS TREE, at common.py:411-425, for these
same two classes -- and I read that comment the same day while diagnosing sgl-project#694
and did not apply it here. The resolution order is now COPIED from there rather
than re-derived, because two spellings of one rule is how this comes back.
WHY NOW. The bug shipped with sgl-project#688's admissibility simulation and needed
usage == 1.00 to bite. sgl-project#696's floor repair shrank the pool by 39,504 tokens, so
full occupancy arrived sooner and the wedge began 12 minutes after that boot.
sgl-project#696 EXPOSED this; it did not cause it.
THE SHAPE TO LEARN: a swallowed exception that yields a PLAUSIBLE value. Zero
is a legal row count, so nothing downstream could tell "the cache holds
nothing" from "the cache was never asked". Every accessor is now tried in turn
and only a genuine absence of all of them yields zero.
Health being 200 for the entire outage is the second lesson: the endpoint
answers while the scheduler builds no batch. It is not a liveness signal.
TESTS: test_post_evict_rows_698.py 5 passed (red-first: pre-fix it fails
"0 not greater than or equal to 150000", reproducing the wedge arithmetic).
managers + mem_cache 64 failed / 1922 passed -- the same 64 baseline.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…er a conservative floor
THE MEASUREMENT IS THE POINT AND THE GATE IS SECONDARY -- that way round, on
purpose, and it is a change from the shape I was asked for.
Three external-kill crashes now, all flip-proximate:
03:22:17 death, flip 03:21:05 -> rank0 lost ~7 s after the flip
03:31:17 death, flip 03:31:06 -> 11 s
03:38:53 death, flip 03:38:07 -> 46 s
Two of the three killed RANK0 ALONE (rank0 contributes 2 log lines while the
peers contribute 60-134 each -- the absent rank is the one that died), and
rank0 is PP0, the 35 G RSS process, the largest on the box. The third took the
whole tree symmetrically. A 21:24 kill is ledger-confirmed as the kernel OOM
killer.
BUT STEADY-STATE HOST HEADROOM IS ~38 G. So a floor small enough to be safe
would never have fired on any of these, and a floor large enough to fire would
defer constantly. I have NO measured projected host transient for the flip, and
I will not invent one to make a gate look decisive -- that is the hand-fitted
input mistake I already made once on sgl-project#702.
So the guard LOGS host availability at every flip, which nobody currently
records, and defers only under a deliberately conservative floor:
* availability collapses at flips -> transient candidate confirmed, and the
floor can then be set FROM DATA rather than from a guess;
* availability stays flat while kills continue -> the flip is exonerated and
lane RSS spikes gain.
The discriminator works whichever way it points, which is the property that
justified shipping ahead of attribution.
FLOOR = 4 GiB, justified from this box, not chosen round: ~10% of the measured
38 G steady headroom -- enough to absorb the lane RSS jitter sharing this
container (pytest/git spikes run under 2 G) without deferring on noise.
Deliberately NOT the 10 G PINNED_HOST_RESERVE_BYTES: that reserve protects
PERMANENT pins; a staging transient is by definition returned.
BOUNDED DEFER, 3 then escalate-and-proceed. A permanent hold is WORSE than the
hazard: the flip is how this instance alternates prefill and decode, so holding
forever converts a POSSIBLE process kill into a CERTAIN half-service outage --
and the kill is recoverable by a restore while the outage is not self-clearing.
The escalation says PROCEEDING WITH EYES OPEN and prints every term.
UNREADABLE HOST RAM STANDS THE GUARD DOWN. pinned_host_memory_bytes returns
(None, None) when there is no honest number, and refusing a flip on a
fabricated figure is worse than not checking -- the refusal is the thing with a
service cost.
sgl-project#696 INTERACTION, NAMED NOT ABSORBED: the defer reason is the shared constant
DEFERRED-HOST-RAM, so SLO / ARM-UNFUNDED accounting sees a host-RAM defer as
its own fact. A flip that did not arm because the HOST was tight is not the
same event as one that could not fund its VRAM seam, and merging them would
hide exactly the signal sgl-project#721 is collecting. It is a named defer reason, not a
new policy: the defer rides the existing unanimous too_small ballot.
Tests, hermetic (CUDA_VISIBLE_DEVICES=""):
healthy allows and does NOT name a defer ..... CAN-FAIL: a guard that
deferred on healthy state would pass every firing test and be useless
low headroom defers; boundary is exactly transient+floor
unreadable stands the guard DOWN
defer is bounded, then ESCALATED and proceeds
every term quoted; defer reason is the shared constant; degenerate inputs
-> 8 passed + 6 subtests; 82 passed + 19 subtests with flip-runtime and 703.
ruff clean in both changed regions.
CAN-FAIL PROVEN BY MUTATION: a never-defer mutant fails 5 of 8; reverted green.
The guard is wrapped so it can never break a flip: on any exception the flip
proceeds UNGUARDED and says so, rather than being refused on an unknown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.