support non-streaming benchmark - #682
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
cen121212
pushed a commit
to cen121212/sglang
that referenced
this pull request
Nov 10, 2025
* migrate daily build and pr * Update daily-build-test-npu-innersource.yml * Update pr-test-npu.yml * Create pr-test-npu-debug.yml * Update pr-test-npu-debug.yml * Update pr-test-npu.yml * Update pr-test-npu-innersource.yml * Update daily-build-test-npu-innersource.yml * migrate daily build and pr * Update npu_ci_install_dependency.sh * Create pyproject_npu.toml * Update npu_ci_install_dependency.sh * Update pr-test-npu-debug.yml * Update pr-test-npu-innersource.yml * Delete python/pyproject_npu.toml * Update pyproject_other.toml * Update nightly-test.yml * Update daily-build-test-npu-innersource.yml * Update npu_ci_install_dependency.sh
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… not hold
MEASURED 2026-08-16 02:07:22, all three ranks, mid-cutover, on
--max-running-requests 4:
PHASE-FLIP POOL CENSUS pre-cutover pp_to_tp: ...
cur_slot_reqs=5 resident_reqs=5 resident_slots=[0, 1, 2]
PHASE-FLIP-CARRY carried 5 resident request(s) ... into the tp phase
ResidentCarryError: running_batch claims 5 resident request(s),
above max_running_requests=4
The carry had already succeeded twice; the raise came from
`resident_req_identity`'s re-harvest, and it took the group down.
THE FIFTH RESIDENT IS NOT A CORRUPTED SET. It is a state the scheduler
creates ON PURPOSE. `_get_new_batch_prefill_raw` suspends the
running-request cap for as long as a chunked prefill is in flight, and its
comment names the reason it must:
# Ignore the check if self.chunked_req is not None.
# In PP case, chunked requests (or dllm requests) can start in one
# microbatch and end in another microbatch, so the max_running_requests
# per microbatch should not be strict. Instead, we should always allow
# chunked requests to be added, otherwise, there will be a memory leak.
So the maintained bound is `max_running_requests + 1`, and the guard was
asserting `max_running_requests`. The competing explanation -- that admission
simply over-admitted -- is excluded by `AdmissionLimiter`'s own contract: the
ceiling "is what the pools were built for and can never be exceeded", so
nothing could reach 5 through the admission limit.
EXACTLY ONE, because the scheduler holds exactly one: `self.chunked_req` is a
single slot, asserted empty before a new one is stashed. `cap + 2` therefore
remains a corrupted resident set and still raises -- this widens defect M's
ceiling by the one the scheduler documents, it does not remove it.
THE ALLOWANCE IS UNCONDITIONAL, AND THAT IS THE GROUP-UNIFORMITY ARGUMENT.
Gating it on `scheduler.chunked_req is not None` would be tighter and would be
a hang: that flag is per-rank scheduler state, the PP ranks sit at different
pipeline positions, and at one cutover instant a peer can hold it while this
rank has just cleared it -- the same resident set legal on one rank and fatal
on another. Deriving the ceiling from `max_running_requests` alone keeps the
verdict replicated: `init_admission_limiter` documents that value as "uniform
across ranks by construction: every input to `ceiling` is min-reduced before
it gets here". No collective added, no second channel.
THE SECOND COPY, WHICH IS WHY FIXING ONE WOULD NOT HAVE HELPED.
`phase_flip_draft_bootstrap.arm_draft_bootstrap` carries an INDEPENDENT
ceiling check -- correctly so, since `committed_slots` is where the
one-tensor-per-request allocation actually happens and it "checks that input
itself rather than trusting every present and future caller". It inherited the
same too-tight bound. The crashing configuration is PP->TP with NEXTN, so that
leg runs: repairing only the carry would have moved the same raise one
function later. Both now import ONE `IN_FLIGHT_CHUNKED_ALLOWANCE`, and
`test_the_two_ceilings_agree` pins that they cannot drift apart -- two guards
asserting two different bounds is the same defect with a longer fuse.
A RECEIPT, BECAUSE THE ATTRIBUTION IS AN INFERENCE. That the fifth resident
was the in-flight chunked prefill is established by ELIMINATION, not by
observation: no chunked request is visible in the 02:07 log, and the nearest
sgl-project#679 park was at 02:04:08, three minutes earlier. The park is NOT required for
this state -- with chunked_prefill_size 512 against 100k-token prompts a
chunked prefill is in flight most of the time -- but that also means the
attribution rests on the cap bypass being the only documented route past the
limit. So the harvest now logs, once, what was actually true when the
allowance was spent, including `chunked_req=SET|CLEAR`. A future excursion
reporting CLEAR means the elimination has a hole and this widening is covering
something else. It is a LOG, never a decision: the verdict reads only
`max_running_requests`, so the line cannot make two ranks disagree whatever it
prints.
CORRECTED IN PASSING: the `IMPLAUSIBLE_RESIDENT_REQS` comment asserted "no
batch on this server holds more requests than max_running_requests (4 on the
production recipe)", which is the same falsified claim one guard down.
TESTS. Red-first on e778276, both legs:
- carry: `test_the_cap_plus_the_in_flight_chunk_is_carried`,
`test_the_error_names_the_effective_ceiling_not_the_raw_cap`,
`test_a_ceiling_of_one_still_admits_its_chunk` (additive, not
proportional -- --max-running-requests 1 is a real configuration) and
the group-uniformity subtests all fail with ResidentCarryError before,
pass after.
- draft: `test_the_chunked_prefill_excursion_arms_instead_of_raising` and
`test_the_two_ceilings_agree` fail before, pass after.
- OPPOSITE DIRECTION, GREEN FROM THE START so the widening cannot
degenerate into removal: `test_a_sixth_resident_still_raises`,
`test_a_resident_set_two_above_the_cap_still_refuses_to_arm`,
`test_the_absurd_length_is_still_refused` (defect M's real shape, 5000),
and the pre-existing defect-M class unchanged. The constant is pinned
from BOTH sides -- 5 accepted, 6 refused -- so its value cannot drift
silently.
- receipt: fires above the cap with the chunked state named, silent at or
below it.
Suites: 81 passed across both guard files; 868 passed, 0 failed across every
phase-flip test in unit/managers.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
… what we remembered
THE ROOT QUESTION IS ANSWERED, AND MY OWN CANDIDATE WAS WRONG. I had named the
flip's "released 1410.0 MiB of weights-arena tail" as the reason the
reservation ends up below `_rows_at_boot`. It is refuted twice over: that is
the WEIGHTS arena, not the KV pool, and the KV reservation cannot move at all.
memory_pool.py:2458 reserved_num_tokens=self.size # at construction
kv_vmm_backing.py:979 self._reserved_num_tokens = int(...) # assigned ONCE
The reservation is pinned to the pool's size at the moment the arena is built
and never assigned again. `size` is NOT immutable -- the sgl-project#330 dial writes it on
every step, which sgl-project#662-F4 already noted one layer up. So a grow target derived
from a remembered or configured row count can sit above a ceiling that never
moves, and `_check_final` refuses it identically, forever.
MEASURED, AND UNCONFOUNDED: 59 times between 02:15:24 and 02:35:26 on
2026-08-16, a steady 3 per minute, once per rank per flip leg --
`recovery to 270646 rows failed: ... reserved=190596`, and the same shape on
the other two ranks (180428/108912, 179466/136140). That window opens before
any test-harness CUDA activity on the rig, so unlike the free-column readings
from 02:29 onward it is not confounded.
WHY IT IS BIGGER THAN THE LINE. Recovery is what LIFTS the backing cap. 59
refusals meant the cap never lifted, the pool stayed shrunk, and every later
`free_up_to` found the backing already at its target and honestly claimed 0
MiB -- which the shrink path then reported as an exhausted ARENA. One
unsatisfiable number, and the corridor guard's only rung above
`allocator-cache` was dead for the whole boot while its diagnostic pointed
somewhere else. That is the shape sgl-project#683 was opened on.
THE REPAIR IS THE SAME CORRECTION AS sgl-project#681 AND sgl-project#682: validate against what the
ACTUATOR can pay, not against the count that proposed it. sgl-project#681 was a token
count against a leaf frontier, sgl-project#682 a guard ceiling against the bound the
scheduler actually holds, and this is a grow target against an immutable
reservation. So the clamp is deliberately NOT conditional on knowing why the
remembered number went stale -- it asks the bound.
CLAMP *AND* RE-DERIVE, because the clamp alone would only convert a loud
failure into a quiet one: `_rows_at_boot` would still name an impossible level
and every later recovery would re-clamp to the same place while believing it
had further to go. Correcting it lets the existing "fully recovered" branch
fire, which clears the remembered rows AND retires the exhaustion marker --
the latch that kept the rung off.
RANK-LOCAL, EXPLICITLY, as the brief asks. A reservation is one card's VA span;
under uneven TP the ranks hold different ones -- 190596 / 136140 / 108912 on
this boot -- so there is no group quantity here to agree on. `recover` takes no
collective, and this commit adds none. The module's collective, the sgl-project#656 C22
cap agreement, is on the SHRINK target and is untouched. The new accessor is
also NOT `_reservation_rows` (the allocator's id space, which does feed
`exposed_rows` and that agreement); the two are cross-referenced in code so a
later reader cannot conflate them.
SAFE DESK-SIDE, AND THE JUDGEMENT IS ASKED FOR, SO HERE IT IS. Two properties
make this shippable without a GPU window:
* the clamp fires ONLY where `rows > ceiling`, which is exactly the path that
currently fails 100% of the time. On any boot where recovery works today
the branch is inert, so there is no working behaviour for it to change.
* it runs AFTER the corridor-affordability bound, so when both bite the
target is the smaller of the two and the clamp can only LOWER it. Raising
it would commit pages the corridor law had already refused -- the failure
that drove rank 1 to 6 MiB free and OOMed inside relief. Pinned by
`test_the_clamp_can_only_lower_the_target_never_raise_it`.
A pool that exposes no reservation keeps its previous behaviour exactly; 0 is
read as "no arena", never as a ceiling of zero, which would be a shrink wearing
a grow's name.
TESTS, red-first. The acceptance pin committed with the verification --
`test_recovery_is_refused_forever_because_nothing_clamps_it` -- was inverted to
the post-fix expectation FIRST and failed, together with the re-derivation pin;
both pass after. It keeps its name: it asserted the defect before the clamp and
asserts the repair after it, which is what an acceptance pin is for. Seven
cases in all, four of which exist so the fix cannot pass by being broken
everywhere: the control (a reservation above the boot rows recovers normally),
the affordability bound still deferring untouched, the clamp/affordability
interaction, and the two backward-compatibility contracts.
Sweep: 2433 passed across unit/managers + unit/mem_ledger + the 681/682 files.
Four failures in that run are pre-existing and unrelated -- same four, same
messages, on the untouched tree (`BudgetHarness` and `_Sched` stubs missing
attributes in test_collective_family_siblings_610 and
test_first_chunk_dynamic_chunking); neither file references anything this
commit touches.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…es the pool forever FOURTH LATCH OF THE NIGHT, and the same cure as the other three: sgl-project#681's eviction count that could not be paid, sgl-project#682's guard ceiling the scheduler never held, sgl-project#684's `_exhausted_at_rows` process-lifetime marker. Each was a number that could only ratchet one way. WHAT IT COST. `corridor_shortfall_bytes` is added straight to the arming floor's load margin -- `(DEFAULT_MARGIN_MIB << 20) + measured` -- and the arming floor is the binding constraint on two of three ranks. On 2026-08-16 the rank-0 record carried 1004 MiB of it while every record written the day before carried 0, and the boot reading it logged NO breach of its own: it was inherited. The event it descends from is almost certainly 02:36:30 on that exact card, where a test harness belonging to this strand held 4.29 GiB and drove free to 76 MiB. A few seconds of intrusion, taxing every subsequent boot. THE OLD SEMANTICS WERE HALF RIGHT, AND THAT HALF IS KEPT. `record_corridor_shortfall` documents itself as "A MONOTONIC MAXIMUM, deliberately -- a shallower breach later does not mean the deeper one cannot recur; the pool must be sized for the worst instant that has ever been seen". Correct WITHIN an observation. Wrong ACROSS boots that never see it again, because "ever" had no end and nothing could retire a number nobody could reproduce. So: monotonic maximum while it is being OBSERVED, geometric decay across boots that observe nothing. A breach that recurs is re-observed and re-raised to its worst on the spot. One that cannot be reproduced is halved by each flip boot that measures its seam without seeing it, and written off to exactly 0 below `SHORTFALL_FORGET_BYTES` so the decay terminates instead of leaving a tail that still moves the floor. 1004 MiB is gone in seven clean boots. "OBSERVED BY THIS PROCESS" IS THE DISCRIMINATOR, and it is a pid rather than a timestamp because both writers live in the same process: the runtime's corridor audit stamps the record mid-run, and `write_seam_reserve` rewrites it at the end of the same boot's flip measurement. Same pid means this boot saw it and the value stands; a different pid means it was inherited, and a boot that completed a seam measurement without its audit firing is evidence against it. Evidence is what retires it. RANK-LOCAL. The record is per (configuration, rank) and the shortfall is one card's own measurement -- 1004 / 0 / 0 on this boot, legitimately different. No collective reads or writes it and this change adds none. TESTS, red-first: 7 cases. The three decay cases failed before and pass after; the four that pin the half worth keeping -- a breach this process observed is preserved, a deeper one still raises, a shallower one does not lower it -- passed from the start, so the fix cannot have been "delete the term". One case drives the full production scenario: a 1004 MiB one-off decaying to zero while the load margin returns to its default, and one that proves a breach observed on every boot is never decayed away.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…parked carrier NOT WIRED YET, AND SAID SO UP FRONT. This is the pure decision core with its pins; the scheduler wiring (park at the finished-prefill boundary, re-admit at TP entry, feed the slot bound into get_num_allocatable_reqs) is a separate change and this module currently has no caller. A module nothing calls measures nothing -- that is why it is labelled rather than claimed. THE WEDGE WAS A COUNTING DEFECT. At 2026-08-16 06:04 the instance held twelve GDN slots with EIGHT FREE, four running against a cap of four, and 403779 tokens of prefill it could not admit. Freeing a GDN slot would have relieved nothing: admission is `min(pp_max_micro_batch_size, limiter.current) - running_bs` then `min(..., req_to_token_pool.available_size())`, and HybridReqToTokenPool does not override available_size, so that second term is the REQUEST-slot count. Neither term sees the GDN pool at all; the mamba allocator is consulted only later, inside alloc_req_slots. What blocked admission was four requests PP is FORBIDDEN to decode being counted against the concurrency cap for the whole residency. NOTHING MOVES IN PHASE 1. The carrier keeps its GDN slot, its KV -- exactly the KV that would have been resident anyway -- and its req_to_token row. That is deliberate: no state movement means no new correctness surface from the sgl-project#450/sgl-project#444 verify-write family, the sgl-project#461 DEVICE_BOUND law, or sgl-project#551 GDN-Vacate x kvso. The blob park is phase 2, gated behind the sgl-project#551 read. BOTH BOUNDS ARE SOLVED FROM BOOT DIMENSIONING and phase 1 raises nothing: parked + running <= the GDN slot pool (12), refused EARLY and by name because alloc_req_slots would refuse it late anyway; and running_bs <= max_running (4) at all times including TP, with re-admission in capture-set-sized batches so every pool stays inside what it was built for. THE ONE BOOKKEEPING EDGE is that a parked request is still RESIDENT -- `resident_ids` exists for the pressure ladder and retract paths, so a parked carrier is neither double-counted (it is out of running_batch) nor invisible (it is in the resident set). A PARK FAILURE DEGRADES TO THE SAFETY NET, pinned: disabled or slot-pool-full, the arithmetic is byte-identical to the pre-change gate, the request stays a carrier and keeps counting, and the sgl-project#677 progress exit still breaks the stall. The failure mode is the behaviour it replaces, never a wedge. A HAZARD THE WIRING MUST ANSWER, found while building this and recorded here rather than discovered on metal: a parked request is outside running_batch, so sgl-project#682's `harvest_resident_batches` and the seam's KV reshard would not see it. Phase 1 keeps its KV, so the seam must still carry it -- `resident_ids` is the hook for that, and the wiring change has to use it or a parked request's KV is left behind in the PP layout at the cutover. 16 hermetic cases, red-first: the 06:04 scenario with four carriers parked and the fifth prefill admitting; a control stating the composition exactly (this is an ADDITIONAL slot-pool bound, not a replacement for the caller's gate); parked+running never exceeding 12; TP re-admission never exceeding 4 and FIFO so no carrier starves; both receipts naming id, set size and binding bound; the disabled path reproducing the old gate; and evacuate() handing every parked request back so a crash strands none.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.