fix: update bench serving by zhyncs · Pull Request #694 · sgl-project/sglang

zhyncs · 2024-07-22T08:09:43Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

Submit issue 381 and resolve the code.

…e two commits to hold MERGE_NOTES_602.md rewritten to cover all 18 commits: what each fixes, its test evidence, whether it touches runtime, and whether an equivalent patch is already on the serving line (verified with `git cherry`, not by message matching -- four are: c41645c, ce60358, 658ea3a, 84b0171). DRY RUN: clean. Merged into `integration/r2` -- the live line, since the serving tree descends from its tip a73a0d8 -- in a throwaway worktree, `--no-commit --no-ff`, then aborted and the worktree dropped. Zero conflicts, zero unmerged paths, so nothing was pre-resolved because nothing needed it. Verified semantically as well as textually: on the MERGED tree, managers 2093 passed / 0 failed and planner 2574 passed / 2 failed, the two being the same pre-existing test_rejected_evidence_pins pair that is already red on the base. THE FACT THE OPERATOR NEEDS, and it is not in the commit count: `7936bc4850` is NOT an ancestor of integration/r2, so merging this branch drags in its whole base lineage -- 115 commits, of which 18 are mine and 97 are the hotfix/677 work (sgl-project#662 x20, [PhasePolicy] x18, sgl-project#677 x8, sgl-project#678 x7, sgl-project#679 x6, ...), 129 files, +22731/-762. Approving this merge is approving that lineage, most of which is not mine to vouch for. If only this work is wanted it must be cherry-picked rather than merged. NOT ATOMIC, and it splits cleanly into four groups with an order: (1) the four already on serving -- merging them only reconciles integration with what is already running; (2) the three sgl-project#624 test-only drift guards, which take managers from 4 failures to 0 and should land early so the line stays green during review; (3) desk tool + docs, all planner/pp_cut.py and markdown, imported by no serving path; (4) hold. HOLD, two commits, both runtime and neither on the serving line: * e21e87f (sgl-project#690) touches the seam hot path and changes the PHASE-FLIP DONE format. Already queued to land on deploy WITH the W=8/W=4 probe after the sgl-project#694 soak verdict; merging it into integration first puts it in front of the soak meant to measure it. * 5301b94 (sgl-project#685) touches the boot sizing path. Announce-only today and abstention-guarded, but unsoaked, and the R' decision it waits on is not made. Nothing else in the chain can move serving behaviour. Docs only; no merge performed, no deploy, scratch worktree removed.

…n retraction disproved Live at 15:00:41: seam staging asked the corridor guard for 5 MiB (tp_to_pp): ok=False, spendable now 1677 MiB against a need of 1682 MiB. The binding rank's inactive-layout arena is the occupant this ask exists to reclaim. That last sentence is false and I wrote it. 0263713 retracted the premise -- arena_fixed_bytes is the tp_to_pp leg's FUTURE commit cost, not memory held idle, and rung 3 already releases the real tail after every refill -- and fixed the label in the staging census. It did not fix this second copy of the same claim in the guard-ask line, so the receipt kept pointing the next reader at a reclaim that does not exist. Fixing one of two copies is how a corrected finding comes back. Replaced with what ok=False actually means here: under must_reclaim the verdict is the measured DELTA, so a False says the ladder released nothing and every registered provider is dry -- which is what the co-reported KV rung line (current=147456 rows, floor=359770, slack=0) independently shows. Receipt-only. Committed, NOT deployed: the sgl-project#694 soak is running on de92bb6 and a wording fix does not justify interrupting it.

… dry run, and the vouch The notes were written at 3c984ad and had gone stale in three ways. Chain: 18 commits at e21e87f -> 22 at 5939d0e. Added sgl-project#697 (8fb86ef), sgl-project#441 (5e0fa1e), sgl-project#524 (5939d0e) and the revision-1 docs commit. Runtime-touching is now 8 of 22, and sgl-project#441 is recorded as NOT runtime despite its ticket -- it adds a handover script and one test guard. Target: integration/r2 has moved a73a0d8 -> a157bf1, so the dry run was re-run rather than carried over. Still zero conflicts. Scope shrank with the target's advance: 115 -> 109 commits, 129 -> 124 files, +22731 -> +22196. Merged-tree tests: managers 2093 passed / 0 failed; planner plus the three new test files 2594 passed / 2 failed. Those 2 are PpWithSpecEvidenceTest in test_rejected_evidence_pins.py and were re-verified as pre-existing by aborting the merge and running that file on clean a157bf1 -- same 2 failed, 2 passed. The chain introduces no new failure. Because the target tip moved, this was re-checked instead of quoting revision 1's finding. Holds: sgl-project#690 (e21e87f) and sgl-project#697 (8fb86ef) are in F4-r4's bundle, so revision 1's objection to landing sgl-project#690 ahead of the sgl-project#694 soak is spent; both lift to group 1 on his boot commit. sgl-project#524 (5939d0e) joins the hold list -- desk-verified only, never on metal, A/B window-gated. sgl-project#685 (5301b94) stays held on the unmade R' decision. Vouch (new section 5): the four F4-r4 cherry-picks are byte-identical to the originals by git patch-id --stable -- f630947/c41645c8c9, 7c58aba/ce6035884d, f1f31d2/658ea3ac11, de92bb6/84b0171fa6 -- so preferring the originals is free and the only real risk is double attribution. 1073702 is excluded: patch-id 8595e66885ac matches nothing here, it is diagnostic scaffolding and not mine to vouch for. Section 6 records that 5af1531 (sgl-project#696) and c738ef5 (sgl-project#689) are F4-r4's line, not this branch, and are listed only because they lift holds; sgl-project#696 is what was DoSing the lanes. Docs only. No merge performed; the dry-run worktree was removed.

…s standing in front of 54 MINUTES OF SILENT OUTAGE, health 200 throughout. From 16:23:10, 325 consecutive lines of: BOTH BLOCKED: nothing can run in the pp layout and the target cannot admit either (0 req resident, 10495392 tok pending) Last real batch 16:23:11: full token usage 1.00, #running-req 0, mamba 0.17. The whole KV pool was radix cache with ZERO resident requests -- every row unlocked and evictable -- while 10.5M tokens queued and three GPUs sat at 0%. ONE SWALLOWED EXCEPTION. _post_evict_rows asked tree_cache.evictable_size(). MambaRadixCache does not return a number from that method; it raises NotImplementedError and says "use full_evictable_size() and mamba_evictable_size() instead". The probe caught it and used 0, so on the class this rig runs it returned `available` ALONE -- the exact error its own docstring warns about, committed three lines below the warning. At usage 1.00 that reads ~0, so every admissibility question answered no: pp could not admit, tp had nothing resident to decode, and sgl-project#688's BOTH BLOCKED branch declined. That branch returns BEFORE alloc_token_slots -- so the allocator was never reached, eviction never ran, and the unlocked cache was never freed. The receipt called it "an evict trigger" while no evict could occur, which is the same counter-vs-actuator shape as sgl-project#681/sgl-project#694 in a third place: a message naming an action nothing performs. CONFIRMED BY ABSENCE in the specimen (WEDGE-2026-08-16T1623Z.txt): zero RADIX SHAPE, zero "Out of memory", zero EVICTION UNDER-DELIVERED. The allocation path was never entered. py-spy shows the ranks spinning the event loop building nothing. THE TRAP IS ALREADY DOCUMENTED IN THIS TREE, at common.py:411-425, for these same two classes -- and I read that comment the same day while diagnosing sgl-project#694 and did not apply it here. The resolution order is now COPIED from there rather than re-derived, because two spellings of one rule is how this comes back. WHY NOW. The bug shipped with sgl-project#688's admissibility simulation and needed usage == 1.00 to bite. sgl-project#696's floor repair shrank the pool by 39,504 tokens, so full occupancy arrived sooner and the wedge began 12 minutes after that boot. sgl-project#696 EXPOSED this; it did not cause it. THE SHAPE TO LEARN: a swallowed exception that yields a PLAUSIBLE value. Zero is a legal row count, so nothing downstream could tell "the cache holds nothing" from "the cache was never asked". Every accessor is now tried in turn and only a genuine absence of all of them yields zero. Health being 200 for the entire outage is the second lesson: the endpoint answers while the scheduler builds no batch. It is not a liveness signal. TESTS: test_post_evict_rows_698.py 5 passed (red-first: pre-fix it fails "0 not greater than or equal to 150000", reproducing the wedge arithmetic). managers + mem_cache 64 failed / 1922 passed -- the same 64 baseline.

…s the ledger the host had ROOT. evict_from_tree_cache gates the eviction on uniform_avail_for_evict(...) < num_tokens. That reads tree_cache.uniform_avail_floor, published ONCE per iteration (scheduler.py:4142-4144) as the group MIN of available_size(). Allocations made later in the same iteration were never charged against it, so late in an iteration the number is stale-OPTIMISTIC: with floor >= num_tokens the eviction is SKIPPED entirely, the alloc then fails against the live pool, and the raise reports a tree full of evictable tokens nothing ever asked for. Not a new class -- the HOST sibling already had this fixed. Its own comment states the reasoning: "a stale floor over-admits ... charging admissions against the floor removes the staleness without a second collective". The DEVICE sibling never got the ledger. This adds it, mirroring sgl-project#645 exactly: uniform_admitted_since_floor, charged on the success path of alloc_token_slots, reset by the scheduler in the same call that publishes the next floor so it never outlives the number it corrects. Sufficient, by sgl-project#645's argument: live availability is at least avail_at_publish - admitted, and avail_at_publish >= floor, so a request clearing floor - admitted fits the real pool. Rank-uniform by construction: num_tokens comes from the replicated batch, so every rank charges the same amount at the same allocation and the predicate stays identical across ranks -- the #616g invariant this must not break, pinned by a test. fundable_extend_tokens reads the same predicate, so admission inherits the correction for free. OVERLAP VERDICT vs sgl-project#701(a), checked in code rather than assumed: NOT one defect, on the available evidence. * evictable and protected are DISJOINT by construction -- inc_lock_ref moves tokens out of evictable_size_ into protected_size_ (radix_cache.py:605-606) and dec_lock_ref moves them back (:622-623). So a reported evictable count never includes protected-prefix pages. * The only specimen present in any accessible log is 66039 (available=273 + full_evictable=65766, 512 requested). Its "Full LRU list evictable size: 65766" matches full_evictable_size EXACTLY. That sanity check is an independent traversal of the eviction list, and divergence is precisely what it exists to detect -- so this specimen's evictable was genuinely reachable, refuting the paper-evictable hypothesis for it. It is also the specimen sgl-project#681 already diagnosed (mamba tombstone leaf) and paid in MambaRadixCache. evict_full. * The "167k evictable" specimen is NOT in any log I can reach, so its decomposition into mamba-recoverable vs paper-only cannot be done and is not inferred. If F4-r4's 1f594e7 instrument catches a recurrence, its skipped-vs-ran line decides it directly. So the staleness defect is proven STRUCTURALLY (the host/device asymmetry) and fixed with a can-fail falsifier; it is not claimed as the cause of a specimen whose instrument output does not exist. One self-inflicted defect caught by the suite: getattr(tree_cache, "uniform_admitted_since_floor", 0) yields a Mock on an unconfigured double, and int(Mock()) is 1, not 0 -- silently shaving a token off the floor and breaking test_a_published_floor_is_returned (499 != 500). Guarded by an isinstance check, with the reason recorded. Third appearance of the sgl-project#624 stub-drift class. The host sibling carries the identical latent exposure and is left untouched here to keep the blast radius small; worth a follow-up. Tests: 7, red first, including the can-fail proof that an uncharged stale floor really does skip. managers 2113 passed / 0 failed. mem_cache is 944 failed / 772 passed both before and after this change -- a large PRE-EXISTING red suite, verified by patch round-trip on the clean tree (945/2870 baseline across managers+mem_cache without the new file). ruff clean on all three files, compared against HEAD.

…t the single chokepoint ## What is wired ChunkedCommitmentLedger is now visible to PrefillAdder, flag-gated DEFAULT ON (chunked_admission_enabled). Off restores the pre-sgl-project#701 arithmetic byte-for-byte, so the flag is the A/B arm for a reviewed window. THE LEDGER IS OWNED BY THE SCHEDULER, NEVER BY THE ADDER. A PrefillAdder is rebuilt every pass, so anything it held itself would forget a resident chunked request's outstanding prefill exactly when the next pass needs it -- which IS defect (b). It is passed in and never constructed there. The subtraction lands at the single chokepoint, rem_total_tokens, via effective_rem_total_tokens(budget, ledger). That covers ALL FOUR sibling sites at once (:1569-1610, :1388-1407, :1230-1276, :1124-1141) because every one of them reads that property rather than recomputing the budget. Four hunks, 29 insertions; :734-737 deliberately UNTOUCHED so the merge train with Slot-2's branch stays clean. ## Falsifier: the two-actor deadlock through the REAL PrefillAdder A resident request commits 80,000 tokens; a FRESH adder (the rebuild) then reports 100,000 unwired and 20,000 wired. Spending returns the budget incrementally, release returns it fully, and a missing ledger is not an error. Red on unwired, green on wired, same adder both ways. ## Defect (a) is NOT wired, deliberately An earlier draft of this commit priced rem_total_tokens against "actually recoverable" rather than full_evictable_size(). THAT HYPOTHESIS IS REFUTED (sgl-project#694, Slot-2 704240c): evictable and protected are DISJOINT BY CONSTRUCTION -- inc_lock_ref moves tokens out of evictable_size_ into protected_size_ (radix_cache.py:605-606) -- and the readable specimen's LRU evictable size matched an independent traversal exactly. The real root was a STALE FLOOR (uniform_avail_for_evict published once per iteration, never charged by later allocations), fixed on Slot-2's branch. The change and its tests are reverted rather than left passing against a mock of a distinction the code does not make. A note in the test file records why, so the idea is not re-derived a third time. Also removed on the way: the harness mocked `mamba_recoverable_size`, which exists NOWHERE in mem_cache/ -- only mamba_evictable_size does. A test built on that mock would pass against itself and fail against the real tree. ## Flip integration: SPEC ONLY, for F4-r4 phase_flip_runtime and the idle-flip integration are his boundary and are not touched. DESIGN_704 carries the contract: (1) ask deferred_head_blocks_idle_flip before arming, since a deferred head is pending work not idleness; (2) count resident chunked requests via effective_running_bs (sgl-project#631 defect O -- a resident-but-batchless request appears in no running_batch.reqs); (3) outstanding_tokens() is non-zero exactly while some prefill is unfinished, and arming across that applies the seam move to KV still being written -- drain or refuse, his call since he owns the window economics. All three are read-only questions against an object the scheduler already holds. ## Third-site sweep Per the standing instruction: no further site found where rem_total_tokens is funded by a quantity the relief path cannot deliver. Had one appeared it would be reported with file:line, not fixed inline. Tests green, hermetic. Lint on the touched test file is unchanged from HEAD (3 pre-existing findings, none introduced); schedule_policy.py deliberately NOT run through ruff format, which reformats unrelated legacy code and would have doubled the diff.

…ling included Third appearance of the stub-drift class earns the consolidation now. Both floor correctors read a per-iteration ledger off the tree cache: the DEVICE one in uniform_avail_for_evict (sgl-project#694) and the HOST one in uniform_host_avail_for_backup (sgl-project#645). Real caches declare the attribute on BasePrefixCache with a 0 default; a test double usually does not, and int(Mock()) is 1, not 0 -- so an unconfigured stand-in silently shaved one token off whichever floor it fed. Measured on both: 499 != 500 against the device ledger (test_a_published_floor_is_returned), and reproduced against the host ledger by a new test that leaves uniform_host_admitted_since_floor unconfigured. The host exposure was real, not theoretical. Rather than a second isinstance check that would drift from the first, both sites now go through _ledger_tokens, whose docstring IS the named list the audit asked for -- there is no separate registry doc to add them to, and a helper both callers must use is a stronger guarantee than a list someone has to remember to read. bool is excluded deliberately: it is an int subclass, so a truthy flag landing there would charge 1 token for exactly the reason a Mock did. Tests: 5 more, red first -- the host case failed with the identical 499 != 500 signature before the fix, which is the can-fail proof that the sibling shared the defect rather than being fixed by coincidence. managers + the sgl-project#645 host suite + the sgl-project#694 suite: 2131 passed, 0 failed. ruff clean.

…authorities only WIRED VS PRICED-ONLY, answered by grep before building: nothing imported coresidency_registry outside its own module, so a tenant COLD event did NOT actuate -- the bridge priced and no one called it. Same for HOT. That is the gap Cut 3 closes. managers/coresidency_policy.py is the caller and introduces NO NEW ACTUATOR. Every move is an existing authority's own API: vram_dial.apply_budget_request (replicated grow/shrink, already enforces the floor, "rejections carry the exact floor arithmetic and change nothing"), vram_dial. verify_pool_reached_capacity (the read-back), GdnSlotRuntime.unbind (sgl-project#364), and the Cut 1/2 bridge. A source pin forbids this module from touching torch.cuda / cuMem / set_active_prefix / runtime_set_ at all. THE TWO DIRECTIONS FAIL DIFFERENTLY AND THE CODE IS SHAPED BY THAT. COLD -- shrink must not strand bytes. Every source drawn on is recorded with what it was asked for and what it REPORTED giving. A source asked that reports nothing is carried as STRANDED: never counted as zero, never dropped. Bytes that left one ledger and entered none are the shape that goes unnoticed for weeks. A delivered zero is an accounting; silence is not, and the result keeps them distinguishable. Per sgl-project#694, totals come only from actuator reports -- nothing increments a "reclaimed" figure from the plan. An unfundable ask refuses and actuates NOTHING rather than drawing what is there, and the refusal states that unavailable bytes did not count toward it. HOT -- grow must not exceed the floor, and that is deliberately NOT enforced here: the dial enforces it. This module's duty is to not paper over the refusal, so a floor refusal is returned unchanged and is NEVER RETRIED SMALLER -- a floor refusal is a statement about the rig, not a negotiation. sgl-project#217 SHAPES THE HOT PATH. A restore that "came back" was measured at 23% of target. So a grow is followed by a read-back and the result reports what was MEASURED, never what was requested. An absent read-back leaves reached_bytes None -- "not measured", distinct from "reached nothing" (sgl-project#606). A read-back that raises REFUSES, because a tenant must not be reported warm on an unverified grow. Tests, hermetic (CUDA_VISIBLE_DEVICES="", real bridge objects, injected actuators, no scheduler, no CUDA): test_coresidency_policy_553.py 16 passed + 4 subtests MUTATIONS: hiding stranding, and reporting the request instead of the measurement, fail 4 test/registered/unit/managers/ 13 failed / 2263 passed -- failures unchanged from this branch's pre-existing 13, passes 2247 -> 2263, exactly the 16 new pins ruff + codespell clean. STILL PRICED-ONLY, and this is the honest seam: the policy takes injected release_fn / grow_fn / measure_fn. Binding them to the live dial and slot runtime needs a scheduler and belongs with the live proof, not a desk fake. WINDOW TICKET: /spinning/GPU_WINDOWS.md. There was NO prior sgl-project#553 entry (grepped; none), so this CREATES one rather than appending to an existing ticket, and says so in the entry. The 142 pre-existing lines are verified byte-identical by sha256 before and after (18e21da5dda7ce55).

The analysis's tenant-COLD row named the gap exactly: parking 'works for the translator (ledger.park_all) and for GDN slots (sgl-project#364). No generic per-tenant mover; no single caller that can address tenant X.' The first half wired the two byte probes so the bridge stops answering all-unavailable; this is the actuator a cold event calls once that bridge has ranked a plan. I CHECKED THE TABLE'S GATING COLUMN RATHER THAN TRUSTING IT, because §10 had just caught the same table lying about numbering. test_ledger.py runs 22 passed under CUDA_VISIBLE_DEVICES=: the ledger's park/restore is already exercised hermetically with CPU tensors, so a mover over the same protocol is desk- provable and the marking holds. Had those tests needed a device this would have been a window item and the table would have been wrong twice. NOT A SECOND LEDGER. AudioAssetLedger already parks and restores and its ParkRoute protocol is already generic; what was missing was never the parking. TenantMover registers TENANTS over that same protocol and reimplements nothing. A parallel asset ledger would have been the two-authorities defect the sgl-project#553 bridge exists to reconcile. The vocabulary is per tenant, which is the whole factoring: the translator's ASR -> talker -> codec order is one tenant's physics, not a property of moving tenants. register(tenant, routes, ranks=...) takes it as configuration, and a test asserts a second tenant restoring y, x -- nonsense for the translator, correct for that tenant. An unranked route sorts last: unlisted is unknown, not urgent. Stranding keeps the shape the policy layer already uses. Released bytes come from what a route RETURNED (sgl-project#694); a route that reports nothing, or raises, is STRANDED and never zero, while a route reporting 0 is an accounting. release_fn() hands cold_event a None in exactly that case, which is how that layer reads 'did not report', so a stranded tenant surfaces there instead of being counted as a delivered zero. parked_bytes_by_device() omits a device with nothing parked rather than reporting 0 (sgl-project#606). An unknown tenant RAISES and names who is registered: 'no such tenant' and 'that tenant had nothing to give' are different answers. A defect in my own test, recorded in the analysis: the first _Route stub used parked_bytes=None for 'not specified', so 'reports nothing' was inexpressible and two stranding tests failed against correct code. A stub whose default and whose absence-case share a value cannot test an absence rule. 12 tests red first, mutation-proven (collapsing stranding to zero reds the two stranding pins; ignoring the supplied ranks reds all three vocabulary pins). 57 passed across the mover, the rung executor, the coresidency policy and the translator ledger -- the ledger's 22 unchanged, it was not touched. ruff clean. Desk only, no boots. sgl-project#553's desk surface is now exhausted: cut 4 is gated on sgl-project#305's multi-model binding, cut 5 is rig-gated by its own text, G1 is filed, and the live bindings plus any live number's correctness remain the named window items.

Both PP void sites released their non-resident requests with `_release_dynamic_chunk_probe`, which hands back KV rows, the mamba slot and the req-pool row by calling the allocator and the pool DIRECTLY. Nothing on that route reaches `cache_finished_req`, and that is where `dec_lock_ref(req.last_node)` lives -- the last two lines of every implementation of it. So the pages went back to the allocator while the radix tree kept its lock on them, and `reset_for_retract` cleared `last_node` one line later, putting the ref out of reach for the life of the process. One leaked lock per voided request. WHY IT EXPLAINS THE BOOTS. `evictable_size_` counts UNLOCKED tokens, so a tree whose nodes are all pinned reports nothing evictable at all. Every funding rung reads its reclaim off that number: "reclaimed 0 MiB from [nothing]" against a full tree is not a pressure reading, it is the statement that nothing in it was ever unlocked (sgl-project#813/sgl-project#694 family). NOT A THIRD MECHANISM. `release_req` (schedule_batch.py) already IS this discipline and both correct retraction paths -- `retract_decode` and `retract_all` -- are its only callers. The void sites join them through `_release_voided_request`, which holds no release logic of its own: it supplies the scheduler's collaborators and this path's never-raise contract, and delegates. Because `release_req` calls `reset_for_retract` as its last act, the call sites no longer reset themselves -- doing both double-counts `retraction_count`, which feeds `retract_decode`'s solo-OOM abort ladder. THE PROBE KEEPS ITS OWN RELEASE, deliberately. The profiler's request is a bare `Req` that is never matched against the tree and holds no ref, and `UnifiedRadixCache.cache_finished_req` decrements `req.last_node` with no None guard. Routing it through the disciplined path would be a decrement with no matching increment -- the same accounting defect mirrored (sgl-project#929). Two premises, not one discipline written twice; both docstrings now say so. SIBLING, SWEPT AND FIXED IN THE SAME PASS. `DecodeKVCacheOffloadManager._release_finished_req` ended with `tree_cache.protected_size_ -= len(req.prefix_indices)`, a hand-rolled imitation of one of the four effects `dec_lock_ref` has per node on the path to root; `lock_ref` was never touched, so the same leak by another route. Reachable with --disaggregation-decode-enable-radix-cache alongside the offload flag. Fixed rather than filed because it is decidable by inspection: under the `ChunkCache` default the old line was wrong in the OTHER direction, since that cache's `inc_lock_ref` never raises the counter it subtracted from. TESTS. test_pp_void_lock_ref_969.py, new, 14 tests + 2 subtests. Red-first, PP void, before the fix: 7 failed / 6 passed -- `lock_ref` stayed 1, zero `dec_lock_ref` calls reached the node. After: all green. Red-first, sibling, with the fix reverted: 2 failed on the lock assertion itself (the fake carries `protected_size_`/`evictable_size_` so the old code RUNS instead of raising -- a red for a thin harness is indistinguishable from a red for a broken product). Gegenrichtung: `test_the_underflow_detector_can_actually_fail` drives a doubled decrement at the detector and requires it to fire, so the two one-decrement-per-request assertions cannot go green through a sgl-project#929 underflow. Source pins run through `ast.unparse`, not raw `getsource`: both void sites NAME the probe helper in prose that stays true after the fix, and a pin that reads the comments is not a pin. test_pp_void_chunked_retracted_798.py: its stub list gains `_release_voided_request`. The stub RESETS the request, because the real one does; a no-op stub would have silently removed the precondition that file's first test asserts. DESK GATE (scripts/gate_tier2_partitioned.py, CUDA_VISIBLE_DEVICES="") BEFORE 4849 passed, 2 genuine (test_collective_family_siblings_610.py x2) 619.34 s AFTER 4863 passed, 2 genuine (the same two) 638.07 s delta +14 = exactly the new module. Count probe: 2 named == 2 in summary, SUBFAILED and ERRORS included, both zero. Lane shift explained, not waved past: wide 3701 -> 3695, serial 946 -> 966. Editing test_798 invalidates its sha256 proof in scripts/gate_partition.tsv, so the runner demoted it to the serial track, where it was measured green -- the table's expiry mechanism working as designed. The new module has no row and was measured in serial for the same reason. Re-proving both rows needs a full serial run plus per-module solo logs; NOT done here, and named as open. ruff: 2 F841 in the touched files, both pre-existing at HEAD (`inc_len`, `carries_flip_arm`), none introduced. codespell clean. NOT CLAIMED: no boot, no metal, no effect proof. The acceptance marker for the window is in TICKET_961_WINDOW.md 8.4 -- evictable_size() sampled right after each void line, under load, printed unconditionally including zero.

…s; delete the load-back delivery chain OPTION A, in the hard form. Two edits that only work together: the memory-axis watchman moves onto the group availability floor FIRST, and only then does the delivery chain come out. sgl-project#1045 -- THE FLOOR IS PUBLISHED UNCONDITIONALLY ON A GROUP. It used to stay None whenever availability happened to be equal that iteration, and `unified_radix_cache`'s load-back fell into a RANK-LOCAL memory decision when it was None, guarded only by the delivery signal `pp_load_back_told`. A watchman on one axis was hanging off the delivery of another, and it stood down exactly when nothing looked wrong. PRICE VERIFIED BEFORE BUILDING, per the no-new-collective rule: at the live path (scheduler.py:6396) `min_avail` and `max_avail` are BOTH harvested from the same already-performed all-reduce (`t[0]`, `t[max_avail_at]`), so dropping the comparison removes work rather than adding it. The single-rank path now publishes the LOCAL value instead of None -- it IS the group min for a group of one -- which is what lets None become a construction violation everywhere downstream. Behaviourally identical on even pools: the published floor equals the live local value, and the sgl-project#694 ledger keeps it tracking through the iteration. FIX-6 IS NOW ONE BRANCH. `floor is None` at the load-back raises, naming rid, kv_tokens and local availability. A rank that cannot answer from a group number must not answer from a local one -- that is how ranks stop agreeing, and the 21:52:25 wedge is the measured price. sgl-project#1046 -- THE DELIVERY CHAIN IS DELETED. Consumption is local: `_pp_load_back_extent` returns this rank's own stamp, chosen at its own match by one shared expression over the same content. Uniformity comes from identical DERIVATION, not from one rank shipping a value. Deleted with it: `apply_pp_load_back_row` and both its call sites, PP0's self-row parking and the one-lap delay, `pp_load_back_told` and all four readers, the `sgl-project#1035` deferral ledger, the fact carriers, `holders_with_unspent_extent`, `_spend_for_entry`, the spend transition, the `sgl-project#1041` population counter and the `sgl-project#1044` ship/recv ledgers. EVERY READER WAS DECIDED, NOT SWEPT ALONG -- the enumeration is the point, because a reader falling silently into a rank-local branch is the class that found the edge in the first place: unified_radix_cache.py:2713 group-fact signal -> moved to the floor (sgl-project#1045) schedule_policy.py:107 delivery mechanics -> local stamp schedule_policy.py:2305 delivery gate -> deleted, no fact to await schedule_policy.py:2360 clamp gate -> KEPT, re-pointed. Its hazard (a GDN anchor adopted at `_applied` while the KV is cut) is created by the load-back itself and is independent of where the extent came from. KEPT AND UNTOUCHED: the admission row, its `prefix_len`/`extend_len` columns and the sgl-project#791 membership and ordering rules -- batch geometry, a different axis. The `load_back_len` FIELD stays on the wire for width tolerance and always travels None. sgl-project#1042's lifecycle contract stays, minus the spend transition that had no consumer left. CHECKS: check_1045_floor_watchman.py proves the guard CAN FIRE (a crash guard that cannot crash is worse than the branch it replaced), that a small floor is an ordinary refusal, that the rank-local fallback is DELETED rather than bypassed, and that publication is unconditional and counted. check_1042 updated: the row build must now LEAVE the field alone -- a build that still cleared it would resurrect the eraser through the back door. check_1040 green. The 1041, 1043 and 1044 checks are DELETED with their subjects; the cell's zombie test picks up their markers (ROW APPLIED, DEFERRED, EXTENT POPULATION, ROW SHIP) so a stale build cannot pass silently. py_compile + import smoke on all five modules; ruff clean. NOT YET BOOTED -- this commit is desk-proven only.

fix: update bench serving

a2eb956

zhyncs merged commit c07526e into sgl-project:main Jul 22, 2024

zhyncs deleted the upd branch July 22, 2024 08:23

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

fix: update bench serving (sgl-project#694)

a1c4196

cherryblo added a commit to cherryblo/sglang-project that referenced this pull request Jul 2, 2026

Merge pull request sgl-project#694 from yangfen0903/testcases

3c7e7a8

Submit issue 381 and resolve the code.

Navigation Menu

Uh oh!

fix: update bench serving - #694

fix: update bench serving#694
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:upd

zhyncs commented Jul 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

zhyncs commented Jul 22, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant