misc: correct the int data type for token ids and indices by xiezhq-hermann · Pull Request #969 · sgl-project/sglang

xiezhq-hermann · 2024-08-07T09:19:38Z

Motivation

The data type for token ids and KV cache indices are supposed to be int32 consistently instead of int64.

Modification

Simply corrected the misuse of torch.int64.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

Both PP void sites released their non-resident requests with `_release_dynamic_chunk_probe`, which hands back KV rows, the mamba slot and the req-pool row by calling the allocator and the pool DIRECTLY. Nothing on that route reaches `cache_finished_req`, and that is where `dec_lock_ref(req.last_node)` lives -- the last two lines of every implementation of it. So the pages went back to the allocator while the radix tree kept its lock on them, and `reset_for_retract` cleared `last_node` one line later, putting the ref out of reach for the life of the process. One leaked lock per voided request. WHY IT EXPLAINS THE BOOTS. `evictable_size_` counts UNLOCKED tokens, so a tree whose nodes are all pinned reports nothing evictable at all. Every funding rung reads its reclaim off that number: "reclaimed 0 MiB from [nothing]" against a full tree is not a pressure reading, it is the statement that nothing in it was ever unlocked (sgl-project#813/sgl-project#694 family). NOT A THIRD MECHANISM. `release_req` (schedule_batch.py) already IS this discipline and both correct retraction paths -- `retract_decode` and `retract_all` -- are its only callers. The void sites join them through `_release_voided_request`, which holds no release logic of its own: it supplies the scheduler's collaborators and this path's never-raise contract, and delegates. Because `release_req` calls `reset_for_retract` as its last act, the call sites no longer reset themselves -- doing both double-counts `retraction_count`, which feeds `retract_decode`'s solo-OOM abort ladder. THE PROBE KEEPS ITS OWN RELEASE, deliberately. The profiler's request is a bare `Req` that is never matched against the tree and holds no ref, and `UnifiedRadixCache.cache_finished_req` decrements `req.last_node` with no None guard. Routing it through the disciplined path would be a decrement with no matching increment -- the same accounting defect mirrored (sgl-project#929). Two premises, not one discipline written twice; both docstrings now say so. SIBLING, SWEPT AND FIXED IN THE SAME PASS. `DecodeKVCacheOffloadManager._release_finished_req` ended with `tree_cache.protected_size_ -= len(req.prefix_indices)`, a hand-rolled imitation of one of the four effects `dec_lock_ref` has per node on the path to root; `lock_ref` was never touched, so the same leak by another route. Reachable with --disaggregation-decode-enable-radix-cache alongside the offload flag. Fixed rather than filed because it is decidable by inspection: under the `ChunkCache` default the old line was wrong in the OTHER direction, since that cache's `inc_lock_ref` never raises the counter it subtracted from. TESTS. test_pp_void_lock_ref_969.py, new, 14 tests + 2 subtests. Red-first, PP void, before the fix: 7 failed / 6 passed -- `lock_ref` stayed 1, zero `dec_lock_ref` calls reached the node. After: all green. Red-first, sibling, with the fix reverted: 2 failed on the lock assertion itself (the fake carries `protected_size_`/`evictable_size_` so the old code RUNS instead of raising -- a red for a thin harness is indistinguishable from a red for a broken product). Gegenrichtung: `test_the_underflow_detector_can_actually_fail` drives a doubled decrement at the detector and requires it to fire, so the two one-decrement-per-request assertions cannot go green through a sgl-project#929 underflow. Source pins run through `ast.unparse`, not raw `getsource`: both void sites NAME the probe helper in prose that stays true after the fix, and a pin that reads the comments is not a pin. test_pp_void_chunked_retracted_798.py: its stub list gains `_release_voided_request`. The stub RESETS the request, because the real one does; a no-op stub would have silently removed the precondition that file's first test asserts. DESK GATE (scripts/gate_tier2_partitioned.py, CUDA_VISIBLE_DEVICES="") BEFORE 4849 passed, 2 genuine (test_collective_family_siblings_610.py x2) 619.34 s AFTER 4863 passed, 2 genuine (the same two) 638.07 s delta +14 = exactly the new module. Count probe: 2 named == 2 in summary, SUBFAILED and ERRORS included, both zero. Lane shift explained, not waved past: wide 3701 -> 3695, serial 946 -> 966. Editing test_798 invalidates its sha256 proof in scripts/gate_partition.tsv, so the runner demoted it to the serial track, where it was measured green -- the table's expiry mechanism working as designed. The new module has no row and was measured in serial for the same reason. Re-proving both rows needs a full serial run plus per-module solo logs; NOT done here, and named as open. ruff: 2 F841 in the touched files, both pre-existing at HEAD (`inc_len`, `carries_flip_arm`), none introduced. codespell clean. NOT CLAIMED: no boot, no metal, no effect proof. The acceptance marker for the window is in TICKET_961_WINDOW.md 8.4 -- evictable_size() sampled right after each void line, under load, printed unconditionally including zero.

…d becomes symmetric Root (R7, boot 6): every void treated the ranks differently. Followers obey the family's own doctrine ('A VoID IS A PARK, NOT A RETRACTION') and keep prefix+pages; rank 0's batch members went through _release_voided_request -> release_req -> reset_for_retract, which clears prefix_indices and returns the pages. The two ends of the ring then disagree about the same request every cycle: 169 told=0 refusals (a FULL re-prefill of computed prefix per voided rid per cycle -- a standing double-prefill-law violation on every void), 333 past-fill refusals (total-token mismatch downstream of the same state), 6 duplicate-rid decisions (dedup-free tail requeue), rotating membership that killed boots 3-6 after every earlier layer was closed. Fix at both void release loops: pp_park_voided_batch_member -- the member is queued UNRESET (prefix_indices, prefix KV pages, req slot, retraction_count survive); only THIS pass's admission-side increments are handed back: the prepared-never-run chunk via _park_chunked_prefill_chunk(pass_allocated=True) and the admission tree-lock ref via pp_give_back_admission_lock_ref. Dedup behind pp_void_keeps_request (ordering load-bearing: reachability scans running_batch, a different slot than running_mbs[mb_id]). _release_voided_request keeps one caller: already-FINISHED members. retract_decode/retract_all untouched -- their passes RAN. Briefing premise falsified by the builder, recorded: keeping the last_node lock ref would have REOPENED sgl-project#969 by a new route -- init_next_round_input re-matches with no dec_lock_ref anywhere on that path, then admission takes a second ref. Pages-versus-claim is the boundary, not keep-both: the prefix pages stay in the tree, the request's claim is returned, which is the ordinary state of a queued request. Both the wrong draft and the correction are in the docstring. Instruments: 'sgl-project#984 VOID-PARK' per member (rid, prefix, kept_pages, chunk_given_back, lock_ref_returned, slot, route) + per-loop census INCLUDING the zero case, outside the sgl-project#801-spin suppression; #791b/#797d now also print parked=/dup-skipped=. Boot 6 measured #791b 514x but #797d 0x -- only the void-output loop is metal-proven; the census closes exactly that blind spot next boot. Execution proof (standing order): the rewired void-output loop IS boot 6's hot path (514 traversals); import smoke green; cold check driving the REAL _pp_void_own_batch body: 23/23 (prefix survives, requeue, dedup, one free = never-run chunk only, one lock-ref give-back, no double free, finished-member path intact). Suites live in the test-agent lane; predicted red there: test_pp_void_lock_ref_969.py encodes the retract premise this commit retires -- its invariant (no leaked ref) is preserved via the give-back. Named posten, not bundled: sibling unreturned admission refs on the #968b route and the kept chunked member; _pending_prefill_tokens_for over-reports a parked member's backlog (honest figure: total - len(prefix_indices)); req-pool kv_committed_len discrepancy duration now unbounded for queued members (worth a test); sgl-project#963 prefix-floor wiring stays its own posten.

…invariant is re-encoded, and a dead can-fail is revived sgl-project#984 makes the void PARK rank 0's members instead of retracting them. Three arms of the 969 suite encoded the old premise. None is deleted; each says in place what changed and why the thing it was protecting still holds. FIRST, THE FAILURE WAS NOT THE PREDICTED ONE. At 8be86f5 all six reds were 'TypeError: lambda got an unexpected keyword argument pass_allocated' -- the harness stub for _park_chunked_prefill_chunk taking two positional args while the fix passes a kwarg. A mechanical signature drift that MASKED the real result: with the stub widened, the truth is 3 failed / 11 passed, and the three are the premise, not the invariant. test_the_request_is_retracted_exactly_once: asserted is_retracted and retraction_count == 1. Now asserts NOT retracted and count 0. What the arm protected is unchanged -- retraction_count feeds the solo-OOM abort ladder in retract_decode, and a request must not be aborted at half the configured retractions. A double count was the hazard when the void retracted; an unasked count is the hazard now. test_the_release_happens_before_the_reset -> renamed test_the_ref_goes_back_while_the_prefix_handle_is_KEPT. Its assertIsNone( req.last_node) encoded an ordering problem of a world where reset_for_retract cleared the handle. sgl-project#984 removes that world and KEEPS last_node on purpose. The invariant is untouched and is now what the arm states: the admission-side inc_lock_ref must not outlive a pass that never ran (exactly one give-back -- not zero, a leak; not two, the sgl-project#929 underflow), while the prefix handle survives so the next offer can report it as executed. test_the_underflow_detector_can_actually_fail: its mutant patched tree_cache.cache_finished_req, reached via release_req. sgl-project#984 does not take that path, so the injected fault could no longer occur at all -- the arm failed 0 != -1 because the mutant was never invoked. A can-fail whose mutant became unreachable reports nothing while looking like a red someone would 'fix' by deleting the assertion. Re-aimed at pp_give_back_admission_lock_ref, which is module-level for exactly this purpose by its own docstring. MEASURED, same file: 646f410 (pre-sgl-project#984): 3 failed, 11 passed 8be86f5 (post) : 14 passed (14 defs, count probe) Two of the three reds are behavioural -- 'None is not <node>' (the reset cleared the handle) and 'True is not false' (the void retracted). The third is an AttributeError on the symbol sgl-project#984 introduces, which is inherent to a can-fail arm aimed at that symbol and is reported as the weaker red it is.

… and the mamba park fence sgl-project#990's ownership guard is right in intent and Boot 10 confirmed it on metal (no underflow). It discriminates the wrong thing: 'is still in the field right now' instead of 'will keep the field'. MEASURED at b27d7c2 by instrumenting the give-back's caller: member IS current chunked_req -> pp_queue_orphaned_chunked_req, guard fires, lock_ref stays 1, request sits in waiting_queue with its admission ref ordinary member -> pp_park_voided_batch_member, guard does not fire, lock_ref -> 0 So in these two shapes the guard's only observed firing is the harmful one. Cause is a statement order: the re-home runs BEFORE the field is overwritten (:7909/:7913, :9261/:9265) and the comment there requires that order for #968b -- 'Re-homed BEFORE the overwrite -- after it, the reference is already gone'. Re-admission then takes a fresh ref: +1 per void cycle, prefix never evictable, sgl-project#969's leak by a new route. Recorded as unittest.expectedFailure, not inverted and not deleted. Inverting would encode the leak as correct; deleting loses the only executable record. When ownership is asked properly this becomes an UNEXPECTED SUCCESS, which is a loud self-clearing signal to drop the marker. The suite stays green. The rid-discriminator arm is withdrawn rather than shipped half-understood: its shape is entangled with the re-home and needs its own study. MAMBA (boot 10's next form, mamba_pool_idx=None at cache_unfinished_req): measured that the void-PARK does NOT release the slot, and fenced that, plus the park's prefix contract beside it so the two cannot be traded off. What produced the None is NOT established and the file says so in an executable arm rather than guessing between the two candidates -- the release route via reset_for_retract, or a re-admission that never re-acquires. Guessing there is exactly this window's recurring class. One of my own arms cited the wrong function for the clearer docstring and went red; the citation is corrected in place with a note, since an arm citing the wrong function is that same class and was caught only by running it. Test: 969 -> 16 defs, 15 passed 1 xfailed; 990b -> 4 defs, 4 passed.

…request Boot 12 (7b855f6) died on PP0 at 21:46:06 with `AssertionError: reusing request must be chunked or have committed KV` (memory_pool.py:395), via get_new_batch_prefill -> prepare_for_extend -> alloc_for_extend -> alloc_req_slots -> HybridReqToTokenPool.alloc. ROOT, from the ledger and not from inference. One second earlier the same rank logged, for the same rid: sgl-project#969 voided-request release failed for 5708abdd579e4f7097ab97ed796481ea: Committed KV cache already freed (self.kv_committed_len=4096) `_release_voided_request`'s never-raise contract caught that, logged it, and fell through to `reset_for_retract`. The contract is right -- an instrument that raises while cleaning up after a divergence turns one defect into two -- but it only decided that this frame does not PROPAGATE the failure. It never decided what the half-released request IS afterwards. The answer was: a request still holding `req_pool_idx`, whose `inflight_middle_chunks` and `kv_committed_len` the reset then zeroes -- and those two fields are the allocator's only evidence that a retained row is legitimate. It was re-queued, re-admitted one second later, and killed rank 0. The assert is correct and is the only thing that caught this. The defect is upstream of it. CLASS: warn-then-continue. The compensator converts a partial failure into a live, corrupt, re-admissible object. This is the catalogued `warn-then-continue swallow` shape, and the cure is not to remove the swallow but to make the post-failure STATE explicit. FIX, at the junction: on a failed release, hand the row back -- mamba slot first, then the req-pool row, the ordering `_release_dynamic_chunk_probe` already documents. The reset below drops prefix, pages and geometry anyway, so the request re-prefills regardless; the row it keeps buys nothing and justifies nothing. `free_slot` carries its own membership scan and REFUSES an already-free row (sgl-project#616), so a release that failed AFTER returning the row is caught by name and logged instead of corrupting the free list -- both outcomes are reported with the rid. NOT CLOSED, and deliberately not guessed at: why `pop_committed_kv_cache` found `kv_committed_freed` already True at `kv_committed_len=4096`. That producer is a second defect. This change contains it and names it in the log (`sgl-project#993 INCOMPLETE RELEASE DISOWNED`), so the next boot measures how often a release fails at all -- a number no boot of this family has ever had. FUTURE CHECK: any new raise inside `release_req` now ends in a disowned row plus a named line, not in an allocator assert three passes later. Evidence: desk. py_compile + import smoke + wiring assertion on `_release_voided_request`. Belegstufe: DESK-BEWIESEN.

…tay sticky until spent Boot boot_855_968pp0lb_..._0831_060725 ran clean -- 0 exceptions, 0 aborts, 0 proxy mismatches, sgl-project#969 EXTENT rank-uniform on all three ranks, so the replacement gate is safe -- but delivered 0 applied load-backs against 12 opportunities. Two defects in the carry, both mine, both measured: 1. init_pp_loop_state cleared BOTH halves at every cutover, by analogy with the pass-scoped voids beside it. That analogy is wrong for this quantity: the voids describe THIS EPOCH'S PIPELINE, while a load-back extent describes the HOST TIER, which is exactly what survives a cutover. And the post-cutover re-admission IS the read-through path, so the clear destroyed the fact with the very event that creates the only chance to use it. Measured: 27 cutovers, and every one of the 12 deferrals fell 1-2 s after a CUTOVER line (06:14:07, 06:16:57, 06:18:24, 06:19:59). The same rid's host hit persisted and GREW across those cutovers (1215 -> 1216 -> 2114), which is the direct evidence that the tier the number describes is durable. 2. The promote replaced `effective` with `pending`, giving every extent a life of exactly ONE pass, while the gap between PP0's offer and the group's next re-admission spans a cutover by construction. A one-pass fact and a next-cutover consumer cannot meet. Fix: `effective` is sticky -- merged, not replaced, and not cleared by a cutover -- and is spent where it is acted on (the load-back site flags the request, the admission loop pops the rid), which bounds the map without it needing to know when a request ends. Same shape as pp_clear_parked_continuation. Staleness is caught where it is cheap: a rank told an extent its host tier no longer covers raises sgl-project#968 LOAD-BACK EXTENT UNHONOURABLE, an epoch-independent check against the tier the number is about. Only `pending` remains per-pass; that is what still keeps PP0 from applying its own offer on the pass it makes it. Desk checks: imports; cutover reset now guarded; promote merges and clears pending; spend set at application and popped by caller; extent helper regression (honest-miss / told / pp_size<=1) unchanged.

…ot end a pass PP0 launched `_pp_void_retracted_pass` let ONE downstream rank decide that the GROUP's pass ran nowhere. Its own docstring named the return trip that made that safe -- "the void output carries the observed local match home, and PP0's guard learns it as a floor" -- and sgl-project#969 CUT V had already deleted that emitter (`_PP_VOID_OUTPUT_KEY`: zero originating senders at ca0ee3a). The verdict therefore travelled downstream only. PP0's `mbs[slot]` stayed set while the last rank's did not, so no output was ever sent and PP0 blocked in `_do_recv` until the deadman -- the exact invariant `_do_recv`'s own comment relies on ("sender and receiver ask one question of one batch"). Measured, twice, and the second is on the stall second itself: 1068cap 07:34:02-09 sgl-project#797 void on rank 1 ONLY (no void/retract line on PP0 or PP2); PP0 parked, PP2 spinning. 1069cohort 08:00:54/55 sgl-project#791 unhonourable on PP1, told=12493 local=8397 then told=13399 local=12493 -- `local` exactly one pass behind `told`. After 08:00:55 only ranks 1 and 2 emit at all (slot_occupant / output_fill / width_agreement run to the takedown at 08:13:32, 4177 and 7973 lines; rank 0 emits nothing). That last measurement also refutes the occupant-sleep node as the halting member: ranks 1 and 2 are alive and turning; the `sgl-project#1000 SLOT-OCCUPANT reasons={'no-statement'}` spin is an INERT probe whose carrier sgl-project#1015 EDIT-F made permanently None. The only halting member is PP0's unbounded output receive. Repairing the return trip would repair a compensation layer for a rank-local verdict, which is the arc the sgl-project#968 order forbids continuing; under upstream-minimal the repair carries the burden of proof and the deletion does not. So the verdict is deleted and the disagreement is DETECTED instead (RAENGE-NIE-UNEINS: a detected divergence stops the group, never a compensating wait): * `_pp_assert_told_honourable` replaces it -- an unhonourable told names rank, slot, rid, told and local and raises. No clamp: clamping to this rank's own local match is rank-local geometry, i.e. sgl-project#631. * The chain-receive throttle arm gets a horizon (SGLANG_PP_OCCUPANT_HORIZON_S, default 90 s, 0 disables) and a named stop. Taking the arm is legitimate and frequent; outliving it never is. * `PpChainReceiver.recv` is bounded the way its sibling `consume_up_to` already was (runaway guard + reported counter), and the launcher now sets SGLANG_PP_CHAIN_RECV_STALL_S=60 -- the sgl-project#824 mechanism has existed since 2026-08-24 and shipped disabled by default, which is why it never fired in either stall. (A-i) is WITHDRAWN rather than built: its counter proof is blind at that site (the rendezvous bumps `sent` only on recv entry) and the DEFER one-shot is itself compensation for the rank-local verdict this commit deletes, so its trigger is removed at the source. Its two red-first tests are replaced by zombie tests for the deletion. Desk evidence: hermetic import + AST (deleted verdict absent, watchman and horizon present, recv bounded); test_968_deletion_falsifiers 24 passed, 1 failed -- test_C_the_void_relay_is_wired_or_deleted_but_never_half_built, which stays red until the relay SYMBOLS are swept out too. The relay is already unreachable at runtime (no rank originates a void any more, and `_pp_absorb_void_output` has no caller in production), so that sweep is a dead-code deletion scheduled beside this commit, not a runtime dependency of it. Belegstufe: DESK-BEWIESEN. Boot pending.

…sends, nobody relays, one rank absorbs sgl-project#969 CUT V deleted the void-output EMITTER on the premise that the batch is the verdict ("sender and receiver ask one question of one batch"). What it left behind was the rest of the arc, and the arc has been dead ever since: _PP_VOID_OUTPUT_KEY: True constructed at exactly ONE site -- inside _pp_absorb_void_output, i.e. only when re-forwarding a void already received. Zero originating senders. _pp_void_forward_payload 1 writer, 0 readers. Computed and never sent. _pp_absorb_void_output 0 call sites (ast over python/sglang). pp_void_forward_payload called only from the absorber. pp_void_relay_stop_rank called only from the forward payload. pp_void_relay_launched_verdict same. pp_first_retracting_rank called only from the forward payload. A closed orphan subtree, whose own docstring states the sgl-project#801 relay invariant it can no longer keep: "a non-last rank that took exactly one message off this wire for a ring generation must put exactly one back on it, void included". WHY DELETED AND NOT WIRED. Wiring it back means rebuilding the second bookkeeping sgl-project#969 CUT V removed, and under the upstream-minimal law (user order 2026-08-29) repair carries the burden of proof that upstream semantics cannot carry the goal. It can: PP0 owns the verdict and the travelling batch is the verdict. Half a relay is the worst of the three states -- PP0 parks on an output the mid-rank void swallowed and its successor throttles on an occupant nobody will speak for, which is the shape both measured stalls took (1068cap 07:34:09 and 1069cohort 08:00:55, roles swapped, same halting node). Tests of the deleted mechanism go with it rather than being patched green: the five files whose SUBJECT is the void relay (sgl-project#797 retracted-pass void, sgl-project#798 chunked-retracted void slot, both sgl-project#801 relay files, #791b output-ring retraction), plus test_pp_continuation_cross_slot_rehome_968b (28 of its 30 tests drive the deleted absorber transitively) and test_pp_void_lock_ref_969 (its harness is the deleted method). Files that merely mentioned it keep their pins; only the individual test that exercised the deleted site is removed. Comments naming a deleted symbol are rewritten rather than left to send the next reader after a ghost. Desk-proven: py_compile + import smoke (scheduler and mixin), the sgl-project#968 deletion falsifiers 25/25 with the wired-or-deleted-but-never-half-built contract now green by the deletion arm, ruff unchanged at 104 pre-existing findings on both touched files (baseline taken from HEAD under the same config).

…oved, not the set the release re-enumerates Boot 9 (boot_855_weg1b9_1116175f6d_0904_164023.log) died at the first pp_to_tp cutover. The measured shape is TEMPORAL, not container-shaped: log:1883/:1886/:1888 at-arm cur_slot_reqs=1 on ALL THREE ranks (cur_slot_reqs IS len(_live_reqs(scheduler))) log:2203/:2210/:2213 one second later, RESIDENTS RELEASED 1 / 0 / 0 log:2189/:2194 PP1+PP2 sgl-project#938 PROTECTED RESIDUE AT DROP: 67 row(s) still locked after a drop that evicted 0 log:2305/:2351 ReqPoolRebindRefused: 1 of 8 rows are still held in the OUTGOING request pool (free=7, rids=[], rows=[]) The same authority, the same rank, two answers one second apart, and nothing reconciled them. Four cuts: 1. LOAD-BEARING. `_enter_armed_state` now opens an armed-window resident ledger (`_armed_residents`) and `on_round` unions each armed round into it; `_release_residents_for_cutover` retracts the release-instant enumeration RECONCILED against that ledger (`cutover_resident_set`). Snapshot rather than a re-run of the quiescence term at :11564, because that point is past the no-return -- PP0's PROCEED has already ridden the request stream -- and a quiescence term there could only be a rank-local verdict where no rank may hold one (sgl-project#969 SS-W3). A snapshot changes WHAT is retracted, never WHETHER the group cuts over. The reconciliation is a FILTER, not a union: a snapshot member is carried only if it names a row, the row is not in the pool's free list, and no live request names the same row. Under-retraction stops the boot loudly at the rebind; over-retraction frees a row its owner still holds and corrupts silently, so the two are not traded symmetrically. 2. `mbs` is now a route of the ONE authority `_live_reqs`, beside `running_mbs`/`last_mbs` (scheduler_pp_mixin.py:7556-7561 builds three distinct arrays; `mbs` is written on all three planning paths). Dedup is by id(), so the scheduler.py:8354-8362 alias costs nothing. 3. `phase_req_pool_binding` drops its own two-container walk and delegates naming to that authority (keeping `waiting_queue`, which the flip has not got), and `census_outgoing_req_pool` now derives the HELD ROW IDS from the pool's own free list. `escapees=1 rids=[] rows=[]` becomes `rows=[N] rids=[...] unnamed=K`. The raise is NOT relaxed -- the module docstring :16-30 is right that a request pool has no disarmed state. 4. phase_flip_runtime.py:1202 cited `orphan_resident_reqs` in `phase_flip_resident_carry.py` in the present tense; sgl-project#969 deleted that module whole. Rewritten in the past tense, naming the deletion. RED FIRST, at parent 1116175, test/registered/unit/managers/ test_1202_arm_release_residency_reconcile.py: 8 failed, 2 passed. The follower test failed with boot 9's verbatim refusal ("1 of 8 rows are still held ... free=7, rids=[], rows=[]"). After the cut: 10 passed. MUTANTS (scratch, all killed): M1 drop the reallocation guard (DANGEROUS: frees a row its new owner holds) -> test_a_row_reallocated_to_a_live_request_is_not_taken_from_it M2 drop the already-free guard (DANGEROUS: returns the same row twice) -> test_a_row_already_returned_is_not_retracted_a_second_time M3 release ignores the armed ledger (boot 9 restored) -> 4 tests M4 remove the mbs route -> test_mbs_is_a_route_of_the_authority M5 census rows from the request list again -> test_held_rows_come_from_the_pools_own_arithmetic VERIFIED: siblings at parity, per file, parent vs cut -- test_phase_flip_runtime.py 30 failed / 38 passed at BOTH, identical failure set (pre-existing); test_1040_req_pool_per_phase + test_seam_order_856 + test_seam_readmission_w31 + test_tree_drop_returns_rows_856: 78 passed at both. ruff check clean on both sources at parent and at cut; ruff-format diff 4 lines at parent and 4 at cut (pre-existing, unchanged). Matched structural check: all three `self._parked_extent = None` statements sweep the ledger too (the sgl-project#746 M5 shape on the request axis). Import smoke on both modules. NOT VERIFIED: nothing was run on metal. No boot, no GPU, no server. Whether the request that held the row on PP1/PP2 in boot 9 was reachable through `mbs` is still PLAUSIBLE only -- an mbs-only resident would have read cur_slot_reqs=0 at arm and the log reads 1. Cut 1 does not depend on that attribution. test_quiescence_no_carry_858.py could not be run: it fails to import at the parent commit too (`cannot import name prefill_runnable_in_current_layout`), a pre-existing breakage.

…eft the nodes naming them ROOT, file:line: python/sglang/srt/mem_cache/unified_radix_cache.py:6069-6250 (UnifiedRadixCache.reclaim_rows_for_drop). The pass walks every node and returns the rows and mamba slots the tree still holds -- the sgl-project#1050 contract for locked nodes `evict` refuses -- but it never took the reference away from the node. Every other release of a node-held anchor goes through MambaComponent.evict_component (mamba_component.py:653-663), which frees AND nulls cd.value AND corrects the size book; this pass had only the first third. THE SIBLING FORM, and why sgl-project#924's own guard was silent: nothing is freed twice. The slot is returned ONCE and the node goes on naming it, so MambaSlotAllocator._refuse_double_free (allocator/mamba.py) has nothing to see (free_list_duplicates=0 in the killer line). The free list and the tree then both own the slot: alloc() hands it to the next request while the tree offers it as a resume anchor -- one GDN state read by two requests. MEASURED (boot 10, /spinning/evidence-665-f1/boot_855_weg1b10_2126a4a1d2_0904_211702.log 21:29:28Z, all three ranks, line 364223 for the census): [mamba] total=20, available=20, evictable=4, withheld=0, double_owned_src=live, free_list_duplicates=0, duplicate_slot_ids=None, free_and_cached=4 TREE CENSUS nodes=5 | MAMBA: tracked_evictable=4 recomputed_evictable=4 DESK REPRODUCTION of that exact line against the real function (two requests through cache_unfinished_req + cache_finished_req, then reclaim_rows_for_drop): before: avail=20 evictable=4 free_and_cached=[1,2,3,4] duplicates=0 after: avail=20 evictable=0 free_and_cached=[] CLASS: "a component's rows are released by a walker over tree nodes instead of through the component's own owner-transfer primitive." SIBLING SWEEP: the FULL half of the same function has the identical omission (_free_full leaves node.component_data[FULL].value standing); fixed and pinned in the same test. FUTURE CHECK: after any pass that returns rows the tree held, free_set & tree_set must be empty for every component -- asserted directly. WHAT CHANGED * unified_radix_cache.py: nodes are carried beside their values; _disown_reclaimed_value(node, ct) is the factored-out other two thirds of evict_component (null the value, debit protected OR evictable per lock_ref, leave the device LRU, join the host LRU when a host copy survives). The FREE stays where it was, differenced against the allocator ledger (sgl-project#1055), because this pass legitimately meets already-free slots; only the DISOWN is new. An already-free slot is now disowned too -- that is the aliasing, not a reason to leave the reference. * invariant_checker.py (_check_mamba_pool): NAMED STOP "sgl-project#924 MAMBA SLOT ALIASING". A negative mamba occupancy (pool.size - available - evictable < 0) or a free list longer than the pool is now a leak verdict with both numbers. Boot 10 printed `mamba usage: -0.10 ... -0.30` five times over eight minutes (first 21:21:20Z, log:3595) and nothing acted on it. Derived at on_idle, on every rank with nothing in flight -- never from a batch line inside the no-return region between collectives (sgl-project#969 SS W3). * allocator/mamba.py: note_924d(), the "#924D" per-(rid, station) trail that decides sgl-project#1190 on boot 11. Stations: alloc (memory_pool.alloc), backup (BACKUP_HOST build), load_back (H->D COW target, with the node's own anchor printed beside it), first_state (the deferred COW execution, with cow_src), free / relinquish (memory_pool). Bounded by construction: one line per (rid, station), global cap 8192, and the cap prints its own suppressed count so a missing rid is never read as "that station was not reached". Line format: #924D station=<s> rid=<rid[:12]> mamba_slot=[<ids>] node=<node id> <extra> If a B probe's first_state slot equals a slot some node still names, sgl-project#1190 IS this defect; if not, sgl-project#1190 is a separate carrier and stays open. EVIDENCE * RED on the parent (2126a4a, scratch tree /tmp/parent924 from git archive): 9 failed / 1 passed. The 1 pass is the can-fail control (test_a_balanced_pool_is_not_a_verdict), which must stay green on both sides. * GREEN after: 10 passed. * MUTANTS, all killed (each against a copy of the source tree): M1 drop the mamba disown (aliased slot survives) -> 4 failed M2 STOP threshold unreachable (STOP cannot fire) -> 2 failed M3 disown without the size-book debit (silent drift) -> 2 failed M4 drop the FULL-half disown (sibling sweep) -> 3 failed M5 discriminator per-call instead of per-rid -> 1 failed * SUITES, the 63 registered unit files touching the changed symbols, run once on each side, hermetic (CUDA_VISIBLE_DEVICES=""): parent 78 failed, 1612 passed, 860 skipped, 195 subtests passed worktree 78 failed, 1622 passed, 860 skipped, 195 subtests passed The failure NAME SETS are byte-identical (diff clean); the +10 is this commit's new file. Zaehlprobe: 78 extracted FAILED/SUBFAIL/ERROR names == 78 in the summary, on both sides. * ruff: new test file clean; the six modified modules carry exactly their parent error counts (43 / 2 / 0 / 0 / 0 / 1), i.e. nothing added. NOT PROVEN: this is DESK-PROVEN, not fixed. The desk shows the mechanism and the exact ledger line; only boot 11 can say whether it was the whole of the boot-10 trajectory, and the #924D trail is what will answer that and sgl-project#1190.

…dules, decided per file WHY: four modules could not be COLLECTED on this tree, so EVERY gate baseline here ended rc=3 / VERDICT: INCONCLUSIVE -- an automated NEW=0/GONE=0 verdict was structurally impossible, which cost half a verdict cycle on 2026-09-11. The exit code is not the tree's redness: gate_tier2_partitioned.py:614-616 is `if inconclusive and rc == 0: rc = 3`. NOT A SKIP (sgl-project#910): a skip can darken a working test; here it darkens nothing -- there is nothing behind these imports to run -- so it would only make rc=3 permanent under a friendlier name. sgl-project#905's form instead: delete a retired promise with its reason, and leave ONE tripwire that fires the day the mechanism returns and NAMES the arms then owed. DECIDED PER FILE, not in bulk: DELETED test_phase_flip_decode_relay_631.py (7 tests). Its only subject was `phase_flip_resident_carry.reseed_decode_input_relay`, deleted by 069f98c "[sgl-project#969 CUT K] Delete the resident carry: the cutover is a re-entry, not object surgery" -- also standing user design (a flip nulls everything and re-admits through HiCache; object surgery at the cutover is the root class of ~15 boot killers). A retired promise. DELETED test_pp_proxy_stamp_631.py. It bound `SchedulerPPMixin.pp_flip_drain_tensor_dicts` at CLASS-BODY level (:334), which is why it failed at import while test_pp_presence_stash_on_real_wire_800.py, which names the same symbol inside a function, still collects. THE RENAME HYPOTHESIS IS REFUTED: `pp_flip_drain_leftover_dicts` is not the same mechanism under a new name. Its own docstring calls it "CORPSE S DONE CORRECTLY" and records that the old drain was kind-blind and ATE AN OWED OUTPUT (PP1, 07:33:30Z), while the successor demultiplexes first; HANDOFF_656.md:1348 says "Do not re-enable pp_flip_drain_tensor_dicts as written". The promise it carried is held by the successor, which six live test modules already cover, so retiring this file loses no coverage. REWROTE test_phase_flip_spec_seam_631.py: dropped the dead `harvest_resident_batches` import and the THREE tests that drove it, two of them sgl-project#905-shape falsifiers whose counterfactual can no longer be constructed. The seven tests over `_reachable_batches` and `clear_spec_info_for_unspeculated_phase` -- both live -- survive unchanged. REWROTE test_phase_flip_draft_bootstrap_631.py, three separate repairs: (a) dropped the dead `ResidentCarryError` import; (b) retired the cap-refusal test and the two-ceilings-agree pin. The product states the reason itself (phase_flip_draft_bootstrap.py:456-461): the defect-M ceiling is gone with the carry, "a count above the cap can no longer be the signature of a corrupted carried set -- it would just be the load", and BOTH copies of IN_FLIGHT_CHUNKED_ALLOWANCE are gone, so there is no second ceiling to agree with; (c) FIXED FIVE STALE-DOUBLE FAILURES that only appeared once the module could be collected again: `retune_carried_batches_for_phase` enumerates through `_harvest(scheduler)` -> `scheduler._resident_batches()`, which sgl-project#969 moved ONTO the Scheduler (scheduler.py:7040, "eight lines replacing harvest_resident_batches ... 911 LOC"), and this file's SimpleNamespace double predates that move. The double now BINDS THE PRODUCT'S OWN METHOD rather than copying its eight lines: the identity dedupe is load-bearing (running_batch is normally an alias of one slot) and a second copy is free to drift. NO production file was touched -- `_resident_batches` exists and has live callers (phase_flip_runtime.py:941/3765), so this was test drift, not a product defect, and I verified that before editing anything. TRIPWIRE test_1347_retired_631_mechanisms_stay_retired.py, three assertions over the retired mechanisms, each naming the arms owed on red and where to recover them (04cd920). CAN-FAIL PROVEN, four mutants, all fire: re-importable resident carry -> red; the kind-blind drain back on the mixin -> red; the SUCCESSOR also gone -> red (so the file cannot pass by the whole family vanishing, the green-by-vacancy direction); the arming allowance back -> red. MEASURED: 47 tests collect and 47 pass across the two rewritten modules plus the tripwire (before: 0 collected, 4 modules uncollectable, 11 pytest ERROR entries). Running them was not optional -- collection alone would have traded rc=3 for five new GENUINE names.

xiezhq-hermann requested a review from hnyls2002 August 7, 2024 09:20

correct the int data type for token ids and indices

3615e5c

xiezhq-hermann force-pushed the main branch from 0fb64cb to 3615e5c Compare August 7, 2024 20:12

xiezhq-hermann requested a review from zhyncs August 7, 2024 20:15

hnyls2002 approved these changes Aug 7, 2024

View reviewed changes

zhyncs merged commit 6db27f7 into sgl-project:main Aug 7, 2024

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

misc: correct the int data type for token ids and indices (sgl-projec…

844b3cc

…t#969)

Navigation Menu

Uh oh!

misc: correct the int data type for token ids and indices - #969

misc: correct the int data type for token ids and indices#969
zhyncs merged 1 commit into
sgl-project:mainfrom
xiezhq-hermann:main

xiezhq-hermann commented Aug 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

xiezhq-hermann commented Aug 7, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants