docs: update model support by zhyncs · Pull Request #760 · sgl-project/sglang

zhyncs · 2024-07-27T04:07:14Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

THE CHECK WAS IN THE WRONG PLACE, NOT MISSING. write() already refuses at ENQUEUE via the sgl-project#718 device-tier disarm. The copy happens later, and the flip rebinds in between, so a write-back queued before a cutover reaches backup_from_device_all_layer carrying a pointer table into the pool it was built from. Both crash specimens died three seconds AFTER a pp_to_tp cutover completed: 14:08:14 -> 14:08:17 (epoch 27) and 07:12:09 -> 07:12:12 (epoch 3), seven hours apart, same direction, same lag. WHY THE SHAPE GUARD COULD NEVER CATCH IT, and why its silence was misread as innocence -- by me, until the operator pushed back. Under layer_first the host layout EQUALS the device layout, so a stale binding is shape-IDENTICAL to the live one and check_shapes passes by construction. That is exactly what sgl-project#760 recorded: KV-TRANSFER-GUARD armed on all three ranks, zero transfers refused, SIGSEGV anyway. Matching shapes plus a crash puts the fault below the Python seam; a generation stamp is what tells "same shape" from "same pool". MEASURED, ONE VARIABLE. HiCache host+disk under sustained 4-way load with the flip REMOVED (plain TP3, which keeps speculation on -- PP+spec requires the flip, server_args.py:18385) survives five minutes with 0 segfaults and 0 admission wedges, and passes the REP gate outright (1 distinct/12, salted 0/6). The fault needs the cutover. STAMPED BY CONSTRUCTION, VERIFIED AT CONSUME. The stamp lives in CacheOperation.__init__ rather than at one enqueue site: an op built by any other path would otherwise be unstamped, and an unstamped op must be refused, which silently dropped legitimate write-backs and broke the staged-dispatch tests until I moved it. A stale op is dropped loudly and counted; its prefix simply misses later, the same cheap failure the sgl-project#718 disarm already accepts. Also folded in, both consequences of earlier sgl-project#767 work rather than new choices: the anchor-protection test that encoded "no interval means no anchors" is updated to the corrected premise it was measured against, and the SECOND-PASS EVICTION line drops from warning to info -- the second pass is documented as legitimate, and anchor eviction was falsified as the drift cause, so it is accounting rather than an alarm. 4 new tests, red-first (a pre-rebind stamp must be refused). mem_cache and managers suites: 4047 passed, 43 failed -- all 43 the pre-existing sgl-project#772 class (PhasePolicyConfig lacks idle_locked_settle_s after the sgl-project#713 revert), one fewer than the 44 baseline because the 747 premise test is now correct. ruff clean on both touched files (0 at HEAD, 0 now).

…nsumed THE ENQUEUE ANSWER WAS RIGHT; NOTHING ASKED AGAIN. write() calls device_tier_disarmed("write") and correctly gets False: the copy is queued while the model computes in PP, which IS the phase these pools are bound to, so queueing it is legitimate. start_writing() then consumes it later, and the cutover lands in between -- both crash specimens died three seconds AFTER a pp_to_tp cutover completed (14:08:14 -> 14:08:17 epoch 27; 07:12:09 -> 07:12:12 epoch 3, seven hours apart, same direction, same lag). WHY THE GENERATION STAMP COULD NOT COVER IT ALONE, which the previous commit assumed it would: with --phase-flip-rebind-hicache off, binding_state() never advances, so every stamp matches by construction and the check is dead code -- measured as 0 write-back refusals on a boot that still took 2 SIGSEGVs. The phase predicate is the one that already knows the answer; it only had to be asked a second time, at the moment the device indices are actually read. Refusing costs a cache MISS later, the same cheap failure the sgl-project#718 disarm already accepts at enqueue. Counted and named so the cost stays answerable. NOT VERIFIED ON METAL. This edit was written but never booted -- the task moves to a successor strand. HANDOVER_760.md carries the repro, the anchors and the ranked hypotheses. ruff: 1 pre-existing F541 in this file at HEAD and after, none added.

…t to redo Stand-down handover for the successor strand. Carries the measured chain (write enqueued legitimately in PP, consumed after the cutover, SIGSEGV in backup_from_device_all_layer), the 3-second cutover correlation measured twice hours apart, the exact boot/load/gate commands, file:line anchors for every guard and refusal touched, the 4 red-first stamp tests, and the ranked open hypotheses. The "what NOT to redo" section is the part with the most value per line: the wheel is sha256-verified against Runbook §2.1 (sgl-project#384 excluded), the transfer kernels DO exist and my earlier "missing" claim was a hasattr-on-package error that would have bought a pointless rebuild, layer_first does not avoid the crash despite sgl-project#760 prescribing it, the flip-off TP3 arm is clean, anchor eviction and dirty carried slots are both falsified, and drift is documented behaviour rather than the defect.

…at outlive their phase TWO HOLES, ONE WINDOW. The sgl-project#718 guard read the parallel_state routing global, which is toggled INSIDE the cutover -- one step among many -- so for the whole seam (waves moving KV rows, movers releasing the outgoing backing, the cutover rebuilding topology) the guard named a phase while pool bytes were in motion. And even a perfect predicate cannot recall a copy that is already riding the controller's private CUDA streams: write() and start_writing() run in the same Python instant (start_writing is called synchronously from write, its only caller), so the previous commit's consume-time re-check re-asks the question at the same moment it was first asked. The torn window is the STREAM's asynchrony, not the queue's: a device->host copy enqueued legitimately in PP outlives its Python call by seconds under load, and the seam releases the pool under it. Both crash specimens died exactly there -- 3 s after a pp_to_tp cutover, inside backup_from_device_all_layer, below the Python seam. THE FIX, in the two halves the window has: 1. AUTHORITY. PhaseFlipRuntime registers itself (weakly) as the phase guard's authority. Its _phase field is what the PHASE-FLIP DONE line reports -- truthful by the 3 s crash correlation -- and it alone knows the seam's extent: hicache_seam_active is raised at the no-return point (after the unanimous-abandon verdict, before the first wave) and cleared after the cutover installs the new phase, with a finally in the caller as insurance. During the seam the guard refuses device-tier I/O for EVERY binding. Outside it, the authority's phase wins over the routing global, and a disagreement logs the sgl-project#754-shape instrument line that settles whether that global was ever stale here. No authority registered (no flip runtime built) falls back to the routing global: non-flipping deployments stay byte-identical. 2. QUIESCE. At the same no-return point the runtime drains the controller's write_stream and load_stream while every pointer they hold still names live memory. Finishing those copies is correct (they become durable cache entries) and bounded (PCIe transfer of the backlog; this thread is the only device-tier producer, so nothing refills behind the drain; rank-local, so it cannot wedge the group). Ordered after the sgl-project#703 flip-writeback hook, whose staging copies are the largest legitimate producer of exactly such in-flight work. Tests: test_flip_seam_guard_760.py, 10 tests, red-first (all 10 fail on the parent commit: seam disarm and authority-wins are impossible there, quiesce/_quiesce_hicache do not exist). With the fix: 14/14 green including the sgl-project#760 stamp tests. Full mem_cache+managers sweep: 4057 passed, 43 failed -- the 43 are the pre-existing sgl-project#772 class (PhasePolicyConfig lacks idle_locked_settle_s), zero regressions, +10 passed vs the parent's 4047. ruff clean on the new code (the 13 E402 in cache_controller.py pre-exist on HEAD); codespell clean. Metal gate next: the 2-SIGSEGV repro arm must hold >=2 cutovers under 4-way load with 0 segfaults and logged seam refusals.

…-- the hole the crash went through THE METAL FALSIFIED THE PREVIOUS COMMIT'S SUFFICIENCY IN 40 SECONDS, and the faulthandler stack it produced is the whole finding. seamfix1 (52df0ff, repro arm, 4-way load) registered the authority on all three ranks, quiesced at two seams, completed pp_to_tp epoch 3 -- and segfaulted seconds later in the TP phase, through a stack the guards never see: unified_radix_cache.cache_finished_req -> insert -> _inc_hit_count -> write_backup -> hybrid_cache/hybrid_cache_controller.write -> start_writing -> backup_from_device_all_layer -> transfer_kv_direct This deployment's tree cache is UnifiedRadixCache driving HybridCacheController -- which INHERITS from HiCacheController (so the seam quiesce worked, it is inherited) but OVERRIDES write() and load() without the device_tier_disarmed checks the base methods carry. Every sgl-project#718/sgl-project#760 metal reading of 'zero disarm hits' on this stack was therefore vacuous: the guarded methods never executed; TP-phase inserts enqueued copies against the PP-bound pools unchecked, and one of them walked released backing. The overrides now ask the guard first, before any pool is touched, with the base contract: refuse -> return None -> the caller books a miss (write_backup returns 0, load-back returns False -- both verified None-tolerant). Tests: two red-first additions to test_flip_seam_guard_760.py pin both overrides (an alloc-must-not-run pool proves the guard runs FIRST); both fail on the parent commit, 16/16 green with the fix. Full mem_cache+managers sweep: 4059 passed, 43 failed -- the pre-existing sgl-project#772 class only, zero regressions. ruff and codespell clean. Metal gate re-run next on this commit: >=2 cutovers under 4-way load, 0 SIGSEGV, and the first genuinely non-zero disarm/refusal counters on this stack.

…ion; give all four consume points one gate Three classes from the cutover-participant sweep. 1. THE PRODUCER CHAIN IS STAMPED, replacing my own one-shot settle as the durable answer. `settle_pending_releases` drained the queue at the rebind instant and cannot see entries that do not exist yet -- and THREE producers keep filling it afterwards: `_drain_revoke`, the prefetch transfer thread, and the direct path. Each manufactures a release naming slots from the pool bound when the OPERATION was opened. `StorageOperation` now stamps itself at construction (where its host slots are allocated) from sgl-project#719's `current_generation` -- one authority, no parallel scheme -- and `BindingState.advance` records generation -> host pool, so a stale release can be freed against the pool it came from. Routed at PRODUCE time, the only point where the generation is still known without changing what the queue carries: a stale batch is freed immediately against its own pool, so nothing on the queue can outlive its binding by construction. A stale batch whose pool is UNKNOWN is refused loudly and neither queued nor freed -- both would corrupt. THE ONE-SHOT SETTLE STAYS, as a documented belt: not every `append_host_mem_release` caller passes a generation yet, so it still covers the unstamped remainder. Decided by reading, not assumed. AND THE OVERRIDE HAD TO BE ROUTED TOO. `HybridCacheController. append_host_mem_release` shadows the base with a different signature; an override that silently dropped `generation` would leave this fix installed and unreachable on the mamba/hybrid lane -- the only lane this rig runs. Same shape as W31/W32/W33, caught at the desk this time. 2. `start_loading` GETS BOTH CONSUME-TIME CHECKS. `load()` asks the phase question at enqueue and is right there; the cutover lands between enqueue and consume, which are separate calls. A stale load fills device rows from host slots this phase does not own and the tree marks the prefix RESIDENT -- attention reads KV nobody wrote, with no assertion anywhere. Checked before a producer is allocated. 3. `HybridCacheController` GETS THE CONSUME HALF in BOTH `start_writing` and `start_loading`. Its `write()`/`load()` carried the enqueue-time checks only, making the live path the one lane where a queued copy is consumed after a cutover -- the shape that file's own sgl-project#760 note describes. ONE AUTHORITY, FOUR CALLERS: the rule now lives in `consume_gate` and the four consume points call it; the base's inline copy is gone. Four copies of one rule is what cost W32. TESTS (11 new, 23 with the W35 release suite): the stamp arm and the disarm arm isolated and BOTH exercised (in a bare process the phase predicate reads "disarmed", which would have hidden the stamp arm -- pinned explicitly); refusals counted by name so a boot can tell clean from blind; and CAN-FAILs that every consume point calls the gate and that none reimplements it inline. Clean SERIAL gate: 8 failed / 4103 passed / 2 skipped -- same pre-existing 8.

…s queued Class 4's consumer half -- the last silent, DURABLE failure in the sweep. `backup_queue` is consumed by an always-running background thread that does not pause across the flip. After a rebind, `_page_backup` reads `mem_pool_host.get_data_page(...)` -- the INCOMING pool -- and writes those bytes to a CONTENT-ADDRESSED store under a hash computed from the tokens the operation was opened with. The hash does not match the payload, every later reader trusts it, and the corruption OUTLIVES THE PROCESS. Unlike the W35 double-free, which was loud, nothing catches this one. REFUSAL, NOT ROUTING, and the asymmetry against class 1 is the point. A stale RELEASE is routed to the pool its generation names, because that pool still owns those slots. A stale BACKUP cannot be: its host slots may belong to a pool that has since been repurposed, so there is no pool whose bytes are the right bytes. Declining is the only safe verb. A declined backup is a correct NON-PERSIST -- the prefix misses later and is recomputed, the same cheap failure the sgl-project#718 disarm and the sgl-project#760 write refusal already accept -- and it is acked either way, because an unacked operation stalls the queue. `operation_is_stale` is the sibling of `consume_gate` and lives beside it: one authority, two shapes (a queued batch at a consume point; a single operation on a background thread). A third copy of the rule is what cost W32. THREAD BOUNDARY: both generations are read EXACTLY ONCE, at the decision point, pinned by a test. The consumer runs on a background thread while the cutover mutates the current generation on another; a second read mid-persist could straddle a rebind and answer two different questions about one operation. TESTS (11) ASSERT ON STORE CONTENT, not on the counter -- a fix that counts and still writes is no fix. Includes the can-fail modelling the pre-fix path (remove the gate and it persists again), the read-once pin, the ack-anyway pin, the unstamped-op compatibility case, and a pin that the hybrid subclass does NOT override this loop -- the standing warning after the `append_host_mem_release` override shadow. NOT CLOSED, and filed rather than rushed: the PREFETCH consumer. Its loop runs `_all_reduce_prefetch_groups`, a COLLECTIVE, so a per-operation refusal placed before it risks splitting the group across ranks. That needs a rank-uniform formulation (most likely routing through the existing revoke path, which is already uniform), and guessing it on a collective is how a silent corruption becomes a hang. It is also the non-durable half: a stale prefetch loads into host memory, it does not persist to the store. Full SERIAL gate: 8 failed / 4114 passed / 2 skipped -- same pre-existing 8.

…oute blind over the anchor W38 acceptance boot died on all three ranks, 2026-08-26 12:54:28Z: IndexError: index 76997 is out of bounds for dimension 0 with size 30518 pool_host/base.py:344 assert self.slot_used[indices_cpu].all() memory_pool_host.py:1724 return self.anchor_entry.host_pool.free(indices) unified_radix_cache.py:2853 mem_pool_host.free(host_indices[:unclaimed_to]) HostPoolGroup.anchor_entry is fixed at construction, so the anchor does not move inside one group -- the GROUP is rebuilt onto a narrower tier at a phase rebind. In-flight prefetch state does not move with it: check_prefetch_progress holds host_indices minted against the previous, wider tier and frees them after the rebind, against a pool whose slot_used is shorter. THE CLASS, not the instance. 322f331 (sgl-project#718/sgl-project#847) fixed the TRANSFER path via _entry_for_transfer and did not sweep one level up. That resolver cannot cover these: it resolves by transfer.name through entry_map, and free / alloc / get_page_buffer_meta / get_data_page / set_from_flat_data_page receive BARE INDICES with no name (it has exactly two callers, :1803 load and :1834 backup). The index axis needs a range guard, not the resolver. Its own docstring asks for this: the raise exists "so that a future producer that bypasses the resolver is LOUD rather than wrong" -- free() is exactly such a producer. Sibling axis of _host_binding_is_stale (sgl-project#760), which guards the POINTER axis (device_buffers captured at construction) and left the INDEX axis unguarded. Same time signature: sgl-project#760's specimens died three seconds after a pp_to_tp cutover completed; this one dies on the free path instead of the write path. free() DROPS a stray, the other four are LOUD. A dropped free is a no-op on a tier torn down wholesale -- nothing leaks. An accessor that returns or writes a page cannot drop: that shortens a result or skips a store and the caller proceeds on data nobody wrote, the wrong-answer-with-no-crash outcome this codebase ranks worse than a crash. StrayHostIndexError subclasses IndexError so existing handlers keep working. THIS IS A BACKSTOP, NOT THE CURE, and the ticket should not be closed on it. The cause-side cut is one of two, both open: refuse at the rebind when the narrower tier cannot cover what is in flight (sgl-project#871, the unmet sgl-project#718/sgl-project#847 precondition), or make ongoing_prefetch a cutover participant with a completeness check (sgl-project#859, cutover_participants.py, present at the pin). What this commit guarantees is only that the failure is named and survivable instead of killing three schedulers. Unlike the binding generation stamp -- dead code whenever the binding does not advance -- this guard compares against the live pool's actual size and cannot be inert. Tests: test/registered/unit/mem_cache/test_stray_host_index_718_class.py, 13 passed. Can-fail proven by two mutations: disabling the stray filter reds 2, making the loud refusal a no-op reds 5; restore returns 13 green.

…n bracket and sgl-project#760 drain honesty: instruments, no sizing change) into the flip train

docs: update model support

5d228b5

zhyncs merged commit c3c74bf into sgl-project:main Jul 27, 2024

zhyncs deleted the doc branch July 27, 2024 04:07

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

docs: update model support (sgl-project#760)

a0c9883

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 27, 2026

Merge fix/917-load-stream-ima (sgl-project#917, the seam census poiso…

a22c4d5

…n bracket and sgl-project#760 drain honesty: instruments, no sizing change) into the flip train

Navigation Menu

Uh oh!

docs: update model support - #760

docs: update model support#760
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:doc

zhyncs commented Jul 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

zhyncs commented Jul 27, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant