Update fused_moe by merrymercy · Pull Request #553 · sgl-project/sglang

merrymercy · 2024-06-17T16:47:52Z

No description provided.

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…and a five-cut list Desk analysis only, no build, as scoped. docs/dev/ANALYSE_553_elastic_coresidence.md. What the four named building blocks actually are, read from the code rather than from their titles: - sgl-project#330 VRAM dial: a REAL mechanism, and its decisive property for sgl-project#553 is that store_bound_rows is a LIFETIME bound baked into captured graphs. So the dial has a FREE BAND (shrink to any floor, re-grow to the boot bound) and a HARD WALL above it. The free band is exactly where CUDA-graph compatibility is preserved. - sgl-project#364 GDN slot ladder: a real, executing hot/cold mover, and graph-neutral by construction (it moves state between slots, not geometry). It is the existence proof that a hot/cold event need not touch capture, and its inventory seam -- the owner enumerates its live set, consumers never read a container directly -- is the pattern the other consumers need (sgl-project#551). - sgl-project#287 KV pressure ladder: PLANNER ONLY. Seventeen NotImplementedError sites; the docstring says so plainly. But the ordering invariants are already enforced in code and the relief rungs reference existing features BY NAME, so the relief half can be made to execute by delegation without new mechanism. - sgl-project#286 register: the right vocabulary, but exactly one ledger exists and it is wired to the translator. There is no generic tenant to raise an event against. The pattern across the event table: both directions are largely built, and what is missing is ARBITRATION. Two registries exist (dial participants, sgl-project#286 asset classes) and neither can see the other, so "free 4 GiB for the video tenant" has no addressee. Cut list, ordered so each cut is provable alone and no cut before the last touches capture: (1) one registry or an explicit bridge; (2) a generic tenant mover factored out of AudioAssetLedger; (3) make sgl-project#287's RELIEF rungs execute by delegation -- the largest block of stubs, and it never approaches the graph wall because relief rungs are declared "no KV layout change"; (4) the arbitration policy on top; (5) LAST and only with the rig, cross the graph wall via the per-rung pre-captured graphs sgl-project#287 already specifies, with cold rungs parked as register class graph_rungs. Graph compatibility is treated as the target state throughout, per directive: every cut states its effect on graph state, and cut 5 carries the explicit route back to full coverage. Until cut 5 lands, sgl-project#553 must state its band -- "elastic within the boot capture bound" -- rather than implying unlimited elasticity. Instrumentation named honestly: SGLANG_KVSO_TICK_TRACE=1 (throttled per-session regulator trace: effective interval, measured tick_cost, binding headroom ratio, host-tail size; pure logging, no control effect). The analysis states what it is NOT -- a regulator trace, not a per-phase breakdown of device time -- and proposes no measurement it has not checked exists. One decision deliberately left open and named as such: whether sgl-project#553 owns the arbitration policy or whether it belongs to sgl-project#363. That fork needs the sgl-project#363 stage feed bound first, since an arbitration policy fed by a table holding one stage would have nothing to choose between -- the same failure sgl-project#363/S8 documents one level up. codespell: clean.

…mposability fixes) Desk strand (fix/spill-composability, included via 2a156c8): six bugs with proven can-fail falsifiers (sgl-project#552 parked-abort leak, sgl-project#552 spec-costs-spill, sgl-project#547 dead interop honesty, sgl-project#551 GDN blob loss + lxcfs-honest host RAM, sgl-project#568 ledger non-persistent buffer corruption), S8 docstring fix, analyses P6/P7/sgl-project#553. Full-sweep regression proof: failing-test sets byte-identical vs base e08946d (911=911), +41 newly passing. GPU strand: 8 matrix cells passed on hardware (kvso under uneven DCP, partial spill, restore gate, kvso x NEXTN, multi-session, ladder actuation, hibernate round trip 1.29x), matrix landed in-tree as docs/dev/ANALYSE_spill_matrix_20260804.md.

The night merge 2a156c8 was cut at b9a9fd6 while the desk strand kept committing; this brings in the missing tail: sgl-project#551 GDN-ladder parked-session fix + honest host RAM, sgl-project#568 generic ledger non-persistent-buffer fix, S8 stage-feed docstring fix + AUDIT_421 B.9, analyses sgl-project#553/P6/P7 and the rejected-register entry. Restores the Absent-means-dead and Round-trip-asymmetry catalog families the earlier merge missed.

… objective Four coordinator answers folded in; two of them close holes in the first draft. R9 (NEW, user 2026-08-05) -- every prefill/draft/decode(verify) cut is itself a load change, and an ENFORCER must actually cause a different layout per load. Flip granularity becomes a PRICED AXIS rather than a blanket rule: per knob, quantify the flip cost and classify by two checkable criteria. A knob is tick-flippable exactly when flipping it costs neither a CUDA-graph recapture nor a byte moved; everything else flips at regime boundaries with hysteresis. Grounded, not asserted: sgl-project#439 cold-expert compute assignment is verified VRAM-neutral in tree (expert_heat_migration.py:31 names "the same size before and after" as the sgl-project#439 sizing latch's invariant), which is precisely the zero-data-movement property that puts it on the tick side. Within a tick the layout is fixed by definition, so the right tools are overlap/behaviour adaptation (sgl-project#128/sgl-project#199, sgl-project#125, sgl-project#274, sgl-project#156) -- not lesser substitutes for flipping, but the correct instrument at that timescale. F9 is the falsifier for the silent error: a wrongly-regime-classified zero-move knob never demonstrates the gain it could have made, so nothing ever surfaces it. F9b guards the opposite mistake (a 3-6 s recapture at tick granularity would spend the whole tick budget on recapture). F9c asserts on ACTUATED state, not on a logged decision, because sgl-project#578 is exactly the case where the decision existed and the actuation did not. SLICE 0 (sgl-project#578) -- verified in tree, not taken on report: regime_runtime.py:911 calls planner_candidates(server_args) without solve_fn, so regime_stages.py:383 takes the None branch, returns [], and the stage table permanently holds the booted stage alone. The docstring at regime_stages.py:357 says it outright -- "it is not broken, it is unfed". The enforcer R9(i) requires therefore EXISTS AS CODE AND ACTUATES NEVER, which made every actuated requirement in the first draft unreachable. Promoted from unknown risk to a named prerequisite slice. The work is wiring plus a SolverAnswer->Stage mapping (key_solver.solve already exists and is objective-aware), but it needs plan_inputs, base plan, per-rank budgets and RigRates -- i.e. a card probe. So slice 0 shares TICKET_582 gate (a) as a prerequisite and should be scheduled into the same window; recorded. R4 -- answered, [CONFIRM] removed. The objective is JOINT (maximum compute AND maximum VRAM throughput/latency at all times), expressed as two axes rather than collapsed into one scalar on the user's behalf. Dominance on disagreement is decided by the running task's phase: decode is memory-bound so bandwidth IS throughput there, prefill is compute-bound. The weights come from the same regime signal R7 consumes. F4 is a pair for the same reason as F7: a fixed scalar objective passes exactly one of the two regimes. sgl-project#551 -- resolved as in-tree sgl-project#553, and my find is confirmed: DESIGN/ANALYSE only, no mechanism. Slice 5 therefore BUILDS elastic co-residency as the planner's tenant-event actuator. Stated plainly in the slice plan and §5 so the plan cannot be misread as driving something that exists. Also recorded: R9's knob inventory is incomplete by construction (only the knobs the addendum named are classified); a full sweep with a cost per knob is part of slice 4b, since an unclassified knob is an unpriced assumption and R9 forbids those. And "3-6 s" for recapture is an observed range, not a model -- R5's movement-cost work must cover it too.

…anularity Fifth addendum. R7 made EVACUATION ORDERING phase-dependent; R10 says the same of the WEIGHTS themselves, and at a finer grain than "the model": individual layers and slices of layers. The named first-class case is the draft/target ping-pong. During DRAFT, target layers or slices may be spilled to any tier for any reason -- including simply that the video and sound pipelines want the VRAM and this is the only way everything is held at once. During PREFILL/VERIFY the draft spills and the target returns. The two models take turns owning VRAM because they take turns computing. (a) Every combination must be EXPRESSIBLE, over the product (layer|slice) x tier x phase. R3 applied to residency: a design that enumerates "draft spilled" and "target spilled" as two supported modes has already failed, because the third thing wanted is a half-spilled target during a long prefill while the diffusion lane holds a card. (b) The planner computes WHAT moves WHEN as a movement-bound decision on R9's priced flip axis -- never a structural cannot. A decline must be arithmetic and recoverable under R8. (c) Executed automatically, and the freed VRAM REFILLED automatically. An empty hole is a DEFECT, not a neutral outcome: it is the "surplus sits idle" failure sgl-project#582 removed from the boot path, reappearing at runtime. F10c therefore asserts on the CLAIM of the freed bytes, not on the free. OVERLAP MUST BE PRICED. While the draft computes the target's layers are idle, so their reload hides behind draft compute and vice versa. The cost model charges the unhidden remainder, effective_cost = max(0, transfer_time - overlappable_compute_time), not raw bytes/bandwidth. Charging raw transfer would price every ping-pong as unaffordable and the planner would never take a swap that is in fact nearly free -- the same error shape as pricing a restore at capture cost (F9e): a cost model wrong upward never exercises the mechanism, so nothing surfaces it. CARRIERS, verified with an honest split. Present: sgl-project#77/sgl-project#123/sgl-project#125 weight streaming really does H2D fetch of weight rows from a host tier (expert_offload.py, expert_heat_migration.py, fused_moe_triton/layer.py, cold_tier_fetch.py with identity-keyed host DRAM segments and zero-copy peer views, plus offload_register/short_term_offload_register/offload_movement). NOT present: it is EXPERT-granular. I found no layer-granular streaming and no per-layer addressing in offload_movement.py, and could not locate the double-buffered prefetch-behind-previous-layer-compute structure as a general mechanism. Slice 5 therefore EXTENDS this carrier rather than driving it -- same posture as sgl-project#553 -- and the overlap pricing in R10 DEPENDS on that structure being real, so slice 5 must confirm or build it. F10 (planned AND executed, not refused), F10b (the verify phase after a spill must be BYTE-IDENTICAL to a never-spilled reference -- residency is a placement decision, never a numerics decision, or the mechanism is unusable under the determinism arm), F10c (freed bytes must be claimed). Slice plan gains 4d (express + price, desk-provable) and extends slice 5. Also recorded in gaps: the 2026-08-05 window's measured out-of-budget demand (2434 MiB on a 3080, 1305 on the 5090) already contradicts the inherited stock activation heuristic (3968 MiB), so R2's "exact" is not yet met on that term. Fixing it is a prerequisite for trusting any objective computed from it.

…dge stops answering all-unavailable DIAL SIDE -- vram_dial.reclaimable_bytes_for(participant, floor_rows). A LIVE read: full_pool_backed_rows (the bound eager launches actually pass, the same quantity verify_pool_reached_capacity checks a commit against) times _pool_row_nbytes (the pool's real per-row K+V bytes), minus the floor. floor_rows is REQUIRED AND NOT DERIVED THERE. There is no per-pool floor authority in that module -- the dial's floor is a card-level NVML measurement (_measure_local_floor_bytes) taken at boot -- and inventing a per-pool one would create a second authority for a number sgl-project#584 says has exactly one. A caller without a floor gets None, which the bridge turns into a named refusal. REGISTER SIDE -- OffloadRegister.reclaimable_bytes(offload_class), mirroring latency_term_ms's lock-and-filter shape so the REGISTER answers about itself rather than the bridge re-deriving its accounting from outside. Resident AND not hot: parked bytes are already gone (counting them promises the same bytes twice), and park() refuses hot items unconditionally, so hot bytes are resident but NOT reclaimable -- including them would hand a caller a figure it cannot spend. An unanswerable hotness predicate counts as HOT: the safe direction is refusing to reclaim, never assuming free to move. THE DISTINCTION THE CUT TURNS ON: ProbeUnavailable vs zero. Zero is a MEASUREMENT ("at its floor" / "nothing resident"); a failed probe is the ABSENCE of one. Both probes return None/raise rather than 0, and the bridge turns that into a NAMED refusal. Collapsing them would remove a real source from an elastic plan while looking like it was considered -- the sgl-project#606 defaulted-measurement defect, which this strand has now refused in four separate places. HONEST LIMIT, unchanged and restated in the analysis: the hermetic proof exercises the PLUMBING with faked dial/register state. No live number's correctness is claimed. In particular the dial probe is only as right as the floor its caller supplies, and no caller supplies one yet -- that is the window item section 4 already names. ALSO ADDED (section 8, SKETCH ONLY, no build): the cold-direction policy that would consume the bridge -- idle event -> query -> refuse rather than take a partial -> GDN vacate first (cheap) -> dial grow below the captured bound -> stop rather than attempt a re-capture. Debounce rationale is #704a's jump price: a rung change costs a full ~1575 ms arena refill, so an idle/hot flap across a rung boundary pays it twice for no net capacity; hysteresis on the event plus a rung-crossing cooldown, the same reasoning SpillCooldownRegistry already carries for the spill/restore pendulum. Anything requiring a geometry flip stays design-only, dependency named (sgl-project#677 arming-floor budget). Tests, hermetic (CUDA_VISIBLE_DEVICES="", no CUDA, no boot): test_coresidency_registry_553.py 30 passed (16 Cut 1 + 14 Cut 2) MUTATION: making a failed probe collapse to 0 fails 3 -- including the pin that a MEASURED zero and an UNMEASURED one stay distinguishable test/registered/unit/managers/ 13 failed / 2247 passed -- failures unchanged from the pre-existing 13-failure baseline, passes 2233 -> 2247, exactly the 14 new pins offload-register suites: 129 passed (I added a method to that class) ruff + codespell clean. Same branch (feat/553-elastic-coresidency). Desk only, no boot, no GPU.

…authorities only WIRED VS PRICED-ONLY, answered by grep before building: nothing imported coresidency_registry outside its own module, so a tenant COLD event did NOT actuate -- the bridge priced and no one called it. Same for HOT. That is the gap Cut 3 closes. managers/coresidency_policy.py is the caller and introduces NO NEW ACTUATOR. Every move is an existing authority's own API: vram_dial.apply_budget_request (replicated grow/shrink, already enforces the floor, "rejections carry the exact floor arithmetic and change nothing"), vram_dial. verify_pool_reached_capacity (the read-back), GdnSlotRuntime.unbind (sgl-project#364), and the Cut 1/2 bridge. A source pin forbids this module from touching torch.cuda / cuMem / set_active_prefix / runtime_set_ at all. THE TWO DIRECTIONS FAIL DIFFERENTLY AND THE CODE IS SHAPED BY THAT. COLD -- shrink must not strand bytes. Every source drawn on is recorded with what it was asked for and what it REPORTED giving. A source asked that reports nothing is carried as STRANDED: never counted as zero, never dropped. Bytes that left one ledger and entered none are the shape that goes unnoticed for weeks. A delivered zero is an accounting; silence is not, and the result keeps them distinguishable. Per sgl-project#694, totals come only from actuator reports -- nothing increments a "reclaimed" figure from the plan. An unfundable ask refuses and actuates NOTHING rather than drawing what is there, and the refusal states that unavailable bytes did not count toward it. HOT -- grow must not exceed the floor, and that is deliberately NOT enforced here: the dial enforces it. This module's duty is to not paper over the refusal, so a floor refusal is returned unchanged and is NEVER RETRIED SMALLER -- a floor refusal is a statement about the rig, not a negotiation. sgl-project#217 SHAPES THE HOT PATH. A restore that "came back" was measured at 23% of target. So a grow is followed by a read-back and the result reports what was MEASURED, never what was requested. An absent read-back leaves reached_bytes None -- "not measured", distinct from "reached nothing" (sgl-project#606). A read-back that raises REFUSES, because a tenant must not be reported warm on an unverified grow. Tests, hermetic (CUDA_VISIBLE_DEVICES="", real bridge objects, injected actuators, no scheduler, no CUDA): test_coresidency_policy_553.py 16 passed + 4 subtests MUTATIONS: hiding stranding, and reporting the request instead of the measurement, fail 4 test/registered/unit/managers/ 13 failed / 2263 passed -- failures unchanged from this branch's pre-existing 13, passes 2247 -> 2263, exactly the 16 new pins ruff + codespell clean. STILL PRICED-ONLY, and this is the honest seam: the policy takes injected release_fn / grow_fn / measure_fn. Binding them to the live dial and slot runtime needs a scheduler and belongs with the live proof, not a desk fake. WINDOW TICKET: /spinning/GPU_WINDOWS.md. There was NO prior sgl-project#553 entry (grepped; none), so this CREATES one rather than appending to an existing ticket, and says so in the entry. The 142 pre-existing lines are verified byte-identical by sha256 before and after (18e21da5dda7ce55).

The analysis's tenant-COLD row named the gap exactly: parking 'works for the translator (ledger.park_all) and for GDN slots (sgl-project#364). No generic per-tenant mover; no single caller that can address tenant X.' The first half wired the two byte probes so the bridge stops answering all-unavailable; this is the actuator a cold event calls once that bridge has ranked a plan. I CHECKED THE TABLE'S GATING COLUMN RATHER THAN TRUSTING IT, because §10 had just caught the same table lying about numbering. test_ledger.py runs 22 passed under CUDA_VISIBLE_DEVICES=: the ledger's park/restore is already exercised hermetically with CPU tensors, so a mover over the same protocol is desk- provable and the marking holds. Had those tests needed a device this would have been a window item and the table would have been wrong twice. NOT A SECOND LEDGER. AudioAssetLedger already parks and restores and its ParkRoute protocol is already generic; what was missing was never the parking. TenantMover registers TENANTS over that same protocol and reimplements nothing. A parallel asset ledger would have been the two-authorities defect the sgl-project#553 bridge exists to reconcile. The vocabulary is per tenant, which is the whole factoring: the translator's ASR -> talker -> codec order is one tenant's physics, not a property of moving tenants. register(tenant, routes, ranks=...) takes it as configuration, and a test asserts a second tenant restoring y, x -- nonsense for the translator, correct for that tenant. An unranked route sorts last: unlisted is unknown, not urgent. Stranding keeps the shape the policy layer already uses. Released bytes come from what a route RETURNED (sgl-project#694); a route that reports nothing, or raises, is STRANDED and never zero, while a route reporting 0 is an accounting. release_fn() hands cold_event a None in exactly that case, which is how that layer reads 'did not report', so a stranded tenant surfaces there instead of being counted as a delivered zero. parked_bytes_by_device() omits a device with nothing parked rather than reporting 0 (sgl-project#606). An unknown tenant RAISES and names who is registered: 'no such tenant' and 'that tenant had nothing to give' are different answers. A defect in my own test, recorded in the analysis: the first _Route stub used parked_bytes=None for 'not specified', so 'reports nothing' was inexpressible and two stranding tests failed against correct code. A stub whose default and whose absence-case share a value cannot test an absence rule. 12 tests red first, mutation-proven (collapsing stranding to zero reds the two stranding pins; ignoring the supplied ranks reds all three vocabulary pins). 57 passed across the mover, the rung executor, the coresidency policy and the translator ledger -- the ledger's 22 unchanged, it was not touched. ruff clean. Desk only, no boots. sgl-project#553's desk surface is now exhausted: cut 4 is gated on sgl-project#305's multi-model binding, cut 5 is rig-gated by its own text, G1 is filed, and the live bindings plus any live number's correctness remain the named window items.

update fused_moe

d6a7c32

merrymercy merged commit 53a7ebd into main Jun 17, 2024

merrymercy deleted the update-moe branch June 17, 2024 16:48

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Update fused_moe (sgl-project#553)

cf21efa

Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026

fix: flush cache hang and update benchmark scripts (sgl-project#553)

e2eb9d7

vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026

Copy OSS code from f3d73b0 on 20251229 (sgl-project#553)

1165ecf

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Navigation Menu

Uh oh!

Update fused_moe - #553

Update fused_moe#553
merrymercy merged 1 commit into
mainfrom
update-moe

merrymercy commented Jun 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

merrymercy commented Jun 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant