Repository navigation
[Perf] [MI355X] deepseek_v41: One launch per reuse layer, the index sources' compressor in K1, the Engram projection in the layer launches, cheaper hand-offs, the Markov block on the package's kernels - #37
Open
heslami wants to merge 12 commits into
Conversation
added 12 commits
November 4, 2026 15:11
…token tile At 6+ rows K1 computes the indexer's head weights (x against weights_proj, 10 K splits of 512 per 16-head tile) on 2 CTAs a token tile, which read 164 KB each, so an index source's K1 ended about 9 us after its other CTAs. Each split pair now runs on its own CTA, a split a wave with every step's loads in flight, on the CTAs past the q / kv norms' (iw_tasks counts them); the first part polls the others' partials and sums all ten in split order, so the head weights stay bit for bit SGLang's.
A reuse layer (not an index source) now runs K1 and K2 in one launch (build_mono_layer, runner.layer). K1's products no longer cross a launch boundary: q, the window records, the key table and the mixes are stored device scope, and every CTA flags its q rows with K2's tag past its wq_b; that flag also tells that the CTA's records, key-table rows and mixes landed. The attention back's split waits on every CTA's q flag of its token's tile and on its token's key-table flag, the combine on the key table's, the FFN seam's slices on the gates' flags; token parallel, the records copied from the other ranks get a flag a row. K1's X8 and the MoE's no longer share a region. The CTA that runs split unit 0 has seen a q flag from every CTA, each stored past that CTA's epoch read, so it moves the epoch on and clears the MoE counter slot there; the end-of-launch fan-in is gone. An index source's K1 and K2 and the FFN launch keep the marks, and the last CTA waits on them off the launch's end. The layer launch loads K2's arguments again from the kernarg segment at K2's start, so they are not live (spilled) through K1. The epoch read at the launch's start is held in a scalar register through K1 into K2. Bit for bit: check_layer.py checks the layer launch against K1 then K2 at every variant and under graph replay.
…fter its reduce A slice of the FFN seam issued its first loads (its rows' first batch and its mix weights) before it waited on the ranks' attention partials. Past 32 rows the reduce of those partials takes more than one pass, and the loads stayed live through all of them, so the K2 and FFN builds there spilled VGPRs, on sgl-project#82's head already. Up to 32 rows the loads still go out ahead of the wait and the code is unchanged; past 32 they go out after the reduce, which runs as a traced loop with one pass's polls in flight at a time. The sums keep their order.
…every width Below 12 rows the router CTAs loaded their gate rows (41 KB a CTA at 6 rows) only after the FFN norm's x flags, so the router started with its weights still in flight: about 2.8 us of exposed load at 4 and 6 rows. The rows do not depend on x, so every width now issues them before the wait, as wide steps already did. Bit for bit.
…unch The verify forward gathered the step's embedding rows (indexSelect, about 6 us) and broadcast them into the four mHC streams (_hc_broadcast_kernel, about 5.5 us) before layer 0; the draft's first stage did the same with a repeat. A reuse layer's launch now takes the whole table kept at load and the step's token ids (MonoBuild.embed): the seam's slices read each stream from the table row of the token's id. Every stream of SGLang's broadcast is the row itself, so the layer's inputs are bit for bit the same; check_layer.py checks the launch on (table, ids) against the launch on the broadcast rows.
… cross-CTA payload device scope Each width's scratch is a one-rank uncached common.peer.PeerBuffer, as glm53_flash's (sgl-project#68), so the hand-offs between CTAs go through no cache line: plain stores, vmcnt(0), then the flag; the consumers read with sc1. The payloads K1 and K2 read behind a flag with the default policy (the X8, q latent, XO and X8B rows, the combine's partials and stats) are now sc1 loads: a default load can hit a stale line in the CU's own L1, uncached memory included. Bit for bit.
A router CTA has its gate rows (41 KB) in flight when it polls the FFN norm's x flags, and a vector poll returns only behind them (the loads return in issue order): up to 1.6 us after it is issued at 6 rows. Each FFN-norm CTA now also stores the launch's tag into its word of every CTA's 128 B box line once its normed row landed, and a router CTA polls its own line with one scalar glc load (lgkmcnt), as glm53_flash's boxes (sgl-project#63) do. The box lives in the uncached scratch. Bit for bit.
…on in K1 At 6+ rows (not token parallel) K1 now also runs the compressor's bf16 projection of its normed rows -- wkv_gate (kv and score, f32) at compress ratio 2, wkv (bf16) at ratio 1 -- a 16-column tile and K quarter a CTA in q's wait, the quarters summed in order. SGLang's fused compression takes it as its projection, in place of its GEMM between K1 and K2i. The sums run in another order than SGLang's GEMM: check_k1 bounds them against fp64 (f32: 1e-5; bf16: 8e-3) and counts the values that differ from SGLang's projection.
…, and the walk plan in K2i Between an index KV source's K1 and its K2i, SGLang ran its fused decode / verify compression in three launches (c2 / c1 decode, wk, index-K store), and a one-CTA Triton launch wrote the walk plan K2i's FULL tasks read before the first FULL or producer source of each compress ratio. KC (indexer/compress.py) does the compression a row at a time from K1's projection: the ratio-2 pair pooling against the state ring (even positions parked) or the ratio-1 kv, the norm, RoPE, the FP4 fake quant and the V4 record store, then the indexer's wk, k_norm, RoPE, both FP4 stages and the split FP4 plane store. It runs in K1's tail: the projection's part-0 CTAs store their columns device-wide and flag each (token tile, column tile), and at K1's end the grid's last CTAs, which finish their front first, run KC's row a CTA once the flags of their token tile and those before are in, with the row's other loads issued while they come in. The spec builds KC's arguments ahead of the index K1 (K2iRunner.kc_args); the KC kernel also builds as its own launch for its test. KC's wk dot reads wk in its threads' order (compress.wk_order, prepared with the K1 weights), so a wave's loads are contiguous, a K quarter a wave. The step's first K2i of a ratio now works out the walk plan itself, a row a lane of every wave, and each CTA writes its row for the ratio's later sources. test_deepseek_v41_kc.py checks the ring, records and keys against SGLang's chain (equal bytes on its cases); the plan equals the host plan (test_deepseek_v41_k2i.py), so the selection is unchanged.
…nches wkv projects the step's retrieved n-gram rows alone, so the reuse layers between two Engram layers can compute it ahead of the gate: each launch takes a range of this rank's rows of wkv (an N shard, all of K), on the CTAs without attention or wo_a units, after their attention work and before their GEMV weights load, so none of the back's registers are live under it. A CTA quantizes the token tile into LDS as SGLang's MXFP8 linear quantizes its input before any wkv weights load, then runs its 16-row tasks as a loop: each K step's registers take the next task's weights once its MFMA has them, so at most one task's weights are in flight and the live registers do not grow with the tasks a CTA runs. It pushes the bf16 outputs to every rank's ENG region; at the launch's end the same CTAs copy the range into the runner's kv for SGLang's gate, in place of the K-split path's quant, copy, GEMM and all-reduce. Layer 0 takes all of layer 1's projection, the reuse layers 1 .. 13 share layer 14's. Each output is one f32 sum over K (the GEMV's step order, the waves in order) rounded to bf16 once, where the K-split path rounded each rank's partial and summed them on the all-reduce: check_layer.py bounds kv against an fp64 product of the MXFP8 operands and checks the launch's other outputs bit for bit as without, kv alike on every rank, and a projection shared by two launches.
… logits SGLang keeps a sampled draft's step logits whole over the vocab, so the Markov block's kernels never ran and every draft step took SGLang's seven launches (two all-gathers and a token broadcast). Each rank now writes its shard's step logits to its exchange buffer, and after the block's last step one launch gathers every rank's shard into the sampled sequences' whole rows; a temperature-0 row's stay as SGLang's buffer holds them.
… shared down's MFMAs A down CTA ran the shared expert's down projection, its input ready long before the routed results, before it issued routed round 0's ug-flag loads and weights. It now issues the shared down's weights, loads the shared rows into LDS, issues round 0's flag loads and weights, and runs the shared MFMAs while they land; the shared down's own loads no longer queue behind round 0's. It does so only where both weight sets fit in registers, a routed intermediate of at most 5 K steps (TP4 and TP8); TP2 keeps the rounds after the shared down. Bit for bit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Depends on sgl-project#82 (
[Perf] deepseek_v41: A line a flag, the Engram K split and the PV hoist; serve at TP8), which merges first. This branch contains sgl-project#82's commits. This PR's own diff is the 12 commits after sgl-project#82's head (44c8515). Every number below is measured against sgl-project#82's head, notmain.DSpark verify cycle on sgl-project#82, at TP4 with one request: about 4.15 ms. Of that, about 3.2 ms is in the 40 layers, 0.27 ms is SGLang glue between the layers' launches, and 0.8 ms is outside the layers. This PR takes time out of all three, as follows.
Two commits change the order of a sum: the compressor projection (8) and the Engram projection (10). Every other commit leaves every value bit for bit as it was.
Result on MI355X, TP4, DSpark (block 5), one request, sgl-project#82's head → this PR (A/B/A, one allocation):
Modifications
Twelve commits, one change each. Each one's gain is in the per-commit table (Speed Tests). Unless a subsection says otherwise, a commit doesn't change any value.
1. An index source's head weights on 10 CTAs a token tile (
attention/index_query.py,attention/plan.py,layer.py)At 6+ rows K1 computes the indexer's head weights (x against
weights_proj: 10 K splits of 512 per 16-head tile) on 2 CTAs a token tile. Each of those CTAs read 164 KB, so an index source's K1 ended about 9 µs after its other CTAs.iwp, a new scratch region) and sums all ten in split order. The head weights stay bit for bit SGLang's.iw_tasks(from [Perf] deepseek_v41: A line a flag, the Engram K split and the PV hoist; serve at TP8 sgl-project/sgl-amd-mono#82) now counts the 10 a tile. Where the grid has no room past the norms (33–47 rows), the head weights start right after the q / kv norms, which run from CTA 0, so the two never share a CTA.2. A reuse layer's K1 and K2 in one launch (
layer.py,runner.py,attention/front.py,attention/back.py,stages/seam.py,spec.py,draft.py,common/epoch.py)A reuse layer (one that isn't an index source) now runs K1 and K2 in one launch (
build_mono_layer,runner.layer).common/epoch.py:epoch_endtakes the CTA that runs it and can skip the marks. Its defaults keep every other model's code as it was.check_layer.py(new) checks the layer launch bit for bit against K1 then K2 at every variant, and under graph replay.test_deepseek_v41_kernels.pyruns it.3. Past 32 rows, the FFN seam slice's loads after its reduce (
stages/seam.py,layer.py) — spill fixA slice of the FFN seam issues its first loads (its rows' first batch and its mix weights) before it waits on the ranks' attention partials. Past 32 rows the reduce of those partials takes more than one pass, and the loads stayed live through all of them, so K2 and the FFN launch spilled VGPRs there, on sgl-project#82's head already.
4. The router's gate rows before x's wait at every width (
stages/moe.py)Below 12 rows the router CTAs loaded their gate rows (41 KB a CTA at 6 rows) only after the FFN norm's x flags, so the router started with its weights still in flight. The rows don't depend on x, so every width now issues them before the wait, as wide steps already did.
5. The first layer's launch reads the embedding table (
layer.py,stages/seam.py,token_par.py,runner.py,spec.py,draft.py)Before layer 0, the verify forward gathered the step's embedding rows (
indexSelect) and broadcast them into the four mHC streams (_hc_broadcast_kernel); the draft's first stage did the same with a repeat.MonoBuild.embed). The seam's slices read each stream from the table row of the token's id.check_layer.pychecks the launch on (table, ids) against the launch on the broadcast rows.6. Uncached launch scratch, every cross-CTA payload read device scope (
runner.py,attention/back.py,attention/gemv.py)common.peer.PeerBuffer, as in glm53_flash ([Perf] [MI355X] glm53_flash: Cheaper hand-offs: uncached scratch, fewer vector loads ahead of the polls, tagged selected tokens sgl-project/sgl-amd-mono#62) and glm53 ([Perf] [MI355X] glm53: Cheaper hand-offs and placement: the shared expert before the routing, uncached scratch, tagged selected tokens (1/2) sgl-project/sgl-amd-mono#68). The hand-offs between CTAs go through no cache line: plain stores,vmcnt(0), then the flag.sc1loads: the X8, q latent, XO and X8B rows, and the combine's partials and stats. A default load can hit a stale line in the CU's own L1, on uncached memory too.7. The router waits on x through a scalar-polled box (
stages/moe.py,layer.py)A router CTA has its gate rows (41 KB) in flight when it polls the FFN norm's x flags. A vector poll returns only behind them, since loads return in issue order: up to 1.6 µs after it is issued at 6 rows.
glcload (lgkmcnt), as glm53_flash's boxes do ([Perf] [MI355X] glm53_flash: Scalar-polled boxes for broadcast rows, and o_proj before the state write-back at 2 rows sgl-project/sgl-amd-mono#63).8. An index KV source's compressor projection computed in K1 (
attention/index_query.py,attention/plan.py,layer.py,runner.py,spec.py) — changes the order of a sumAt 6+ rows (not token parallel), K1 now also runs the compressor's bf16 projection of its normed rows:
wkv_gate(kv and score, f32 out) at compress ratio 2, andwkv(bf16 out) at ratio 1;_low_ratio_compress_fused(projected=)), in place of its GEMM between K1 and K2i.The sums run in another order than SGLang's GEMM (Accuracy Tests).
check_k1bounds them against fp64 (k1.cp.rel1e-5 for f32,k1.cp_bf16.rel8e-3 for bf16) and now also counts the values that differ from SGLang's own projection.9. An index KV source's compressor in K1's tail, and the walk plan in K2i (
indexer/compress.py(new),indexer/k2i.py,indexer/walk_plan.py,indexer/score_fp4.py,indexer/runner.py,attention/index_query.py,attention/plan.py,layer.py,runner.py,spec.py)Between an index KV source's K1 and its K2i, SGLang ran its fused decode / verify compression in three launches: the c2 / c1 decode,
wkand the index-K store. Before the first FULL or producer source of each compress ratio, a one-CTA Triton launch also wrote the walk plan that K2i's FULL tasks read.indexer/compress.py) does the compression a row at a time, from K1's projection (commit 8):wk,k_norm, RoPE, both FP4 stages and the split FP4 plane store.K2iRunner.kc_args). The KC kernel also builds as a launch of its own, for its test.wkdot readswkin its threads' order (compress.wk_order, prepared with the K1 weights), so a wave's loads are contiguous, a K quarter a wave.test_deepseek_v41_kc.py(new) checks the ring, records and keys against SGLang's chain, including itswkGEMM: equal bytes on every case. The plan equals the host plan (test_deepseek_v41_k2i.py), so the selection is unchanged.10. The Engram rows projected in the reuse layers' launches (
stages/engram.py(new),attention/back.py,layer.py,token_par.py,runner.py,spec.py) — changes the order of a sumwkvprojects only the step's retrieved n-gram rows, so the reuse layers between two Engram layers can compute it before the gate.wkv(an N shard, all of K). It runs on the CTAs without attention or wo_a units, after their attention work and before their GEMV weights load (run_back(idle=)), so none of the back's registers are live under it.wkvweights load. It then runs its 16-row tasks as a loop: each K step's registers take the next task's weights once its MFMA has them, so at most one task's weights are in flight and the live registers don't grow with the tasks a CTA runs. It pushes the bf16 outputs to every rank's ENG peer region as tagged pairs. At the launch's end the same CTAs copy the range into the runner's kv, which SGLang's gate reads.check_layer.pybounds kv against an fp64 product of the MXFP8 operands. It also checks that the launch's other outputs are bit for bit as without the projection, that kv is alike on every rank, and that a projection shared by two launches equals one launch's.11. The sampled Markov draft block on whole step logits (
draft_sampler.py,markov/,check_markov.py)The image's SGLang keeps a sampled draft's step logits whole over the vocab, so sgl-project#95's Markov kernels never ran: every sampled draft step took SGLang's seven launches (two all-gathers and a token broadcast).
draft_markov_block.draft_sampler.pyno longer readsdraft_markov_block(sharded_accept), which this SGLang no longer has; that read failedtest_contracton [Perf] deepseek_v41: A line a flag, the Engram K split and the PV hoist; serve at TP8 sgl-project/sgl-amd-mono#82's head.12. Down round 0's flags and weights under the shared expert's down MFMAs (
stages/moe.py)A down CTA ran the shared expert's down projection before it issued routed round 0's ug-flag loads and weights. It now:
The shared down's own loads no longer queue behind round 0's.
It does so only where both weight sets fit in registers: a routed intermediate of at most 5 K steps (
DOWN_EARLY_STEPS), i.e. TP4 and TP8. TP2 keeps the routed rounds after the shared down.Accuracy Tests
Numerics
Every commit except the compressor projection (8) and the Engram projection (10) leaves every value bit for bit as it was. The kernel checks show it per launch:
check_layer.py(new in commit 2), TP4 at 1 / 4 / 6 / 12 / 17 / 24 / 36 / 42 / 48 rows and TP2 at 1 / 6 / 12 / 24 / 48. The layer launch matches K1 then K2 bit for bit in every variant: fold, compressed keys, the embedding table, and token parallel from 12 rows. It also holds under 20 graph replays.check_k1.py, TP4 at 1 / 6 / 12 / 24 / 30 / 36 / 42 / 48 rows and TP2 at 1 / 6 / 12 / 24 / 36 / 48. Every seam output, the KV records and the index query (iq,iqs,iw) differ from SGLang's chain in 0 bytes, as on [Perf] deepseek_v41: A line a flag, the Engram K split and the PV hoist; serve at TP8 sgl-project/sgl-amd-mono#82's head. That covers the head weights' 10-way split (commit 1) and its new placement at 33–47 rows.check_k2.pyat TP4 (1 / 6 / 12 rows, 20 replays) and TP2 (5 replays): the FFN stages bit for bit.test_deepseek_v41_kc.pycompares the state ring, compressed records, FP4 index plane and scales byte for byte against SGLang's chain, including itswkGEMM. Every case matches.test_deepseek_v41_k2i.py. The plan K2i lays out equals the host plan on 240 random ragged steps, and the selection tests pass.check_markov.pygives 0 differing tokens at batch 1–4, and the startup check compares against SGLang on every rank.Commit 8, the compressor projection. K1's projection is summed in another order than SGLang's GEMM.
check_k1on this PR's head at 6 / 12 / 24 / 48 rows (the SGLang comparison at TP4, 2 trials; the fp64 bound at TP4 and TP2):wkv_gate)wkv)At ratio 2 nearly every f32 value differs in its last bits. The compressor then rounds the pooled kv to bf16, so a key's FP4 code can flip where a value sits on a rounding edge.
Commit 10, the Engram projection. Each kv element is now one f32 sum over K, rounded to bf16 once. sgl-project#82's K-split path rounded each rank's 1536-wide partial to bf16 and summed the four on the custom all-reduce.
check_layer.pyat TP4 (1–48 rows) and TP2 (1–48 rows) gives, against the fp64 product of the MXFP8 operands:End to end (TP4, real acceptance, the e2e test's server flags)
test/e2e/test_deepseek_v41_server.py, all 9 cases on this PR's head: DSpark verify (monokernel batches, GSM8K past the largest width, decode vs prefill KL, long prompts vs prefill at 18K–27K tokens, passkey past the candidate span), decode (the same three), and the stalled-rank fault. 9 passed (55 min 51 s), withHSA_DISABLE_COREDUMP_ON_EXCEPTION=1.44c8515) pass@1 ± SEMThe PR's runs are on earlier heads of this branch:
2361a0f(runs 1 and 2) and6631639(run 3). Both also carried four commits that were dropped later: the seam's 16 B LIN loads, SGLang's DSpark folds, the verify logits in two launches and the producer's block maxima. Each of those was bitwise, to SGLang or to the code before it, by its own check.The kernels are equal: every eager layer, K1 and K2 launch of
check_layer.pyat TP4, 6–48 rows by 6, digested on every rank, is bitwise the same on6631639and on this head. That is 1,344 launches, 0 differing. The same tree run twice also differs in 0, the control.majority@16 and pass@16 are 1.0 in every run. Every truncation is a repetition loop at the token cap, on either head.
GSM8K with SGLang's
few_shot_gsm8k: 8-shot, greedy, all 1,319 questions, 16 in parallel, on the same servers.Greedy DSpark serving isn't reproducible run to run on this stack, sgl-project#82's head included: its three runs span 1.3 points. These are the same servers as the AIME runs.
Speed Tests and Profiling
Setup
lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261028, imported as a squashfs with sha256f04d57dce53ca5d6aafe6d9bc92b8d81a6b316fde7e55e38564c857f60bc2252(the registry digest was not recorded at import). It ships SGLang 0.5.22.dev20261028 (5d8e90a0f9), AITER285579b6, FlyDSL 0.4.0, torch 2.9.1+rocm7.2.0 and ROCm 7.2.0. The plugin tree is mounted over the image's.deepseek-ai/DeepSeek-V4.1-Flashat314113f8a7dadb388215396ba5fbee2c610a04d6.test/e2e/deepseek_v41_server.py:--tp 4 --trust-remote-code --enforce-shared-experts-fusion --fp8-gemm-backend aiter --mem-fraction-static 0.7, plus--enable-amd-monokernel. The e2e test passes--monokernel-backend amd, which this SGLang takes as a deprecated alias of it.--speculative-algorithm DSPARK --speculative-dspark-block-size 5 --cuda-graph-max-bs-decode 16 --max-running-requests 16, i.e. 6 verify rows a request.--cuda-graph-max-bs-decode 64 --max-running-requests 56.SGLANG_USE_AITER=1 SGLANG_USE_ROCM700A=0 AITER_BF16_FP8_MOE_BOUND=0.--tp 2/--tp 8. TP2 also setsSGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank, as InferenceX's TP2 recipes do./generate: an exact 8,192-token (or 327,680-token) prompt of ShareGPT text, its own slice so nothing is cached, and 1,024 greedy output tokens with EOS ignored. 24 prompts follow a warm-up request; the 320K point is one prompt.spec_verify_ct, median over the prompts. It is the robust number: the accept length of greedy requests moves by about ±0.1 between servers on the same prompts.aa45727) against [Perf] deepseek_v41: A line a flag, the Engram K split and the PV hoist; serve at TP8 sgl-project/sgl-amd-mono#82's head (44c8515), unless a table says otherwise.This PR against sgl-project#82's head (TP4)
DSpark, one request: A/B/A in one allocation, one server per column.
DSpark off, one request, 8 prompts at 8K, A/B/A:
Each commit on its own (TP4)
TP2 and TP8
TP2 (2 GPUs, the Engram host table
per_rank), one request at 8K, A/B/A in one allocation for each table:TP8 (8 GPUs), one DSpark request at 8K, A/B/A, then the one-launch commit against the commit before it, in the same allocation:
AgentX (TP4)
semianalysis_cc_traces_weka_062126at23f152f6, scenarioinferencex-agentx-mvp), in InferenceX's fast mode: a 1,200 s replay after one warm-up request a lane, seed 42.SGLANG_SIMULATE_ACC_LEN=3.51), so tok/s/user is 1 / ITL. The measured accept length is 3.51 on every server.--reasoning-parser auto --tool-call-parser auto --enable-cache-report --chunked-prefill-size 16384 --prefill-decode-interval 192 --schedule-policy lpm --swa-prefix-tails 128(512 at C = 8)--enable-decoder-swa-bounded-replay --cuda-graph-backend-prefill disabled, withSGLANG_DEFAULT_THINKING=1 SGLANG_DSV41_REASONING_EFFORT=high.Startup
Every launch of every width is built at startup. On an empty FlyDSL cache, this PR builds about five times as much as sgl-project#82's head. Each width adds the layer launch's variants (fold × compressed keys × the Engram share it carries, and layer 0 on the embedding table) and the index sources' K1 with the compressor.
TP4, DSpark off (widths 1–48), measured in one allocation; the build time is the spec's
built inline:Registers
A compile-only build of every key the spec builds:
spec._buildderives them, and read withllvm-readelf --notes;Largest VGPR spill / scratch bytes a lane, over the builds of each class:
Tests
5d8e90a0):minimax/spec.py,minimax_m31/spec.pyand the threemuse_sparkmodules, andtest_deepseek_v41_spec.py::test_model_reasons.deepseek_v41/draft_sampler.py: it readsdraft_markov_block(sharded_accept), which this SGLang no longer has. The Markov commit removes that read.GPU_ARCHS=gfx950set they collect: 513 passed, 7 failed on [Perf] deepseek_v41: A line a flag, the Engram K split and the PV hoist; serve at TP8 sgl-project/sgl-amd-mono#82's head; 514 passed, 6 failed on this PR. The extra failure on both istest_qwen38_flash_next_spec.py::test_the_draft_worker_reads_index_sharing_off_after_the_build;test_model_reasonsthen passes.test_deepseek_v41_kernels.pyat TP2 and TP4, withtest_deepseek_v41_k2i.py,test_deepseek_v41_kc.pyandtest_guard_multi_rank.py: 32 passed (1 h 28 min, 4 GPUs).test_deepseek_v41_kernels.pyat TP2, TP4 and TP8 (8 GPUs): all 9 pass. One run passed seven of them at TP2, TP4 and TP8, and the sampled Markov block at TP2, then hit its time limit in the block at TP4. The block at TP8 and the step glue then ran on their own and pass; the block at TP4 is in the 4-GPU run above.Relation to other PRs
iw_tasks, which the head-weights commit now sizes to 10 CTAs a tile.dsv41: Allocate the launch scratch uncached) makes the same allocation change as the uncached-scratch commit, inDSV41MonoRunner.scratch(). The two conflict there textually.hipMemset. This PR doesn't: every width's scratch is allocated while the spec builds the launches, before the first step runs, as glm53 and glm53_flash allocate theirs.sc1, which [Perf] [MI355X] deepseek_v41: One-tile ug placement on every CTA, uncached launch scratch sgl-project/sgl-amd-mono#72 doesn't.layer.pyandrunner.pywith the one-launch and down-round commits.Checklist
pre-commit run --all-filespasses.test/cover the change (check_layer.pyandtest_deepseek_v41_kc.pyare new;check_k1.py,check_markov.pyandtest_deepseek_v41_k2i.pyare extended)..github/CODEOWNERSthat names its owner. (No new model.)