Repository navigation
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
heslami
requested review from
Fridge003,
Qiaolin-Yu,
Ying1123,
hnyls2002,
ispobock,
merrymercy and
xiezhq-hermann
as code owners
November 3, 2026 20:10
heslami
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
alphabetc1,
celve,
hanming-lu,
huangtingwei9988,
hzh0425,
yizhang2077 and
yuan-luo
as code owners
November 3, 2026 20:10
jhinpan
pushed a commit
to sgl-project/sgl-amd-mono
that referenced
this pull request
Nov 3, 2026
…ows, keys straight into registers, the scoring layout ahead, first digits from the scoring (#86) ## Motivation In long-context decode the indexer launch is what grows with context. It scores every cached key of each row and then selects each row's exact top-2048, and it runs on 18 of a step's 75 MoE layer launches, on layer 0, on the dense layers 1–2 and in every NextN draft step. In AgentX-style serving most decode steps sit at 256K keys and above. There the scoring scan and the selection's first radix digit dominate the launch: - **The scan is bound by per-wave compute, not by bytes in flight.** One 8-page batch a CTA in flight already keeps the key loads streaming; the scoring waves spend the time in address math, FP8 → bf16 conversion, the head-half reduction and LDS traffic. At 4 rows (one request's MTP verify rows) each key was read and converted once per row. - **The scan copied every batch through LDS.** At 4 rows and more, each 8-page batch went through an LDS copy and barriers before any wave scored it. - **The scoring stage started with a chain of loads.** Each scoring CTA first worked out which rows score together (each row's bound, and its page-table entry against the previous row's slot) before it could poll its query. - **The selection's first digit re-read every score.** A selection slice of up to 8,192 keys built its first 64-bin histogram in a local pass, and most scores share the one or two bins of their common exponent, so the LDS atomics serialized. This PR shortens all four. **Every output, every write, every score and every selection stays bitwise equal to the base PR**, at TP8 and TP4, every width, on the bf16 and E4M3 MLA cache, at 8K to 512K keys. The one addition is a new scratch mailbox, `score_hist`. Result on MI355X, the base PR → this PR (TP8, FP8 E4M3 KV, MTP 3-1-4 at the golden accept length 2.99), measured on the same commits in an earlier stack, whose kernel files are these PRs' except the base PR's bounded code warm (Speed Tests): - **8K:** 6.018 → 5.985 ms at one request (−0.5%), 8.479 → 8.443 ms at two (−0.4%). - **128K:** 6.313 → 6.134 ms at one request (−2.8%), 9.372 → 9.173 ms at two (−2.1%). - **512K, kernel-only on this branch:** the indexer launch at 4 / 8 rows 132.5 → 108.7 / 218.0 → 185.6 µs (−18% / −15%), the 75-launch model order 76.8 → 70.8 / 117.5 → 110.2 µs; about 0.52 ms a step less at one request (Speed Tests). ## Modifications Four commits, one per change, `kernel.py` and `layout.py`. The head is bitwise against the base PR. | # | Commit | Change | Numerics | |---|---|---|---| | 1 | two-row scoring from 4 rows | A scoring wave's 16-key group addresses come from its page and first key in scalar math. A row's two head halves reduce across lane rows in three permlane swaps, and two rows share one score store. From 4 rows (was: above 4) a wave scores two rows, reading and converting each key once for both, with the rows' queries staged in MFMA operand order; a run's keys load nontemporal, long rows' runs split evenly, and every run stages its scores in LDS and stores them as pairs after each batch. A lone live row (2 rows: a request and its pad row) is scored by all 8 waves | bitwise | | 2 | key groups straight into registers | From 4 rows each wave of a row set takes every fourth 16-key group of the CTA's run and loads its lane's words straight into registers as MFMA B operands, two groups ahead (four 8-byte loads and the scale a group). The scores are stored as pairs from the registers, the store's mask dropping keys at or past a row's bound. No LDS copy of the run, no batch barriers, no score stage; the direct loads are plain (commit 1's nontemporal loads go with the batch path). A key this launch writes is taken from its row's `index_k8` pairs, for every row of the request in the launch. Below 4 rows the batch path reads all its block ids before its first load and its loads are unconditional | bitwise | | 3 | the scoring layout ahead | Above 4 rows, with the paged indexer, the CTAs past the q_norm ones work out the scoring layout once their front tile is done, from scalar loads of each row's slot, position and page-table row: the row groups, their requests' first rows, and each row's index bound and page-table row. They keep it in LDS ahead of the arena, and the scoring stage reads it there. Up to 4 rows the scoring stage still computes it | bitwise | | 4 | first digits from the scoring | From 4 rows each scoring wave adds the first radix digit (`radix_key >> 26`) of every score it stores to LDS bins per row and slice segment. A request's rows join a group only at the next position, so a run spans at most two of each row's slices. The run publishes each live row's two 64-bin histograms in the new `score_hist` mailbox. A selection slice polls its runs' segments (one run lookup and a compare), publishes their sum as its first-digit histogram before reading its scores, and its first exchange skips the local pass; its first chunk of scores loads while the segments are summed. Above 4 rows the q_norm CTAs, which score nothing, plan the scoring layout at the scoring stage, since their selection slices read it | bitwise; new `score_hist` mailbox | - **Layout:** `score_hist` holds, per row and scoring CTA, two 64-bin histograms as tagged pairs: 1 MB at 8 rows, 512 KB at 4. Above 4 rows the scoring layout takes `(5 + 2 S + 3) // 4 * 4` LDS words ahead of the arena. - **What changes in #74's code:** - The scoring (`score_group`): from 4 rows the paired path replaces the batched one; below 4 rows the batch path stays, with its block ids read first. The new-key patch keeps #74's rule: every new key of the request's rows in the launch, from its first row on. - `score_groups` moves to the launch level and takes the rows' bounds and page-table lookups as arguments, so the front and the scoring stage share it. From 4 rows a row joins its group only at the next position. - The selection (`index_slice`): from 4 rows the first digit's local pass is replaced by the sum of the slice's runs' segments. The later digits, the counts and the placement are unchanged. - **Unchanged:** the shared, NextN shared and tail launches' code objects. ## Accuracy Tests The checks, on MI355X under torchrun on 8 GPUs (TP8) and 4 GPUs (TP4), each against the base PR: - **Dumps, bitwise:** on each tree, for each kind:launches in `shared:3 index:3 model:8 dense:3 embed:3` and each kv in `bf16 fp8`, `torchrun --nproc-per-node 8 test/kernels/glm53/check_layer.py --kind K --launches N --rows 2,4,8 --rowsets p0,p1,v4 --kv KV --dump DIR/<tree>/tp8` (and `--nproc-per-node 4` into `tp4`); then `python3 test/kernels/glm53/check_layer.py --compare DIR/base/tp8 DIR/pr/tp8`. With #76's `peer.py` applied, 0 of 33,552 digests differ at TP8 and 0 of 1,944 at TP4. Without it, only a few 2-row configs differ, which is the known 2-row run-to-run variation that #76 fixes (TP8: bf16 model and fp8 dense, index and embed; TP4: bf16 model and fp8 embed). - **Top-k exact at 8K to 512K, scores and selections bitwise:** `torchrun --nproc-per-node 8 test/kernels/glm53/check_topk.py --pos 8447,131071,262143,524287 --rows 2,4,8 --dump DIR/topk/<tree>/tp8` on the base PR and on this PR (and `--nproc-per-node 4`), then `check_layer.py --compare` of the two digest sets. The selections are exact on both trees, on both caches and every rowset. Scores and selections are bitwise: 0 of 41,984 digests differ at TP8 and 0 of 20,992 at TP4. The `check_layer.py` dump set stays under 9K keys, so this is the check of the multi-batch runs, the even split, the direct loads and the first digits. - **NextN launch = plain launch fed its own x, bitwise:** `torchrun --nproc-per-node 8 test/kernels/glm53/check_nextn.py --rows 2,4,8 --kv fp8 --only entry` (and `--kv bf16`, and `--nproc-per-node 4`). Passes at TP8 and TP4 on both caches. - AIME 2025, 30 problems × 16 samples, with the model card's sampling (temperature 1.0, top_p 0.95, thinking on, up to 163,840 tokens). TP8, FP8 KV, MTP 3-1-4. Generations average about 23K tokens and some reach the cap, so the indexer runs over long KV. | | Accuracy | 95% CI | Accept length | |---|---|---|---| | SGLang without monokernels | 99.17% | 98.1–100 | 3.21 | | sgl-amd-mono before #85 | 99.17% | 98.3–99.8 | 3.23 | | With #85, this PR, sgl-project/sglang#47324 and sgl-project/sglang#48224 | 98.33% | 96.9–99.6 | 3.23 | The stack has 8 wrong answers out of 480, against 4 for each of the others (Fisher p = 0.39; against the build before #85, paired by problem, p = 0.36). Samples are independent draws. The reference served 16 requests at a time, the monokernel builds 2. - Decode drift at long context: KL of decode against SGLang's stock prefill of the same tokens (top-20), and top-1 agreement. TP8, FP8 KV, MTP off; 128 prompts × 256 greedy tokens at each of 8K, 32K, 64K and 128K, cut from the AIME reasoning traces; two servers per build. | | KL decode ‖ prefill | Top-1 agreement | |---|---|---| | SGLang without monokernels | 0.0249 | 95.94% | | sgl-amd-mono before #85 | 0.0199 | 96.34% | | With #85, this PR, sgl-project/sglang#47324 and sgl-project/sglang#48224 | 0.0203 | 96.20% | The stack against the build before #85: KL +1.9% (95% CI −1.0 to +5.0), top-1 −0.14 points, both within the spread between two servers of the same build (up to 4% in KL and 0.25 points in top-1). With MTP on at 64K: KL −5.7% (−12.8 to +2.1). ## Speed Tests and Profiling Setup as #75: image `lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261025` (squashfs `sha256:fb07e0ef78fe`) with SGLang's `python/sglang` overlaid at `5044779f`, AITER 285579b6, FlyDSL 0.4.0, `zai-org/GLM-5.3` at `ba32f8c1`, the e2e test's server flags with `--kv-cache-dtype fp8_e4m3`. TP8, MTP 3-1-4 at golden AL 2.99, FP8 E4M3 KV, cycle in ms: | Context | Requests | the base PR | This PR | Change | | --- | --- | --- | --- | --- | | 8K | 1 / 2 | 6.018 / 8.479 | 5.985 / 8.443 | −0.5% / −0.4% | | 128K | 1 / 2 | 6.313 / 9.372 | 6.134 / 9.173 | −2.8% / −2.1% | - Measured on the same commits in an earlier stack (the base PR's head as `1b93ec8`, this PR's as `522a7ed`): golden AL 2.99, one server per arm, the cycle at two requests the last-prefilled request's. The gain grows with context: the scoring scan and the first digit scale with the keys a row reads. **Kernel-only, on this branch** (`time_layers.py --kv fp8 --similar 1.0` under `torchrun --nproc-per-node 8`, E4M3 cache, µs a launch, the mean of 2 rounds in alternating order, each round the median of 7 trials of 50 replays; the rounds agree within 0.3 µs). `model` is the mean over the 75 MoE launches in model order, run one width a process: | Launch | Rows | 8K | 128K | 512K | | --- | --- | --- | --- | --- | | index | 2 | | | 96.85 → 90.97 (−5.9) | | index | 4 | 79.56 → 77.94 (−1.6) | 92.09 → 85.59 (−6.5) | 132.48 → 108.70 (−23.8) | | index | 8 | 111.78 → 108.41 (−3.4) | 147.81 → 140.51 (−7.3) | 217.98 → 185.56 (−32.4) | | dense | 2 | | | 89.83 → 83.41 (−6.4) | | dense | 4 | 63.08 → 62.09 (−1.0) | 75.53 → 69.66 (−5.9) | 115.72 → 92.02 (−23.7) | | dense | 8 | 79.52 → 75.80 (−3.7) | 113.53 → 105.53 (−8.0) | 181.71 → 149.72 (−32.0) | | model | 4 | 63.88 → 63.57 (−0.3) | 67.06 → 65.55 (−1.5) | 76.84 → 70.81 (−6.0) | | model | 8 | 92.10 → 91.80 (−0.3) | 101.22 → 99.67 (−1.6) | 117.50 → 110.19 (−7.3) | - From these, the 75 MoE launches and the 3 dense ones of a step take about 0.52 ms less at 512K and one request (4 rows), 0.13 ms at 128K and 0.03 ms at 8K. - Commands: `torchrun --nproc-per-node 8 test/kernels/glm53/time_layers.py --kind {index,dense} --rows 2,4,8 --pos 524287 --kv fp8 --similar 1.0`, `--kind model --rows 4` and `--rows 8` (one width a process), and the same at `--pos 131071` and `--pos 8447` with `--rows 4,8`. **AgentX** (InferenceX `inferencex-agentx-mvp`, seed 42, traces in InferenceX order) with AMD's InferenceX recipe: TP8, FP8 KV, MTP 3-1-4 at simulated accept length 2.99, one hour per point. The build before #85 and the stack ran back to back on the same node. | Concurrency | | P90 interactivity (tok/s/user) | Throughput (tok/s/GPU) | ITL p50 (ms) | TTFT p50 (s) | | --- | --- | --- | --- | --- | --- | | 1 | SGLang without monokernels | 181.5 | 1,944 | 4.808 | 0.351 | | | sgl-amd-mono before #85 | 291.1 | 2,890 | 2.484 | 0.372 | | | With #85 and this PR | 345.3 (+18.6%) | 3,135 (+8.5%) | 2.207 (−11.2%) | 0.375 | | 2 | sgl-amd-mono before #85 | 212.2 | 2,911 | 2.491 | 0.318 | | | With #85 and this PR | 282.9 (+33.3%) | 3,160 (+8.5%) | 2.219 (−10.9%) | 0.318 | | 4 | sgl-amd-mono before #85 | 167.1 | 4,069 | 2.706 | 0.256 | | | With #85 and this PR | 181.0 (+8.4%) | 4,369 (+7.4%) | 2.326 (−14.1%) | 0.263 | - Paired by request (same conversation and turn), ITL falls 11.6% at concurrency 1 (95% CI 9.0–12.3), 10.9% at 2 (8.9–11.6) and 10.0% at 4 (9.1–11.9); TTFT is unchanged at every point. Throughput is a fixed one-hour window, so a faster server gets further into the traces. - SGLang without monokernels ran at concurrency 1 only, on the same node. Paired by request, ITL is 49.0% lower before #85 (95% CI 48.1–50.4) and 54.7% lower with the stack (54.4–54.8). ### Registers and lint From the code objects of the spec's builds (`stop=True`): `/opt/rocm/llvm/bin/llvm-readelf --notes <key>.co`, fields `.vgpr_count`, `.vgpr_spill_count`, `.sgpr_count`, `.sgpr_spill_count` and `.private_segment_fixed_size`. Two figures a cell are bf16 / E4M3 MLA cache. **the base PR → this PR:** - TP8 index, dense and NextN index at 8 rows lose their last VGPR spills (1–6 → 0). - TP4 at 8 rows: index 10 → 1 / 2, dense and embed 3–5 → 1. - Two builds gain one: TP4 index at 4 rows 0 → 1 (8 B), and TP8 NextN index at 4 rows, bf16 cache, 0 → 1. - The shared, NextN shared and tail objects are unchanged. **Lint:** #84's `test/kernels/test_kernarg_lint.py` (the GLM-5.3 index and shared launches at TP4 and TP8, widths 2, 4 and 8, both caches): 24 / 24 pass. ## Reviewer notes - **New keys, as #74 has them.** A key this launch writes is scored from its row's `index_k8` pairs for every row of the request in the launch from its first row on (`c0`), not only the rows of the scoring group. That matters when one request has more rows than a group (an 8-row draft chain: two groups of 4). The direct path keeps the batch path's rule, over the S rows (`new_bounds`), and the scoring layout of commit 3 carries the requests' first rows (`firsts`) for it. `check_topk.py --rowsets chain` at 8 rows covers it. - **The join rule.** From 4 rows a row joins its request's group only at the next position (`bounds[r] == bounds[r - 1] + 1`), so a run spans at most two of each row's selection slices. A request's rows in an MTP verify step or a draft chain are always consecutive; other rows start a new group, which keeps `firsts` for the new keys. - **`score_hist`.** A scoring CTA publishes, per live row of its group, the 64 first-digit counts of each of its two slice segments as tagged pairs. A slice sums the segments of the runs over its key range; the runs' layout it recomputes is the scoring's own (`score_groups`, `select_slices`, the even split). - **Two scoring paths remain.** Below 4 rows the batch path (one row a wave, LDS batches) stays; from 4 rows the paired direct path. A runtime switch by key count was not taken: both paths in one object grew it and cost at short context. - **The scoring layout ahead (commit 3)** applies above 4 rows only; at 4 rows it does not pay. It assumes the atom KV cache layout (asserted), as the paged indexer does. - **Commit 4's trade.** It moves the first digit's counting into the scoring CTAs: a small cost per scoring CTA at short context (8K), and the slice's local pass saved at long context. Commit 4 can be dropped alone. - `kernel.py` grows by 317 lines (+587 / −270); `layout.py` +3. ## Checklist - [ ] `pre-commit run --all-files` passes. (ruff, ruff-format, isort and codespell are clean on the touched files) - [x] Tests under `test/` cover the change. (`check_topk.py` and the kernel tests) - [x] The SGLang commit or PR this was tested against is named above. - [ ] Accuracy and speed results are attached when kernel or forward code changes. (Speed: serving at 8K and 128K, above.) - [ ] A new model adds one line to `.github/CODEOWNERS` that names its owner. (No new model.) --------- Co-authored-by: Hassan Eslami <hassan@marlowe.ai>
added 7 commits
November 3, 2026 14:19
draft_topk1_postprocess reduced each row in a split pass and a finalize launch. Each split now folds its (value, lowest index) into a per-row 64-bit key with an atomic max, and the last split of a row to arrive writes the token, the draft token column and the position, and resets the row's state: one launch per top-1, three a GLM-5.3 MTP cycle fewer. Ties still resolve to the lowest index and NaN still reads as -1e30. draft_topk1_argmax_only no longer advances a scratch positions buffer (positions=None).
A greedy EAGLE verify with topk 1 filled the predicts and accept index, reduced the logits rows with torch.argmax and walked the chains in verify_tree_greedy, then broadcast the three results on HIP. greedy_chain_verify does the reduction and the walk in one launch: each CTA folds a vocabulary split's (value, lowest index) into its row's key with an atomic max, and the last CTA to arrive writes the target tokens and walks every request's chain, writing every predict and accept index entry. Its results are views of one buffer, which the HIP greedy branch broadcasts once. Outputs equal row_argmax then verify_tree_greedy's (kernel test).
… relayed lengths and the draft extend layout A top-1 chain's verify ran three small ops for values the launches around them already hold: - The accept lengths (num_correct_drafts + 1): greedy_chain_verify and simulated_accept now also write them (accept_lens=). - The overlap scheduler's relay of the new lengths (FutureMap.publish's index_put): the commit launch writes them into the relay (prepare_verify_commit_outputs(relay=...)). The EAGLE worker offers FutureRelayBuffers for the FutureMap to adopt, as DSpark's verify does, and marks the lengths published. - The draft extend's layout (prepare_draft_extend_layout): build_chain_tree writes it over the same window (with_extend_layout=True), carried on the verify input to prepare_for_draft_extend. Three fewer launches a decode step; every output equals the op it replaces.
…olumn 0 With more than one draft step, the chain's column 0 (the step's input tokens) was copied in before the steps. The first step's top-1 launch now writes it (draft_topk1_postprocess(first_tokens=...)), one copy fewer a decode step.
…-planned inputs The verify input's tokens and positions are loaded into the graph's buffers when the verify batch is planned, which run_eagle_verify starts only after the draft has written them. The replay then copied the same tensors again. Like DSpark's pre-planned verify (SGLANG_OPT_DSV41_DSPARK_SKIP_PREFILLED_INPUTS), EAGLE's skips that copy when the replayed batch holds the planned tensors: two launches a step.
The draft decode graph ended with positions.sub_ to undo the steps' advance of its positions buffer. Every replay loads the positions again before it runs, so the restore is one launch a step for nothing.
Two kinds of device copies a decode step that read or write nothing new: - Each step of EAGLE's draft graph ran the eager runner's input load, which copied the batch's tensors into the eager buffers, though the draft graph's steps read its own capture buffers at fixed addresses. The draft graph's body now runs its steps under eager_inputs_in_place(), so their load passes the tensors through, as SGLANG_EAGER_INPUT_NO_COPY does. Every other eager forward, captured or not, copies as before. - The draft graph runners copied the hidden states and the DSA seed with copy_() apart from the grouped small copy. Fields up to MAX_FIELD_NUMEL elements (64K, raised from 8K) now ride in the grouped launch; larger ones keep the DMA copy.
added 3 commits
November 3, 2026 14:30
…s KV-length copies only when allocating Two device ops a decode step that no one reads: - FutureMap.publish records an event every step, but with the GPU-only relayed lengths (HIP, no backend that needs the CPU lengths) the next step gathers them in forward-stream order and never waits on it. Only a mixed chunk's tails and the confidence relay do, so the scheduler keeps the event for those alone (FutureMap.record_publish). - eagle_prepare_for_decode copies the current and next KV lengths to the device every step, though only an allocation reads them; with whole pages most steps allocate nothing. The copies now go with the allocation.
SGLANG_SIMULATE_ACC_LEN benchmarks on HIP built the simulated accept index with about a dozen torch ops a verify step (fill, arange, index writes, copies), because the fused simulated_accept kernel was gated to CUDA. It is Triton, so it now runs on HIP too. A kernel test checks it against those torch ops.
FutureMap.stash wrote each relayed buffer with its own index_put (bonus tokens with a dtype copy, top-k values and indices, hidden states, the DSA top-k seed): six launches every speculative step. scatter_spec_extras, the mirror of gather_spec_extras, writes all of them in one launch, casting to each buffer's dtype; off the GPU it keeps the index_puts. The resolve's gather also takes the DSA top-k seed, one indexing launch fewer. The stash is also the overlap relay of a non-speculative decode step, which stashes only its tokens: the default decode path takes the one-launch write too, one index_put becoming one kernel there.
heslami
force-pushed
the
up/spec-eagle-relay-stash
branch
from
November 3, 2026 23:13
fccdf39 to
d0f4bd0
Compare
4 of 5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #47324; only the last commit is new here.
Motivation
With the overlap scheduler, every speculative decode step ends with
FutureMap.stash, which hands the step's outputs to the next step's resolve. It wrote each relayed buffer with its ownindex_put: bonus tokens with a dtype copy, top-k values and indices, hidden states, draft probabilities and the DSA seed. At one request each of those launches sits on the critical path between two steps.Modifications
scatter_spec_extras(gather_spec_extras.py), the mirror ofgather_spec_extras, writes every relayed buffer in one launch, casting each to its buffer's dtype. It shares the gather's row kernel. Off the GPU it keeps theindex_puts.FutureMap.stash(overlap_utils.py) calls it once. Buffers the step does not relay are skipped, as before. A source with fewer rows than indices raises, as theindex_putdid.Six fewer launches per speculative step. On by default; no new flag.
stashis also the relay of every non-speculative overlap decode step, which hands over only the next tokens. There it now writes them with the same kernel, one launch for one.Tests:
test_gather_spec_extras.py:index_puts it replaces: mixed dtypes, a float downcast, the seed, non-contiguous rows, bonus tokens alone (the non-speculative relay);test_dspark_step_inputs_once.py: the CPU stash path.Related:
stash. This PR reuses the gather's row kernel and keepsstasha single call on every device.Accuracy Tests
The build for these runs also carried an opt-in change that is not part of this PR. It was off, and when off it adds no device work, so these runs exercise this PR's stash and resolve.
The stack has 8 wrong answers out of 480, against 4 for each of the others (Fisher p = 0.39; against the build before #85, paired by problem, p = 0.36). Samples are independent draws. The reference served 16 requests at a time, the monokernel builds 2.
The stack against the build before #85: KL +1.9% (95% CI −1.0 to +5.0), top-1 −0.14 points, both within the spread between two servers of the same build (up to 4% in KL and 0.25 points in top-1). With MTP on at 64K: KL −5.7% (−12.8 to +2.1).
test_gather_spec_extras(15 passed, 36 subtests),test_disaggregation_wire(63 passed), and the FutureMap relay teststest_next_verify_prep,test_dspark_step_inputs_once,test_spec_registry,test_spec_cpu_overlap_constraint,test_schedule_batch_req_pool_indices,test_auxiliary_output,test_fork,test_pd_eagle_draft_proposal(284 passed).Speed Tests and Profiling
Setup:
SGLANG_SIMULATE_ACC_LEN).--enable-amd-monokernel, sgl-project/sgl-amd-mono with Can the stop tokens be retained? #85 and OOM Error on A40 GPU [Jupyter notebook] #86). The forward runs in the GLM-5.3 monokernels, so the launches between them are a larger share of the cycle than on the stock path.AgentX (InferenceX
inferencex-agentx-mvp, seed 42, traces in InferenceX order) with AMD's InferenceX recipe: TP8, FP8 KV, MTP 3-1-4 at a forced accept length of 2.99, concurrency 1, one hour per arm, all arms on the same node. The SGLang base is main + #47101 + #47102; with the merged sgl-amd-mono plugin those two change nothing on these steps (decode cycle +0.24%, 95% CI −0.32 to +0.63).Checklist
CI States
Latest PR Test (Base): ❌ Run #41737463344
Latest PR Test (Extra): ❌ Run #41737463349
Latest PR Test (AMD ROCm 10): ❌ Run #41737463394
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: