feat(moe_wna16): 2-bit experts, expert streaming, N-contiguous GEMV - #37796
Draft
HaberstrohSystems wants to merge 2 commits into
Draft
HaberstrohSystems wants to merge 2 commits into
HaberstrohSystems wants to merge 2 commits into
Conversation
## Motivation
Qwen3.8-Flash-Next (Qwen4-Exp: 176B total / 6B active, 48 layers =
36 GatedDeltaNet + 12 Qwen sparse attention, 512 experts top-10, a 51 GB
PLE n-gram embedding table) only fits a 24 GB card with `--cpu-offload-gb`.
On the base commit that path does not work: weight loading fails on a
device mismatch in `GemmaRMSNorm`, `functional_call` rejects the tied
`A_log` tensors, the GDN convolution reads a stale `.view()` of a
parameter whose `.data` the offloader has replaced (GDN output exactly
zero, NaN logits), checkpoints saved through transformers carry the layer
type `qwen_sparse_attention`, the fused `in_proj_ba` lands at the wrong bit
width because `packed_modules_mapping` never reaches the quant config, and
the PLE table is allocated in full (51.2 GB as fp8, 51,200,245,760 B;
102.4 GB if materialised as bf16, derived as 2 x 51,200,245,760 B) before
any offload can act. These are the nine findings of `WRITEUP.md` section 3
("Nine findings in SGLang"), with finding 8 (the silent NaN) analysed in
section 4 and finding 9 (26 GB of PCIe traffic per token) in section 5.
The reference machine is one RTX PRO 4000 Blackwell (24 GB, sm_120) with
32 GB host RAM, serving a 2.572 bpw AutoRound checkpoint (model card,
"Quantization").
Measurements and logs: https://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglang
(`docs/CAMPAIGN.md`, an append-only log cited below by its dated entries;
`docs/ELASTIC_MEMORY.md`; `docs/WRITEUP.md`; `docs/DECODE_PERF_PLAN.md`;
`tools/bench_speed.py`) and the model card
https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang
(cited by section name).
## Modifications
Correctness under `--cpu-offload-gb` (general, not Qwen4-specific):
* `layers/layernorm.py`: `GemmaRMSNorm` re-derives `gemma_weight` on the
device of the parameter (`_weight_loader`) and of the input
(`_gemma_weight_for`); the non-persistent buffer was left on the GPU
while the parameter went to the CPU.
* `utils/offloader.py`: `functional_call(..., tie_weights=False)` in both
hook variants; `_CpuParamOffloader` keeps a hard reference to the pinned
host storage (`cpu_data`). With `SGLANG_MOE_EXPERT_STREAM=1` only expert
tensors (`w13_qweight`, `w2_qweight`, `w13_scales`, `w2_scales`,
`w13_qzeros`, `w2_qzeros`) are offloaded, and the forward hook is not
installed when every offloaded parameter of a module is a streamed expert
parameter (the hook would reparametrize the module with tensors that are
already on the device). The streamer itself is part 2 of this series.
* `layers/radix_linear_attention.py`, `models/qwen3_5.py`: the GDN module
passes its `nn.Conv1d` instead of a `.view()` of its weight;
`RadixLinearAttention.conv_weights` is a property that builds the 2D view
from the current parameter on every access. Tensors and tuples pass
through unchanged, so KDA / ShortConv / Lightning are not affected.
Qwen4-Exp specific:
* `models/qwen4_exp.py`: accept the layer type `qwen_sparse_attention`
(transformers renames `full_attention` on load; loading otherwise fails
with `KeyError: 'qwen_sparse_attention'`). The matching
`layers_block_type` alias in `configs/qwen4_exp.py` is already on
`qwen4-main-squashed` since sgl-project#36772 and is not repeated here;
`Qwen4ExpForConditionalGeneration.__init__` calls
`quant_config.update_packed_modules_mapping(...)` like `deepseek_v2.py`
does for Quark.
* `server_args.py`: `Qwen4ExpForConditionalGeneration` in
`LANGUAGE_MODEL_ONLY_ARCHITECTURES` (the class inherits the
`language_model_only` implementation from `Qwen3VLForConditionalGeneration`;
only the entry was missing). Saves the vision tower (0.84 GiB) and the
multimodal reservation in the KV budget for text-only serving.
* `models/qwen4_exp.py`: `Qwen4ExpMmapEmbedding` serves the PLE table from
a memory-mapped file (`SGLANG_QWEN4_PLE_MMAP=<dir>` with `ple.f8_e4m3.bin`
and `ple.json`; `ple.json` carries `file`, `rows`, `dim`, `dtype` and
`weight_scale`, the file is `rows x dim x itemsize` bytes and is checked).
The `VocabParallelEmbedding` is created on the `meta` device in that mode
so nothing is allocated first. Decode fetches its rows with parallel
`os.pread` (16 threads, GIL released; the numpy fancy-index into the
memmap held the GIL for the whole cold row fetch, measured at 2.56-3.09
ms per token; `DECODE_PERF_PLAN.md`, "2. Parallelise the PLE row
fetch"), bulk gathers go through the memmap. `MADV_RANDOM` on the
mapping and `POSIX_FADV_RANDOM` on the fd stop the page-cache
read-around per 160-byte row (`CAMPAIGN.md` 2026-09-02 14:12);
`POSIX_FADV_DONTNEED` after bulk gathers of >= 512 ids drops the pages
again (random text otherwise churns ~1M page-cache pages per long
prompt; `ELASTIC_MEMORY.md`, "What the reference host taught").
* `models/qwen4_exp.py`: `Qwen4ExpMmapEmbedding.forward` is wrapped with
`eager_on_graph(True)`, so decode runs under
`--cuda-graph-backend-decode breakable` with the PLE lookup as the one
eager break.
* `model_executor/runner_backend/breakable_cuda_graph_backend.py`: the four
output-structure helpers understand the `LogitsProcessorOutput`
dataclass; capture died with "Unsupported BCG output type" for any graph
body that returns it.
* `layers/attention/qsa/qsa_indexer.py`: `_qsa_ensure_rope` hoists the
`positions.max().item()` device sync (once per QSA layer, 12 per token;
`DECODE_PERF_PLAN.md`, "5. Two micro-fixes") and pre-sizes the rotary
cos/sin cache from `context_length`.
* `mem_cache/memory_pool.py`: fp8 write path in both `set_kv_buffer`
variants skips the no-op `div_` by a unit scale and saturates to +-448
before the e4m3 cast (the fp32/bf16 -> e4m3fn cast returns NaN beyond
the range). Consumed by the fp8 read path of part 4.
No new server flags; the two switches are `SGLANG_MOE_EXPERT_STREAM` and
`SGLANG_QWEN4_PLE_MMAP`. No tests are added in this part.
## Accuracy Tests
Exactness oracle: teacher-forced logprobs on a fixed 10k-token prompt,
mean and max |dlogprob| per token against the previous configuration;
noise floor mean 0.002, threshold for host-side changes mean <= 0.01,
max <= 0.5 (model card, "Serving fidelity"; `CAMPAIGN.md` 2026-09-02
11:25).
* Host fixes (offloader hook skip, indexer hoist, PLE pread; together with
the streamer items of part 2): oracle max 0.078 / mean 0.0019, equivalent
(`CAMPAIGN.md` 2026-09-02 09:30).
* Breakable decode graphs + `LogitsProcessorOutput` support: oracle max
0.079 / mean 0.0022, equivalent (`CAMPAIGN.md` 2026-09-02 11:48).
* PLE page-cache fixes involve no numerics; the 248k-token needle test on
the same server retrieved 5/5 codes after the page-drop fix
(`CAMPAIGN.md` 2026-09-02 20:00).
* fp8 KV write path: a 1-token prompt still crashes in the QSA indexer
prefill in `fp8_e4m3` mode; >= 101-token prompts work (`CAMPAIGN.md`
2026-09-02 15:26). Open issue, documented here.
## Speed Tests and Profiling
Streaming bench (`tools/bench_speed.py`, 200 streamed tokens, single
request, ~10k context; model card, "Performance"):
* host fixes: decode 15.4 -> 19.4 tok/s (`CAMPAIGN.md` 2026-09-02 09:30;
includes the streamer items of part 2);
* breakable decode graphs: decode 21.8 -> 40.0 tok/s at prefill 2249
tok/s, graph capture 3.97 s / 0.12 GB (`CAMPAIGN.md` 2026-09-02 11:33
and 11:48);
* fp8 write path: `fp8_e4m3` mode at parity with bf16, decode 55.9-57.2 /
prefill 2303-2339 tok/s (`CAMPAIGN.md` 2026-09-02 15:12);
* PLE read-around: before `MADV_RANDOM` / `POSIX_FADV_RANDOM` long
prompts (~55k tokens) drove host memory pressure past systemd-oomd
(`CAMPAIGN.md` 2026-09-02 14:12); the 248k random-text needle run
survived only after `POSIX_FADV_DONTNEED` (2026-09-02 19:37 and 20:00).
Part 1/5 of the Qwen3.8-Flash-Next 24 GB serving series. The series is
based on 73a2552, the first commit of the branch qwen4-main-squashed
(PR sgl-project#36497); there is no part 0.
## Motivation `moe_wna16` accepts 4 and 8 bits only, so a symmetric 2-bit AutoRound / GPTQ MoE checkpoint cannot be loaded at all. For Qwen3.8-Flash-Next (512 experts, top-10, `moe_intermediate_size` 640) the 2-bit experts are what makes the model fit next to a 24 GB card: the served checkpoint is 2.572 bpw (model card, "Quantization"). Two more things are needed once the experts are offloaded: `--cpu-offload-gb` copies a whole module's `state_dict` to the device on every forward (measured ~26 GB per token at 2.3 tok/s, while 10 x 48 x 1.31 MB = 0.63 GB of expert rows are actually needed; `WRITEUP.md` section 5), and the byte-row `[E, N, K/4]` layout reads 128-byte chunks at 640-byte stride, which limits an in-place decode GEMV to 73 GB/s from device memory and 5 GB/s from pinned host memory (`CAMPAIGN.md` 2026-09-02 08:29-08:31). Measurements and logs: https://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglang (`docs/CAMPAIGN.md`, an append-only log cited below by its dated entries; `docs/WRITEUP.md`; `tools/bench_speed.py`) and the model card https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang (cited by section name). The reference machine is one RTX PRO 4000 Blackwell (24 GB, sm_120) with 32 GB host RAM. ## Modifications 2-bit path (general for symmetric 2-bit `moe_wna16` checkpoints): * `kernels/ops/moe/fused_moe_triton_kernels.py`: 2-bit unpack in `fused_moe_kernel_gptq_awq` as the generalization of the 4-bit branch (`offs_k // 4`, shift `(offs_k % 4) * 2`, mask `0x3`, zero point 2). New `fused_moe_kernel_gptq_awq_word`: the same kernel reading the N-contiguous int32-word layout (one coalesced 128-byte line per warp load instead of 32 distinct lines). The layout is derived from the tensor dtype (int2 + `int32` = word layout, int2 + `uint8` = byte layout), so nothing crosses the custom-op schema. * `moe_runner/triton.py`, `triton_utils/fused_moe.py`, `triton_utils/fused_moe_triton_config.py`: `use_int2_w2a16` plumbing, shape handling for the word layout, config dtype name `int2_w2a16` so a tuned int4 config is never picked up by accident (no int2 config files ship; the tuned configs used for the measurements below live outside the tree, `SGLANG_MOE_CONFIG_DIR`). * `layers/quantization/moe_wna16.py`: 2-bit loader; asymmetric `qzeros` raise `NotImplementedError` by design. `process_weights_after_loading` re-lays every 2-bit expert tensor once from `[E, N, K/4]` uint8 to `[E, K/16, N]` int32 and scales to `[E, K/128, N]` (same bytes; `SGLANG_MOE_NCONTIG=0` keeps the byte layout); pinned host tensors are re-pinned after the copy, which transiently doubles one layer's host footprint. * `layers/moe/expert_gemv.py` (new): batch-1 int2 GEMV (`moe_gemv_int2_tab`, used for M <= 16 in `apply()`, `SGLANG_MOE_GEMV=0` disables) that reads experts in place through int64 address tables; an entry may point into device or pinned host memory, the kernel indexes with the original expert ids. `to_word_ncontig`, `make_tables`. * `layers/moe/expert_stream.py` (new, `SGLANG_MOE_EXPERT_STREAM=1`): gathers exactly the routed experts of a forward into one shared, shape-keyed staging buffer and renumbers the top-k ids onto it; fully GPU-resident layers skip the gather and run on the original ids; `arange`/cast tensors are memoized (no per-token allocations). * `layers/quantization/moe_wna16.py`: deferred placement pass at the last layer (`SGLANG_MOE_PLACEMENT=<expert_freq.pt>` with `{"mass": [L, E]}`, `SGLANG_MOE_PLACEMENT_S`, default 184): the hottest S experts of every layer are resident on the GPU, cold rows of GPU layers take the pinned slots that host layers' hot rows vacate, memory-neutral on both sides. With `SGLANG_MOE_ELASTIC=1` the pass hands the layers to `ExpertElastic` (part 3) instead. * `test/registered/unit/layers/quantization/test_moe_wna16_int2.py` (new, `register_cuda_ci`, synthetic data, no checkpoint): the exact 2-bit unpack expressions of `fused_moe_kernel_gptq_awq` against a torch reference (bit-exact, including the real expert dims 2560 x 640 g128); `to_word_ncontig` byte -> word layout round trip (bit-exact and invertible, CPU); `fused_moe(use_int2_w2a16=True)` on the byte layout (symmetric and with per-group `qzeros`) and on the word layout against a torch MoE reference on the dequantized weights; `moe_gemv_int2_tab` through `make_tables` addresses against an fp32 reference (w13 and w2 forms, fp16 and bf16 scales, N not a multiple of the block; a bad block is rejected). * `test/manual/test_triton_moe_wna16.py`: `w2a16` / `w2a16b2` cases and the 2-bit packing. This stays a manual test: its 8-bit path fails the tolerance check with the unmodified upstream kernel on the reference machine (129 of 144 cases in the 8-bit / group-128 subset, identical on the unpatched tree), so it is not a usable gate; the 2-bit path is covered by `test_moe_wna16_int2.py` instead. ## Accuracy Tests * `test_moe_wna16_int2.py` (7 tests, 34 subtests, pass; run as `PYTHONPATH=python python test/registered/unit/layers/quantization/test_moe_wna16_int2.py` with ninja and an nvcc that supports the GPU architecture on PATH): packing order, the 2-bit unpack expressions and the word-layout round trip are bit-exact; `fused_moe` on both layouts and the GEMV agree with the dequantized references within atol 1e-3 / rtol 2e-2. The original A/B of the same kernel path on real shapes (dequantized bf16 vs packed 2-bit through the same MoE machinery) was bit-identical on small shapes and 1e-5 relative on large ones including e=64, n=640, k=2560, g128 (`WRITEUP.md` section 8). * Word layout and GEMV: A/B through the patched `invoke_fused_moe_kernel` at 64/64/32, 16/16/128, 16/32/128 identical (2e-5) (`CAMPAIGN.md` 2026-09-02 11:04); per-layer dump A/B with identical inputs 5.1e-5 (M=6) / 1.7e-4 (M=1) relative, i.e. bf16 output rounding from a different fp32 accumulation order, verdict numerically equivalent (2026-09-02 11:25). End-to-end oracle on the 10k prompt (teacher-forced logprobs, model card, "Serving fidelity") max 0.275 / mean 0.0125, inside the kernel-class band (2026-09-02 11:30). * Placement is exact by construction (rows move, values do not): oracle max 0.057 / mean 0.0013 (`CAMPAIGN.md` 2026-09-02 12:10). * Tuned int2 configs: oracle max 0.086 / mean 0.0018 (2026-09-02 09:13). ## Speed Tests and Profiling Streaming bench, single request, ~10k context (`tools/bench_speed.py`; model card, "Performance"; `CAMPAIGN.md`): * PCIe traffic per decoded token 26 GB -> 0.31 GB with expert-only offload and the streamer (`WRITEUP.md` section 5; first working path 2.24 tok/s -> 15.5 tok/s, sections 4 and 5). * Tuned int2 configs: decode 13.4 -> 15.2 tok/s (2026-09-02 09:13). * N-contiguous GEMV micro-benchmark on real layer-5 tensors: 320 GB/s from device memory, 51 GB/s (PCIe line rate) from pinned host memory, 2.1 ms/token all-device; the byte-shuffled table variant reached 157 / 14 GB/s and was rejected (2026-09-02 08:29-08:31 and 09:45). * N-contiguous layout + GEMV end to end: decode 19.4 -> 21.8 tok/s, prefill 1443 -> 2249 tok/s (2026-09-02 11:30). * Routing-mass placement: the top 171 of 512 experts per layer cover 82 % of routing mass on a 2496-token, 3-domain probe (2026-09-02 09:48-09:53); decode 40.0 -> 48.4 tok/s, prefill 2249 -> 2334 tok/s (2026-09-02 12:10). Part 2/5 of the Qwen3.8-Flash-Next 24 GB serving series; stacked on part 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft: stacked on #37793 (part 1); parts precede it in the series.
Part 2/5 of the Qwen3.8-Flash-Next 24 GB serving series (RFC issue: #37792). Patch:
upstream/series-q4head/0002-feat-moe_wna16-2-bit-experts-expert-streaming-N-cont.patchin the companion repository (9 files, +1,583 / -33). Nothing in this part is Qwen4-specific.
Motivation
moe_wna16accepts 4 and 8 bits only, so a symmetric 2-bit AutoRound / GPTQ MoE checkpointcannot be loaded at all. For Qwen3.8-Flash-Next (512 experts, top-10,
moe_intermediate_size640) the 2-bit experts are what makes the model fit next to a 24 GB card: the served checkpoint
is 2.572 bpw (model card, "Quantization"). Two more things are needed once the experts are
offloaded:
--cpu-offload-gbcopies a whole module'sstate_dictto the device on every forward(measured ~26 GB per token at 2.3 tok/s, while 10 x 48 x 1.31 MB = 0.63 GB of expert rows are
actually needed;
docs/WRITEUP.mdsection 5), and the byte-row[E, N, K/4]layout reads128-byte chunks at 640-byte stride, which limits an in-place decode GEMV to 73 GB/s from device
memory and 5 GB/s from pinned host memory (CAMPAIGN 2026-09-02 08:29-08:31).
Sources: dated entries of
docs/CAMPAIGN.mdanddocs/WRITEUP.mdinhttps://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglang and the model card
https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang.
Reference machine: one RTX PRO 4000 Blackwell (24 GB, sm_120) with 32 GB host RAM.
Modifications
kernels/ops/moe/fused_moe_triton_kernels.py: 2-bit unpack infused_moe_kernel_gptq_awqasthe generalization of the 4-bit branch (
offs_k // 4, shift(offs_k % 4) * 2, mask0x3,zero point 2). New
fused_moe_kernel_gptq_awq_word: the same kernel reading the N-contiguousint32-word layout (one coalesced 128-byte line per warp load instead of 32 distinct lines). The
layout is derived from the tensor dtype (int2 +
int32= word layout, int2 +uint8= bytelayout), so nothing crosses the custom-op schema. Reviewer's choice: keep the second Triton
function or fold it into a
constexprbranch of the existing kernel.moe_runner/triton.py,triton_utils/fused_moe.py,triton_utils/fused_moe_triton_config.py:use_int2_w2a16plumbing, shape handling for the word layout, config dtype nameint2_w2a16so a tuned int4 config is never picked up by accident. No int2 config files ship; the tuned
configs used below (for
E={10,512}, N=160on the RTX PRO 4000 Blackwell) live in thecompanion repository under
assets/moe_configsand are selected throughSGLANG_MOE_CONFIG_DIR.layers/quantization/moe_wna16.py: 2-bit loader; asymmetricqzerosraiseNotImplementedErrorby design.process_weights_after_loadingre-lays every 2-bit experttensor once from
[E, N, K/4]uint8 to[E, K/16, N]int32 and scales to[E, K/128, N](same bytes;
SGLANG_MOE_NCONTIG=0keeps the byte layout); pinned host tensors are re-pinnedafter the copy, which transiently doubles one layer's host footprint. Deferred placement pass
at the last layer (
SGLANG_MOE_PLACEMENT=<expert_freq.pt>with{"mass": [L, E]},SGLANG_MOE_PLACEMENT_S, default 184): the hottest S experts of every layer are resident onthe GPU, cold rows of GPU layers take the pinned slots that host layers' hot rows vacate,
memory-neutral on both sides. With
SGLANG_MOE_ELASTIC=1the pass hands the layers toExpertElastic(part 3).layers/moe/expert_gemv.py(new): batch-1 int2 GEMV (moe_gemv_int2_tab, used for M <= 16in
apply(),SGLANG_MOE_GEMV=0disables) that reads experts in place through int64 addresstables; an entry may point into device or pinned host memory, the kernel indexes with the
original expert ids.
to_word_ncontig,make_tables.layers/moe/expert_stream.py(new,SGLANG_MOE_EXPERT_STREAM=1): gathers exactly the routedexperts of a forward into one shared, shape-keyed staging buffer and renumbers the top-k ids
onto it; fully GPU-resident layers skip the gather;
arange/cast tensors are memoized.test/registered/unit/layers/quantization/test_moe_wna16_int2.py(new,register_cuda_ci,est_time=60, stagebase-b, runner1-gpu-small; synthetic data, no checkpoint): the exact2-bit unpack expressions against a torch reference (bit-exact, including the real expert dims
2560 x 640 g128);
to_word_ncontigbyte -> word round trip (bit-exact and invertible, CPU);fused_moe(use_int2_w2a16=True)on the byte layout (symmetric and with per-groupqzeros)and on the word layout against a torch MoE reference on dequantized weights;
moe_gemv_int2_tabthroughmake_tablesaddresses against an fp32 reference (w13 and w2forms, fp16 and bf16 scales, N not a multiple of the block; a bad block is rejected).
test/manual/test_triton_moe_wna16.py:w2a16/w2a16b2cases and the 2-bit packing. Thisstays a manual test: its 8-bit path fails the tolerance check with the unmodified upstream
kernel on the reference machine (129 of 144 cases in the 8-bit / group-128 subset, identical on
the unpatched tree), so it is not a usable gate.
Accuracy Tests
test_moe_wna16_int2.py: 7 tests, 34 subtests, pass (packing order, unpack expressions andthe word-layout round trip bit-exact;
fused_moeon both layouts and the GEMV withinatol 1e-3 / rtol 2e-2 of the dequantized references, measured ~4e-3 relative). Also passes on
the finished 5-commit branch.
the same MoE machinery): bit-identical on small shapes, 1e-5 relative on large ones including
e=64, n=640, k=2560, g128 (
docs/WRITEUP.mdsection 8).invoke_fused_moe_kernelat 64/64/32,16/16/128, 16/32/128 identical (2e-5) (CAMPAIGN 2026-09-02 11:04); per-layer dump A/B with
identical inputs 5.1e-5 (M=6) / 1.7e-4 (M=1) relative, i.e. bf16 output rounding from a
different fp32 accumulation order (CAMPAIGN 2026-09-02 11:25). End-to-end oracle on the 10k
prompt max 0.275 / mean 0.0125, inside the kernel-class band (CAMPAIGN 2026-09-02 11:30).
(CAMPAIGN 2026-09-02 12:10). Tuned int2 configs: max 0.086 / mean 0.0018 (2026-09-02 09:13).
Reproduction (needs
ninjaonPATH(otherwiseFileNotFoundError: ninja) and an nvcc thatsupports the GPU architecture at
CUDA_HOME(compute_120a on the reference machine; a systemnvcc 12.0 does not, the virtualenv's 13.3 does)):
Speed Tests and Profiling
Streaming bench, single request, ~10k context (
tools/bench_speed.py; card, "Performance"),measured on
73a255206fwith the flat patch:(
docs/WRITEUP.mdsection 5; first working path 2.24 -> 15.5 tok/s, sections 4 and 5).51 GB/s (PCIe line rate) from pinned host memory, 2.1 ms/token all-device; the byte-shuffled
table variant reached 157 / 14 GB/s and was rejected (CAMPAIGN 2026-09-02 08:29-08:31, 09:45).
(CAMPAIGN 2026-09-02 11:30).
2,496-token, 3-domain probe (CAMPAIGN 2026-09-02 09:48-09:53); decode 40.0 -> 48.4 tok/s,
prefill 2,249 -> 2,334 tok/s (CAMPAIGN 2026-09-02 12:10).
Not re-measured on
78c5024e9d(see the RFC, open question 3). The placement profile(
assets/expert_freq.pt, built bytools/expert_freq.pyfrom a routing dump) and the tunedconfigs are per model and per GPU.
Checklist
F401,F821,UP037, codespell andpy_compileclean on the commit.)test/registered/unit/layers/quantization/test_moe_wna16_int2.py, 7 tests / 34 subtests,register_cuda_ci.)fused_experts, the placement-profile format, the environment variables.)Suggested reviewers
python/sglang/kernels(Triton MoE kernels): @BBuf (Kernel).python/sglang/srt/layers/moe,layers/quantization/moe_wna16.py: no Merge Oncall area islisted for these paths; CODEOWNERS will be requested automatically. @JustinTong0323 as author
of Introduce Qwen 3.8 Flash Next #36497.
CI States
Latest PR Test (Base): ❌ Run #33750768338
Latest PR Test (Extra): ❌ Run #33750768055
Latest PR Test (AMD ROCm 10): ❌ Run #33750768353