feat(qsa): quantized KV pools with dequant-on-gather (fp8/int8/int4) - #37798
Draft
HaberstrohSystems wants to merge 4 commits into
Draft
HaberstrohSystems wants to merge 4 commits into
HaberstrohSystems wants to merge 4 commits into
Conversation
## Motivation
Qwen3.8-Flash-Next (Qwen4-Exp: 176B total / 6B active, 48 layers =
36 GatedDeltaNet + 12 Qwen sparse attention, 512 experts top-10, a 51 GB
PLE n-gram embedding table) only fits a 24 GB card with `--cpu-offload-gb`.
On the base commit that path does not work: weight loading fails on a
device mismatch in `GemmaRMSNorm`, `functional_call` rejects the tied
`A_log` tensors, the GDN convolution reads a stale `.view()` of a
parameter whose `.data` the offloader has replaced (GDN output exactly
zero, NaN logits), checkpoints saved through transformers carry the layer
type `qwen_sparse_attention`, the fused `in_proj_ba` lands at the wrong bit
width because `packed_modules_mapping` never reaches the quant config, and
the PLE table is allocated in full (51.2 GB as fp8, 51,200,245,760 B;
102.4 GB if materialised as bf16, derived as 2 x 51,200,245,760 B) before
any offload can act. These are the nine findings of `WRITEUP.md` section 3
("Nine findings in SGLang"), with finding 8 (the silent NaN) analysed in
section 4 and finding 9 (26 GB of PCIe traffic per token) in section 5.
The reference machine is one RTX PRO 4000 Blackwell (24 GB, sm_120) with
32 GB host RAM, serving a 2.572 bpw AutoRound checkpoint (model card,
"Quantization").
Measurements and logs: https://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglang
(`docs/CAMPAIGN.md`, an append-only log cited below by its dated entries;
`docs/ELASTIC_MEMORY.md`; `docs/WRITEUP.md`; `docs/DECODE_PERF_PLAN.md`;
`tools/bench_speed.py`) and the model card
https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang
(cited by section name).
## Modifications
Correctness under `--cpu-offload-gb` (general, not Qwen4-specific):
* `layers/layernorm.py`: `GemmaRMSNorm` re-derives `gemma_weight` on the
device of the parameter (`_weight_loader`) and of the input
(`_gemma_weight_for`); the non-persistent buffer was left on the GPU
while the parameter went to the CPU.
* `utils/offloader.py`: `functional_call(..., tie_weights=False)` in both
hook variants; `_CpuParamOffloader` keeps a hard reference to the pinned
host storage (`cpu_data`). With `SGLANG_MOE_EXPERT_STREAM=1` only expert
tensors (`w13_qweight`, `w2_qweight`, `w13_scales`, `w2_scales`,
`w13_qzeros`, `w2_qzeros`) are offloaded, and the forward hook is not
installed when every offloaded parameter of a module is a streamed expert
parameter (the hook would reparametrize the module with tensors that are
already on the device). The streamer itself is part 2 of this series.
* `layers/radix_linear_attention.py`, `models/qwen3_5.py`: the GDN module
passes its `nn.Conv1d` instead of a `.view()` of its weight;
`RadixLinearAttention.conv_weights` is a property that builds the 2D view
from the current parameter on every access. Tensors and tuples pass
through unchanged, so KDA / ShortConv / Lightning are not affected.
Qwen4-Exp specific:
* `models/qwen4_exp.py`: accept the layer type `qwen_sparse_attention`
(transformers renames `full_attention` on load; loading otherwise fails
with `KeyError: 'qwen_sparse_attention'`). The matching
`layers_block_type` alias in `configs/qwen4_exp.py` is already on
`qwen4-main-squashed` since sgl-project#36772 and is not repeated here;
`Qwen4ExpForConditionalGeneration.__init__` calls
`quant_config.update_packed_modules_mapping(...)` like `deepseek_v2.py`
does for Quark.
* `server_args.py`: `Qwen4ExpForConditionalGeneration` in
`LANGUAGE_MODEL_ONLY_ARCHITECTURES` (the class inherits the
`language_model_only` implementation from `Qwen3VLForConditionalGeneration`;
only the entry was missing). Saves the vision tower (0.84 GiB) and the
multimodal reservation in the KV budget for text-only serving.
* `models/qwen4_exp.py`: `Qwen4ExpMmapEmbedding` serves the PLE table from
a memory-mapped file (`SGLANG_QWEN4_PLE_MMAP=<dir>` with `ple.f8_e4m3.bin`
and `ple.json`; `ple.json` carries `file`, `rows`, `dim`, `dtype` and
`weight_scale`, the file is `rows x dim x itemsize` bytes and is checked).
The `VocabParallelEmbedding` is created on the `meta` device in that mode
so nothing is allocated first. Decode fetches its rows with parallel
`os.pread` (16 threads, GIL released; the numpy fancy-index into the
memmap held the GIL for the whole cold row fetch, measured at 2.56-3.09
ms per token; `DECODE_PERF_PLAN.md`, "2. Parallelise the PLE row
fetch"), bulk gathers go through the memmap. `MADV_RANDOM` on the
mapping and `POSIX_FADV_RANDOM` on the fd stop the page-cache
read-around per 160-byte row (`CAMPAIGN.md` 2026-09-02 14:12);
`POSIX_FADV_DONTNEED` after bulk gathers of >= 512 ids drops the pages
again (random text otherwise churns ~1M page-cache pages per long
prompt; `ELASTIC_MEMORY.md`, "What the reference host taught").
* `models/qwen4_exp.py`: `Qwen4ExpMmapEmbedding.forward` is wrapped with
`eager_on_graph(True)`, so decode runs under
`--cuda-graph-backend-decode breakable` with the PLE lookup as the one
eager break.
* `model_executor/runner_backend/breakable_cuda_graph_backend.py`: the four
output-structure helpers understand the `LogitsProcessorOutput`
dataclass; capture died with "Unsupported BCG output type" for any graph
body that returns it.
* `layers/attention/qsa/qsa_indexer.py`: `_qsa_ensure_rope` hoists the
`positions.max().item()` device sync (once per QSA layer, 12 per token;
`DECODE_PERF_PLAN.md`, "5. Two micro-fixes") and pre-sizes the rotary
cos/sin cache from `context_length`.
* `mem_cache/memory_pool.py`: fp8 write path in both `set_kv_buffer`
variants skips the no-op `div_` by a unit scale and saturates to +-448
before the e4m3 cast (the fp32/bf16 -> e4m3fn cast returns NaN beyond
the range). Consumed by the fp8 read path of part 4.
No new server flags; the two switches are `SGLANG_MOE_EXPERT_STREAM` and
`SGLANG_QWEN4_PLE_MMAP`. No tests are added in this part.
## Accuracy Tests
Exactness oracle: teacher-forced logprobs on a fixed 10k-token prompt,
mean and max |dlogprob| per token against the previous configuration;
noise floor mean 0.002, threshold for host-side changes mean <= 0.01,
max <= 0.5 (model card, "Serving fidelity"; `CAMPAIGN.md` 2026-09-02
11:25).
* Host fixes (offloader hook skip, indexer hoist, PLE pread; together with
the streamer items of part 2): oracle max 0.078 / mean 0.0019, equivalent
(`CAMPAIGN.md` 2026-09-02 09:30).
* Breakable decode graphs + `LogitsProcessorOutput` support: oracle max
0.079 / mean 0.0022, equivalent (`CAMPAIGN.md` 2026-09-02 11:48).
* PLE page-cache fixes involve no numerics; the 248k-token needle test on
the same server retrieved 5/5 codes after the page-drop fix
(`CAMPAIGN.md` 2026-09-02 20:00).
* fp8 KV write path: a 1-token prompt still crashes in the QSA indexer
prefill in `fp8_e4m3` mode; >= 101-token prompts work (`CAMPAIGN.md`
2026-09-02 15:26). Open issue, documented here.
## Speed Tests and Profiling
Streaming bench (`tools/bench_speed.py`, 200 streamed tokens, single
request, ~10k context; model card, "Performance"):
* host fixes: decode 15.4 -> 19.4 tok/s (`CAMPAIGN.md` 2026-09-02 09:30;
includes the streamer items of part 2);
* breakable decode graphs: decode 21.8 -> 40.0 tok/s at prefill 2249
tok/s, graph capture 3.97 s / 0.12 GB (`CAMPAIGN.md` 2026-09-02 11:33
and 11:48);
* fp8 write path: `fp8_e4m3` mode at parity with bf16, decode 55.9-57.2 /
prefill 2303-2339 tok/s (`CAMPAIGN.md` 2026-09-02 15:12);
* PLE read-around: before `MADV_RANDOM` / `POSIX_FADV_RANDOM` long
prompts (~55k tokens) drove host memory pressure past systemd-oomd
(`CAMPAIGN.md` 2026-09-02 14:12); the 248k random-text needle run
survived only after `POSIX_FADV_DONTNEED` (2026-09-02 19:37 and 20:00).
Part 1/5 of the Qwen3.8-Flash-Next 24 GB serving series. The series is
based on 73a2552, the first commit of the branch qwen4-main-squashed
(PR sgl-project#36497); there is no part 0.
## Motivation `moe_wna16` accepts 4 and 8 bits only, so a symmetric 2-bit AutoRound / GPTQ MoE checkpoint cannot be loaded at all. For Qwen3.8-Flash-Next (512 experts, top-10, `moe_intermediate_size` 640) the 2-bit experts are what makes the model fit next to a 24 GB card: the served checkpoint is 2.572 bpw (model card, "Quantization"). Two more things are needed once the experts are offloaded: `--cpu-offload-gb` copies a whole module's `state_dict` to the device on every forward (measured ~26 GB per token at 2.3 tok/s, while 10 x 48 x 1.31 MB = 0.63 GB of expert rows are actually needed; `WRITEUP.md` section 5), and the byte-row `[E, N, K/4]` layout reads 128-byte chunks at 640-byte stride, which limits an in-place decode GEMV to 73 GB/s from device memory and 5 GB/s from pinned host memory (`CAMPAIGN.md` 2026-09-02 08:29-08:31). Measurements and logs: https://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglang (`docs/CAMPAIGN.md`, an append-only log cited below by its dated entries; `docs/WRITEUP.md`; `tools/bench_speed.py`) and the model card https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang (cited by section name). The reference machine is one RTX PRO 4000 Blackwell (24 GB, sm_120) with 32 GB host RAM. ## Modifications 2-bit path (general for symmetric 2-bit `moe_wna16` checkpoints): * `kernels/ops/moe/fused_moe_triton_kernels.py`: 2-bit unpack in `fused_moe_kernel_gptq_awq` as the generalization of the 4-bit branch (`offs_k // 4`, shift `(offs_k % 4) * 2`, mask `0x3`, zero point 2). New `fused_moe_kernel_gptq_awq_word`: the same kernel reading the N-contiguous int32-word layout (one coalesced 128-byte line per warp load instead of 32 distinct lines). The layout is derived from the tensor dtype (int2 + `int32` = word layout, int2 + `uint8` = byte layout), so nothing crosses the custom-op schema. * `moe_runner/triton.py`, `triton_utils/fused_moe.py`, `triton_utils/fused_moe_triton_config.py`: `use_int2_w2a16` plumbing, shape handling for the word layout, config dtype name `int2_w2a16` so a tuned int4 config is never picked up by accident (no int2 config files ship; the tuned configs used for the measurements below live outside the tree, `SGLANG_MOE_CONFIG_DIR`). * `layers/quantization/moe_wna16.py`: 2-bit loader; asymmetric `qzeros` raise `NotImplementedError` by design. `process_weights_after_loading` re-lays every 2-bit expert tensor once from `[E, N, K/4]` uint8 to `[E, K/16, N]` int32 and scales to `[E, K/128, N]` (same bytes; `SGLANG_MOE_NCONTIG=0` keeps the byte layout); pinned host tensors are re-pinned after the copy, which transiently doubles one layer's host footprint. * `layers/moe/expert_gemv.py` (new): batch-1 int2 GEMV (`moe_gemv_int2_tab`, used for M <= 16 in `apply()`, `SGLANG_MOE_GEMV=0` disables) that reads experts in place through int64 address tables; an entry may point into device or pinned host memory, the kernel indexes with the original expert ids. `to_word_ncontig`, `make_tables`. * `layers/moe/expert_stream.py` (new, `SGLANG_MOE_EXPERT_STREAM=1`): gathers exactly the routed experts of a forward into one shared, shape-keyed staging buffer and renumbers the top-k ids onto it; fully GPU-resident layers skip the gather and run on the original ids; `arange`/cast tensors are memoized (no per-token allocations). * `layers/quantization/moe_wna16.py`: deferred placement pass at the last layer (`SGLANG_MOE_PLACEMENT=<expert_freq.pt>` with `{"mass": [L, E]}`, `SGLANG_MOE_PLACEMENT_S`, default 184): the hottest S experts of every layer are resident on the GPU, cold rows of GPU layers take the pinned slots that host layers' hot rows vacate, memory-neutral on both sides. With `SGLANG_MOE_ELASTIC=1` the pass hands the layers to `ExpertElastic` (part 3) instead. * `test/registered/unit/layers/quantization/test_moe_wna16_int2.py` (new, `register_cuda_ci`, synthetic data, no checkpoint): the exact 2-bit unpack expressions of `fused_moe_kernel_gptq_awq` against a torch reference (bit-exact, including the real expert dims 2560 x 640 g128); `to_word_ncontig` byte -> word layout round trip (bit-exact and invertible, CPU); `fused_moe(use_int2_w2a16=True)` on the byte layout (symmetric and with per-group `qzeros`) and on the word layout against a torch MoE reference on the dequantized weights; `moe_gemv_int2_tab` through `make_tables` addresses against an fp32 reference (w13 and w2 forms, fp16 and bf16 scales, N not a multiple of the block; a bad block is rejected). * `test/manual/test_triton_moe_wna16.py`: `w2a16` / `w2a16b2` cases and the 2-bit packing. This stays a manual test: its 8-bit path fails the tolerance check with the unmodified upstream kernel on the reference machine (129 of 144 cases in the 8-bit / group-128 subset, identical on the unpatched tree), so it is not a usable gate; the 2-bit path is covered by `test_moe_wna16_int2.py` instead. ## Accuracy Tests * `test_moe_wna16_int2.py` (7 tests, 34 subtests, pass; run as `PYTHONPATH=python python test/registered/unit/layers/quantization/test_moe_wna16_int2.py` with ninja and an nvcc that supports the GPU architecture on PATH): packing order, the 2-bit unpack expressions and the word-layout round trip are bit-exact; `fused_moe` on both layouts and the GEMV agree with the dequantized references within atol 1e-3 / rtol 2e-2. The original A/B of the same kernel path on real shapes (dequantized bf16 vs packed 2-bit through the same MoE machinery) was bit-identical on small shapes and 1e-5 relative on large ones including e=64, n=640, k=2560, g128 (`WRITEUP.md` section 8). * Word layout and GEMV: A/B through the patched `invoke_fused_moe_kernel` at 64/64/32, 16/16/128, 16/32/128 identical (2e-5) (`CAMPAIGN.md` 2026-09-02 11:04); per-layer dump A/B with identical inputs 5.1e-5 (M=6) / 1.7e-4 (M=1) relative, i.e. bf16 output rounding from a different fp32 accumulation order, verdict numerically equivalent (2026-09-02 11:25). End-to-end oracle on the 10k prompt (teacher-forced logprobs, model card, "Serving fidelity") max 0.275 / mean 0.0125, inside the kernel-class band (2026-09-02 11:30). * Placement is exact by construction (rows move, values do not): oracle max 0.057 / mean 0.0013 (`CAMPAIGN.md` 2026-09-02 12:10). * Tuned int2 configs: oracle max 0.086 / mean 0.0018 (2026-09-02 09:13). ## Speed Tests and Profiling Streaming bench, single request, ~10k context (`tools/bench_speed.py`; model card, "Performance"; `CAMPAIGN.md`): * PCIe traffic per decoded token 26 GB -> 0.31 GB with expert-only offload and the streamer (`WRITEUP.md` section 5; first working path 2.24 tok/s -> 15.5 tok/s, sections 4 and 5). * Tuned int2 configs: decode 13.4 -> 15.2 tok/s (2026-09-02 09:13). * N-contiguous GEMV micro-benchmark on real layer-5 tensors: 320 GB/s from device memory, 51 GB/s (PCIe line rate) from pinned host memory, 2.1 ms/token all-device; the byte-shuffled table variant reached 157 / 14 GB/s and was rejected (2026-09-02 08:29-08:31 and 09:45). * N-contiguous layout + GEMV end to end: decode 19.4 -> 21.8 tok/s, prefill 1443 -> 2249 tok/s (2026-09-02 11:30). * Routing-mass placement: the top 171 of 512 experts per layer cover 82 % of routing mass on a 2496-token, 3-domain probe (2026-09-02 09:48-09:53); decode 40.0 -> 48.4 tok/s, prefill 2249 -> 2334 tok/s (2026-09-02 12:10). Part 2/5 of the Qwen3.8-Flash-Next 24 GB serving series; stacked on part 1.
## Motivation On a 24 GB card that also has to hold a 2-bit MoE, VRAM is shared between resident expert rows and the KV cache, and the split that is right for a short chat is wrong for a 256k-token prompt. Two facts make a static split unnecessary: expert weights are immutable, so an expert's GPU residency is a pure cache (eviction = table write, admission = one row copy + table write); and SGLang already keeps the KV cache in a CUDA VMM arena (`KvVmmBufferOwner`), but commits it monotonically at startup (`CAMPAIGN.md` design note 2026-09-02 12:40). Host RAM is the wall on the reference machine (RTX PRO 4000 Blackwell 24 GB, 32 GB host RAM): 31 offloaded layers pin 24-26 GB of the 32 GB, so no host mirror can be added at runtime (`CAMPAIGN.md` 2026-09-02 12:49; model card, "Scope and limitations"). Measurements and logs: https://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglang (`docs/CAMPAIGN.md`, an append-only log cited below by its dated entries; `docs/ELASTIC_MEMORY.md`; `tools/bench_speed.py`) and the model card https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang (cited by section name). ## Modifications Elastic expert cache (`SGLANG_MOE_ELASTIC=1`, wired to the placement pass of part 2): * `layers/moe/row_arena.py` (new): `RowArena` reserves virtual address space for `max_rows` once (`cuMemAddressReserve`) and backs a rank-ordered prefix with physical chunks on demand (`cuMemCreate` + `cuMemMap`, 4 MiB chunks aligned to the 2 MiB device granularity). Shrinking is a tail unmap that really returns the memory to the driver while every row address stays fixed for the process lifetime, so kernels that read rows through an address table, and CUDA graphs that captured them, need no recapture. Uses the driver plumbing in `cuda_vmm_utils`. The module's former `__main__` self-test is replaced by the registered unit test below. * `layers/moe/expert_elastic.py` (new): one arena per (layer, tensor kind) in routing-mass rank order; the int64 table `addr[e]` points into the arena for rank(e) < S and at the row's pinned host slot otherwise. Grow = table-driven host -> arena gather + table rewrite; shrink = D2H copy of the tail ranks into pool slots + table rewrite + unmap. Host memory is conserved through a slot pool (a host layer's hot rows vacate their slots, a GPU layer's cold rows take them); `free()` never pins at runtime, so S_floor is what the pool can absorb (184 rows per layer on the reference machine; `SGLANG_MOE_ELASTIC_PIN_MB` grants a pinned fallback budget, `SGLANG_MOE_ELASTIC_FILL_MB` / `_RESERVE_ROWS` steer the startup fill). A control file (`SGLANG_MOE_ELASTIC_CTL`: `S <n>`, `fill <MB>`, `free <MB>`, `status`) is polled from the MoE apply path outside graph replay and writes a status file. * `test/registered/unit/layers/moe/test_row_arena.py` (new, `register_cuda_ci`, ~150 MB of VRAM): arena geometry (chunk / VA alignment, `chunks_for_rows`, `bytes_to_reach`, `rows_for_bytes`); `ensure_rows` backs a prefix and driver-free memory drops by exactly the mapped bytes; a Triton gather through an address table that mixes arena and pinned-host rows reads every row; a CUDA graph captured against the arena addresses replays after `shrink_rows` (memory returned, evicted rows repointed to the host row) and after growing back; `close` returns the memory; `ArenaOOM` is raised when a chunk cannot be created. Lazy KV backing (`SGLANG_KV_LAZY=1`): * `mem_cache/kv_vmm_backing.py`: `KvVmmArena.uncommit_beyond`, `KvVmmBufferOwner.release_beyond`, `backed_tokens`, `bytes_per_token`. * `mem_cache/memory_pool.py`: with `SGLANG_KV_LAZY=1` the full-attention pool is allocated through the VMM owner in the classic flow as well, with `SGLANG_KV_LAZY_FLOOR` (4096) tokens backed at start. `lazy_ensure` commits in `SGLANG_KV_LAZY_MARGIN` (2048) token steps as pages are handed out and keeps `SGLANG_KV_LAZY_HEADROOM_MB` (1536) driver-free after every commit: below the watermark it shrinks the expert cache first (through `ExpertElastic.free`), and `empty_cache()` is only forced when the cache can still shrink, otherwise rate-limited to once per 30 s. `lazy_release` unmaps beyond the floor when the pool goes idle and regrows the expert cache at that point (no forward in flight). * `mem_cache/allocator/paged.py`, `allocator/token.py`: `_lazy_hook` before pages are consumed; an allocation whose commit fails returns `None` (refused) instead of crashing; `_lazy_idle_check` in `_release_page_ids` / `free`. * `mem_cache/kv_cache_configurator.py`: virtual capacity `SGLANG_KV_LAZY_TOKENS` above the profiled value, admitted capacity `min(requested, SGLANG_KV_LAZY_SAFETY x profiled)` (default 0.85), so a prompt that could not be backed is refused at admission. All switches are environment variables in this part; server arguments, a registered callback instead of the `ExpertElastic` import inside `lazy_ensure`, and an HTTP endpoint instead of the control file are the obvious follow-ups. Known limits: `lazy_release` fires when the whole pool is idle (validated at `--max-running-requests 1`); the scheduler does not survive `alloc_extend` returning `None` mid-prefill, which is why the admission cap exists (`CAMPAIGN.md` 2026-09-02 14:56 and 14:58). ## Accuracy Tests Both mechanisms move bytes without changing values; the exactness oracle (teacher-forced logprobs on the 10k prompt, model card, "Serving fidelity") confirms it: elastic cache max 0.060 / mean 0.0016 (`CAMPAIGN.md` 2026-09-02 13:17); lazy KV max 0.057 / mean 0.0014 (2026-09-02 13:42); 128k configuration mean 0.0021 (2026-09-02 13:53). `test_row_arena.py` on the idle GPU (3 tests pass; run as `PYTHONPATH=python python test/registered/unit/layers/moe/test_row_arena.py` with ninja and an nvcc that supports the GPU architecture on PATH): 2 MiB granularity, `ensure_rows(184)` maps 36 MiB, `shrink_rows(64)` returns 20 MiB to the driver (`mem_get_info` confirms), a CUDA graph captured against arena addresses replays correctly after shrink and after regrow, `ArenaOOM` raised cleanly (the same steps as the original self-test, `CAMPAIGN.md` 2026-09-02 12:44). ## Speed Tests and Profiling Streaming bench, single request, ~10k context (`tools/bench_speed.py`; model card, "Performance"): * elastic expert cache: decode 56.2 tok/s, prefill 2335 tok/s (the previous configuration measured 56.0 / 2362 after the bench measurement fix of 2026-09-02 12:58; 13:17). Live S sweep without restart: S=184 arena 11.06 GB, 1.73 GB free, 84.3 % of routing mass, decode 56-57.6; S=200 12.19 GB / 0.62 GB free / 86.7 % / 55.6-57.9: the dial buys +2-3 % decode per ~1.7 GB, its value is VRAM on demand for the KV cache (14:20). * lazy KV: decode 55.4 / prefill 2323 (13:42); the log shows 288 MB backed at 10k tokens (24 KB/token bf16) and release to the 4096-token floor (144 MB) at idle (14:45); KV committed at startup 0.8 GB -> 0.1 GB (`ELASTIC_MEMORY.md`, "Result"). * 128k context: decode 55.0-57.5, prefill 2270-2340 re-measured on the live server (13:53). * watermark rule: a 60k-token prompt first crashed because every commit ate the prefill's working memory (driver free 0.07 GB; 13:55); with the headroom rule a 68,905-token bf16 prompt ran at prefill 48.9 s (1408 tok/s), decode 53.3 tok/s, expert cache 192 -> 184 during the request and regrown afterwards (14:54). * rate-limited `empty_cache`: 257,905-token prompt (tiered pool of part 4) prefill 171.0 s -> 165.3 s (1508 -> 1560 tok/s), decode 52.3 (2026-09-02 22:19; 2026-09-03 00:55). Part 3/5 of the Qwen3.8-Flash-Next 24 GB serving series; stacked on part 2.
## Motivation The Qwen sparse attention (QSA) kernels read bf16/fp16 K/V only, so the stock `fp8_e4m3` KV option cannot be consumed by Qwen4-Exp, and a bf16 KV cache (24 KB per token over the 12 QSA layers) stops a 24 GB card at about 80k tokens once the expert cache is at its floor (`CAMPAIGN.md` 2026-09-02 13:55). QSA has exactly two places where every K/V row is gathered anyway (the decode/verify compaction into the FlashAttention scratch and the prefix-chunk row gather), so dequantization is free there and the attention kernels stay untouched. The formats were chosen from measured K/V statistics of this model (~19k tokens): simulated relative RMS error e4m3 2.66 %, int8 per-token 1.3 %, int8 with 64-channel groups 0.9 % (`CAMPAIGN.md` 2026-09-02 15:50; `ELASTIC_MEMORY.md`, "The three mechanisms", 3). Measurements and logs: https://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglang (`docs/CAMPAIGN.md`, an append-only log cited below by its dated entries; `docs/ELASTIC_MEMORY.md`; `tools/bench_speed.py`) and the model card https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang (cited by section name). The reference machine is one RTX PRO 4000 Blackwell (24 GB, sm_120) with 32 GB host RAM. ## Modifications New `--kv-cache-dtype` values `int8_g64`, `int4_g32`, `int8ring_int4` (`server_args.py` with help text, `mem_cache/kv_cache_dtype.py`, `mem_cache/kv_cache_configurator.py` pool-class selection, `model_executor/pool_configurator.py` cell sizes). Pool classes are subclasses of `MHATokenToKVPool`; the payload and the fp16 group scales are extra `KvBufferDesc`s on the same (lazy, part 3) VMM owner, and writing is one fused quantize + scatter Triton launch. * `mem_cache/int8_kv_pool.py` (new): int8 K/V `[rows, H, D]` plus one fp16 absmax/127 scale per (token, kv-head, 64-channel group), the first group being the rotary dimensions; 12.4 KB per token over the 12 QSA layers. `kv_bits = 8` is the backend's dispatch key. * `mem_cache/int4_kv_pool.py` (new): nibble-packed K/V `[rows, H, D/2]` (low nibble = even channel, offset-binary q + 8, q in [-7, 7]) plus fp16 absmax/7 scales per 32-channel group; 6.75 KB per token. `kv_bits = 4`. * `mem_cache/tiered_kv_pool.py` (new, `int8ring_int4`): every token is written twice, int8-g64 into a ring of `SGLANG_KV_TIERS_W` (8192) slots with an int32 owner table, and int4-g32 into the full-context pool. Readers test `owner[slot & (R-1)] == slot` on the device and read the int8 ring row (hot) or the int4 row (cold) with `tl.where`: CUDA-graph safe, no compactor, no unmapping mid-request. The ring is allocated once before capture and is not a `KvBufferDesc`, so `bytes_per_token()` and the lazy backing stay exact. 7,308 B per token at 256k (6,912 int4 + 12,672 x 8192/262,144), ring 103.8 MB + 32 KB owner table. * `layers/attention/qsa/sparse_attn.py`: `_compact_kv_fp8` (uint8 -> fp8 bitcast -> bf16 into the FA2 scratch), `_quant_store_kv_int8` / `_compact_kv_int8` / `_gather_dequant_rows_int8` (the last replaces `index_select + cat` for prefix chunks), the int4 siblings, `_stamp_ring_owner`, `_quant_store_kv_tiered`, `_compact_kv_tiered`, `_gather_dequant_rows_tiered` and their wrappers; fp16 scale clamp to 65504 in every write kernel. * `layers/attention/qwen_sparse_attn_backend.py`: dispatch on `pool.kv_bits` / `pool.kv_tiered` (`_kv_bits`, `_kv_head_dim`, `_kv_scratch_dtype`, `_int8_gather_kwargs`, `_kv_tier_kwargs`), bf16 scratch keyed by the query dtype, prefix-chunk gather on the uint8 view for fp8; the CPU fallback raises `NotImplementedError` for int8/int4 pools. * `mem_cache/memory_pool.py`: `HybridLinearKVPool.get_kv_smooth_buffer`, `get_kv_ring_buffer`, `get_kv_ring_owner` forwarders. Per-channel smoothing constants (`sm_k` / `sm_v`) are plumbed but identity: a static per-channel smoothing A/B on the fake-quantized int4-g32 pool was mixed (mean NLL -0.005 nats, per-token deviation larger) and was not adopted (`CAMPAIGN.md` 2026-09-03 00:40). * Registered unit tests (`register_cuda_ci`, 30-50 MB of VRAM each, no server, torch references): - `test/registered/unit/layers/attention/qsa/test_sparse_attn_fp8_gather.py`: the fp8 compaction gather equals `pool.to(fp8).to(bf16)` bit for bit; e4m3 relative RMS error band on N(0, 3). - `test/registered/unit/mem_cache/test_int8_kv_pool.py`: int8-g64 quantize + scatter (int32 and int64 loc), scale index arithmetic, compaction gather-dequant (3 requests, permuted `req_to_token`, invalid positions untouched), prefix row gather-dequant with interior gaps, all bit-exact; relative RMS error; `MHATokenToKVPoolInt8` eager and lazy-VMM paths. - `test/registered/unit/mem_cache/test_int4_kv_pool.py`: int4-g32 nibble packing and unpack round trip, quantize + scatter with the .5-tie rounding of `rint`, scale index and nibble order, compaction gather including the trtllm strided layout, prefix gather with gaps, fp16-max scale clamp, all bit-exact; `MHATokenToKVPoolInt4` eager and lazy-VMM paths. - `test/registered/unit/mem_cache/test_tiered_kv_pool.py`: tiered ring owner stamping (R = 64 over 4096 slots, ring wrap), hot/cold boundary and same-launch ring-row collisions, hot/cold selection in the compaction and prefix gathers (stale owner -> int4 path), fp16-max clamp on the ring, all-cold / all-hot equality with the single-tier kernels; `MHATokenToKVPoolTiered` eager and lazy-VMM paths, ring reset on `lazy_release`, a bad `SGLANG_KV_TIERS_W` rejected. Decode routing on `qwen4-main-squashed` (sgl-project#36806, sgl-project#36845): `_forward_paged_attention` first tries `_resolve_trtllm_sparse_decode()`, which returns FlashInfer's `trtllm_batch_decode_with_kv_cache` on exact SM120 as well as SM100, and otherwise uses `_resolve_flash_attn_varlen_func()` (the KDA kernel on SM121, else FA2, else FA4). Both routes gather the selected rows through `qwen_sparse_kv_extraction_compact_triton`, which this commit extends with the scale / smoothing / ring arguments, so a quantized pool is dequantized into the bf16 scratch before either attention kernel runs, and the prefix-chunk gather in `forward_extend` is not routed by device at all; no device-specific guard is added. What does change on SM120 relative to the base the numbers below were measured on (73a2552, where the FlashInfer route was gated on SM100 only): the attention over the gathered scratch runs FlashInfer's XQA kernel (JIT-compiled at the first sparse decode; it needs an nvcc >= 12.9 at `CUDA_HOME`, otherwise FlashInfer raises `RuntimeError: No supported CUDA architectures found`) instead of FA2 `flash_attn_varlen_func`; the bound is FlashInfer's (`flashinfer/compilation_context.py`, `_normalize_cuda_arch`: SM 12.x requires CUDA >= 12.9). A standalone call of that kernel with the backend's arguments (24 query heads, 2 KV heads, head_dim 256, page 64, topk 2051, which is `indexer_budget` 2048 + `indexer_compress_ratio` 4 - 1, the index-row width `token_topk + compress_ratio - 1` of `qsa_indexer.py`; batch 1 and 4, plus topk 130 at batch 3; bf16) on the reference RTX PRO 4000 Blackwell with flashinfer 0.6.17 and nvcc 13.3 matches a torch softmax reference within bf16 precision (max relative error 3.9e-3, 2.5e-3, 2.5e-3; no NaN). Probe and output: `tools/probe_trtllm_sm120.py` and `docs/logs/probe_trtllm_sm120.log` in the companion repository. The speed numbers below were measured with the FA2 route and have not been re-measured on this base. Scope and known limits: the read path exists for `qwen_sparse_attn_backend` only; any other GPU backend that meets one of these pools today would read raw bytes, so a generic guard or a `kv_bits`-aware materialisation is the follow-up. `TORCH_DTYPE_TO_KV_CACHE_STR` maps `torch.uint8` to `int4_g32`, which the tiered mode shares. `fp8_e4m3` mode still crashes on 1-token prompts (`CAMPAIGN.md` 2026-09-02 15:26). Prefix-chunk prefill materialises the prefix as bf16 per layer and chunk; a paged prefix kernel reading the pools directly was built, verified within 2 row-ulps and rejected as 4-5x slower (`ELASTIC_MEMORY.md`, "Where the long-context prefill time goes"). On-disk layouts are documented in the module docstrings. ## Accuracy Tests Protocol (model card, "Serving fidelity"; `ELASTIC_MEMORY.md`, "Quality protocol"): prompts shorter than one prefill chunk never read the KV cache, so the decisive test scores every position from the second chunk on (8561 positions of a 9.6k-token text, bf16 pool as reference, run-to-run noise NLL +0.0002 / mean |dlogprob| 0.059), plus the 512-window test (noise 0.099 / +-0.008 NLL), the 10k logprob oracle, and needle retrieval with `ignore_eos`. * The four registered tests above pass on the reference machine (`test_sparse_attn_fp8_gather.py` 2, `test_int8_kv_pool.py` 10, `test_int4_kv_pool.py` 14, `test_tiered_kv_pool.py` 15 tests; run as `PYTHONPATH=python python <file>` with ninja and an nvcc that supports the GPU architecture on PATH, which the lazy-VMM cases need for their `load_inline` stub). They establish the bit-exactness of quantize + scatter, compaction gather-dequant and prefix gather-dequant for every pool against the torch references, and the relative RMS error 0.62 % of int8-g64 on N(0,3) (`CAMPAIGN.md` 2026-09-02 16:08 and 16:26 for the original int8 run and its review round). * Fake-quant ladder on the bf16 pool, 512 window, mean |dlogprob|: noise 0.099, int8_g64 0.099, int8 per-token 0.100, int8_g32 0.106, e4m3 0.110 (2026-09-02 16:17). * INT8-G64 server: short NLL -0.001, 512-window 0.094 vs noise 0.099, NLL +0.010 (noise +-0.008), oracle 0.0019; all-position +0.001 / 0.059; needle 41k 5/5 (2026-09-02 16:45 and 21:05). * INT4-G32 server: all-position NLL +0.0088 / 0.138 (fake-quant int4_g32 +0.0078 / 0.139), oracle 0.0015; needle 5/5 at 247,629 tokens (2026-09-02 19:40 and 20:00). * Tiered default: short NLL -0.0002, 512-window 0.118, all-position NLL -0.0001 / 0.074, oracle 0.0019; needles 5/5 at 41,370 and 5/5 at 247,629 tokens (2026-09-02 21:05). * Controls that make the ladder trustworthy: fake int2_g16 +0.30 / 0.62 (the cliff), K/V := 0 +3.70 / 3.92. The ladder was re-measured after the write-path fake-quant hook was found to be bypassed by the int8 pool subclass; the K/V := 0 control caught it (`CAMPAIGN.md` 2026-09-02 19:40). ## Speed Tests and Profiling Streaming bench, single request, ~10k context (`tools/bench_speed.py`; model card, "Performance"; `CAMPAIGN.md`): * fp8 read path: decode 55.9-57.2 / prefill 2303-2339 tok/s, parity with bf16 (2026-09-02 15:12). * INT8-G64: decode 56.1-57.4 / prefill 2301-2316; 115,560-token prompt prefill 71.8 s (1610 tok/s), decode 50.8 (16:45); the longest prompt admitted on the reference machine 162,215 tokens at prefill 95.8 s (1694 tok/s), decode 51.2, VRAM free bottomed at 1.18 GB, a ~179k prompt refused at admission (17:00). * INT4-G32: decode 55-57, prefill 1493 at 10k (int8 2316: the unpack gather); 257,905-token prompt prefill 182.2 s (1415 tok/s), decode 51.9 (19:40 and 19:34); needle haystack of 247,629 random-word tokens prefill 269 s (920 tok/s, PLE rows from NVMe), decode 51.4 (20:00). * Tiered default: decode 54-57 (56.2 / 54.3 / 56.8 / 55.7 / 54.5 at context 101 / 421 / 1701 / 6821 / 10001), prefill 2271 at 10k; 257,905-token prompt prefill 171 s (1508 tok/s), decode 51.8; lazy backing 366,976 tokens profiled -> 262,144 admitted (21:05); with the rate-limited `empty_cache` of part 3: 165.3 s (1560 tok/s), decode 52.3 (2026-09-03 00:55). Per-chunk prefill time at 258k fits 615 ms + 0.49 us x prefix tokens; the O(prefix) term is the QSA indexer, not the KV gather (`ELASTIC_MEMORY.md`, "Where the long-context prefill time goes"). Part 4/5 of the Qwen3.8-Flash-Next 24 GB serving series; stacked on part 3.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft: stacked on #37793 (part 1); parts 2, 3 precede it in the series.
Part 4/5 of the Qwen3.8-Flash-Next 24 GB serving series (RFC issue: #37792). Patch:
upstream/series-q4head/0004-feat-qsa-quantized-KV-pools-with-dequant-on-gather-f.patchin the companion repository (14 files, +4,424 / -25). The pool classes are general; the read
path is specific to the Qwen sparse attention backend.
Motivation
The Qwen sparse attention (QSA) kernels read bf16/fp16 K/V only, so the stock
fp8_e4m3KVoption cannot be consumed by Qwen4-Exp, and a bf16 KV cache (24 KB per token over the 12 QSA
layers) stops a 24 GB card at about 80k tokens once the expert cache is at its floor (CAMPAIGN
2026-09-02 13:55). QSA has exactly two places where every K/V row is gathered anyway (the
decode/verify compaction into the attention scratch and the prefix-chunk row gather), so
dequantization is free there and the attention kernels stay untouched. The formats were chosen
from measured K/V statistics of this model (~19k tokens): simulated relative RMS error e4m3
2.66 %, int8 per-token 1.3 %, int8 with 64-channel groups 0.9 % (CAMPAIGN 2026-09-02 15:50;
docs/ELASTIC_MEMORY.md, "The three mechanisms", 3).Sources: dated entries of
docs/CAMPAIGN.md,docs/ELASTIC_MEMORY.mdanddocs/logs/tiers-validate.login https://github.com/HaberstrohSystems/qwen3.8-flash-next-24gb-sglangand the model card
https://huggingface.co/HaberstrohSystems/Qwen3.8-Flash-Next-int2-mixed-AutoRound-24GB-SGLang.
Reference machine: one RTX PRO 4000 Blackwell (24 GB, sm_120) with 32 GB host RAM.
Modifications
New
--kv-cache-dtypevaluesint8_g64,int4_g32,int8ring_int4(server_args.pywithhelp text,
mem_cache/kv_cache_dtype.py,mem_cache/kv_cache_configurator.pypool-classselection,
model_executor/pool_configurator.pycell sizes). Pool classes are subclasses ofMHATokenToKVPool; the payload and the fp16 group scales are extraKvBufferDescs on the same(lazy, part 3) VMM owner, and writing is one fused quantize + scatter Triton launch.
mem_cache/int8_kv_pool.py(new): int8 K/V[rows, H, D]plus one fp16 absmax/127 scale per(token, kv-head, 64-channel group), the first group being the rotary dimensions; 12.4 KB per
token over the 12 QSA layers.
kv_bits = 8is the backend's dispatch key.mem_cache/int4_kv_pool.py(new): nibble-packed K/V[rows, H, D/2](low nibble = evenchannel, offset-binary q + 8, q in [-7, 7]) plus fp16 absmax/7 scales per 32-channel group;
6.75 KB per token.
kv_bits = 4.mem_cache/tiered_kv_pool.py(new,int8ring_int4): every token is written twice, int8-g64into a ring of
SGLANG_KV_TIERS_W(8192) slots with an int32 owner table, and int4-g32 intothe full-context pool. Readers test
owner[slot & (R-1)] == sloton the device and read theint8 ring row (hot) or the int4 row (cold) with
tl.where: CUDA-graph safe, no compactor, nounmapping mid-request. 7,308 B per token at 256k (6,912 int4 + 12,672 x 8192/262,144), ring
103.8 MB + 32 KB owner table.
layers/attention/qsa/sparse_attn.py:_compact_kv_fp8(uint8 -> fp8 bitcast -> bf16 intothe attention scratch),
_quant_store_kv_int8/_compact_kv_int8/_gather_dequant_rows_int8(the last replacesindex_select + catfor prefix chunks), theint4 siblings,
_stamp_ring_owner,_quant_store_kv_tiered,_compact_kv_tiered,_gather_dequant_rows_tieredand their wrappers; fp16 scale clamp to 65504 in every writekernel. On-disk layouts (nibble order, scale index arithmetic, owner-table semantics) are in
the module docstrings.
layers/attention/qwen_sparse_attn_backend.py: dispatch onpool.kv_bits/pool.kv_tiered(
_kv_bits,_kv_head_dim,_kv_scratch_dtype,_int8_gather_kwargs,_kv_tier_kwargs),bf16 scratch keyed by the query dtype, prefix-chunk gather on the uint8 view for fp8; the CPU
fallback raises
NotImplementedErrorfor int8/int4 pools.mem_cache/memory_pool.py:HybridLinearKVPool.get_kv_smooth_buffer,get_kv_ring_buffer,get_kv_ring_ownerforwarders. Per-channel smoothing constants are plumbed but identity: astatic smoothing A/B on the fake-quantized int4-g32 pool was mixed and was not adopted
(CAMPAIGN 2026-09-03 00:40).
register_cuda_ci, stagebase-b, runner1-gpu-small, 30-50 MB ofVRAM each, torch references, no server):
test/registered/unit/layers/attention/qsa/test_sparse_attn_fp8_gather.py(est_time=10),test/registered/unit/mem_cache/test_int8_kv_pool.py(20),test_int4_kv_pool.py(20),test_tiered_kv_pool.py(40); contents under Accuracy Tests.Decode routing on
qwen4-main-squashed(#36806, #36845):_forward_paged_attentionfirst tries_resolve_trtllm_sparse_decode(), which returns FlashInfer'strtllm_batch_decode_with_kv_cacheon exact SM120 as well as SM100, and otherwise uses
_resolve_flash_attn_varlen_func()(the KDAkernel on SM121, else FA2, else FA4). Both routes gather the selected rows through
qwen_sparse_kv_extraction_compact_triton, which this PR extends with the scale / smoothing /ring arguments, so a quantized pool is dequantized into the bf16 scratch before either attention
kernel runs, and the prefix-chunk gather in
forward_extendis not routed by device at all; nodevice-specific guard is added. What does change on SM120 relative to the base the numbers below
were measured on (
73a255206f, FlashInfer route gated on SM100 only): the attention over thegathered scratch runs FlashInfer's XQA kernel (JIT-compiled at the first sparse decode; it needs
an nvcc >= 12.9 at
CUDA_HOME, FlashInfer's bound inflashinfer/compilation_context.py(
_normalize_cuda_arch: SM 12.x requires CUDA >= 12.9); otherwise FlashInfer raisesRuntimeError: No supported CUDA architectures found) instead of FA2flash_attn_varlen_func.A standalone call of that kernel with the backend's arguments (24 query heads, 2 KV heads,
head_dim 256, page 64, topk 2,051 =
indexer_budget2048 +indexer_compress_ratio4 - 1, theindex-row width
token_topk + compress_ratio - 1ofqsa_indexer.py; batch 1 and 4, plus topk130 at batch 3; bf16) on the RTX PRO 4000 with flashinfer 0.6.17 and nvcc 13.3 matches a torch
softmax reference within bf16 precision (max relative error 3.9e-3, 2.5e-3, 2.5e-3; no NaN):
tools/probe_trtllm_sm120.pyand its outputdocs/logs/probe_trtllm_sm120.login the companionrepository. See the RFC, open question 3.
Scope and known limits, disclosed: the read path exists for
qwen_sparse_attn_backendonly; anyother GPU backend that meets one of these pools today would read raw bytes, so a generic guard or
a
kv_bits-aware materialisation is the follow-up (RFC, open question 2).TORCH_DTYPE_TO_KV_CACHE_STRmapstorch.uint8toint4_g32, which the tiered mode shares.fp8_e4m3mode still crashes on 1-token prompts (CAMPAIGN 2026-09-02 15:26). Prefix-chunkprefill materialises the prefix as bf16 per layer and chunk; a paged prefix kernel reading the
pools directly was built, verified within 2 row-ulps and rejected as 4-5x slower
(
docs/ELASTIC_MEMORY.md, "Where the long-context prefill time goes").SGLANG_KV_TIERS_Wshould become a server argument.
Accuracy Tests
Protocol (card, "Serving fidelity";
docs/ELASTIC_MEMORY.md, "Quality protocol"): promptsshorter than one prefill chunk never read the KV cache, so the decisive test scores every
position from the second chunk on (8,561 positions of a 9.6k-token text, bf16 pool as reference,
run-to-run noise NLL +0.0002 / mean |dlogprob| 0.059), plus the 512-window test (noise 0.099 /
+-0.008 NLL), the 10k logprob oracle, and needle retrieval with
ignore_eos.test_sparse_attn_fp8_gather.py(2): the fp8 compaction gather equalspool.to(fp8).to(bf16)bit for bit; e4m3 relative RMS error in (0.01, 0.04) on N(0, 3).
test_int8_kv_pool.py(10 + 2 subtests): int8-g64 quantize + scatter (int32 and int64loc), scale index arithmetic, compaction gather-dequant (3 requests, permuted
req_to_token,invalid positions untouched), prefix row gather-dequant with interior gaps, all bit-exact;
relative RMS error < 1.2 %;
MHATokenToKVPoolInt8eager and lazy-VMM paths.test_int4_kv_pool.py(14 + 2): nibble packing and unpack round trip, quantize + scatterwith
rinttie rounding, scale index and nibble order, compaction gather including thetrtllm strided layout (a packed-width scratch is rejected), prefix gather with gaps,
fp16-max scale clamp, all bit-exact; relative RMS error 8-11.5 %;
MHATokenToKVPoolInt4eager and lazy-VMM paths;
kv_bitskeys.test_tiered_kv_pool.py(15 + 2): ring owner stamping (R = 64 over 4,096 slots, ring wrap),hot/cold boundary and same-launch ring-row collisions (x20), hot/cold selection in the
compaction and prefix gathers (stale owner and no owner -> int4 path), fp16-max clamp on the
ring, all-cold == int4 and all-hot == int8 kernels;
MHATokenToKVPoolTieredeager andlazy-VMM paths, ring reset on
lazy_release, a badSGLANG_KV_TIERS_Wrejected.int8 per-token 0.100, int8_g32 0.106, e4m3 0.110 (CAMPAIGN 2026-09-02 16:17).
+-0.008), oracle 0.0019; all-position +0.001 / 0.059; needle 41k 5/5 (CAMPAIGN 2026-09-02
16:45 and 21:05).
oracle 0.0015; needle 5/5 at 247,629 tokens (CAMPAIGN 2026-09-02 19:40 and 20:00).
oracle 0.0019; needles 5/5 at 41,370 and 5/5 at 247,629 tokens (CAMPAIGN 2026-09-02 21:05).
re-measured after the write-path fake-quant hook was found to be bypassed by the int8 pool
subclass; the K/V := 0 control caught it (CAMPAIGN 2026-09-02 19:40).
Reproduction (needs
ninjaonPATH(otherwiseFileNotFoundError: ninja) and an nvcc thatsupports the GPU architecture at
CUDA_HOME(compute_120a on the reference machine; a systemnvcc 12.0 does not, the virtualenv's 13.3 does); the lazy-VMM cases build a
load_inlinestub):Speed Tests and Profiling
Streaming bench, single request, ~10k context (
tools/bench_speed.py; card, "Performance"),measured on
73a255206fwith the flat patch and the FA2 decode route:2026-09-02 15:12).
(1,610 tok/s), decode 50.8 (16:45); the longest prompt admitted on the reference machine
162,215 tokens at prefill 95.8 s (1,694 tok/s), decode 51.2, VRAM free bottomed at 1.18 GB, a
~179k prompt refused at admission (17:00).
prompt prefill 182.2 s (1,415 tok/s), decode 51.9 (19:40 and 19:34); needle haystack of
247,629 random-word tokens prefill 269 s (920 tok/s, PLE rows from NVMe), decode 51.4 (20:00).
6,821 / 10,001), prefill 2,271 at 10k; 257,905-token prompt prefill 171 s (1,508 tok/s), decode
51.8; lazy backing 366,976 tokens profiled -> 262,144 admitted (21:05;
tiers-validate.log);with the rate-limited
empty_cacheof part 3: 165.3 s (1,560 tok/s), decode 52.3 (2026-09-0300:55). Per-chunk prefill time at 258k fits 615 ms + 0.49 us x prefix tokens; the O(prefix)
term is the QSA indexer, not the KV gather (
docs/ELASTIC_MEMORY.md, "Where the long-contextprefill time goes").
Not re-measured on
78c5024e9d, where SM120 decode runs FlashInfer XQA after the gather (seeModifications).
Checklist
F401,F821,UP037, codespell andpy_compileclean on the commit.)register_cuda_ci. Missing: an end-to-end test with a model, which needs a checkpoint.)--kv-cache-dtypevalues in the server-arguments documentation,SGLANG_KV_TIERS_W, the on-disk layouts outside the docstrings.)kv_bits/kv_tieredasgetattrclass attributes,SGLANG_KV_TIERS_Was an environment variable.)Suggested reviewers
python/sglang/srt/layers/attention(QSA kernels and backend): @Fridge003, @ishandhanani,@Qiaolin-Yu (NV and model-specific optimizations); @JustinTong0323 as author of Introduce Qwen 3.8 Flash Next #36497 and of
the SM120 routing changes (fix(qsa): route exact SM120 to FlashInfer sparse decode #36806, fix(qsa): restore SM121 correctness with Humanize and Kernel Design Agent #36845).
python/sglang/srt/mem_cache(pools, dtype, configurator): @ispobock, @xiezhq-hermann(KV Cache).
python/sglang/srt/model_executor(pool_configurator.py): @merrymercy, @hnyls2002, @cctry(Scheduler).
CI States
Latest PR Test (Base): ❌ Run #33750783183
Latest PR Test (Extra): ❌ Run #33750782869
Latest PR Test (AMD ROCm 10): ❌ Run #33750782984