[akhilg-nv]

[by Claude Code]

Motivation

In every GLM-5.2 decoder layer, the attention output and the MLP output (an MoE in 75 of the 78 layers) are each summed across the tensor-parallel ranks (an all-reduce), added to the residual and normalized (RMSNorm). All-reduce fusion runs these three steps as one kernel. On one Blackwell node at TP4, SGLang turns it on by itself with --flashinfer-allreduce-fusion-backend auto, which resolves to FlashInfer's mnnvl backend (MNNVL is multi-node NVLink; these kernels exchange data over the NVLink fabric). That fusion covers batches of up to 2048 tokens; larger prefill chunks fall back to an NCCL all-reduce and a separate RMSNorm. The opt-in cutedsl backend uses FlashInfer kernels written in CuTe DSL (CUTLASS's Python kernel language) over the same fabric. It fuses batches up to --max-prefill-tokens (8192 in this deployment) and, for batches of up to 192 tokens, also folds in the MoE finalize (the weighted sum of the selected experts' outputs) and the shared-expert add.

This PR adds a Configuration Tips bullet to the GLM-5.2 cookbook that recommends cutedsl for one deployment: GB300 NVFP4, aggregated TP4, concurrency 1 (c1), a point of the page's Agentic Long-Context section. That section lists commands measured with AgentX, SemiAnalysis InferenceX's agentic-coding benchmark, which replays recorded multi-turn coding-agent sessions against the server. The bullet gives the measured effect, the builds the flag needs (those with #45565, which raised the default prefill CUDA-graph size for MLA models from 2048 tokens to min(8192, chunked-prefill size)), the limit of the evidence to c1, and a memory note. Defaults and runtime code do not change; the default stays auto because only this one deployment was measured. The PR also corrects the server-arguments row for this flag, which does not list cutedsl and says auto picks mnnvl on SM90 (the code picks trtllm).

Impact:

• Decode (measured): time per output token about 3.4% lower in three runs: −3.45% [95% CI −3.65, −3.18] (default vs default +0.04%), −3.41% [−3.59, −3.16] in a replicate, and −3.47% [−3.84, −2.99] in a shorter third run. GLM-5.2 NVFP4, one GB300 node, TP4, c1, AgentX replay, 148 paired requests.
• TTFT (measured): not worse. p90 −3.94% [95% CI −16.55, +4.43] and −4.63% [−16.22, −0.68] in the two 1,200-s runs, and +2.49% [−11.17, +67.82], within its noise, in the 900-s third run; p50 not worse in any run.
• Throughput (measured): +1.45% input + output tokens/s per GPU against the mean of the two default arms (replicate +1.38%).
• Accuracy (measured): GPQA-Diamond 0.9116 → 0.9003, −1.14 points [95% CI −2.78, +0.51], no significant change; MTP accept length 2.729 → 2.742.
• Scope (measured): c1 only. In a GB300 microbenchmark over 4–256 tokens per call, the CuTe DSL kernel was faster than mnnvl only at 4–12 tokens, and 1.05–1.13× slower at 16–20 and 1.05–1.10× slower at 96–256 tokens, so the bullet tells readers to re-measure before raising concurrency.
• Builds without [Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192 #45565 (measured): on nightly-dev-20261016-a93f16f0 the flag raised p50 TTFT by 9.0% (decode still −3.24%), so the bullet names a minimum build.
• Memory (measured): with the flag, PyTorch's caching allocator held about 8 GiB more idle cache per GPU while serving at --mem-fraction-static 0.83, leaving 0.45 GiB free for allocations outside PyTorch; emptying the cache brought both backends to the same level (within 0.8 GiB). No request failed.

Modifications

+3 / −2 docs (2 files); no runtime code, no tests. Base: main at f2d31889258d22214d9541e40b1e1e6961ce95e7.

• GLM-5.2 Configuration Tips (docs/cookbook/autoregressive/GLM/GLM-5.2.mdx): one new bullet. It says what the flag changes and where it was measured, gives the effect, names the build requirement ([Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192 #45565; the image the Agentic Long-Context section lists for this point, nightly-dev-20261021-c1c287a4, predates it), the older-build counter-example, the c1-only scope with the microbenchmark caveat and the memory note, and says that the Deploy panel cells do not set the flag. The bullet names nightly-dev-20261026-8252e3af (measured) or newer; the first nightly built after [Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192 #45565 merged, nightly-dev-20261024-5994c559, was not measured.

• Server-arguments row (docs/docs/advanced_features/server_arguments.mdx, --flashinfer-allreduce-fusion-backend): the description follows the field's help string on main and adds facts read from the code: when SGLang turns the fusion on by itself (including the DeepSeek-V4.1 TP4 exception), the 2048-token limit of trtllm and mnnvl, and for cutedsl its TP sizes, forward modes and workspace bound, and that it turns the other fusion off for the whole process, so speculative draft layers run unfused. The options column gains cutedsl. The old row's "Defaults to auto" and "mnnvl on SM90" did not match the code (auto picks trtllm on SM90).

  Where each fact in the row comes from (code on main)

  • Help string: python/sglang/srt/arg_groups/fields/exec_.py; the options match the field's Literal, and the default stays None.
  • Auto-enable: TP size above 1, no DP attention, no MoE all-to-all backend, multi-node only on Blackwell, and not for DeepSeek-V4.1 (hidden size 5120) at TP4 on one Blackwell node, which keeps its custom all-reduce (_flashinfer_allreduce_fusion_auto_enable and prefer_custom_dsv41 in arg_groups/overrides.py).
  • auto resolves to mnnvl on Blackwell and trtllm on SM90 single-node (_resolve_backend in layers/flashinfer_comm_fusion.py).
  • The 2048-token limit of the trtllm and mnnvl fusion: FUSE_ALLREDUCE_MAX_BATCH_SIZE in layers/layer_boundary/residual/add_norm.py.
  • cutedsl: TP sizes 2, 4, 8 and 16 (SUPPORTED_TP_SIZES in layers/flashinfer_mnnvl_cutedsl.py); decode, extend and target-verify batches only (_is_supported_forward_mode in layers/layer_boundary/fusions/cutedsl.py); workspace bound, the largest of --max-prefill-tokens, --max-running-requests, the CUDA graph batch sizes and the decode graph batch size × draft tokens (_resolve_max_m, cutedsl_moe_max_num_tokens); the other fusion is off for the whole process (uses_cutedsl_ar_fusion), and draft layers never get the CuTe DSL fusion (install_cutedsl_fusion returns None in speculative scope), so they run unfused.

• Not changed: the AgentX panel data (docs/src/snippets/agentx/zai-org/glm-5.2.jsx) and the Deploy panel. Each AgentX entry mirrors an InferenceX submission and links to the run that measured it. The GB300 Agg TP4 entry's run used nightly-dev-20261021-c1c287a4 (before [Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192 #45565) without this flag, so editing that entry here would attach a command to a run that did not measure it. The flag can enter that entry through the regular AgentX sync once an InferenceX run with it exists.

Checks.

• test/registered/unit/server_args/test_server_args_docs_coverage.py (the CPU CI test for server_arguments.mdx): 4 of 4 pass with this row on a CPU checkout of main at cb55d77d, and a misspelled flag in the new row makes test_no_stale_rows fail, so the test does read the row. Since then, main has changed none of the code this row describes (exec_.py, overrides.py and the fusion files have no all-reduce-fusion changes up to the base commit); CI reruns the test on this base.
• The pre-commit hooks that apply to .mdx files: no trailing whitespace, final newlines present, no merge markers, codespell clean on the added lines. The full pre-commit run was not completed locally.
• mint validate and mint broken-links in docs/

Accuracy Tests

GPQA-Diamond (measured). sgl-eval 0.1.3 (the version pinned in python/pyproject.toml): 198 questions × 4 repeats, thinking on, temperature 1.0, top_p 0.95 (the checkpoint's recommended sampling), max tokens 200,000, 32 client threads. Real MTP acceptance (no simulation). Server: the launch command under Speed Tests without the SGLANG_SIMULATE_ACC_* variables, plus --max-running-requests 32 --cuda-graph-max-bs-decode 32 so that the 32 threads run concurrently. The two arms differ only in the flag. Build: the replicate build described under Speed Tests (Builds). The two arms ran on different GB300 nodes on different days.

	default (auto → mnnvl)	cutedsl
GPQA-Diamond pass@1 (792 samples)	0.9116	0.9003
Paired difference per question (bootstrap, 10,000 draws)		−1.14 points, 95% CI [−2.78, +0.51]
MTP accept length (server counters)	2.7293	2.7420 (+0.47%)
Truncated responses	0.13%	0.38%
Mean completion tokens	23,542	23,951

The interval contains 0, so there is no significant change. At this sample size a drop of about 2.3 points is detectable with 80% power, so a loss of about 1 point cannot be excluded.

GPQA command and dataset

sgl-eval run gpqa --base-url http://127.0.0.1:30000/v1 --model nvidia/GLM-5.2-NVFP4 \
  --n-repeats 4 --num-threads 32 --temperature 1.0 --top-p 0.95 --max-tokens 200000 \
  --chat-template-kwarg enable_thinking=true --from-dataset gpqa_diamond_public.jsonl

The Hugging Face GPQA set is gated, so both arms read the same local file: the 198 GPQA-Diamond questions from the public OpenAI simple-evals CSV, formatted with sgl-eval's own GPQA prompt formatting (seed 42) and passed with --from-dataset. Absolute scores may therefore differ slightly from runs on sgl-eval's own copy; the comparison is paired.

Precision (code reading of SGLang 8252e3af and FlashInfer 0.7.2, plus server logs)

• No dtype or quantization change. The two CuTe DSL kernels SGLang calls (all-reduce + residual + RMSNorm, and MoE finalize + all-reduce + residual + RMSNorm) read and write BF16 and accumulate in FP32; neither has an FP8 or FP4 output.
• One narrow change. In a cutedsl process SGLang turns off its older MNNVL all-reduce for the whole process, so two non-fused all-reduces move: the target model's last-layer all-reduce (it feeds the final norm, the logits and the draft's input hidden states) and the draft (MTP) layer's all-reduces. Up to 1,365 tokens they now use SGLang's custom all-reduce, which also accumulates in FP32. For 1,366–2,048-token prefill chunks they use NCCL, which SGLang runs with NCCL_NVLS_ENABLE=0 (Ring/Tree, BF16 rounding at each hop) instead of MNNVL with FP32 accumulation. The default backend already sends every chunk above 2,048 tokens through that same NCCL path.
• The draft runs as shipped. The MTP layer never uses the CuTe DSL fusion (install_cutedsl_fusion returns None in speculative scope, and GLM-5.2's NextN layer skips it; the logs show 78 of 78 target layers fused on each rank), and SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set.
• Outputs are not bitwise identical to the default backend, because rounding points and summation order differ.

Speed Tests and Profiling

Which point, and why. The low-latency end: GB300 NVFP4, aggregated TP4, c1 (the GLM-5.2 GB300 TP4 point of the Agentic Long-Context section). SGLang applies this fusion only to TP deployments without DP attention or a MoE all-to-all backend, so the high-throughput DP-attention points do not use it at all. At c1 a decode step usually carries one request with 5 draft tokens, so 5–15 tokens per all-reduce. Its gain most likely comes from faster small all-reduces (inferred: in the microbenchmark below, the CuTe DSL kernel takes 0.86–0.89× the time of mnnvl at 4–12 tokens). The same microbenchmark shows it slower at 16–20 and 96–256 tokens, which higher concurrency reaches (for example 4 or more requests with 5 draft tokens), so this PR recommends the flag only for c1.

Why an AgentX replay rather than a static workload. TTFT depends on prefix-cache hits and on the sizes of the prefill chunks, and this flag's effect on TTFT depends on chunk size: the regression on the older build came from chunks above 2,048 tokens. In this trace 95.5% of prompt tokens were read from the prefix cache, and 2,049–8,192-token chunks were 203 of 290 prefill forwards and 94.5% of new prefill tokens (default arm; counts include the client's warm-up phase). A fixed-length random workload would not reproduce that mix. Decode time per token was measured in the same runs.

Setup (measured). One GB300 node, 4 GPUs, image lmsysorg/sglang:nightly-dev-20261026-8252e3af, model nvidia/GLM-5.2-NVFP4, the AgentX command for this point launched with sglang.launch_server (no Dynamo frontend), AgentX client 1.0.7 at c1, simulated MTP acceptance 3.33 (InferenceX's golden value). One node per run, fresh server per arm, 1,200 s windows, order: warm-up, default, cutedsl, default. Run 1 used the image's own SGLang; the replicate and the GPQA runs used a newer SGLang main tree laid over the same image, whose files on the flag's path are identical to the image's except for an opt-in row cap that was left unset (Builds, below).

Setup details, builds, launch and client commands

• One GB300 node, 4 GPUs. Image lmsysorg/sglang:nightly-dev-20261026-8252e3af (sha256:b4e4abb9e1dc038f7e14c85805be97e4f74abb46e541bd4edd31401d50a2e593), FlashInfer 0.7.2, PyTorch 2.14.1+cu130. Model nvidia/GLM-5.2-NVFP4.
• Builds. Run 1 used the image's own SGLang (8252e3af). The replicate and both GPQA arms used the same image with SGLang's Python tree replaced by a newer main (316ff4dce2, 47 commits after 8252e3af) plus one unmerged commit that adds an opt-in environment variable capping the rows the CuTe DSL fusion takes; the variable was unset in these runs, so behaviour is unchanged (code reading). On the flag's path, six files are identical in 316ff4dce2 and 8252e3af: layers/flashinfer_mnnvl_cutedsl.py, layers/flashinfer_comm_fusion.py, layers/layer_boundary/fusions/cutedsl.py, layers/layer_boundary/residual/add_norm.py, model_executor/runner/base_runner.py and distributed/bootstrap.py; the unmerged commit adds the cap to five of them. models/deepseek_v2.py, where GLM-5.2 installs the fusion, differs between the two only by a hasattr → is not None change from [Fix] deepseek-v4: Fix DeepSeek-V4 non-EP TBO for attention TP > 1 #33250.
• Client: AgentX agentic-coding trace replay (AgentX client 1.0.7), c1; requests carry enable_thinking=true, temperature 1 and top_p 0.95.
• Each arm starts a fresh server and measures a 1,200 s window. Order on one node: a warm-up arm (default), then default, cutedsl, default.
• MTP acceptance is simulated at 3.33, InferenceX's golden value for this model and draft length.
• Differences from the AgentX command for this point:
  • --flashinfer-allreduce-fusion-backend cutedsl (the cutedsl arm only);
  • sglang.launch_server serves the client directly, instead of dynamo.frontend in front of dynamo.sglang. With one worker the frontend has no routing choice to make; that it adds only host-side request handling is an estimate, not measured;
  • 1,200 s windows instead of InferenceX's 3,600 s; AgentX client 1.0.7 (InferenceX's published run used 1.0.12);
  • image nightly-dev-20261026-8252e3af instead of the listed nightly-dev-20261021-c1c287a4, which predates [Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192 #45565;
  • NCCL_MNNVL_ENABLE=1, which InferenceX's GB300 workers also had from their environment (the recipe does not set it);
  • --enable-metrics --enable-cache-report (logging only) and the SGLANG_SIMULATE_ACC_* variables, which InferenceX's harness also sets.
• Every arm: exit code 0, no errors in the server log, identical KV-cache size (2,571,968 tokens), and mean accept length 3.332 / 3.332 / 3.335. Every 2,049–8,192-token prefill chunk ran in a CUDA graph (203 / 204 / 203, including warm-up). For this model the prefill graphs are "breakable": captured segments replay as graphs and the few operations that cannot be captured, such as sparse attention, run eagerly between them; the all-reduce sites are inside the captured segments. Default arms log Auto-enabling FlashInfer AllReduce Fusion (auto) for GlmMoeDsaForCausalLM and backend=mnnvl. The cutedsl arm logs Prepared the FlashInfer MNNVL CuTe DSL fusion workspace for M_max=8192 (78 fused layers) on each of the 4 ranks and no mnnvl workspace.
• The cutedsl arm also logs MNNVL CuTe DSL: no shipped profile for tp=4 hidden=6144 top_k=8; re-targeting the GB300 presets at this shape ... benchmark against --flashinfer-allreduce-fusion-backend mnnvl before deploying a shape that matters. This A/B is that benchmark.
• The node also exported InfiniBand and socket NCCL settings, the same in every arm.

Launch command (the cutedsl arm; the default arms omit the last line):

NCCL_CUMEM_ENABLE=1 \
NCCL_MNNVL_ENABLE=1 \
SGLANG_TIMEOUT_KEEP_ALIVE=900 \
TORCH_DISTRIBUTED_DEFAULT_TIMEOUT=1800 \
SGLANG_SIMULATE_ACC_LEN=3.33 \
SGLANG_SIMULATE_ACC_METHOD=match-expected \
SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token \
python3 -m sglang.launch_server \
  --model-path nvidia/GLM-5.2-NVFP4 --served-model-name nvidia/GLM-5.2-NVFP4 \
  --host 127.0.0.1 --port 30000 \
  --trust-remote-code --quantization modelopt_fp4 \
  --tensor-parallel-size 4 --expert-parallel-size 1 \
  --kv-cache-dtype fp8_e4m3 --bf16-gemm-backend cutedsl \
  --chunked-prefill-size 8192 --max-prefill-tokens 8192 --context-length 1048576 \
  --mem-fraction-static 0.83 --max-running-requests 8 --cuda-graph-max-bs-decode 8 \
  --speculative-algorithm EAGLE --speculative-num-steps 4 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
  --enable-hierarchical-cache --hicache-size 100 --hicache-write-policy write_back \
  --hicache-io-backend direct --hicache-mem-layout page_first_direct \
  --tool-call-parser glm47 --reasoning-parser glm45 \
  --weight-loader-prefetch-checkpoints \
  --model-loader-extra-config '{"enable_multithread_load":true}' \
  --watchdog-timeout 1800 --enable-metrics --enable-cache-report \
  --flashinfer-allreduce-fusion-backend cutedsl

Client command (each arm):

AIPERF_REQUIRED_SERVER_METRIC_PREFIX=sglang: \
AIPERF_HTTP_TCP_USER_TIMEOUT=900000 \
agentx profile --url http://127.0.0.1:30000 --model nvidia/GLM-5.2-NVFP4 \
  --tokenizer nvidia/GLM-5.2-NVFP4 --tokenizer-trust-remote-code \
  --benchmark-duration 1200 --concurrency 1 \
  --extra-inputs '{"chat_template_kwargs":{"enable_thinking":true},"temperature":1,"top_p":0.95}' \
  --warmup-grace-period 1800 --public-dataset semianalysis_cc_traces_weka_062126

Results, run 1 (measured; the image's own SGLang). Each arm completed 149 requests; the 148 with at least 2 output tokens are paired by conversation and turn. Decode ms/token = (latency − TTFT) / (output tokens − 1). How each row compares cutedsl with the defaults:

• decode, per-request ratio: the median over requests of cutedsl / default, pooled over both default arms (296 ratios);
• percentile rows (decode p50 / p90, p90 interactivity, TTFT): the percentile of the 148 cutedsl requests against the same percentile of the 296 requests of both default arms taken together;
• throughput: the whole window, against the mean of the two default arms.

Intervals are 95% paired-bootstrap intervals that resample requests (2,000 draws; 10,000 for the per-request decode ratio, where each draw keeps both of a request's ratios). Noise = second default arm vs first. A change counts as beyond noise when it is larger than the noise and both cutedsl-vs-default pairs have the same sign.

Metric	default	cutedsl	default (repeat)	cutedsl vs defaults [95% CI]	Noise
Decode ms/token, per-request ratio (median)				−3.45% [−3.65, −3.18]	+0.04%
Decode ms/token p50	2.964	2.857	2.965	−3.62% [−4.08, −2.87]	+0.05%
Decode ms/token p90	3.694	3.618	3.704	−2.07% [−4.94, −1.24]	+0.28%
p90 interactivity, 1000 / p90 ITL (tok/s per user)	272.3	280.2	272.2	+3.06% [+1.83, +5.74]	−0.03%
TTFT p50 (ms)	245	226	236	−4.10% [−7.25, +1.97]	−3.98%
TTFT p90 (ms)	1,386	1,325	1,379	−3.94% [−16.55, +4.43]	−0.52%
TTFT p99 (ms)	4,114	3,995	4,132	−3.34% [−10.84, +0.32]	+0.44%
Throughput, whole window (input + output tok/s/GPU)	5,151	5,228	5,156	+1.45%	+0.10%

• Decode: both pairs agree (−3.43% and −3.46% per request), far outside the +0.04% noise.
• p90 TTFT: −3.94% [−16.55, +4.43]; both pairs are negative (−4.44% and −3.94%), beyond the 0.52% noise. The interval is wide because the p90 of 148 requests is set by the slowest 15 or so.
• p50 TTFT: the −4.10% is about the size of the difference between the two default arms (−3.98%), so we claim only that p50 TTFT is not worse.
• By prefill size (uncached prompt tokens), the per-request median TTFT change against the mean of the two default arms is negative in every bin, from −1.95% to −4.27%; above 8,192 new tokens it is −3.56%, with 0 of 28 requests slower.
• For reference, InferenceX's published result for this point (image nightly-dev-20261021-c1c287a4, Dynamo frontend, 3,600 s, client 1.0.12) is 5,173 tok/s/GPU at 279.3 tok/s/user p90 interactivity; the default arms here measured 5,151 and 5,156 tok/s/GPU at 272.3 and 272.2 (shorter window, no Dynamo frontend, older client).

Replicate, run 2 (measured; newer main tree)

Same image, flags, client and arm order, on another GB300 node, with the newer main tree described under Builds. 148 paired requests.

• Decode ms/token, per-request ratio: −3.41% [−3.59, −3.16] (noise +0.03%).
• p90 interactivity: +3.63% [+0.99, +11.80] (noise −0.05%).
• TTFT p90: −4.63% [−16.22, −0.68] (noise −0.21%).
• TTFT p50: −4.08% [−6.89, +2.97] (noise −3.89%, so again only "not worse").
• Throughput: 5,148 / 5,218 / 5,146 tok/s/GPU, +1.38% against the mean of the two default arms.

Run 3: memory check (measured; 900-s windows)

Same image (its own SGLang), flags and arm order on a third GB300 node, with 900-s windows and no warm-up, to test the memory note below; 106 paired requests. Its perf numbers are reported but were not a decision input.

• Decode ms/token, per-request ratio: −3.47% [−3.84, −2.99].
• TTFT p50: −5.02% (noise 0.32%). TTFT p90: +2.49% [−11.17, +67.82], within its noise (4.21%); the pairs disagree in sign (+3.17% and −1.00%).
• Throughput: 5,691 / 5,824 / 5,691 tok/s/GPU (+2.3%).

An earlier run, excluded

An earlier run on the same image was discarded: a shared-filesystem outage on the test cluster froze one default-backend arm for about 14 minutes. The server was waiting on a read from its Triton kernel cache, which sat on that filesystem; an unrelated process on the same node froze over the same interval, and both resumed when the storage recovered (attributed from timing and file-access records; no stack sample was taken). SGLang is not implicated. Over the 74 requests that arm completed, the run's valid default-vs-cutedsl pair agreed with the runs above (p50 decode time per token −3.24%).

Counter-example on an older build (measured). On nightly-dev-20261016-a93f16f0 (which reports SGLang c15edf31, the image's build commit; FlashInfer 0.7.1), prefill CUDA graphs stopped at 2,048 tokens, so larger chunks ran eagerly. The same A/B (default / cutedsl / default, 1,200 s each, one node) gave decode −3.24% [−3.50, −3.03] (per-request ratio), but raised TTFT p50 from 274.7 ms to 299.5 ms (+9.0%; +9.6% over the paired requests). #45565 raised the MLA prefill CUDA-graph cap to min(8192, chunked-prefill size). With it, every 2,049–8,192-token chunk in runs 1 and 2 ran in a CUDA graph, and p50 TTFT no longer rises. The mechanism is not isolated: the two builds also differ in other commits (for example #43216, which changes NCCL graph registration) and in FlashInfer version.

All-reduce microbenchmark on GB300 (measured)

One GB300 node, TP4, hidden size 6144, BF16, in the image under test with SGLang's own wrappers. Every arm runs in a CUDA graph; the time is the maximum over ranks, legs are interleaved, and the result is the median of leg medians. CuTe DSL all-reduce + residual + RMSNorm against the mnnvl fused kernel, with the CuTe DSL workspace sized for 256 tokens (at 256 tokens the 8,192-token workspace gives the same ratio, 1.08):

Tokens per call	4	8	12	16	20	24–64	96	128	160	192	224	256
CuTe DSL / mnnvl time	0.89	0.87	0.86	1.13	1.05	0.98–1.02	1.06	1.05	1.09	1.10	1.10	1.08

With the 8,192-token workspace, CuTe DSL took 0.92–1.00× the time of mnnvl at 384–2,048 tokens, and 0.71–0.77× the time of NCCL + RMSNorm (what the default runs there) at 3,072–8,192 tokens.

GPU memory headroom (measured)

• In each of the four cutedsl arms on this image (runs 1-3 and the excluded run), GPU memory in use while serving (nvidia-smi) settled about 8 GiB per GPU above the default arms: about 276.2 GiB against 268.2 GiB, out of 277.5 GiB. It then stayed flat for the rest of each window.
• It is PyTorch's cache. In run 3, after each arm's window we paused the server and called /continue_generation with torch_empty_cache. PyTorch's reserved memory went from 271.8 to 231.3 GiB with the flag and from 263.0 to 231.3 GiB without it (each rank logs it), and nvidia-smi then read 235.7 GiB with the flag against 236.5 GiB for both default arms. So the extra 8 GiB was idle cache, which PyTorch reuses and frees when an allocation fails; at startup the flag uses about 0.8 GiB less.
• What this does not show: whether the flag's peak working set during a forward is larger. Growth of the cache means only that the allocator once needed more segments, by peak or by fragmentation.
• Allocations outside PyTorch cannot use that cache. While it is held, about 0.45 GiB is free, and each cutedsl arm logged 64 warnings when Triton kernels were first loaded during serving (16 loads of 8 distinct kernels on each of 4 ranks, most at the first moment two requests were running): Triton kernel '...' device-loaded after serving started (free device mem: 0.45 GiB). Pre-load it during engine init to avoid CUDA OOM. No default arm logged it (threshold 1 GiB). Every load and request succeeded.
• The likely source of the cache (code reading, not tested): long-context prefills allocate large FP32 logits scratch blocks for the DSA indexer (GLM-5.2's sparse-attention token selector) in eager code, and which blocks the allocator keeps depends on the history of earlier allocations; the CuTe DSL workspace itself is about 267 MiB per rank.
• Only --mem-fraction-static 0.83 at c1 was tested with this flag. A lower fraction may not restore headroom, because the indexer's scratch budget during prefill graph replay is 0.2 × (1 − mem-fraction-static) × GPU memory (code reading), so the cache can grow into the freed memory too.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests. Not applicable: docs only; the existing docs-coverage test passes (see Checks).
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

AI assistance: this PR was prepared with Claude Code; claude-opus-5-5 was the only model used (the lead agent and all subagents).

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ✅ Run #40880328379
Latest PR Test (Extra): ❌ Run #40880328449
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mintlify]

✅ Preview approved

@nvpohanh approved a Mintlify preview for commit 67d6ad2.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
sglang-doc	🟢 Ready	View Preview	Nov 4, 2026, 11:34 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[nvpohanh]

@akhilg-nv why don't we make this the default backend?