[Docs] GLM-5.2: move GB300 AgentX recipe to the agentic HiCache section and drop no-op settings by nvpohanh · Pull Request #42167 · sgl-project/sglang

nvpohanh · 2026-10-02T01:35:03Z

[by Claude Code]

Motivation

#42109 added the InferenceX GB300 GLM-5.2 NVFP4 AgentX settings to the general-purpose GB300 cells of the command generator. Most of the added env vars and flags were no-ops or repeated SGLang's defaults, and an agentic HiCache recipe fits better in §3.6 Agentic Long-Context with HiCache DRAM Offload (NVFP4, MTP), next to the existing B300/B200 recipes (the same layout as DeepSeek-V4 §3.6). Follows up on the review comments in #42109.

Modifications

Revert [Docs] GLM-5.2 GB300 NVFP4: add env vars from InferenceX AgentX recipe #42109 in docs/src/snippets/configs/zai-org/glm-5.2.jsx. The three GB300 NVFP4 single-node cells go back to their earlier state.
Add a GB300 (TP4, single node) block to §3.6 of docs/cookbook/autoregressive/GLM/GLM-5.2.mdx with three commands, all using HiCache DRAM offload (--hicache-size 135, write_back, direct):
- Low-latency: TP4, MTP 4-1-5, max 16 running requests, --mem-fraction-static 0.8 (the [Docs] GLM-5.2 GB300 NVFP4: add env vars from InferenceX AgentX recipe #42109 low-latency cell).
- Balanced: TP4 + DP attention, MTP 2-1-3, max 256 running requests.
- High-throughput: TP4 + DP attention, no MTP, max 512 running requests.

Env vars and flags from #42109 left out, because they do nothing on current main or repeat the default for this model on GB300:

Setting	Why it is not needed on `main`
`SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK`, `SGLANG_ENABLE_THINKING`	Deprecated aliases removed in #40795; nothing reads them.
`SGLANG_HICACHE_DEBUG_LOG`, `SGLANG_HICACHE_DEBUG_SAMPLE_RATE`	Not read anywhere in the codebase.
`SGLANG_ENABLE_UNIFIED_RADIX_TREE`	Deprecated; the unified radix tree is the default since #35081.
`SGLANG_MOE_NVFP4_DISPATCH`	No effect without an all-to-all backend: the standard dispatcher only uses the FP4 input scale on the `flashinfer_cutlass` all-gather path, and the `flashinfer_trtllm` runner quantizes its own input.
`--kv-cache-dtype fp8_e4m3`	Default for DSA models on SM10x (`_dsa_kv_cache_dtype_default`).
`--dsa-prefill-backend trtllm`, `--dsa-decode-backend trtllm`	Default for DSA with an FP8 KV cache on SM10x.
`--moe-runner-backend flashinfer_trtllm`	Default for `modelopt_fp4` DeepSeek-family MoE on Blackwell with no all-to-all backend.
`--disable-shared-experts-fusion`	Set automatically for the `flashinfer_trtllm` MoE runner.
`--flashinfer-allreduce-fusion-backend auto`	Enabled automatically for `GlmMoeDsaForCausalLM` at TP>1 without DP attention on SM90/SM10x.

Kept: --fp4-gemm-backend flashinfer_trtllm in the low-latency command, because the default on SM10x is flashinfer_cutedsl.

Accuracy Tests

N/A — docs-only change.

Speed Tests and Profiling

N/A — docs-only change. I have not re-run the §3.6 GB300 commands as written. The low-latency settings come from the InferenceX GB300 AgentX recipe (SemiAnalysisAI/InferenceX#3178). The balanced and high-throughput commands are the existing generator cells with HiCache added.

Checklist

Format your code according to the Format code with pre-commit. (All hooks pass except the three Rust hooks, which could not run because no compatible Rust toolchain is available locally; this PR changes no Rust files.)
Add unit tests according to the Run and add unit tests. (N/A, docs only)
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed. (N/A, docs only)
Follow the SGLang code style guidance.

🤖 Generated with Claude Code

CI States

Latest PR Test (Base): ✅ Run #37084394597
Latest PR Test (Extra): ❌ Run #37084394519
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

…tX recipe (sgl-project#42109)" This reverts commit 8acd36b.

Add a GB300 (TP4, single node) block with low-latency, balanced, and high-throughput commands to "Agentic Long-Context with HiCache DRAM Offload (NVFP4, MTP)". The commands omit env vars and flags that only repeat SGLang defaults (FP8 KV cache and TRT-LLM DSA backends for DSA models on Blackwell, flashinfer_trtllm MoE runner, shared-expert fusion off, FlashInfer allreduce fusion) and keep --fp4-gemm-backend flashinfer_trtllm, which differs from the flashinfer_cutedsl default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nvpohanh · 2026-10-02T01:45:26Z

cc @faradawn

faradawn · 2026-10-02T19:04:11Z

+
+**GB300 (TP4, single node):** one 4-GPU GB300 node serves the whole model. All three commands offload KV cache to host DRAM with `--hicache-size 135` (GB of host memory per rank, 540 GB per node). GLM-5.2 is a DSA model, so on Blackwell SGLang already defaults to an FP8 E4M3 KV cache and TRT-LLM DSA attention; the commands do not need to set them.
+
+Low-latency (TP4, MTP 4-1-5, up to 16 concurrent requests):
+```bash Command
+sglang serve \
+  --model-path nvidia/GLM-5.2-NVFP4 \
+  --trust-remote-code \
+  --tp 4 \
+  --quantization modelopt_fp4 \
+  --fp4-gemm-backend flashinfer_trtllm \
+  --max-running-requests 16 \
+  --cuda-graph-max-bs-decode 16 \
+  --chunked-prefill-size 8192 \
+  --max-prefill-tokens 8192 \
+  --mem-fraction-static 0.8 \
+  --tool-call-parser glm47 \
+  --reasoning-parser glm45 \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 4 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 5 \
+  --enable-hierarchical-cache \
+  --hicache-size 135 \
+  --hicache-write-policy write_back \
+  --hicache-io-backend direct
+```
+
+Balanced (DP attention, MTP 2-1-3, up to 256 concurrent requests):
+```bash Command
+sglang serve \
+  --model-path nvidia/GLM-5.2-NVFP4 \
+  --trust-remote-code \
+  --tp 4 \
+  --attn-dp-size 4 \
+  --quantization modelopt_fp4 \
+  --max-running-requests 256 \
+  --chunked-prefill-size 8192 \
+  --mem-fraction-static 0.92 \
+  --tool-call-parser glm47 \
+  --reasoning-parser glm45 \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 2 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 3 \
+  --enable-hierarchical-cache \
+  --hicache-size 135 \
+  --hicache-write-policy write_back \
+  --hicache-io-backend direct
+```
+
+High-throughput (DP attention, no MTP, up to 512 concurrent requests):
+```bash Command
+sglang serve \
+  --model-path nvidia/GLM-5.2-NVFP4 \
+  --trust-remote-code \
+  --tp 4 \
+  --attn-dp-size 4 \
+  --quantization modelopt_fp4 \
+  --max-running-requests 512 \
+  --chunked-prefill-size 8192 \
+  --mem-fraction-static 0.92 \
+  --tool-call-parser glm47 \
+  --reasoning-parser glm45 \
+  --enable-hierarchical-cache \
+  --hicache-size 135 \
+  --hicache-write-policy write_back \
+  --hicache-io-backend direct
+```
+
+With `--attn-dp-size 4`, `--chunked-prefill-size` is a global budget divided across the attention DP ranks, so the balanced and high-throughput commands prefill 2048 tokens per rank per chunk.


The flags looks great. Do we want to move these configs into the command builder?

If you prefer it to be in this static markdown section, I'm also fine.

The InferenceX GB300 AgentX aggregate points (concurrency 1, 2 and 4) passed with EAGLE 5-1-6 (SemiAnalysisAI/InferenceX#3659). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nvpohanh · 2026-10-06T01:40:25Z

Merging to unblock SA submission. Let me know if anyone has concerns.

nvpohanh and others added 2 commits October 1, 2026 18:29

Revert "[Docs] GLM-5.2 GB300 NVFP4: add env vars from InferenceX Agen…

4aa2088

…tX recipe (sgl-project#42109)" This reverts commit 8acd36b.

nvpohanh requested review from JustinTong0323, sogalin, wisclmy0611 and zijiexia as code owners October 2, 2026 01:35

github-actions Bot added the documentation Improvements or additions to documentation label Oct 2, 2026

faradawn reviewed Oct 2, 2026

View reviewed changes

[Docs] GLM-5.2 GB300 low-latency: use MTP 5-1-6

287b189

The InferenceX GB300 AgentX aggregate points (concurrency 1, 2 and 4) passed with EAGLE 5-1-6 (SemiAnalysisAI/InferenceX#3659). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

faradawn approved these changes Oct 5, 2026

View reviewed changes

nvpohanh merged commit f2f0be7 into sgl-project:main Oct 6, 2026
97 of 101 checks passed

Ankur-singh mentioned this pull request Oct 6, 2026

docs: clarify recipe coverage across topologies and Dynamo / 澄清各拓扑及 Dynamo 的 recipe 文档要求 SemiAnalysisAI/InferenceX#3743

Merged

5 of 11 tasks

Navigation Menu

Repository navigation

[Docs] GLM-5.2: move GB300 AgentX recipe to the agentic HiCache section and drop no-op settings - #42167

[Docs] GLM-5.2: move GB300 AgentX recipe to the agentic HiCache section and drop no-op settings#42167
nvpohanh merged 3 commits into
sgl-project:mainfrom
nvpohanh:docs/glm5.2-gb300-agentic-hicache

nvpohanh commented Oct 2, 2026 •

edited by github-actions Bot

Loading

Uh oh!

nvpohanh commented Oct 2, 2026

Uh oh!

faradawn Oct 2, 2026

Uh oh!

nvpohanh commented Oct 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nvpohanh commented Oct 2, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

CI States

Uh oh!

nvpohanh commented Oct 2, 2026

Uh oh!

faradawn Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

nvpohanh commented Oct 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nvpohanh commented Oct 2, 2026 •

edited by github-actions Bot

Loading