diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx index 04531d84173a..9ac990c01cca 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx @@ -178,6 +178,7 @@ import { agentx } from "/src/snippets/agentx/zai-org/glm-5.2.jsx"; - **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points. - **PD Disaggregation (prefill/decode).** GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP. - **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider. +- **GB300 NVFP4, TP4, concurrency 1: add `--flashinfer-allreduce-fusion-backend cutedsl`.** It moves the fused all-reduce + residual add + RMSNorm from the default FlashInfer `mnnvl` kernels, which stop at 2048 tokens, to FlashInfer's CuTe DSL kernels, which also fuse larger prefill chunks. Added to the GB300 NVFP4 Agg TP4 concurrency-1 command in the [Agentic Long-Context section](#agentic-long-context-deployment), it lowered decode time per output token by about 3.4% in three runs without making TTFT worse, and did not significantly change GPQA-Diamond. It needs a build with [#45565](https://github.com/sgl-project/sglang/pull/45565), such as `nightly-dev-20261026-8252e3af` (tested) or newer; the image that section lists for this point, `nightly-dev-20261021-c1c287a4`, is older. On the older `nightly-dev-20261016-a93f16f0`, the flag raised p50 TTFT by about 9%. Re-measure before raising concurrency: in a microbenchmark the CuTe DSL kernel was slower than `mnnvl` at 16–20 and 96–256 tokens per call. With the flag, PyTorch kept about 8 GiB more cached memory per GPU while serving, leaving 0.45 GiB free for allocations outside PyTorch; only `--mem-fraction-static 0.83` was tested. The Deploy panel cells do not set this flag. - **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2). - **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the DSA triton backends (`--dsa-prefill-backend triton --dsa-decode-backend triton`, also SGLang's ROCm default), and the same `--chunked-prefill-size` and `--watchdog-timeout` as the FP8/BF16 recipes above. At `tp=4` this shape — 16 query heads per rank on an FP8 KV cache — is the one the gfx950 sparse-MLA kernels are tuned for; the `tp=8` recipes above stay on tilelang, which is what their published numbers were measured with. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified. diff --git a/docs/docs/advanced_features/server_arguments.mdx b/docs/docs/advanced_features/server_arguments.mdx index 84f07b035464..4263934e84db 100644 --- a/docs/docs/advanced_features/server_arguments.mdx +++ b/docs/docs/advanced_features/server_arguments.mdx @@ -1998,9 +1998,9 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec `--flashinfer-allreduce-fusion-backend` - Enable FlashInfer allreduce fusion and choose backend. Defaults to auto. 'auto': choose mnnvl on SM90 single-node systems and SM100/SM103 single-node or multi-node systems; choose trtllm otherwise. 'trtllm': available on single-node systems only. 'mnnvl': available on SM90 single-node systems and SM100/SM103 single-node or multi-node systems via MNNVL fabric. Fuses allreduce with Residual + RMSNorm for supported MoE models. + Enable FlashInfer allreduce fusion and choose backend. Requires SM90 or SM10X NVIDIA GPUs. When unspecified, SGLang turns it on for supported models only when the TP size is above 1, DP attention is off and no MoE all-to-all backend is set (multi-node only on Blackwell): supported MiMo configurations use cutedsl, DeepSeek-V4.1 (hidden size 5120) at TP4 on one Blackwell node keeps its custom all-reduce, and other supported models use auto. 'auto': choose mnnvl on Blackwell (SM100/SM103) systems (single- and multi-node) and trtllm on SM90 single-node systems. 'trtllm': available on single-node systems only. 'mnnvl': available on SM90 single-node systems and SM100/SM103 single-node or multi-node systems via MNNVL fabric. trtllm and mnnvl fuse batches of up to 2048 tokens. 'cutedsl': Blackwell-only bf16 MNNVL CuTe DSL backend for TP sizes 2, 4, 8 and 16; also fuses the MoE finalize and the shared-expert add into the collective when the MoE runner can defer them. It fuses decode, extend and target-verify batches up to its workspace size, the largest of `--max-prefill-tokens`, `--max-running-requests`, the CUDA graph batch sizes and the decode CUDA graph batch size × draft tokens. It turns off the trtllm/mnnvl fusion for the whole process, so speculative draft layers and other batches run unfused. Fuses allreduce with Residual + RMSNorm for supported MoE models. `None` - auto, trtllm, mnnvl + auto, trtllm, mnnvl, cutedsl `--enforce-disable-flashinfer-allreduce-fusion`