Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs_new/cookbook/autoregressive/GLM/GLM-5.2.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: GLM-5.2
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, and GB300."
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, GB300, and AMD MI300X/MI325X/MI355X."
tag: NEW
---

Expand Down Expand Up @@ -100,6 +100,8 @@ import { Playground } from "/src/snippets/_playground.jsx";
- **BF16 weights need more GPUs (unverified).** The full-precision build (`zai-org/GLM-5.2`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM); on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). The BF16 recipes in the panel are **proposed/inferred, not yet benchmarked** (`verified: false`) — FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8.
- **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider.

- **AMD GPUs (MI300X / MI325X / MI355X).** Both FP8 (`zai-org/GLM-5.2-FP8`) and BF16 (`zai-org/GLM-5.2`) run single-node at `tp=8`. Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). **MTP / EAGLE speculative decoding is not currently usable on AMD for GLM-5.2** — the spec-decode kernel does not yet build for gfx950 — so omit the `--speculative-*` flags and serve without MTP. The Deploy panel's AMD cells already reflect this (no MTP). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2).

## 3. Advanced Usage

### 3.1 Reasoning
Expand Down
300 changes: 299 additions & 1 deletion docs_new/src/snippets/configs/zai-org/glm-5.2.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ export const config = {

supportedHardware: [
"h200", "b200", "gb300", "b300",
"mi355x", "mi325x", "mi300x",
],

// Single released checkpoint — no size/mode split.
Expand Down Expand Up @@ -90,6 +91,9 @@ sgl-eval run aime25 \\
b200: "lmsysorg/sglang:latest",
gb300: "lmsysorg/sglang:latest",
b300: "lmsysorg/sglang:latest",
mi355x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi35x-20260616",
mi325x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616",
mi300x: "lmsysorg/sglang-rocm:v0.5.13.post1-rocm700-mi30x-20260616",
},

github: {
Expand All @@ -106,7 +110,7 @@ sgl-eval run aime25 \\
knobs: [
{ id: "tp", label: "TP", values: [null, 4, 8] },
{ id: "cp", label: "CP (DSA prefill)", values: [null, 1, 2, 4, 8],
disable: { hw: ["b200", "gb300", "b300"] },
disable: { hw: ["b200", "gb300", "b300", "mi355x", "mi325x", "mi300x"] },
disableReason: "DSA prefill Context Parallel is verified on Hopper (H200); the Blackwell sm100 DSA-CP FP8 rope kernel is not yet adapted." },

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add disable reason for AMD?

{ id: "dpAttn", label: "DP-Attention",
values: [null, false, 4, 8],
Expand Down Expand Up @@ -628,5 +632,299 @@ sgl-eval run aime25 \\
"--port {{PORT}}",
],
},

// ====================================================================
// AMD MI300X / MI325X / MI355X (ROCm) — TP8, DSA tilelang backend.
// No MTP: EAGLE spec-decode kernel does not yet build for gfx950.
// Mirrors the GLM-5.1 ROCm recipe (model path swapped).
// ====================================================================
{
match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
verified: true,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
verified: true,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi355x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
verified: true,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi355x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi325x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi325x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi300x", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the bf16 fit in single node mi300x? might need multinodes

verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "high-throughput", nodes: "single" },
verified: false,
env: [],
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dsa-prefill-backend tilelang",
"--dsa-decode-backend tilelang",
"--chunked-prefill-size 131072",
"--mem-fraction-static 0.80",
"--watchdog-timeout 1200",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
],
};
Loading