Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ docker run \

For **MI355X → Pro / Pro Official → FP4 → Multi-Nodes**, the panel generates a **1P1D deployment**: one prefill node and one decode node, each running an independent worker. Select **Router**, **Prefill**, or **Decode** to copy its command. Start the [UMBP tier](#3-8-unified-cache-external-linker-umbp) on the prefill node, then both workers, then the router. Send client requests to the router on port `8000`.

The MI355X multi-node strategies follow the [InferenceX AgentX submission](https://github.com/SemiAnalysisAI/InferenceX/pull/3664): **Low-Latency**, **Balanced**, and **High-Throughput** use fixed concurrency presets of **16**, **48**, and **256**, respectively. **Pro** uses EAGLE/MTP; **Pro Official (0813)** uses DSpark, matching the InferenceX checkpoint. Both enable UMBP on prefill by default. The Playground inherits the selected worker, including its speculative decoder and cache settings.
The MI355X multi-node strategies **Low-Latency**, **Balanced**, and **High-Throughput** use fixed concurrency presets of **16**, **48**, and **256**, respectively. **Pro** uses EAGLE/MTP; **Pro Official (0813)** uses DSpark. Both enable UMBP on prefill by default. The Playground inherits the selected worker, including its speculative decoder and cache settings.

Run these recipes' **Python** commands inside the ROCm image shown in the **Docker** tab, where Aiter is already enabled. Add explicit parser and network-device settings through the Playground's **Parsers** and **PD Disagg → IB Device** controls.

Expand Down Expand Up @@ -950,17 +950,17 @@ docker run \
--ulimit nofile=1048576:1048576 \
--network host --ipc=host --shm-size 32g \
-v ~/.cache/huggingface:/root/.cache/huggingface \
lmsysorg/sglang-rocm:v0.5.21-rocm724-mi35x-20261001 \
lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261019 \
python3 -m sglang.launch_server <role args below>
```

The Deployment and Playground Docker outputs retain this `v0.5.21` base image for the MI355X Pro and Pro Official FP4 multi-node recipes. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts.
Use this image for the **MI355X Pro / Pro Official FP4 multi-node** workers and router, including the additional concurrency points below. The Deployment and Playground Docker outputs select it for all three PD roles. It supports the documented `SGLANG_DSV4_DENSE_MXSCALE` and `SGLANG_ROUTER_P2C_LOAD_METRIC` settings. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts.

For the **MI355X Pro / Pro Official FP4** presets, **High-Throughput** sets `MORI_IO_QP_MAX_SEND_WR=32767` and `MORI_IO_QP_MAX_CQE=32768` on both MORI roles. Low-Latency and Balanced use the transport's default queue sizes. The send-queue backoff timeout uses MORI's default of 5 seconds.

#### Role-based configuration

Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. The commands use the [submitted recipe at `df5e433239d7`](https://github.com/SemiAnalysisAI/InferenceX/blob/df5e433239d72106529ece4dd97cc36e4c3adf4c/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/mi355x-fp4/agentx/disagg-umbp-dspark.yaml), with cluster-specific paths replaced by local placeholders and benchmark acceptance simulation removed. The Pro Official commands additionally apply the kernel and routing updates described below.
Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. Replace the host and model-path placeholders with your deployment's values. The Pro Official commands use the kernel and routing settings described below.

| Strategy | Concurrency | Prefill → Decode | `--max-running-requests` | DSpark block size |
|---|---|---|---|---|
Expand All @@ -981,7 +981,7 @@ Selecting a TP-only role forces **DP-Attention** off and greys the control. `--m
</Note>

<Warning>
**Speculative decoding changes how you size `--max-running-requests`.** For a target concurrency of N, set it to `N*2` on **both** roles; at N the served batch stays below the concurrency you are aiming for. This holds for DSpark and EAGLE/MTP alike. The table above follows it: 32 serves a concurrency of 16, 96 serves 48, and 512 serves the high-throughput concurrency of 256.
**Speculative decoding changes how you size `--max-running-requests`.** For the strategy presets above, a target concurrency of N uses `N*2` on **both** roles; at N the served batch stays below the concurrency you are aiming for. This holds for DSpark and EAGLE/MTP alike. The table above follows it: 32 serves a concurrency of 16, 96 serves 48, and 512 serves the high-throughput concurrency of 256. The Pro Official c1 override below is an exception: retain 8 request slots on each role even though client concurrency is 1.

The decode role's `--cuda-graph-bs-decode` then has to cover `N*2 / attn_dp_size`, the per-rank batch. The strategy presets apply both rules: decode graphs cover 1–32, 1–96, and 1–64, respectively.
</Warning>
Expand Down Expand Up @@ -1143,7 +1143,7 @@ The multi-node router presets retain the default request timeout. The standalone

The panel keeps its c16 / c48 / c256 strategy presets. Apply the following overrides to the **Multi-Nodes** commands to reproduce the optimized recipe's endpoints. Here `cN` means a target concurrency of N.

**c1 (replaces c4):** start from Low-Latency. Keep prefill TP4, set decode to `--tp 4`, and set `--max-running-requests 2` on both roles. Replace decode's graph list with `--cuda-graph-bs-decode 1 2`. Keep DSpark block size 6 (7 draft tokens), `MORI_MAX_DISPATCH_TOKENS_DECODE=448`, and both kernel switches on both roles. The source also sets `SGLANG_MLA_HCA_KV_SPLITS=8` on both roles; the FP8 KV path does not read this setting. Benchmark with concurrency 1.
**c1 (replaces c4):** start from Low-Latency. Keep prefill TP4, set decode to `--tp 4`, and set `--max-running-requests 8` on both roles. Replace decode's graph list with `--cuda-graph-bs-decode 1 2 3 4 5 6 7 8`. Keep DSpark block size 6 (7 draft tokens), `MORI_MAX_DISPATCH_TOKENS_DECODE=448`, and both kernel switches on both roles. The source also sets `SGLANG_MLA_HCA_KV_SPLITS=8` on both roles; the FP8 KV path does not read this setting. Keep these 8 server slots and graph sizes when benchmarking with client concurrency 1; do not reduce them to 2 using the strategy presets' `N*2` rule.

**c32:** start from Low-Latency, keep TP4 → TP8 and block size 6, set both admission limits to 64, and capture decode graph sizes 1–64. Both roles use FP8 KV and mxscale, as at c16 and c48.

Expand Down
12 changes: 5 additions & 7 deletions docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,9 @@ export const config = {
{ id: "multi-2", label: "Multi-Nodes" },
],

// AgentX 1P1D recipe: InferenceX PR #3664, commit df5e433239d7.
// The strategy presets use c16 / c48 / c256. Pro uses its MTP head;
// Pro Official is the InferenceX checkpoint. Acceptance simulation is omitted.
// Pro Official kernel/routing updates follow the optimized AgentX recipe;
// see section 3.7 for the c1 / c512 overrides.
// MI355X 1P1D strategy presets use c16 / c48 / c256.
// Pro uses its MTP head; Pro Official uses DSpark.
// See section 3.7 for the Pro Official c1 / c512 overrides.
overlayDims: [
{ id: "pdMode", title: "PD role", default: "prefill",
showWhen: (s) => s.hw === "mi355x" && s.quant === "fp4"
Expand All @@ -73,7 +71,7 @@ export const config = {
const official = sel.variant === "pro-official";
const gamma = sel.strategy === "low-latency" ? 6 : 3;
const tp = dp || !prefill ? 8 : 4;
const image = "lmsysorg/sglang-rocm:v0.5.21-rocm724-mi35x-20261001";
const image = "lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261019";
const router = [
...(official && dp ? ["env SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests"] : []),
"python3 -m sglang_router.launch_router",
Expand All @@ -93,7 +91,7 @@ export const config = {
router: { port: 8000, command: router } };
if (role === "router") return { ...common,
commands: { python: router, docker: `docker run --network host --rm ${image} ${router}` } };
// Emit overrides only. Defaults checked against the pinned image:
// Emit overrides only. Baseline defaults originally checked against
// SGLang 3b2ad1c6ae, MORI 879983bdbd8c. ROCm model setup enables the
// Aiter indexer; MegaMoE MTPR defaults to 8192 and static heap is implicit.
// This Aiter MegaMoEV2 uses mori_shmem_create_tensor, not the CCO allocator.
Expand Down
Loading