Skip to content

docs(DeepSeek-V4): verify H200 Pro max-throughput recipe #23726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yhyang201 wants to merge 2 commits into from
Closed

docs(DeepSeek-V4): verify H200 Pro max-throughput recipe #23726

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
991e476
docs(DeepSeek-V4): verify H200 Pro max-throughput recipe
yhyang201 Apr 25, 2026
c4ece72
docs(DeepSeek-V4): add H200 multinode env var hints
yhyang201 Apr 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 27 additions & 5 deletions docs_new/src/snippets/autoregressive/deepseek-v4-deployment.jsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ export const DeepSeekV4Deployment = () => {
"gb300|small|max-throughput",
"h200|small|cp",
"h200|small|pd-disagg",
"h200|big|max-throughput",
"h200|big|pd-disagg",
"gb300|small|cp",
"gb300|big|cp",
Expand Down Expand Up @@ -262,7 +263,9 @@ export const DeepSeekV4Deployment = () => {
}
} else if (recipe === "max-throughput") {
if (hardware === "h200") {
recipeEnv.push("SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256");
recipeEnv.push(isBig
? "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128"
: "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256");
} else {
recipeEnv.push(isBig
? "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"
Expand Down Expand Up @@ -351,16 +354,22 @@ export const DeepSeekV4Deployment = () => {
if (!multinode) flags.push(DEEPEP_LARGE_SMS_FLAG);
} else if (recipe === "max-throughput") {
// allinone max-throughput: TP + DP + DP-attn + DeepEP (NO MTP).
// H200 small: cg=128 max-run=256 | H200 big: cg=128 max-run=256 (same)
// H200 small: cg=128 max-run=256 | H200 big: max-run=64 mem-frac=0.875
// B200 small: no cg/max-run | B200 big: cg=64 max-run=256
// GB300 small: no cg/max-run | GB300 big: cg=128 max-run=256
flags.push(` --tp ${tp}`);
flags.push(` --dp ${tp}`);
flags.push(" --enable-dp-attention");
if (multinode) flags.push(...multiNodeFlags(nnodes));
flags.push(" --moe-a2a-backend deepep");
if (isBig) flags.push(" --mem-fraction-static 0.82");
if (hardware === "h200") {
if (isBig && hardware === "h200") {
flags.push(" --mem-fraction-static 0.875");
} else if (isBig) {
flags.push(" --mem-fraction-static 0.82");
}
if (hardware === "h200" && isBig) {
flags.push(" --max-running-requests 64");
} else if (hardware === "h200") {
flags.push(" --cuda-graph-max-bs 128");
flags.push(" --max-running-requests 256");
Comment on lines +365 to 374
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Apr 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The logic for adding flags in the max-throughput recipe is slightly fragmented across multiple if blocks. While functional, consolidating the hardware === "h200" && isBig check would improve readability and maintainability, especially as more hardware-specific overrides are added.

      if (hardware === "h200" && isBig) {
        flags.push("  --mem-fraction-static 0.875");
        flags.push("  --max-running-requests 64");
      } else {
        if (isBig) flags.push("  --mem-fraction-static 0.82");
        if (hardware === "h200") {
          flags.push("  --cuda-graph-max-bs 128");
          flags.push("  --max-running-requests 256");
        } else if (isBig && hardware === "b200") {
          flags.push("  --cuda-graph-max-bs 64");
          flags.push("  --max-running-requests 256");
        } else if (isBig && hardware === "gb300") {
          flags.push("  --cuda-graph-max-bs 128");
          flags.push("  --max-running-requests 256");
        }
      }

} else if (isBig && hardware === "b200") {
Expand Down Expand Up @@ -416,7 +425,20 @@ export const DeepSeekV4Deployment = () => {
const envAll = [...HW_ENV, ...recipeEnv, ...COMMON_ENV];
const envBlock = envAll.length ? envAll.join(" \\\n") + " \\\n" : "";
const base = `${envBlock}sglang serve \\\n${flags.join(" \\\n")}`;
const withMultinode = multinode ? prependMultiNodeNote(base, nnodes) : base;
// H200 big is multinode and may need machine-specific NVSHMEM / NCCL / Gloo
// env vars; emit them as commented hints above the env block.
let cmd = base;
if (hardware === "h200" && multinode) {
cmd =
`# The following env vars may be needed depending on your cluster:\n` +
`# NVSHMEM_ENABLE_NIC_PE_MAPPING=1\n` +
`# NVSHMEM_HCA_LIST=<your-hca-list>\n` +
`# GLOO_SOCKET_IFNAME=<your-nic>\n` +
`# NCCL_SOCKET_IFNAME=<your-nic>\n` +
`# NCCL_IB_HCA=<your-hca-list>\n` +
cmd;
}
const withMultinode = multinode ? prependMultiNodeNote(cmd, nnodes) : cmd;
const verifyKey = `${hardware}|${modelSize}|${recipe}`;
if (TBD_RECIPES.has(verifyKey)) return TBD_PLACEHOLDER;
return VERIFIED_RECIPES.has(verifyKey)
Expand Down
Loading