[NV] Add GLM-5 NVFP4 GB200 disaggregated Dynamo TensorRT-LLM MTP benchmarks / [NV] 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TensorRT-LLM MTP 基准测试

[xinli-sw]

Opened on behalf of @richardhuo-nv. / 代表 @richardhuo-nv 提交。

Summary

• Add the glm5-fp4-gb200-dynamo-trt-mtp multi-node configuration for GLM-5 NVFP4 on GB200 using disaggregated TensorRT-LLM through Dynamo.
• Add 23 checked-in srt-slurm recipes: 13 ISL 1K / OSL 1K configurations and 10 ISL 8K / OSL 1K configurations.
• Pin the GB200 launcher to NVIDIA/srt-slurm v1.0.26 and overlay the checked-in recipes/trtllm/glm5 recipes.
• Use nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13 with nvidia/GLM-5-NVFP4.
• Enable chat-template benchmarking for every MTP recipe.
• Validated by Run Sweep #29443720859: all 23 throughput jobs, 8 evaluation jobs, collection jobs, and comparison jobs passed.
• The upstream-engine-first ordering requirement is satisfied by the merged SGLang submission in PR #2115.

中文说明

• 新增 glm5-fp4-gb200-dynamo-trt-mtp 多节点配置，在 GB200 上通过 Dynamo 运行分离式 TensorRT-LLM，为 GLM-5 NVFP4 提供 MTP 基准测试。
• 新增 23 个已检入的 srt-slurm recipe：13 个 ISL 1K / OSL 1K 配置和 10 个 ISL 8K / OSL 1K 配置。
• GB200 启动器固定使用 NVIDIA/srt-slurm v1.0.26，并覆盖使用仓库内的 recipes/trtllm/glm5 recipe。
• 使用 nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13 镜像和 nvidia/GLM-5-NVFP4 模型。
• 所有 MTP recipe 均启用 chat template 基准测试。
• 已通过 Run Sweep #29443720859 验证：23 个吞吐量任务、8 个评估任务、结果收集任务和结果对比任务全部通过。
• 上游推理引擎优先的顺序要求已由合并的 SGLang PR #2115 满足。

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1bda6cd. Configure here.}

[cursor]

GB200 launcher rejects glm5 TRT

High Severity

The new entry wires model-prefix glm5, runner gb200, and framework dynamo-trt, but runners/launch_gb200-nv.sh only supports gptoss, dsr1, and kimik2.5 for that framework and exits for glm5. GB200 sweep jobs for this config fail immediately, so none of the added MTP scenarios can run.

 

^{Reviewed by Cursor Bugbot for commit 1bda6cd. Configure here.}

[functionstackx]

waiting on glm5 nvfp4 gb200 mtp sgl PR first

would like prefer if @SemiAnalysisAI/core quick whip up "best effort/good faith attempt" at an sgl GLM gb200 disagg mtp config rn and then merge then first in so then we can merge in trtllm?

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27664500375
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27664500375

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29053553616
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29053553616

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29055719605
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29055719605

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29061622425
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29061622425

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29061911005
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29061911005

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29061911005
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29061911005

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29443720859
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29443720859

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29443720859
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29443720859

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29443720859
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29443720859

[Ankur-singh]

/reuse-sweep-run 29443720859

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Run Sweep #29443720859
• Verified that this PR passes evals. Run Sweep #29443720859
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• Run Sweep #29443720859 passed all 23 throughput jobs, 8 evaluation jobs, collection jobs, and comparison jobs for the exact PR head. / Run Sweep #29443720859 在当前 PR 的准确 head 上通过了全部 23 个吞吐量任务、8 个评估任务、结果收集任务和结果对比任务。
• All 23 MTP recipes set use_chat_template: true. This is a fixed-sequence multi-node submission rather than an AgentX workload, so the AgentX synthetic-acceptance and golden-AL requirement is not applicable. / 全部 23 个 MTP recipe 均设置了 use_chat_template: true。本 PR 是固定序列长度的多节点提交，并非 AgentX 工作负载，因此 AgentX 的模拟接受率和金标准 AL 要求不适用。
• The prior CHANGES_REQUESTED review asked for an SGLang-first submission. That condition is now satisfied by merged PR #2115, although GitHub still retains the earlier review state. / 之前的 CHANGES_REQUESTED review 要求优先提交 SGLang 配置。合并的 PR #2115 已满足该条件，但 GitHub 仍保留之前的 review 状态。

Signed: Ankur-singh

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — signer Ankur-singh is a named owner of configs/nvidia-master.yaml; all other changed paths fall only under the catch-all, which any recognized CODEOWNER satisfies.
✅ Check 1 (green sweep on in-PR commit): PASS — PR head 851bd20 (unchanged since sign-off) carries success, non-skipped executed check-runs: 23× multi-node 1k1k/8k1k / and 8× multi-node eval / from run 29443720859 (single-node/eval lanes skipped as expected — this PR has no single-node configs).
✅ Check 2 (evals pass): PASS — downloaded agg_eval_all.json: 8/8 GSM8K evals for GLM-5-NVFP4, em_strict 0.963–0.970, run on this PR's image nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0-dev.1-cuda13.
➖ Check 3 (recipe linked & merged): N/A — disaggregated/multi-node submission (all recipes under benchmarks/multi_node/srt-slurm-recipes/**; master entry has multinode: true, disagg: true, framework: dynamo-trt); the recipe-link requirement applies to single-node recipes only.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 29443720859 posted by Ankur-singh (COLLABORATOR).
✅ Check 5 (latest checklist template): PASS — sign-off matches the current docs/PR_REVIEW_CHECKLIST.md item-for-item, all checked.
✅ Check 6 (upstream images / engine-first): PASS — entry is framework: dynamo-trt so the upstream vLLM/SGLang image rule doesn't apply to it; engine-first ordering satisfied by merged SGLang entries for the same model+SKU (glm5-fp4-gb200-dynamo-sglang / -mtp, from PR #2115, merged 2026-07-08).
✅ Check 7 (no architecture hacks): PASS — no hf-overrides / model-config edits anywhere in the diff; MTP uses the model's native num_nextn_predict_layers.
✅ Check 8 (spec-decode via chat template): PASS — all 23 MTP recipes set use_chat_template: true in the benchmark client.
✅ Check 9 (no engine patches): PASS — no .patch/sed/heredoc/wheel installs; the launcher change only pins the srt-slurm orchestration repo at v1.0.26 and copies this PR's checked-in recipe YAMLs into it — the TRT-LLM container runs as shipped.
➖ Check 10 (agentic golden AL): N/A — spec-decode but fixed-seq-len (no agentic configs), and no simulated/synthetic-acceptance knobs are introduced.