Add GLM-5 NVFP4 GB200 disaggregated Dynamo TRT-LLM STP benchmarks / 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TRT-LLM STP 基准测试 - #1803
Conversation
|
Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you
PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers. If additional help is needed, PR authors can reach out to core maintainers over Slack. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| export SRT_SLURM_MODEL_PREFIX="nvidia/GLM-5-NVFP4" | ||
| else | ||
| echo "Unsupported model prefix: $MODEL_PREFIX. Supported prefixes are: gptoss, dsr1, or kimik2.5" | ||
| echo "Unsupported model prefix: $MODEL_PREFIX. Supported prefixes are: gptoss, dsr1, kimik2.5, or glm5" |
There was a problem hiding this comment.
GLM5 missing srt-slurm checkout
High Severity
The new glm5-fp4-gb200-dynamo-trt sweep uses CONFIG_FILE paths under recipes/GLM5/... on NVIDIA srt-slurm sa-submission-q2-2026, but launch_gb200-nv.sh only selects that repo for kimik2.5 dynamo-trt. glm5 falls through to the cquil11/srt-slurm-nv clone, so recipe files are missing and the launcher exits when validating CONFIG_FILE.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit e99f46e. Configure here.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0137194. Configure here.
| - "Align MiniMax-M3 B200 vLLM fixed-sequence serving with MiniMax-M2.5 FP8 B200 settings by setting VLLM_FLOAT32_MATMUL_PRECISION=high and restoring max cudagraph capture size 2048." | ||
| - "Add TP4+EP4 coverage for MiniMax-M3 B200: DP-attention rows for 1k1k/8k1k and the missing non-DP-attention row for 8k1k." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1779 | ||
|
|
There was a problem hiding this comment.
Duplicate changelog triggers wrong sweep
Medium Severity
This commit appends a second perf-changelog.yaml block for minimaxm3-fp8-b200-vllm that matches the entry already at lines 3853–3858. process_changelog.py builds the PR sweep from added diff lines only, so that duplicate is treated as a new changelog contribution and schedules minimaxm3-fp8-b200-vllm benchmarks alongside glm5-fp4-gb200-dynamo-trt.
Reviewed by Cursor Bugbot for commit 0137194. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27664455130 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27664515704 |
…non-MTP)
Adds `glm5-fp4-gb200-dynamo-trt` covering 13 ISL1K/OSL1K + 10 ISL8K/OSL1K STP
recipes from NVIDIA/srt-slurm `sa-submission-q2-2026` under
`recipes/GLM5/disagg/trtllm_dynamo/gb200_nvfp4/{ISL1K_OSL1K,ISL8K_OSL1K}/STP`.
Model naming aligned with the GLM-5.1 convention already used by the GB200
dynamo-sglang entry: `model: nvidia/GLM-5.1-NVFP4`, `model-prefix: glm5.1`,
`MODEL_PATH=/mnt/lustre01/models/GLM-5.1-NVFP4`. `SRT_SLURM_MODEL_PREFIX`
matches the upstream recipe's `model.path` alias (`nvidia/GLM-5-NVFP4`).
`launch_gb200-nv.sh` gains a `glm5.1 + fp4` case in the dynamo-trt
dispatch block and a matching srt-slurm clone entry.
b097ba8 to
04e5a87
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28975489919 |
…runtime Ports the 23 STP recipes (13 ISL1K_OSL1K + 10 ISL8K_OSL1K) from NVIDIA/srt-slurm sa-submission-q2-2026 into benchmarks/multi_node/srt-slurm-recipes/GLM5/... so that the version consumed by the sweep is pinned to this PR rather than tracking upstream drift. Mirrors the same overlay convention PR #1780 used for the dynamo-sglang GLM5.1 recipes. launch_gb200-nv.sh adds `mkdir -p recipes/GLM5 && cp -rT ...` to the existing glm5.1 dynamo-trt clone case so the local tree overrides the upstream copy in the srt-slurm checkout before srtctl reads any recipe.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28980396520 |
- nvidia-master.yaml: model nvidia/GLM-5-NVFP4, model-prefix glm5 - launch_gb200-nv.sh: dynamo-trt glm5/fp4 case uses MODEL_PATH=/home/sa-shared/glm-model/GLM-5-NVFP4, SERVED_MODEL_NAME=glm-5-nvfp4; matching srt-slurm overlay branch keyed on MODEL_PREFIX=glm5 - perf-changelog: describe as GLM-5 with the new model path
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28980396520 |
Prior /home/sa-shared path was not present on the watchtower login node (the launcher's model-path pre-check surfaced FileNotFoundError immediately). Move MODEL_PATH to /mnt/lustre01/slurm-shared/glm-model/GLM-5-NVFP4, which is on Lustre and cross-mounted to every watchtower compute node.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29048764289 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29055234997 |
- Rename benchmarks/multi_node/srt-slurm-recipes/GLM5/disagg/trtllm_dynamo/ gb200_nvfp4/** -> benchmarks/multi_node/srt-slurm-recipes/trtllm/glm5/... . - launch_gb200-nv.sh dynamo-trt+glm5 branch: git checkout v1.0.17, mkdir -p recipes/trtllm/glm5, cp -rT src/dst updated. - configs/nvidia-master.yaml: 23 CONFIG_FILE= paths under glm5-fp4-gb200-dynamo-trt retargeted to recipes/trtllm/glm5/... ; matching doc-URL comments repointed at v1.0.17. - perf-changelog: PR 1803 description updated to v1.0.17 + new overlay path. gb300 sibling entries (recipes/GLM5/.../gb300_nvfp4/**) intentionally untouched — separate PR.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29055342205 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29060697451 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29061830343 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29061830343 |
Picks up NVIDIA/srt-slurm#261, which sets the dynamo.trtllm publish_events + metrics flags to false by default. Applies the bump in three places: the git checkout in launch_gb200-nv.sh, the 23 source-URL comments in configs/nvidia-master.yaml, and the perf-changelog description bullets for glm5-fp4-gb200-dynamo-trt. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29287890247 |
3 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29287890247 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29287890247 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29287890247 |
|
/reuse-sweep-run |
Ankur-singh
changed the title
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
Set the top-level kv-p2p-transfer field required by disaggregated matrix validation. The TRT-LLM recipes retain UCX as their internal cache transceiver backend.\n\n中文：为 GLM-5 分离式配置声明顶层 NIXL KV 传输方式，以满足矩阵校验要求；TRT-LLM recipe 继续使用 UCX 作为内部缓存传输后端。
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
6 participants
submitted on behalf of @richardhuo-nv
Note
Low Risk
Benchmark and launcher configuration only; no application auth, data, or inference runtime code changes.
Overview
Adds
glm5-fp4-gb200-dynamo-trtto the NVIDIA master benchmark matrix: multinode disaggregated GLM-5 NVFP4 on GB200 using Dynamo + TensorRT-LLM (STP, non-MTP), with fixed-seq-len scenarios at 1K/1K and 8K/1K and many concurrency-specific prefill/decode search points wired to NVIDIA/srt-slurm recipe YAMLs viaCONFIG_FILE.launch_gb200-nv.shgains adynamo-trtbranch forglm5+fp4, setting Lustre model path, served name, andSRT_SLURM_MODEL_PREFIX=nvidia/GLM-5-NVFP4, and updates the unsupported-prefix error message.perf-changelog.yamlrecords the new config key and launch-script change for PR #1803.Reviewed by Cursor Bugbot for commit b097ba8. Bugbot is set up for automated code reviews on this repo. Configure here.
中文说明
代表 @richardhuo-nv 提交。
Note
低风险
本 PR 仅涉及基准测试、recipe（配方）和启动器配置，不修改应用认证、数据处理或推理运行时代码。
概述
在 NVIDIA 主基准测试矩阵中新增
glm5-fp4-gb200-dynamo-trt：使用 Dynamo + TensorRT-LLM 在 GB200 上运行 GLM-5 NVFP4 多节点分离式 STP（非 MTP）基准测试。配置覆盖 1K/1K 和 8K/1K 固定序列长度场景，并通过CONFIG_FILE关联多组按并发度划分的预填充/解码 NVIDIA/srt-slurm recipe。launch_gb200-nv.sh新增glm5+fp4的dynamo-trt分支，配置 Lustre 模型路径、服务名称和SRT_SLURM_MODEL_PREFIX=nvidia/GLM-5-NVFP4，并更新不受支持的模型前缀错误提示。perf-changelog.yaml记录 PR #1803 新增的配置键及启动脚本变更。