config(dsv4): add GB300 Dynamo-TRT AgentX points / 添加 GB300 Dynamo-TRT AgentX 配置点 - #2690
Conversation
添加 DeepSeek-V4-Pro GB300 Dynamo TensorRT-LLM AgentX 配置点，并关闭预填充和解码工作进程的 return_perf_metrics。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
将 DeepSeek-V4-Pro GB300 AgentX 变更日志条目关联到 PR 2690。
恢复历史变更日志条目，并仅将新添加的 DeepSeek-V4-Pro GB300 AgentX 条目关联到 PR 2690。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32332884204 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| git checkout main | ||
| mkdir -p recipes/vllm/kimi-k2.5-fp4 | ||
| cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k2.5-fp4" recipes/vllm/kimi-k2.5-fp4 | ||
| elif [[ "$IS_AGENTIC" == "1" && $FRAMEWORK == "dynamo-trt" && $MODEL_PREFIX == "dsv4" ]]; then | ||
| SRT_SLURM_MODEL_PREFIX="deepseek-ai/DeepSeek-V4-Pro" | ||
| git clone --branch v1.0.50 --single-branch https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" || exit 1 | ||
| cd "$SRT_REPO_DIR" || exit 1 | ||
|
|
||
| mkdir -p benchmarks/multi_node/srt-slurm-recipes/trtllm/deepseek-v4 || exit 1 | ||
| cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/trtllm/deepseek-v4" \ | ||
| benchmarks/multi_node/srt-slurm-recipes/trtllm/deepseek-v4 || exit 1 | ||
| if [[ "${EVAL_ONLY:-false}" == "true" ]]; then | ||
| # srt-slurm v1.0.50 launches lm-eval on the allocation head and uses | ||
| # localhost:8000. Keep AgentX frontends on first_decode for throughput, | ||
| # but co-locate the eval-only frontend with lm-eval so loopback resolves. | ||
| find benchmarks/multi_node/srt-slurm-recipes/trtllm/deepseek-v4 -name "*.yaml" \ | ||
| -exec sed -i \ | ||
| -e '/TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS/d' \ | ||
| -e 's/^ orchestrator_placement: first_decode$/ orchestrator_placement: head/' \ | ||
| {} + | ||
| fi | ||
| elif [[ $FRAMEWORK == "dynamo-trt" && $MODEL_PREFIX == "dsv4" ]]; then | ||
| # DSv4 dynamo-trt recipes use the HuggingFace model ID as model.path, | ||
| # so override SRT_SLURM_MODEL_PREFIX to match the recipe's model path key. |
There was a problem hiding this comment.
🔴 The new elif [[ "$IS_AGENTIC" == "1" && $FRAMEWORK == "dynamo-trt" && $MODEL_PREFIX == "dsv4" ]] branch (added at line 359) sits after the generic catch-all elif [[ "$IS_AGENTIC" == "1" ]]; then at line 253 in the same if/elif chain, so bash's first-match semantics route every dsv4-fp4-gb300-dynamo-trt-agentx run into the generic branch instead. That branch clones srt-slurm v1.0.36 and stages unrelated recipes/vllm/deepseek-v4/agentic/recipes/vllm/kimi-k3/agentic directories instead of v1.0.50 and benchmarks/multi_node/srt-slurm-recipes/trtllm/deepseek-v4, so the new AgentX recipes''' CONFIG_FILE and load_balancer paths will not exist in the checkout. Fix: move the new branch above line 253, matching the ordering of the other model/framework-specific IS_AGENTIC branches (qwen3.5 dynamo-trt, dynamo-sglang qwen3.5/dsv4/glm5.2).
Extended reasoning...
The bug: runners/launch_gb300-nv.sh has one long if/elif/.../else/fi chain starting at line 197. This PR adds a new, more-specific branch at line 359:
elif [[ "$IS_AGENTIC" == "1" && $FRAMEWORK == "dynamo-trt" && $MODEL_PREFIX == "dsv4" ]]; then
SRT_SLURM_MODEL_PREFIX="deepseek-ai/DeepSeek-V4-Pro"
git clone --branch v1.0.50 --single-branch https://github.com/NVIDIA/srt-slurm.git ...
cp -rT ".../trtllm/deepseek-v4" benchmarks/multi_node/srt-slurm-recipes/trtllm/deepseek-v4But an existing, much more general branch already sits earlier in the same chain, at line 253:
elif [[ "$IS_AGENTIC" == "1" ]]; then
git clone --branch v1.0.36 ...
cp -rT .../recipes/vllm/deepseek-v4/agentic ...
cp -rT .../recipes/vllm/kimi-k3/agentic ...This line-253 branch only checks IS_AGENTIC == \"1\" — it does not check FRAMEWORK or MODEL_PREFIX at all. Bash elif chains take the first branch whose condition is true, so for the new dsv4-fp4-gb300-dynamo-trt-agentx scenario added to configs/nvidia-master.yaml in this PR (IS_AGENTIC=1, FRAMEWORK=dynamo-trt, MODEL_PREFIX=dsv4), execution never reaches the new line-359 branch — it is unconditionally intercepted at line 253.
Why nothing catches this today: every other model/framework-specific IS_AGENTIC branch in this same file (dynamo-trt qwen3.5 at line 197, dynamo-sglang qwen3.5/dsv4/glm5.2 at 209/230/243) is already placed before the generic line-253 fallback, which is exactly why they work. Only the newly-added branch breaks that established ordering convention by being appended after the fallback instead of before it — nothing in the diff itself would fail to apply or fail a syntax check, since both branches are individually valid bash; the chain simply never dispatches to the new code.
Concrete walkthrough:
- Scenario
dsv4-fp4-gb300-dynamo-trt-agentx(agentic-coding type) runs on GB300 withIS_AGENTIC=1,FRAMEWORK=dynamo-trt,MODEL_PREFIX=dsv4. - The script evaluates the elif chain top-to-bottom: lines 197, 209, 230, 243 don't match (different framework/model combos), so it reaches line 253's
[[ "$IS_AGENTIC" == "1" ]], which does match. - Bash executes that branch's body and skips all remaining
elifbranches, including the new one at line 359 — it is truly unreachable dead code for this scenario. - The job checks out srt-slurm tag
v1.0.36(not thev1.0.50this PR's recipes were validated against) and stagesrecipes/vllm/deepseek-v4/agentic+recipes/vllm/kimi-k3/agentic(unrelated vLLM recipes) instead ofbenchmarks/multi_node/srt-slurm-recipes/trtllm/deepseek-v4. - The new recipe YAMLs'
additional-settings: CONFIG_FILE=.../trtllm/deepseek-v4/agentx-gb300-20260811/*.yamlandmoe_config.load_balancer: .../eplb_confs/*.yamlpaths reference files that were never copied into the srt-slurm checkout, andSRT_SLURM_MODEL_PREFIXis never set to the DeepSeek-V4-Pro HF path. The job will fail to find its config/eplb files.
Impact: this is not a cosmetic issue — the entire new AgentX config added by this PR (all four concurrency points: 4, 24, 736, 2626) is non-functional as merged, since it can never reach the code path written specifically for it.
Fix: move the new elif [[ "$IS_AGENTIC" == "1" && $FRAMEWORK == "dynamo-trt" && $MODEL_PREFIX == "dsv4" ]] block from line 359 to before the generic elif [[ "$IS_AGENTIC" == "1" ]] block at line 253, consistent with how the existing model/framework-specific IS_AGENTIC branches are ordered.
| check_env_vars \ | ||
| MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION \ | ||
| SRT_FRONTEND_HOST SRT_FRONTEND_PORT | ||
|
|
||
| export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" | ||
| echo "Using srt-slurm frontend endpoint: $AIPERF_SERVER_URL" |
There was a problem hiding this comment.
🔴 This script is invoked by ~80 pre-existing multi-node agentic recipes (kimi-k3, minimax-m3, glm5.2, qwen3.5, deepseek-v4 vllm/sglang, etc.), none of which set SRT_FRONTEND_HOST/SRT_FRONTEND_PORT. Adding these to check_env_vars (which hard-exits with no fallback) means any recipe whose srt-slurm pin doesn't export them will fail immediately, before ever reaching resolve_trace_source. The fix is to mirror the safe ${AIPERF_SERVER_URL:-http://localhost:$PORT} fallback pattern already used in the sibling change to benchmark_lib.sh.
Extended reasoning...
benchmarks/multi_node/agentic_srt.sh is the shared client entrypoint invoked by command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh from roughly 80 pre-existing multi-node agentic recipes across vllm/sglang/trtllm (kimi-k3, minimax-m3, glm5.2, qwen3.5, deepseek-v4, etc.), spanning gb200-nv, gb300-nv, and mi355x launchers. This PR adds SRT_FRONTEND_HOST and SRT_FRONTEND_PORT to the check_env_vars call at line 14, and check_env_vars (benchmarks/benchmark_lib.sh:328) exits the whole job with status 1 the moment any listed variable is unset. Line 16 then builds AIPERF_SERVER_URL exclusively from those two new variables, with no fallback in this script.
Before this PR, the frontend URL was always http://localhost:$PORT, which worked because the benchmark client and frontend were implicitly co-located for every existing recipe. A repo-wide grep shows SRT_FRONTEND_HOST/SRT_FRONTEND_PORT have exactly one in-repo consumer — this script — so they must be injected by srt-slurm itself at runtime, and only if the pinned srt-slurm version supports it. The existing ~80 agentic recipes clone many different srt-slurm branches/pins (default branch, v1.0.31, v1.0.36, a pinned fork SHA, etc.) — none of them set these vars in their recipe yaml, and there's no evidence any of those pins export them. Only this PR's new dsv4 gb300 dynamo-trt recipes pin srt-slurm v1.0.50 and use orchestrator_placement/client_placement fields that differ from the frontend's node — that's the actual motivation for moving off localhost, but it was applied unconditionally to the shared script rather than being scoped to the new recipe.
The result: for every existing recipe whose srt-slurm pin doesn't export SRT_FRONTEND_HOST/SRT_FRONTEND_PORT, the job hits check_env_vars and exits with "required environment variables are not set: SRT_FRONTEND_HOST SRT_FRONTEND_PORT" before ever reaching resolve_trace_source/install_agentic_deps. This regression is latent with respect to this PR's own CI — the PR only sweeps its own new dsv4-trt recipe, so a breakage in the other ~80 recipes would not surface until they're run (e.g. on the next scheduled sweep or a re-run of an existing recipe).
Proof by example: take any pre-existing recipe, e.g. benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/*.yaml, which invokes this same shared agentic_srt.sh. Its srt-slurm pin (an older branch than v1.0.50) does not export SRT_FRONTEND_HOST/SRT_FRONTEND_PORT. When the job runs: (1) agentic_srt.sh sources benchmark_lib.sh and calls check_env_vars with the full list including the two new vars; (2) check_env_vars iterates the list and finds SRT_FRONTEND_HOST unset; (3) it prints "required environment variables are not set: SRT_FRONTEND_HOST SRT_FRONTEND_PORT" and calls exit 1; (4) the job dies before running any workload, with no fallback to the old localhost:$PORT behavior.
The sibling change in this same PR, benchmark_lib.sh:1999, demonstrates the fix is trivial and was already the codebase's chosen idiom: it changed --url http://localhost:$PORT to --url ${AIPERF_SERVER_URL:-http://localhost:$PORT}. agentic_srt.sh should apply the same fallback pattern — build AIPERF_SERVER_URL from SRT_FRONTEND_HOST/SRT_FRONTEND_PORT only when both are set, falling back to http://localhost:$PORT otherwise — and drop the two new vars from the hard-required list in check_env_vars. That preserves the new dsv4-trt recipe's need for an explicit frontend host while not breaking the ~80 existing recipes that rely on the old co-located default.
|
|
||
| - config-keys: | ||
| - dsv4-fp4-gb300-dynamo-trt-agentx | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Add DeepSeek-V4-Pro GB300 Dynamo-TensorRT-LLM AgentX recipes for concurrencies 4, 24, 736, and 2626." | ||
| - "Disable TensorRT-LLM return_perf_metrics in the prefill and decode workers." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX |
There was a problem hiding this comment.
🟡 The newly appended perf-changelog.yaml entry uses a literal placeholder pull/XXX instead of this PR's actual number. Since this PR is #2690, the pr-link should read https://github.com/SemiAnalysisAI/InferenceX/pull/2690, matching the pattern of every prior entry in the file.
Extended reasoning...
The perf-changelog.yaml entry appended at the tail of the file (for the new dsv4-fp4-gb300-dynamo-trt-agentx config) sets:
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXXXX is a literal, un-substituted placeholder rather than a real PR number. This PR is #2690, so the link should read https://github.com/SemiAnalysisAI/InferenceX/pull/2690.
Code path: this is a pure data/documentation entry — nothing in the benchmark pipeline reads or validates pr-link, so it does not affect ingestion, scoring, or any recipe execution. It's caught only by manual review, which is exactly why it's easy to miss: nothing in CI currently checks that the trailing changelog entry's link resolves to a real PR.
Why it happened: every other entry in the file (e.g. the immediately preceding entries referencing PRs 2686, 2677, 2660, 2639) has the real numeric PR numbers filled in, confirming the convention is to substitute the actual PR number once it's known — which is easy to forget when drafting the changelog entry before the PR is opened.
Impact: low — this is purely a traceability/documentation issue. A reader trying to trace this changelog entry back to its originating PR via the link will hit a 404 (github.com/.../pull/XXX doesn't resolve to anything), defeating the purpose of the pr-link field, but it doesn't break any benchmark run, sweep, or ingestion path.
Proof / step-by-step:
- Open
perf-changelog.yamland look at the very last entry (appended by this PR). - Its
pr-linkfield readshttps://github.com/SemiAnalysisAI/InferenceX/pull/XXX. - Compare against the entry immediately above it, which reads
https://github.com/SemiAnalysisAI/InferenceX/pull/2686— a real, resolvable PR number. - Navigating to
.../pull/XXXin a browser returns a 404/invalid PR, whereas.../pull/2686resolves correctly. Since this PR's actual number is 2690, the fix is to replaceXXXwith2690.
Fix: change the last line of the appended entry from pull/XXX to pull/2690.
将 DeepSeek-V4-Pro Dynamo TensorRT-LLM AgentX 启动分支移到通用 Agentic 分支之前，确保选择正确的 srt-slurm 版本和 TRT-LLM 配置。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32336041718 |
扩展 DeepSeek-V4-Pro GB300 AgentX 配置点，移除 EPLB 覆盖，并将作业路由到 batch_3。
合并 origin/main，并将本分支的性能变更日志条目保留在文件末尾。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32403083041 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32403083041 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32403083041 |
|
/reuse-sweep-run 32403083041 |
kedarpotdar-nv
left a comment
kedarpotdar-nv
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Full sweep workflow
- Verified that this PR passes evals. Full sweep and eval workflow
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- Reviewed PR head
25c59b762db329544fceb8bc33f9c8ab5bf23ad3; all six multi-node AgentX sweep points and all configured eval jobs passed in the linked workflow. - The recipes enable chat templating and use a three-token MTP draft with TensorRT-LLM's forced accepted-draft-token value of
1.49, corresponding to the committed DeepSeek-V4-Prothinking_ongolden total AL of2.49. - This is a multi-node Dynamo–TensorRT-LLM submission. The single-node vLLM/SGLang recipe-publication requirement and
append-onlyrequirement are not applicable. Existing upstream-image vLLM and SGLang DeepSeek-V4 submissions precede this additional framework submission. - No inference-engine or serving-stack patch is introduced.
Signed: kedarpotdar-nv
将 main 合并到 PR 分支，并保留双方新增的配置及性能变更日志条目。
3 participants
Description
Add a DeepSeek-V4-Pro FP4 Dynamo–TensorRT-LLM AgentX configuration for GB300.
return_perf_metricsin the prefill and decode TensorRT-LLM workers.为 GB300 添加 DeepSeek-V4-Pro FP4 Dynamo–TensorRT-LLM AgentX 配置。
return_perf_metrics。Related Issue
N/A / 不适用
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.