feat(dsv4): add GB300 Dynamo-TensorRT-LLM AgentX recipes / 添加 GB300 Dynamo-TensorRT-LLM AgentX 配置 - #2595
feat(dsv4): add GB300 Dynamo-TensorRT-LLM AgentX recipes / 添加 GB300 Dynamo-TensorRT-LLM AgentX 配置#2595RohitNagraj wants to merge 22 commits into
Conversation
添加 DeepSeek-V4-Pro GB300 Dynamo TensorRT-LLM AgentX 配置与配套配方。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
在性能变更日志中补充 PR 2595 链接。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Add a DeepSeek-V4-Pro FP4 Dynamo-TensorRT-LLM MTP AgentX configuration for GB300 with ten disaggregated topologies." | ||
| - "Use NIXL KV transfer, deterministic MTP acceptance settings, and in-repo EPLB placement tables." | ||
| - "Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX |
There was a problem hiding this comment.
🟡 The new perf-changelog.yaml entry for dsv4-fp4-gb300-dynamo-trt-agentx ends with a placeholder pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX instead of this PR's real number. Since this is PR #2595, the line should read pull/2595; TBD is the only sanctioned pre-PR placeholder per docs/configuration-procedures.md.
Extended reasoning...
The last entry appended to perf-changelog.yaml (for the new dsv4-fp4-gb300-dynamo-trt-agentx config-key introduced by this PR) ends with:
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
XXX is a literal, hardcoded string — it isn't a template variable that gets substituted anywhere, so it resolves to a broken GitHub URL that points at a PR which doesn't exist (there is no PR #"XXX").
docs/configuration-procedures.md documents the expected changelog convention: the pr-link field must follow the .../pull/<number> format, and the only acceptable placeholder before a PR exists is the literal string pr-link: TBD — which must be replaced with the real PR URL immediately once the PR is created. pull/XXX doesn't match either of these: it's neither a valid link nor the sanctioned TBD placeholder, so it violates the documented convention outright.
The code path that "triggers" this is simply: whoever authored the recipe copy-pasted a previous changelog entry as a template and forgot to fill in the actual PR number before generating the diff. Nothing in the repo's tooling currently validates this field at PR-creation time, so a malformed pull/XXX slips through unless a reviewer or CI check catches it — which is presumably why the docs explicitly call out this exact failure mode.
Impact is limited to traceability/documentation: a maintainer or anyone browsing the changelog later to find "which PR introduced this recipe / regression" will follow a dead link instead of landing on the actual PR. It has zero effect on the recipe YAML, the launcher, or benchmark behavior — this is purely a metadata defect in the changelog.
Step-by-step proof:
- Open
perf-changelog.yamland look at the newest (physically last) entry, whoseconfig-keyslist containsdsv4-fp4-gb300-dynamo-trt-agentx(the recipe this PR adds). - Its final field reads
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. - Per the PR metadata, this change is submitted as PR feat(dsv4): add GB300 Dynamo-TensorRT-LLM AgentX recipes / 添加 GB300 Dynamo-TensorRT-LLM AgentX 配置 #2595.
- Compare against sibling entries in the same file (e.g.
pull/2580,pull/2578,pull/2577for the three most-recently-merged PRs before this one) — every other entry uses a real numeric PR id, confirmingpull/XXXis an unintentional leftover rather than an intentional convention. - Fix: change the line to
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2595.
This is a simple one-line fix with no functional risk, so it doesn't block merging, but it should be corrected since the PR's own checklist explicitly calls out maintaining accurate perf-changelog entries.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31752586653 |
将 AIPerf 请求和指标查询路由到实际的 srt-slurm 前端，并在预热失败后限时清理挂起进程。评测模式会移除强制接受令牌数设置。
合并最新的 main，并保留其完整性能变更日志后，将本分支的条目重新追加到文件末尾。
将 AgentX 修复范围限定为前端路由，并恢复原有的 AIPerf 执行与清理流程。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31827243370 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31866510964 |
AgentX warmup 404'd for the DSv4 GB300 dynamo-trt recipe: the frontend
registers the model as "DeepSeek-V4-Pro" (per the recipe's
SERVED_MODEL_NAME), but build_replay_cmd always passed the raw $MODEL
HF id ("deepseek-ai/DeepSeek-V4-Pro") to aiperf, which killed warmup
before profiling started. No other AgentX recipe sets
SERVED_MODEL_NAME, so this is a no-op for them.
Run 22805 showed the dynamo-trt frontend registers the qwen3.5 model
as "Qwen3.5-397B-A17B-NVFP4-V2" (no nvidia/ prefix) regardless of
DYN_TRTLLM_SERVED_MODEL_NAME, while build_replay_cmd passed the raw
$MODEL HF id ("nvidia/Qwen3.5-397B-A17B-NVFP4-V2") to aiperf --model,
404ing on every warmup request before profiling could start (same
root cause as the DSv4 GB300 dynamo-trt case in PR #2595).
Add SERVED_MODEL_NAME to all 6 qwen3.5 agentx recipes and make
build_replay_cmd prefer it over $MODEL, consistent with PR #2595.
…-trt-agentx-recipes # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31892147288 |
aiperf's dataset manager resolves the tokenizer from --model by default. Once --model became SERVED_MODEL_NAME (a wire name, not necessarily a valid HF repo id), tokenizer loading 404'd during Configure Profiling (e.g. "Qwen3.5-397B-A17B-NVFP4-V2" is not a repo on HF Hub; only "nvidia/Qwen3.5-397B-A17B-NVFP4-V2" is). Pass --tokenizer $MODEL explicitly so routing and tokenizer resolution use the correct name for each.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31909326740 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31910356591 |
) * Add Qwen3.5-397B-A17B-NVFP4-V2 GB300 TRT-LLM disaggregated AgentX * fix perf change log * address comments * fix: honor SERVED_MODEL_NAME for qwen3.5 AgentX to fix warmup 404s Run 22805 showed the dynamo-trt frontend registers the qwen3.5 model as "Qwen3.5-397B-A17B-NVFP4-V2" (no nvidia/ prefix) regardless of DYN_TRTLLM_SERVED_MODEL_NAME, while build_replay_cmd passed the raw $MODEL HF id ("nvidia/Qwen3.5-397B-A17B-NVFP4-V2") to aiperf --model, 404ing on every warmup request before profiling could start (same root cause as the DSv4 GB300 dynamo-trt case in PR #2595). Add SERVED_MODEL_NAME to all 6 qwen3.5 agentx recipes and make build_replay_cmd prefer it over $MODEL, consistent with PR #2595. * fix: pass real HF id as --tokenizer alongside SERVED_MODEL_NAME override Run 22866 (qwen3.5 GB300 trtllm) died at Configure Profiling: aiperf's dataset manager resolves the tokenizer from --model, and --model is now SERVED_MODEL_NAME ("Qwen3.5-397B-A17B-NVFP4-V2"), which isn't a valid HF repo id — only "nvidia/Qwen3.5-397B-A17B-NVFP4-V2" is. Pass --tokenizer $MODEL explicitly so routing (--model) and tokenizer resolution (--tokenizer) use the correct name for each. * adjust dataset * Update perf-changelog.yaml --------- Co-authored-by: Xin Li <xinli@nvidia.com> Co-authored-by: Ankur-singh <ankusingh@nvidia.com> Co-authored-by: Cameron Quilici <cjquilici@gmail.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31910356591 |
同步 AgentX 客户端放置、显式运行时环境、服务模型标识、标准作业时限和 AIPerf 依赖。
将 DeepSeek-V4-Pro AgentX 配方的 TensorRT-LLM 合成接受目标与选定的黄金接受曲线对齐。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31996573498 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31998082917 |
恢复 DeepSeek-V4-Pro AgentX 配方的思考启用合成接受目标，同时保持服务器默认思考模式禁用。
在仅评测模式下将 AgentX 前端与 lm-eval 放置在同一节点，确保本地回环地址能够正确连接。
将 AgentX 客户端使用的模型名称与 Dynamo TRT-LLM 服务端注册名称保持一致。
合并 origin/main，并将本 PR 的性能变更日志条目重新追加到文件末尾。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32055067187 |
移除 AgentX 配方中的结果文件名覆盖，并为 DSV4 AgentX 作业恢复八小时的 Slurm 时限；同时同步主分支并解决性能变更日志冲突。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32080403273 |
恢复使用仓库默认的 AIPerf 子模块版本，移除此配置分支上的独立版本变更。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32096792037 |
合并最新的 main 分支，并解决性能变更日志冲突。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32166592085 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32171407952 |
3 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32171407952 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32171407952 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32171407952 |
|
/reuse-sweep-run 32171407952 |
| TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: '1.49' | ||
| UCX_TLS: cuda_ipc,cuda_copy,sm,self,tcp | ||
| DYN_ENGINE_CONV_AFFINITY: '1' | ||
| DYN_DEFAULT_THINKING_MODE: disabled |
There was a problem hiding this comment.
Thinking should on right? Mismatch with TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS
There was a problem hiding this comment.
Thanks Bryan
- good news is that
TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENSis using the right thinking-on AL's consistent with vLLM & SGLang so I think perf should be the same - thinking mode on server side should be same from perf standpoint and client thinking_mode should still work as expected, and I think we set OSL to the values provided by dataset anyway
- will update as a quick follow submission just to be consist if that's OK :)
|
/stage-results 32171407952 |
|
@cquil11 staging run 32171407952 failed. Inspect the staging workflow. |
| enable_block_reuse: true | ||
| event_buffer_max_size: 0 | ||
| free_gpu_memory_fraction: 0.8 | ||
| host_cache_size: 193273528320 |
There was a problem hiding this comment.
DRAM offloading is set but not specified in the master yaml, meaning this information will not be surfaced to the frontend which is not ideal
we can manually correct it this time but in the future, please remember to ask codex / claude to verify these values map correctly (I know it's a bit annoying but easier than trying to auto detect it)
There was a problem hiding this comment.
There was a problem hiding this comment.
tysm for understanding, will do!
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2592 | ||
|
|
||
| - config-keys: | ||
| - dsv4-fp4-gb300-dynamo-trt-agentx |
There was a problem hiding this comment.
Can you fix this? Not sure why there are like 5 separate entries
There was a problem hiding this comment.
My bad. Fixed!
| MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION \ | ||
| SRT_FRONTEND_HOST SRT_FRONTEND_PORT | ||
|
|
||
| export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" |
There was a problem hiding this comment.
This isn't correct. AIPerf needs the server urls, not the router url
There was a problem hiding this comment.
Normally "http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" resolves to localhost:port , but in some cases, aiperf client can be located on a different node than frontend, therefore using the env var is more versatile.
将 PR 2595 的性能变更日志合并为单个配置条目，避免重复触发基准测试。
合并最新的 main，并将 PR 2595 的单个性能变更日志条目重新追加到文件末尾。
5 participants
Description
Add a DeepSeek-V4-Pro FP4 Dynamo–TensorRT-LLM AgentX configuration for GB300.
dsv4-fp4-gb300-dynamo-trt-agentxmaster configuration with NIXL KV transfer./scratch/models/DeepSeek-V4-Pro.Validation:
中文说明
为 GB300 添加 DeepSeek-V4-Pro FP4 Dynamo–TensorRT-LLM AgentX 配置。
dsv4-fp4-gb300-dynamo-trt-agentx主配置。/scratch/models/DeepSeek-V4-Pro模型路径。验证：
Related Issue
N/A / 不适用
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.