Repository navigation
config(dsv4): add GB300 Dynamo-TRT AgentX points / 添加 GB300 Dynamo-TRT AgentX 配置点 #2690
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Oseltamivir
merged 7 commits into
main
from
dsv4-fp4-gb300-dynamo-trt-agentx-recipes-v2
Aug 21, 2026
Merged
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
87530f9
feat(config): add DSV4 GB300 Dynamo TRT AgentX points
RohitNagraj b04a962
chore(changelog): link PR 2690
RohitNagraj f69fa40
fix(changelog): link the DSV4 entry to PR 2690
RohitNagraj b7bd4de
fix(gb300): prioritize DSV4 AgentX launcher
RohitNagraj 05c7a18
feat(config): expand DSV4 GB300 AgentX recipes
RohitNagraj 25c59b7
Merge origin/main and resolve perf changelog
RohitNagraj 9312aa9
chore: merge main into PR branch
Oseltamivir File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
187 changes: 187 additions & 0 deletions
187
...lm/deepseek-v4/agentx-gb300-20260811/dynamo-disagg-gb300-1p1d-dep8-dep32-c388-b4-mtp.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,187 @@ | ||
| name: dynamo-disagg-gb300-1p1d-dep8-dep32-c388-b4-mtp | ||
| model: | ||
| path: deepseek-ai/DeepSeek-V4-Pro | ||
| container: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc24 | ||
| precision: fp4 | ||
| identity: | ||
| model: | ||
| repo: deepseek-ai/DeepSeek-V4-Pro | ||
| container: | ||
| image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24 | ||
| frameworks: | ||
| dynamo: 1.4.0.dev20260807 | ||
| tensorrt_llm: 1.3.0rc24 | ||
| dynamo: | ||
| install: true | ||
| wheel: 1.4.0.dev20260807 | ||
| request_plane: tcp | ||
| health_check: | ||
| max_attempts: 1440 | ||
| interval_seconds: 10 | ||
| resources: | ||
| gpu_type: gb300 | ||
| gpus_per_node: 4 | ||
| prefill_nodes: 2 | ||
| prefill_workers: 1 | ||
| gpus_per_prefill: 8 | ||
| decode_nodes: 8 | ||
| decode_workers: 1 | ||
| gpus_per_decode: 32 | ||
| backend: | ||
| type: trtllm | ||
| prefill_environment: | ||
| TLLM_LOG_LEVEL: INFO | ||
| TRTLLM_SERVER_DISABLE_GC: '1' | ||
| TRTLLM_WORKER_DISABLE_GC: '1' | ||
| NCCL_GRAPH_MIXING_SUPPORT: '0' | ||
| TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: '1.49' | ||
| UCX_TLS: cuda_ipc,cuda_copy,sm,self,tcp | ||
| DYN_ENGINE_CONV_AFFINITY: '1' | ||
| DYN_DEFAULT_THINKING_MODE: disabled | ||
| MIMALLOC_ARENA_RESERVE: '0' | ||
| TRTLLM_PINNED_WEIGHT_STAGING: '1' | ||
| PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True | ||
| CUDA_SCALE_LAUNCH_QUEUES: 4x | ||
| decode_environment: | ||
| TLLM_LOG_LEVEL: INFO | ||
| TRTLLM_SERVER_DISABLE_GC: '1' | ||
| TRTLLM_WORKER_DISABLE_GC: '1' | ||
| NCCL_GRAPH_MIXING_SUPPORT: '0' | ||
| TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: '1.49' | ||
| UCX_TLS: cuda_ipc,cuda_copy,sm,self,tcp | ||
| DYN_ENGINE_CONV_AFFINITY: '1' | ||
| DYN_DEFAULT_THINKING_MODE: disabled | ||
| MIMALLOC_ARENA_RESERVE: '0' | ||
| TRTLLM_PINNED_WEIGHT_STAGING: '1' | ||
| trtllm_config: | ||
| prefill: | ||
| attention_dp_config: | ||
| kv_cache_routing_conversation_affinity: true | ||
| cache_transceiver_config: | ||
| backend: NIXL | ||
| kv_transfer_timeout_ms: 600000 | ||
| transceiver_runtime: PYTHON | ||
| cuda_graph_config: null | ||
| custom_tokenizer: deepseek_v4 | ||
| disable_overlap_scheduler: false | ||
| enable_attention_dp: true | ||
| enable_chunked_prefill: true | ||
| kv_cache_config: | ||
| dtype: fp8 | ||
| enable_block_reuse: true | ||
| event_buffer_max_size: 0 | ||
| free_gpu_memory_fraction: 0.8 | ||
| host_cache_size: 193273528320 | ||
| pool_ratio: | ||
| - 0.55 | ||
| - 0.22 | ||
| - 0.23 | ||
| tokens_per_block: 128 | ||
| block_reuse_config: | ||
| policy: per_conversation | ||
| max_num_turns: 5 | ||
| max_batch_size: 256 | ||
| max_num_tokens: 16384 | ||
| max_seq_len: 990016 | ||
| moe_config: | ||
| backend: TRTLLM | ||
| moe_expert_parallel_size: 8 | ||
| pipeline_parallel_size: 1 | ||
| print_iter_log: true | ||
| return_perf_metrics: false | ||
| scheduler_config: | ||
| capacity_scheduler_policy: MAX_UTILIZATION | ||
| speculative_config: | ||
| decoding_type: MTP | ||
| max_draft_len: 3 | ||
| tensor_parallel_size: 8 | ||
| decode: | ||
| cache_transceiver_config: | ||
| backend: NIXL | ||
| kv_transfer_timeout_ms: 600000 | ||
| transceiver_runtime: PYTHON | ||
| cuda_graph_config: | ||
| batch_sizes: | ||
| - 1 | ||
| - 2 | ||
| - 4 | ||
| enable_padding: true | ||
| custom_tokenizer: deepseek_v4 | ||
| enable_attention_dp: true | ||
| enable_lm_head_tp_in_adp: true | ||
| kv_cache_config: | ||
| avg_seq_len: 200000 | ||
| dtype: fp8 | ||
| enable_block_reuse: false | ||
| event_buffer_max_size: 0 | ||
| free_gpu_memory_fraction: 0.7 | ||
| host_cache_size: 0 | ||
| tokens_per_block: 128 | ||
| max_batch_size: 4 | ||
| max_num_tokens: 16 | ||
| max_seq_len: 996595 | ||
| moe_config: | ||
| backend: MEGAMOE_DEEPGEMM | ||
| use_low_precision_moe_combine: true | ||
| moe_expert_parallel_size: 32 | ||
| num_postprocess_workers: 4 | ||
| pipeline_parallel_size: 1 | ||
| print_iter_log: true | ||
| return_perf_metrics: false | ||
| sparse_attention_config: | ||
| algorithm: deepseek_v4 | ||
| enable_heuristic_topk: true | ||
| speculative_config: | ||
| decoding_type: MTP | ||
| max_draft_len: 3 | ||
| stream_interval: 20 | ||
| tensor_parallel_size: 32 | ||
| publish_events_and_metrics: false | ||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| env: | ||
| MODEL: deepseek-ai/DeepSeek-V4-Pro | ||
| MODEL_PREFIX: dsv4 | ||
| FRAMEWORK: dynamo-trt | ||
| PRECISION: fp4 | ||
| CONC: '388' | ||
| DURATION: '3600' | ||
| KV_OFFLOADING: none | ||
| ETCD_LEASE_TTL: '120' | ||
| DYN_ROUTER_QUEUE_THRESHOLD: None | ||
| DYN_TOKENIZER_CACHE: '1' | ||
| DYN_TOKENIZER_CACHE_BYTES: '8000000000' | ||
| DYN_TOKENIZER: fastokens | ||
| DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400' | ||
| args: | ||
| router-mode: kv | ||
| no-kv-events: true | ||
| active-decode-blocks-threshold: None | ||
| active-prefill-tokens-threshold: None | ||
| active-prefill-tokens-threshold-frac: None | ||
| orchestrator_placement: first_decode | ||
| benchmark: | ||
| client_placement: last_decode | ||
| type: custom | ||
| use_chat_template: true | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| MODEL: deepseek-ai/DeepSeek-V4-Pro | ||
| MODEL_PREFIX: dsv4 | ||
| FRAMEWORK: dynamo-trt | ||
| PRECISION: fp4 | ||
| CONC: '388' | ||
| DURATION: '3600' | ||
| KV_OFFLOADING: none | ||
| INFMAX_CONTAINER_WORKSPACE: /infmax-workspace | ||
| RESULT_DIR: /logs/agentic | ||
| PORT: '8000' | ||
| IS_MULTINODE: 'true' | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: '14400' | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true' | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0' | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache | ||
| HF_HUB_CACHE: /hf_hub_cache | ||
| WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126 | ||
| SERVED_MODEL_NAME: DeepSeek-V4-Pro |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 This script is invoked by ~80 pre-existing multi-node agentic recipes (kimi-k3, minimax-m3, glm5.2, qwen3.5, deepseek-v4 vllm/sglang, etc.), none of which set
SRT_FRONTEND_HOST/SRT_FRONTEND_PORT. Adding these tocheck_env_vars(which hard-exits with no fallback) means any recipe whose srt-slurm pin doesn't export them will fail immediately, before ever reachingresolve_trace_source. The fix is to mirror the safe${AIPERF_SERVER_URL:-http://localhost:$PORT}fallback pattern already used in the sibling change tobenchmark_lib.sh.Extended reasoning...
benchmarks/multi_node/agentic_srt.shis the shared client entrypoint invoked bycommand: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.shfrom roughly 80 pre-existing multi-node agentic recipes across vllm/sglang/trtllm (kimi-k3, minimax-m3, glm5.2, qwen3.5, deepseek-v4, etc.), spanning gb200-nv, gb300-nv, and mi355x launchers. This PR addsSRT_FRONTEND_HOSTandSRT_FRONTEND_PORTto thecheck_env_varscall at line 14, andcheck_env_vars(benchmarks/benchmark_lib.sh:328) exits the whole job with status 1 the moment any listed variable is unset. Line 16 then buildsAIPERF_SERVER_URLexclusively from those two new variables, with no fallback in this script.Before this PR, the frontend URL was always
http://localhost:$PORT, which worked because the benchmark client and frontend were implicitly co-located for every existing recipe. A repo-wide grep showsSRT_FRONTEND_HOST/SRT_FRONTEND_PORThave exactly one in-repo consumer — this script — so they must be injected by srt-slurm itself at runtime, and only if the pinned srt-slurm version supports it. The existing ~80 agentic recipes clone many different srt-slurm branches/pins (default branch, v1.0.31, v1.0.36, a pinned fork SHA, etc.) — none of them set these vars in their recipe yaml, and there's no evidence any of those pins export them. Only this PR's new dsv4 gb300 dynamo-trt recipes pin srt-slurmv1.0.50and useorchestrator_placement/client_placementfields that differ from the frontend's node — that's the actual motivation for moving offlocalhost, but it was applied unconditionally to the shared script rather than being scoped to the new recipe.The result: for every existing recipe whose srt-slurm pin doesn't export
SRT_FRONTEND_HOST/SRT_FRONTEND_PORT, the job hitscheck_env_varsand exits with "required environment variables are not set: SRT_FRONTEND_HOST SRT_FRONTEND_PORT" before ever reachingresolve_trace_source/install_agentic_deps. This regression is latent with respect to this PR's own CI — the PR only sweeps its own new dsv4-trt recipe, so a breakage in the other ~80 recipes would not surface until they're run (e.g. on the next scheduled sweep or a re-run of an existing recipe).Proof by example: take any pre-existing recipe, e.g.
benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/*.yaml, which invokes this same sharedagentic_srt.sh. Its srt-slurm pin (an older branch than v1.0.50) does not exportSRT_FRONTEND_HOST/SRT_FRONTEND_PORT. When the job runs: (1)agentic_srt.shsourcesbenchmark_lib.shand callscheck_env_varswith the full list including the two new vars; (2)check_env_varsiterates the list and findsSRT_FRONTEND_HOSTunset; (3) it prints "required environment variables are not set: SRT_FRONTEND_HOST SRT_FRONTEND_PORT" and callsexit 1; (4) the job dies before running any workload, with no fallback to the oldlocalhost:$PORTbehavior.The sibling change in this same PR,
benchmark_lib.sh:1999, demonstrates the fix is trivial and was already the codebase's chosen idiom: it changed--url http://localhost:$PORTto--url ${AIPERF_SERVER_URL:-http://localhost:$PORT}.agentic_srt.shshould apply the same fallback pattern — buildAIPERF_SERVER_URLfromSRT_FRONTEND_HOST/SRT_FRONTEND_PORTonly when both are set, falling back tohttp://localhost:$PORTotherwise — and drop the two new vars from the hard-required list incheck_env_vars. That preserves the new dsv4-trt recipe's need for an explicit frontend host while not breaking the ~80 existing recipes that rely on the old co-located default.