Update DeepSeek-V4-Pro (DSV4) FP4 GB300 Dynamo-SGLang AgentX MTP configs / 更新 DeepSeek-V4-Pro（DSV4）FP4 GB300 Dynamo-SGLang AgentX MTP 配置 #2644
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| name: "agg-gb300-tp4-mtp-lowlatency" | ||
|
|
||
| # Low-latency AgentX aggregate topology: one TP4 worker occupies one | ||
| # four-GPU GB300 node and serves both prefill and decode. | ||
|
|
||
| model: | ||
| path: "deepseek-v4-pro" | ||
| container: "dynamo-sglang" | ||
| precision: "fp4" | ||
|
|
||
| identity: | ||
| model: | ||
| repo: "deepseek-ai/DeepSeek-V4-Pro" | ||
| container: | ||
| image: "lmsysorg/sglang:nightly-dev-cu13-20260821-f825d729" | ||
| frameworks: | ||
| dynamo: "1.4.0" | ||
|
|
||
| dynamo: | ||
| install: true | ||
| wheel: "1.4.0" | ||
|
|
||
| health_check: | ||
| max_attempts: 1440 | ||
| interval_seconds: 10 | ||
|
|
||
| resources: | ||
| gpu_type: "gb300" | ||
| gpus_per_node: 4 | ||
| agg_nodes: 1 | ||
| agg_workers: 1 | ||
| gpus_per_agg: 4 | ||
|
|
||
| infra: | ||
| etcd_nats_dedicated_node: false | ||
| nats_max_payload_mb: 32 | ||
|
|
||
| frontend: | ||
| type: dynamo | ||
| nginx_session_affinity: true | ||
| nginx_session_affinity_header: X-Dynamo-Session-ID | ||
| enable_multiple_frontends: false | ||
| env: | ||
| PIP_BREAK_SYSTEM_PACKAGES: "1" | ||
| DYN_NATS_REQUEST_TIMEOUT_SECS: "1800" | ||
| args: | ||
| router-mode: "kv" | ||
| router-session-affinity-ttl-secs: "3600" | ||
| active-decode-blocks-threshold: "None" | ||
| active-prefill-tokens-threshold: "None" | ||
| active-prefill-tokens-threshold-frac: "None" | ||
|
|
||
| backend: | ||
| type: sglang | ||
| aggregated_environment: | ||
| SGLANG_DEFAULT_THINKING: "1" | ||
| SGLANG_DSV4_REASONING_EFFORT: high | ||
| PIP_BREAK_SYSTEM_PACKAGES: "1" | ||
| SGLANG_JIT_DEEPGEMM_PRECOMPILE: "1" | ||
| SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1" | ||
| SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: "1" | ||
| SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2: "1" | ||
| SGLANG_OPT_USE_ONLINE_COMPRESS: "0" | ||
| SGLANG_OPT_USE_JIT_INDEXER_METADATA: "1" | ||
| SGLANG_OPT_USE_JIT_NORM: "1" | ||
| SGLANG_OPT_USE_TOPK_V2: "True" | ||
|
|
||
| sglang_config: | ||
| aggregated: | ||
| served-model-name: "deepseek-ai/DeepSeek-V4-Pro" | ||
| enable-metrics: true | ||
| trust-remote-code: true | ||
| stream-interval: 10 | ||
| watchdog-timeout: 1000000 | ||
| mem-fraction-static: 0.94 | ||
| chunked-prefill-size: 8192 | ||
| max-prefill-tokens: 8192 | ||
| moe-runner-backend: "flashinfer_mxfp4" | ||
| disable-flashinfer-autotune: true | ||
| swa-full-tokens-ratio: 0.1 | ||
| max-running-requests: 32 | ||
| cuda-graph-max-bs-decode: 32 | ||
| scheduler-recv-interval: 30 | ||
| dp-size: 1 | ||
| tp-size: 4 | ||
| ep-size: 1 | ||
| speculative-algorithm: EAGLE | ||
| speculative-num-steps: 3 | ||
| speculative-eagle-topk: 1 | ||
| speculative-num-draft-tokens: 4 | ||
|
|
||
| sbatch_directives: | ||
| mem: "0" | ||
| cpus-per-task: "144" | ||
|
|
||
| srun_options: | ||
| mem: "0" | ||
| container-remap-root: "" | ||
|
|
||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| IS_MULTINODE: "false" | ||
| TP: "4" | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| name: "agg-gb300-tp8-mtp-lowlatency" | ||
|
|
||
| # Low-latency AgentX aggregate topology: one TP8 worker spans two | ||
| # four-GPU GB300 nodes and serves both prefill and decode. | ||
|
|
||
| model: | ||
| path: "deepseek-v4-pro" | ||
| container: "dynamo-sglang" | ||
| precision: "fp4" | ||
|
|
||
| identity: | ||
| model: | ||
| repo: "deepseek-ai/DeepSeek-V4-Pro" | ||
| container: | ||
| image: "lmsysorg/sglang:nightly-dev-cu13-20260821-f825d729" | ||
| frameworks: | ||
| dynamo: "1.4.0" | ||
|
|
||
| dynamo: | ||
| install: true | ||
| wheel: "1.4.0" | ||
|
|
||
| health_check: | ||
| max_attempts: 1440 | ||
| interval_seconds: 10 | ||
|
|
||
| resources: | ||
| gpu_type: "gb300" | ||
| gpus_per_node: 4 | ||
| agg_nodes: 2 | ||
| agg_workers: 1 | ||
| gpus_per_agg: 8 | ||
|
|
||
| infra: | ||
| etcd_nats_dedicated_node: false | ||
| nats_max_payload_mb: 32 | ||
|
|
||
| frontend: | ||
| type: dynamo | ||
| nginx_session_affinity: true | ||
| nginx_session_affinity_header: X-Dynamo-Session-ID | ||
| enable_multiple_frontends: false | ||
| env: | ||
| PIP_BREAK_SYSTEM_PACKAGES: "1" | ||
| DYN_NATS_REQUEST_TIMEOUT_SECS: "1800" | ||
| args: | ||
| router-mode: "kv" | ||
| router-session-affinity-ttl-secs: "3600" | ||
| active-decode-blocks-threshold: "None" | ||
| active-prefill-tokens-threshold: "None" | ||
| active-prefill-tokens-threshold-frac: "None" | ||
|
|
||
| backend: | ||
| type: sglang | ||
| aggregated_environment: | ||
| SGLANG_DEFAULT_THINKING: "1" | ||
| SGLANG_DSV4_REASONING_EFFORT: high | ||
| PIP_BREAK_SYSTEM_PACKAGES: "1" | ||
| SGLANG_JIT_DEEPGEMM_PRECOMPILE: "1" | ||
| SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT: "1" | ||
| SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: "1" | ||
| SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2: "1" | ||
| SGLANG_OPT_USE_ONLINE_COMPRESS: "0" | ||
| SGLANG_OPT_USE_JIT_INDEXER_METADATA: "1" | ||
| SGLANG_OPT_USE_JIT_NORM: "1" | ||
| SGLANG_OPT_USE_TOPK_V2: "True" | ||
|
|
||
| sglang_config: | ||
| aggregated: | ||
| served-model-name: "deepseek-ai/DeepSeek-V4-Pro" | ||
| enable-metrics: true | ||
| trust-remote-code: true | ||
| stream-interval: 10 | ||
| watchdog-timeout: 1000000 | ||
| mem-fraction-static: 0.94 | ||
| chunked-prefill-size: 8192 | ||
| max-prefill-tokens: 8192 | ||
| moe-runner-backend: "flashinfer_mxfp4" | ||
| disable-flashinfer-autotune: true | ||
| swa-full-tokens-ratio: 0.1 | ||
| max-running-requests: 4 | ||
| cuda-graph-max-bs-decode: 4 | ||
| scheduler-recv-interval: 30 | ||
| dp-size: 1 | ||
| tp-size: 8 | ||
| ep-size: 1 | ||
| speculative-algorithm: EAGLE | ||
| speculative-num-steps: 3 | ||
| speculative-eagle-topk: 1 | ||
| speculative-num-draft-tokens: 4 | ||
|
|
||
| sbatch_directives: | ||
| mem: "0" | ||
| cpus-per-task: "144" | ||
|
|
||
| srun_options: | ||
| mem: "0" | ||
| container-remap-root: "" | ||
|
|
||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| IS_MULTINODE: "false" | ||
| TP: "8" | ||
|
There was a problem hiding this comment. TP8 aggregate forces single-node pathMedium Severity The new TP8 low-latency recipe spans two GB300 nodes (   Reviewed by Cursor Bugbot for commit 6219021. Configure here. |
||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TP8 aggregate missing multi-node NCCL
High Severity
The new TP8 low-latency aggregate recipe places one
tp-size: 8worker acrossagg_nodes: 2, butaggregated_environmentomits the GB300 multi-node interconnect settings used by sibling SGLang recipes (NCCL_MNNVL_ENABLE,NCCL_CUMEM_ENABLE,SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK). Cross-node tensor parallel init can fail or hang, so the concurrency 1/4 aggregate matrix points may never serve traffic.Additional Locations (1)
benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/agentic/agg-gb300-tp8-mtp-lowlatency.yaml#L26-L32Reviewed by Cursor Bugbot for commit c36d371. Configure here.