Repository navigation
[AgentX] MiniMax-M3.1-Flash GB300 SGLang: update config / [AgentX] MiniMax-M3.1-Flash GB300 SGLang：更新配置 #4686
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,202 @@ | ||
| schema: 2 | ||
|
|
||
| base: | ||
| name: minimaxm31flash-fp4-gb300-dynamo-sglang-agentx-disagg | ||
| identity: | ||
| model: {} | ||
| frameworks: | ||
| dynamo: "1.7.0.dev20261030" | ||
| sglang: "0.0.0.dev1+ge3718813e" | ||
| dynamo: | ||
| install: false | ||
| request_plane: tcp | ||
| event_plane: zmq | ||
| environment: | ||
| HF_HUB_CACHE: /hf_hub_cache | ||
| # Tachometer's per-node exporters slow decode steps and add host memory; | ||
| # InferenceX does not read its output. | ||
| observability: | ||
| tachometer: | ||
| enabled: false | ||
| slurm: | ||
| time_limit: "6:00:00" | ||
| health_check: | ||
| max_attempts: 1440 | ||
| interval_seconds: 10 | ||
| resources: | ||
| gpu_type: gb300 | ||
| gpus_per_node: 4 | ||
| services: | ||
| - name: etcd | ||
| type: etcd | ||
| placement: | ||
| node: infra | ||
| # Disables Dynamo's per-request lifecycle span export; InferenceX does not read it. | ||
| - name: otel-collector | ||
| type: otel-collector | ||
| enabled: false | ||
| # Keeps tokenize, route and stream CPU work off the engine trays. | ||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
| placement: | ||
| node: dedicated | ||
| env: | ||
| DYN_ROUTER_TEMPERATURE: "0" | ||
| DYN_ROUTER_QUEUE_THRESHOLD: None | ||
| DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: "3600" | ||
| DYN_TCP_CONNECT_TIMEOUT: "120" | ||
| # The default frontend tokenizer disables its prefix cache for this model, | ||
| # so every turn would re-tokenize the full prompt. | ||
| DYN_TOKENIZER: "fastokens" | ||
| DYN_TOKENIZER_CACHE: "1" | ||
| DYN_TOKENIZER_CACHE_BYTES: "8000000000" | ||
| args: | ||
| router-mode: kv | ||
| router-session-affinity-ttl-secs: "14400" | ||
| active-decode-blocks-threshold: "None" | ||
| active-prefill-tokens-threshold: "None" | ||
| active-prefill-tokens-threshold-frac: "None" | ||
| engine: sglang | ||
| roles: | ||
| prefill: | ||
| nodes: 3 | ||
| workers: 6 | ||
| gpus: 2 | ||
| env: | ||
| PYTHONUNBUFFERED: "1" | ||
| PYTHONNOUSERSITE: "1" | ||
| FMHA_SM100_ALLOW_JIT: "1" | ||
| SGLANG_TIMEOUT_KEEP_ALIVE: "1800" | ||
| SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600" | ||
| SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600" | ||
| SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True" | ||
| SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "32768" | ||
| NCCL_MNNVL_ENABLE: "1" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| MC_FORCE_MNNVL: "1" | ||
| MC_ENABLE_DEST_DEVICE_AFFINITY: "1" | ||
| # Some GB300 nodes list a BMC interface with the same address on every node first. | ||
| MC_TCP_BIND_ADDRESS: "{node}" | ||
| WITH_NVIDIA_PEERMEM: "0" | ||
| # Dynamo 1.7 backend admission rejects turns with 529 once a DP rank is full; | ||
| # the SGLang scheduler queues them instead. | ||
| DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT: "10000" | ||
| DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT: "10000" | ||
| DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT: "10000" | ||
| args: | ||
|
Comment on lines
+67
to
+87
Contributor
There was a problem hiding this comment. 🔴 Turns can get 529 rejections under load if Dynamo 1.7's default backend admission limits apply. That happens when one DP rank fills before the engine queue. The prefill and decode env blocks never raise the DYN_BACKEND_ADMISSION_* limits. Every other Dynamo 1.7 SGLang AgentX recipe raises them. Fix: add DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT and DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT to both role envs so the SGLang scheduler queues work instead of Dynamo rejecting it. Why this was flaggedThis applies if Dynamo 1.7.0.dev20261030 (identity.frameworks.dynamo, disagg.yaml:13) keeps the admission behaviour that dsv41flash/sglang/gb300-fp4/agentx/disagg-dep16-hicache-variants.yaml:87-91 describes: "Dynamo 1.7 backend admission rejects turns with 529 once a DP rank is full". dsv41flash agg-variants.yaml:69-73 says the same. Both recipes set the three limits to "10000", and the minimaxm31flash vLLM gb300 recipes raise them too. The new recipe sets none of them in the prefill env (disagg.yaml:76-91) or the decode env (disagg.yaml:133-148). Decode at conc 360 runs 48 slots per DP rank (max-running-requests 384 over 8 ranks), with load-balance-method total_tokens. That is not request-count balancing, so a single rank can fill while others still have room. The AgentX client would then see failed turns, which can fail the sweep or skew its results. Verification: The new recipe pins |
||
| served-model-name: MiniMaxAI/MiniMax-M3.1-Flash | ||
| trust-remote-code: true | ||
| reasoning-parser: minimax-m3 | ||
| tool-call-parser: minimax-m3 | ||
| tensor-parallel-size: 2 | ||
| data-parallel-size: 2 | ||
| expert-parallel-size: 2 | ||
| enable-dp-attention: true | ||
| enable-dp-lm-head: true | ||
| # flashinfer_trtllm_routed needs attention TP to match EP, which attention DP breaks. | ||
| moe-a2a-backend: megamoe | ||
| disaggregation-transfer-backend: mooncake | ||
| chunked-prefill-size: 8192 | ||
| # Shortest uncached prefill first: short cached turns stop queueing | ||
| # behind long uncached prompts, which cuts the TTFT tail. | ||
| schedule-policy: shortest-prefill-first | ||
| # 64 per DP rank. | ||
| max-running-requests: 128 | ||
| mem-fraction-static: 0.85 | ||
| enable-hierarchical-cache: true | ||
| # 200 GB per rank (400 of the 482 GB per-worker allowance): at 60 | ||
| # sessions per prefill worker a 150 GB host pool thrashes and the prefix | ||
| # hit rate collapses. | ||
| hicache-size: 200 | ||
| hicache-write-policy: write_back | ||
| speculative-algorithm: DSPARK | ||
| speculative-dspark-block-size: 7 | ||
| speculative-use-block-verification: true | ||
| speculative-draft-kv-cache-dtype: fp8_e4m3 | ||
|
Contributor
There was a problem hiding this comment. 🔴 This new config runs the DSpark draft with an FP8 KV cache while the target keeps its default bf16 KV, which CONTRIBUTING.md forbids. Both roles set Why this was flaggedCONTRIBUTING.md:64-66 allows only the framework's default draft KV dtype, or an upstream-supported dtype applied to both target and draft. CONTRIBUTING.md:90 forbids draft-only dtype overrides that lower precision below the default. The prefill (disagg.yaml:119) and decode (disagg.yaml:169) args set Verification: The prefill args (disagg.yaml:116-119) and the decode args (disagg.yaml:166-169) both set |
||
| # A session's turns stay on one prefill worker. | ||
| router-mode: kv | ||
| router-session-affinity-ttl-secs: "14400" | ||
| watchdog-timeout: 3600 | ||
| enable-metrics: true | ||
| enable-cache-report: true | ||
| kv_events: true | ||
| decode: | ||
| nodes: 2 | ||
| workers: 1 | ||
| gpus: 8 | ||
| env: | ||
| PYTHONUNBUFFERED: "1" | ||
| PYTHONNOUSERSITE: "1" | ||
| FMHA_SM100_ALLOW_JIT: "1" | ||
| SGLANG_TIMEOUT_KEEP_ALIVE: "1800" | ||
| DYN_SGL_GATEWAY_WORKERS: "8" | ||
| SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600" | ||
| SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600" | ||
| SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True" | ||
| SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "16384" | ||
| NCCL_MNNVL_ENABLE: "1" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| MC_FORCE_MNNVL: "1" | ||
| MC_ENABLE_DEST_DEVICE_AFFINITY: "1" | ||
| MC_TCP_BIND_ADDRESS: "{node}" | ||
| WITH_NVIDIA_PEERMEM: "0" | ||
| DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT: "10000" | ||
| DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT: "10000" | ||
| DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT: "10000" | ||
| args: | ||
| served-model-name: MiniMaxAI/MiniMax-M3.1-Flash | ||
| trust-remote-code: true | ||
| reasoning-parser: minimax-m3 | ||
| tool-call-parser: minimax-m3 | ||
| tensor-parallel-size: 8 | ||
| data-parallel-size: 8 | ||
| expert-parallel-size: 8 | ||
| enable-dp-attention: true | ||
| enable-dp-lm-head: true | ||
| moe-a2a-backend: megamoe | ||
| disaggregation-transfer-backend: mooncake | ||
| # The frontend picks the decode DP rank; least-loaded sends each request to | ||
| # the rank with the fewest in flight, where round-robin left the busiest rank | ||
| # at 1.45x the mean and every DP rank waits for it each step. | ||
| router-mode: least-loaded | ||
| load-balance-method: total_tokens | ||
| # 48 per DP rank. | ||
| max-running-requests: 384 | ||
| cuda-graph-max-bs-decode: 48 | ||
| mem-fraction-static: 0.85 | ||
| speculative-algorithm: DSPARK | ||
| speculative-dspark-block-size: 7 | ||
| speculative-use-block-verification: true | ||
| speculative-draft-kv-cache-dtype: fp8_e4m3 | ||
| watchdog-timeout: 3600 | ||
| enable-metrics: true | ||
| kv_events: false | ||
| sbatch_directives: | ||
| mem: "0" | ||
| exclusive: "" | ||
| srun_options: | ||
| container-remap-root: "" | ||
| mem: "0" | ||
| benchmark: | ||
| placement: | ||
| node: dedicated | ||
| env: | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:" | ||
|
|
||
| override_6p_dep2_dep8: | ||
| name: minimaxm31flash-fp4-gb300-dynamo-sglang-agentx-disagg-6p-dep2-dep8 | ||
|
|
||
| override_6p_dep2_dep16: | ||
| name: minimaxm31flash-fp4-gb300-dynamo-sglang-agentx-disagg-6p-dep2-dep16 | ||
| roles: | ||
| decode: | ||
| nodes: 4 | ||
| gpus: 16 | ||
| args: | ||
| tensor-parallel-size: 16 | ||
| data-parallel-size: 16 | ||
| expert-parallel-size: 16 | ||
| max-running-requests: 768 | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 nit (optional): a two-row bundle is named
disagg.yaml, but the recipe guide requires<agg|disagg>[-<axis>]-variants.yamlfor multi-row entries. Every sibling sglang AgentX disagg bundle follows that rule. Fix: rename todisagg-variants.yamland update bothsrt-recipe:selectors in nvidia-master.yaml.Why this was flagged
RECIPES.md:23 says an entry with two or more rows of one design keeps them in one
<agg|disagg>[-<axis>]-variants.yaml. AGENTS.md:90 repeats that rows of one design share a*-variants.yaml. The master entry selects two rows from this file:disagg.yaml:override_6p_dep2_dep8anddisagg.yaml:override_6p_dep2_dep16. Sibling sglang AgentX bundles, for example dsv41flash/sglang/gb300-fp4/agentx/disagg-dep16-hicache-variants.yaml and glm5.2/sglang/gb300-fp4/agentx/disagg-variants.yaml, use the -variants suffix. The plain name is reserved for single-row recipes, so this bundle breaks the convention that tells the two apart.Verification: nit. The new master entry takes two rows of one design from the bundle,
srt-recipe: disagg.yaml:override_6p_dep2_dep8andsrt-recipe: disagg.yaml:override_6p_dep2_dep16. AGENTS.md:90 says: "Rows of one design share a*-variants.yaml". The sibling glm5.2 hasdisagg-variants.yaml. No code in the repo checks the filename, so nothing fails at runtime.