Repository navigation
Refresh GB200 Qwen3.5 AgentX Pareto / 刷新 GB200 Qwen3.5 AgentX Pareto 配置 #2660
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| name: qwen35-gb200-sglang-agentic-mtp-agg-tp2ep2-hicache | ||
|
|
||
| model: { path: qwen3.5-fp4, container: lmsysorg/sglang:nightly-dev-20260818-c0b6474b, precision: fp4 } | ||
| identity: | ||
| model: { repo: nvidia/Qwen3.5-397B-A17B-NVFP4-V2 } | ||
| container: { image: lmsysorg/sglang:nightly-dev-20260818-c0b6474b } | ||
| slurm: { time_limit: "8:00:00" } | ||
| health_check: { max_attempts: 2160, interval_seconds: 10 } | ||
| resources: { gpu_type: gb200, gpus_per_node: 4, agg_nodes: 1, agg_workers: 1, gpus_per_agg: 2 } | ||
| infra: { nats_max_payload_mb: 8 } | ||
| frontend: | ||
| type: sglang | ||
| args: | ||
| worker-startup-timeout-secs: 3600 | ||
|
|
||
| backend: | ||
| type: sglang | ||
| aggregated_environment: | ||
| PYTHONNOUSERSITE: "1" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| NCCL_MNNVL_ENABLE: "1" | ||
| NCCL_NVLS_ENABLE: "1" | ||
| SGLANG_ENABLE_FLASHINFER_GEMM: "true" | ||
| SGLANG_ENABLE_SPEC_V2: "1" | ||
| SGL_ENABLE_JIT_DEEPGEMM: "false" | ||
| TORCH_CUDA_ARCH_LIST: "10.0" | ||
| sglang_config: | ||
| aggregated: | ||
| served-model-name: nvidia/Qwen3.5-397B-A17B-NVFP4-V2 | ||
| model-path: /model/ | ||
| trust-remote-code: true | ||
| tensor-parallel-size: 2 | ||
| pipeline-parallel-size: 1 | ||
| data-parallel-size: 1 | ||
| expert-parallel-size: 2 | ||
| enable-dp-attention: false | ||
| enable-dp-lm-head: false | ||
| enable-symm-mem: false | ||
| quantization: modelopt_fp4 | ||
| fp4-gemm-backend: flashinfer_cutlass | ||
| kv-cache-dtype: fp8_e4m3 | ||
| mamba-ssm-dtype: bfloat16 | ||
| mamba-radix-cache-strategy: extra_buffer_lazy | ||
| mamba-track-interval: 1048576 | ||
| attention-backend: trtllm_mha | ||
| linear-attn-prefill-backend: flashinfer | ||
| linear-attn-decode-backend: flashinfer | ||
| moe-runner-backend: flashinfer_trtllm | ||
| speculative-moe-runner-backend: flashinfer_trtllm | ||
| speculative-algorithm: NEXTN | ||
| speculative-num-steps: 3 | ||
| speculative-eagle-topk: 1 | ||
| speculative-num-draft-tokens: 4 | ||
| enable-linear-replayssm-spec: true | ||
| linear-replayssm-cache-len: 8 | ||
| cuda-graph-max-bs: 64 | ||
| max-running-requests: 80 | ||
| max-prefill-tokens: 16384 | ||
| chunked-prefill-size: 16384 | ||
| mem-fraction-static: 0.88 | ||
| # extra_buffer_lazy uses four physical Mamba state slots per running request. | ||
| # C40 remains healthy while C48 reaches the device-KV capacity cliff. | ||
| max-mamba-cache-size: 320 | ||
| allow-auto-truncate: true | ||
| stream-interval: 50 | ||
| scheduler-recv-interval: 10 | ||
| tokenizer-worker-num: 6 | ||
| page-size: 64 | ||
| enable-hierarchical-cache: true | ||
| hicache-ratio: 2.5 | ||
| hicache-io-backend: direct | ||
| hicache-mem-layout: page_first_direct | ||
| hicache-write-policy: write_through_selective | ||
| mamba-max-states-per-path: 1 | ||
| weight-loader-prefetch-checkpoints: true | ||
| weight-loader-prefetch-num-threads: 4 | ||
| weight-loader-drop-cache-after-load: true | ||
| watchdog-timeout: 1000000 | ||
| enable-metrics: true | ||
| enable-cache-report: true | ||
|
|
||
| sbatch_directives: { mem: "0", cpus-per-task: "144" } | ||
| srun_options: { mem: "0", container-remap-root: "" } | ||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: /infmax-workspace | ||
| RESULT_DIR: /logs/agentic | ||
| PORT: "8000" | ||
| IS_MULTINODE: "false" | ||
| TP: "2" | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126_256k | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache | ||
| HF_HUB_CACHE: /hf_hub_cache | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6086,3 +6086,13 @@ | |
| - "Use TP8/EP1 concurrency 2 through 24, published FlashInfer 0.6.17 CUDA 13 wheels, and golden MTP acceptance length 3.39." | ||
| - "Use the lmsysorg/sglang:nightly-dev-cu13-20260815-a5ba081f image." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2592 | ||
|
|
||
| - config-keys: | ||
| - qwen3.5-fp4-gb200-dynamo-sglang-agentic-mtp | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Refresh GB200 Qwen3.5 NVFP4-V2 AgentX on lmsysorg/sglang:nightly-dev-20260818-c0b6474b without engine patches." | ||
| - "Add a TP2/EP2 HiCache branch and update the resident recipes with extra_buffer_lazy, ReplaySSM speculative verification, and checkpoint page-cache release." | ||
| - "Select TP4 C1/C2/C4/C8/C12/C20/C24, TP2/EP2 resident C4/C8/C12/C20, and TP2/EP2 HiCache C22/C24/C28/C32/C40 with golden synthetic acceptance length 3.39." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX | ||
|
Check warning on line 6098 in perf-changelog.yaml
|
||
|
Contributor
There was a problem hiding this comment. 🟡 The new changelog entry's Extended reasoning...Bug: The changelog entry appended at the end of pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXThis is PR #2660 (per the PR metadata), so the link should read Documented convention: Why it's not caught elsewhere: The changelog-key generation and CI ingest logic only read Impact: Low — this doesn't affect benchmark behavior, CI, or sweep execution. It's a traceability/documentation issue: anyone auditing Proof / walkthrough:
Fix: Change line 6098 to |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 This PR removes the last reference to
agg-gb200-tp4-mtp-hicache.yamlfromconfigs/nvidia-master.yaml(the TP4 HiCache search-space branch is replaced by the new TP2/EP2 resident + HiCache branches), leaving that recipe file orphaned and unable to pick up the new nightly image/flags its siblings received. Since unreferenced recipe files are common in this directory already, this is a minor nit — consider deleting it (or noting it as intentionally parked) as part of this cleanup.Extended reasoning...
What changed:
configs/nvidia-master.yamlpreviously wired one of theagentic-codingsearch-space branches underqwen3.5-fp4-gb200-dynamo-sglang-agentic-mtptoCONFIG_FILE=recipes/sglang/qwen3.5/gb200-fp4/agentic/agg-gb200-tp4-mtp-hicache.yaml. This PR removes that branch entirely and replaces it with two new branches — a TP2/EP2 resident arm and a TP2/EP2 HiCache arm — pointing atagg-gb200-tp2ep2-mtp.yamland the newly-addedagg-gb200-tp2ep2-mtp-hicache.yaml.The orphan:
benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb200-fp4/agentic/agg-gb200-tp4-mtp-hicache.yaml(added in PR #2568) is not touched by this PR's diff at all. A repo-wide grep foragg-gb200-tp4-mtp-hicacheafter this diff applies returns zero matches anywhere inconfigs/or the recipe tree — the file simply stops being reachable from the master config. It still carries the oldlmsysorg/sglang:v0.5.17-cu130image and the pre-refresh flag set (noextra_buffer_lazy, no ReplaySSM, no checkpoint page-cache release) that its siblings (agg-gb200-tp4-mtp.yaml,agg-gb200-tp2ep2-mtp.yaml) just received in this same PR.Why this isn't caught elsewhere: there's no CI check that cross-references recipe files against master-config
CONFIG_FILE=references, so nothing flags an orphaned recipe automatically — it just sits in-tree, increasingly stale relative to its siblings, until someone notices.Counter-consideration (from a reviewer who refuted this as a blocking issue): a repo-wide check shows roughly 191 of 423 recipe YAMLs under
srt-slurm-recipes/are already unreferenced by any master-configCONFIG_FILE=, so parking an inactive recipe in-tree is the established norm for this directory (it functions as a reusable recipe library, not a strict 1:1 mirror of the active sweep). That's a fair point, and it's why this is filed as a nit rather than a blocker — leaving the file is consistent with existing practice, and a future TP4-HiCache revival could reuse it as a starting point. The residual concern is narrower: this file was actively wired moments before this PR and became orphaned by this PR's own change (not pre-existing drift), and it now silently diverges from the nightly image/flag refresh its direct siblings received in the same diff, which could mislead a future reader into assuming the TP4-HiCache path is still validated against current settings.Proof of orphaning:
configs/nvidia-master.yamlline ~7266 (pre-diff) containsCONFIG_FILE=recipes/sglang/qwen3.5/gb200-fp4/agentic/agg-gb200-tp4-mtp-hicache.yamlunder thekv-offloading: dram/kv-offload-backend: hicachebranch withtp: 4.configs/nvidia-master.yaml:7232-7244shows it replaced by atp: 2, ep: 2, kv-offloading: nonebranch and atp: 2, ep: 2, kv-offloading: dram/hicachebranch, both pointing atagg-gb200-tp2ep2-mtp*.yaml.grep -r "agg-gb200-tp4-mtp-hicache" .post-diff → no hits outside the file's own path.container: { image: lmsysorg/sglang:v0.5.17-cu130 }while its siblings now readlmsysorg/sglang:nightly-dev-20260818-c0b6474b— confirming the drift.Suggested fix: delete
benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb200-fp4/agentic/agg-gb200-tp4-mtp-hicache.yamlas part of this cleanup, or leave a one-line note if it's intentionally kept for a future TP4-HiCache revival.