Validate vLLM Router on GB200: DEP4, DEP8, and 1P/2D - #2549
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
cquil11
added
agentx-fast
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31425550346 |
…00-smoke # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31426276922 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31430860441 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31431838034 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31434053812 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31436843969 |
cquil11
changed the title
|
Exact-head GB200 Router validation is complete: workflow 31436843969 is green for aggregate DEP4, multinode aggregate DEP8, and disaggregated 1P/2D DEP4. The PR description now contains the per-job request counts, coverage, metrics artifacts, consistent-hash affinity audit, P/D two-stage evidence, and uploaded Router-log evidence. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31502477920 |
|
Closing because the project scope is now limited to porting active, nondeprecated AMD multi-node configurations to srt-slurm. This PR is outside that scope. |
|
Reopening: this PR predates the recent out-of-scope configuration-porting detour and was closed by mistake during an overly broad cleanup. Its prior state is being restored. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 66ea40c. Configure here.
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" |
There was a problem hiding this comment.
B200 Router recipes omit session affinity
Medium Severity
The new B200 vllm-router AgentX recipes never set Router consistent-hash policy or AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID. Follow-up turns will not pin to a DP rank, so prefix-cache reuse and disagg P/D affinity break and the AgentX numbers will not match the session-aware GB200 recipes.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 66ea40c. Configure here.
| additional-settings: | ||
| - "SRT_SLURM_REPOSITORY=https://github.com/SemiAnalysisAI/srt-slurm.git" | ||
| - "SRT_SLURM_REF=ab98030f0993bdb1595acc25e883b59d76ad7791" | ||
| - "CONFIG_FILE=recipes/vllm/deepseek-v4/agentic/disagg-b200-1p1d-dep8-mtp-vllm-router-agentic.yaml" |
There was a problem hiding this comment.
B200 Router pins unfixed readiness SHA
Medium Severity
The B200 vllm-router configs pin srt-slurm ab98030f, which still treats Router as ready after any one healthy host. The all-base /health barrier that stops a multi-host job from starting while an advertised worker is down lives on b4bf6eb0, so these aggregate DEP8 and 1P/1D jobs can begin AIPerf against a partial pool.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 66ea40c. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline findings, I also looked at the new drain-wait branch in wait_for_agentic_servers_idle (benchmarks/multi_node/agentic_srt.sh) for sgl-router/vllm-router jobs, where frontend_metrics_url is now left empty — the PR's own validated runs show AIPERF_SERVER_METRICS_URLS (worker_urls) populated end-to-end for these frameworks, so the idle poll still has a real signal source rather than none at all.
Extended reasoning...
This run's inline findings (the gb200 vllm-router success-gate ordering relative to log bundling, and the perf-changelog entry omitting the new dsv4/glm5.2 sgl-router and vllm-router master-config keys) are already queued to post as inline comments and are not restated here. Beyond those, I independently re-examined the candidate concern that wait_for_agentic_servers_idle has no drain signal at all for non-dynamo frameworks now that frontend_metrics_url is scoped to dynamo-*. Tracing AIPERF_SERVER_METRICS_URLS/worker_urls through benchmark_lib.sh confirms it is a general multi-node-launcher-supplied per-worker metrics list independent of the frontend type, and the PR's documented live GB200/B200 runs show backend metrics collectors reachable and required vllm:-prefixed metrics validated for the vllm-router/sgl-router jobs, which is strong evidence the worker-side signal is actually present in practice for these frameworks, not absent. Given the two confirmed findings are substantive (a success-gate ordering bug and a changelog/master-config coverage gap) and this PR already has a prior review from this same automated system with unresolved threads whose fix status can't be independently confirmed from metadata alone, a human review is still warranted; I am not adding a new top-level concern beyond acknowledging this one ruled-out item.
|
|
||
| stream_slurm_job_log "$JOB_ID" "$LOG_FILE" || exit 1 | ||
|
|
||
| if [[ "$FRAMEWORK" == "vllm-router" ]]; then | ||
| wait_for_slurm_job_success "$JOB_ID" || exit 1 |
There was a problem hiding this comment.
🔴 New vllm-router success gate exits before log bundling on gb200; cleanup_srt_job trap (lines 719-725, pre-existing) only runs scancel, never bundle_server_logs. The new wait_for_slurm_job_success "$JOB_ID" || exit 1 at line 737 fires before the unconditional LOGS_DIR copy / bundle_server_logs call at line 743-755, so a failed vllm-router job loses its log artifacts on exit via that trap.
Extended reasoning...
Before this diff, stream_slurm_job_log (runners/slurm_utils.sh:64-87) only fails if the log file never appears; once it exists, wait always returns 0 regardless of job outcome, so 'Collecting results' (LOGS_DIR copy + bundle_server_logs) always ran, even for a failed dynamo job. This PR adds if [[ "$FRAMEWORK" == "vllm-router" ]]; then wait_for_slurm_job_success "$JOB_ID" || exit 1; fi right before that section. When a GB200 vllm-router Slurm job ends FAILED/TIMEOUT/OOM/CANCELLED, this now exits early through the EXIT trap cleanup_srt_job, which only does scancel — never bundle_server_logs — so the workflow uploads no server logs or LOGS/ directory for the failed run, unlike every other framework and unlike the sibling launcher runners/launch_h200-dgxc-slurm.sh, which was updated (line 251) to move bundle_server_logs into its EXIT trap specifically to survive this new early-exit path. Fix: add bundle_server_logs to gb200-nv.sh's cleanup_srt_job trap (mirroring launch_h200-dgxc-slurm.sh) so failed-router-job diagnostics are still uploaded.
Verification: normal. New gate at runners/launch_gb200-nv.sh:736-738 (if [[ "$FRAMEWORK" == "vllm-router" ]]; then wait_for_slurm_job_success "$JOB_ID" || exit 1; fi) runs before the log-collection block at lines 743-758 (cp -r "$LOGS_DIR" "$GITHUB_WORKSPACE/LOGS" and bundle_server_logs "$LOGS_DIR" "$GITHUB_WORKSPACE/multinode_server_logs.tar.gz"). wait_for_slurm_job_success returns 1 for…
| - config-keys: | ||
| - dsr1-fp4-gb200-vllm-router-agentic-agg | ||
| - dsr1-fp4-gb200-vllm-router-agentic-agg-dep8 | ||
| - dsr1-fp4-gb200-vllm-router-agentic-disagg | ||
| - dsr1-fp4-gb200-vllm-router-agentic-disagg-2p2d | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Validate first-class vLLM Router 0.1.15 through srt-slurm on GB200 without a Dynamo request plane." | ||
| - "Exercise three focused AgentX-fast topologies: single-node aggregate DEP4, two-node aggregate DEP8, and disaggregated one-prefill/two-decode with DEP4 on every logical endpoint." | ||
| - "Follow the official vLLM DeepSeek R1 Blackwell recipe: the staged nvidia/DeepSeek-R1-0528-NVFP4-v2 checkpoint, vLLM 0.25.1, DEP4 with expert parallelism, FP8 KV cache, and explicit prefix caching." | ||
| - "Pin the test deployment to exact SemiAnalysisAI/srt-slurm PR head b4bf6eb0abefa16c5108b339dabb903214a4c806, preserve each node-local backend Prometheus endpoint for AIPerf, and require the vllm: metric family before accepting results." | ||
| - "Launch one native vLLM hybrid data-parallel server per node-local pool and let the official Router expand each base URL into DP ranks through intra-node-data-parallel-size, matching vLLM's supported X-Data-Parallel-Rank contract." | ||
| - "Use consistent-hash routing for aggregate traffic and for both prefill and decode selection in the 1P/2D case; map AIPerf's stable conversation correlation ID to X-Session-ID so every turn returns to the same DP rank and reuses its conversation KV cache." | ||
| - "Use the 062126-256k AgentX corpus while enforcing the checkpoint's supported 163,840-token maximum in vLLM and through an explicit AIPERF_MAX_CONTEXT_LENGTH recipe contract, so trace selection cannot dispatch a context the server must reject." | ||
| - "Reserve 15% of GB200 HBM outside vLLM's KV-cache budget so long-context AgentX prefills retain activation workspace; the initial 0.90 setting left only 0.9-2.1 GiB free and reproducibly OOMed on 2.4-4.0 GiB runtime allocations." | ||
| - "Remove unsupported NCCL symmetric-memory and UCX overrides that invalidated CUDA graph capture before the Router could be exercised; retain the official FlashInfer FP4 MoE path." | ||
| - "Enable vLLM prompt-token details on both native Router deployments so AgentX captures response-level prompt-cache usage in addition to each logical backend's Prometheus metrics." | ||
| - "Verify the terminal Slurm state and exit code after log streaming so a failed benchmark cannot be reported as a successful GitHub job merely because its log tail ended cleanly." | ||
| - "Classify sgl-router and vllm-router as explicit runtime criteria in the CI scheduler, matching the router-specific framework keys before their broader engine prefixes." | ||
| - "Drain sequential native-router AgentX points from all explicit logical worker metrics without requiring a Dynamo frontend gauge; recognize both vllm:num_requests_* and sglang:num_requests_* worker families." | ||
| - "Require every logical vLLM backend URL advertised to Router to pass direct /health readiness after Router's expanded /workers count succeeds, closing the startup race where Router reports all configured ranks after only one healthy host per P/D pool." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2549 |
There was a problem hiding this comment.
🟡 New sweep config keys dsv4-fp4-b200-vllm-router-agentic-agg, dsv4-fp4-b200-vllm-router-agentic-disagg (configs/nvidia-master.yaml:969,999), dsv4-fp4-b200-sgl-router-agentic-agg (line 1155), and glm5.2-fp8-h200-sgl-router-agentic-mtp-2p2d (line 7997) are added to configs/nvidia-master.yaml but have no perf-changelog.yaml entry; the single new changelog entry added by this diff only lists the four dsr1-fp4-gb200-vllm-router-* keys.
Extended reasoning...
AGENTS.md requires every recipe addition to get a perf-changelog entry, but no automated gate cross-checks master-config keys against changelog coverage (validate_perf_changelog.py/process_changelog.py only check that changelog config-keys exist in master config, not the reverse), so these four new recipes merge with zero changelog history explaining their AIPERF_MAX_CONTEXT_LENGTH cap, router version, or GPU-memory-utilization rationale — future reviewers/bisectors lose the audit trail for these specific lanes.
Verification: Severity: nit (convention/audit-trail gap; no runtime failure, crash, data loss, or security issue — nothing that works on the base branch breaks). The factual claim is real and reachable. Four new master-config keys are added by this diff with no perf-changelog entry: - configs/nvidia-master.yaml:969 dsv4-fp4-b200-vllm-router-agentic-agg - configs/nvidia-master.yaml:999… | nit…
1 participant
Summary
frontend.type: vllm-routerfrom SemiAnalysisAI/srt-slurm commitb4bf6eb0abefa16c5108b339dabb903214a4c806nvidia/DeepSeek-R1-0528-NVFP4-v2, and consistent-hash routing keyed byX-Session-IDgpu-memory-utilization: 0.85) while retaining ample KV capacityInferenceX changes
AIPERF_MAX_CONTEXT_LENGTHpath to the shared AgentX replay builder; recipes that do not set it retain existing behaviorX-Session-IDand configure official Router consistent hashingvllm:metrics and pass one explicit metrics URL per logical vLLM worker leaderEarlier hardware validation
Earlier tested InferenceX head:
395d57f0e1355628e84e12b9826d0ee634d9c631Full workflow: 31436843969 — success
COMPLETED 0:0COMPLETED 0:0COMPLETED 0:0Every job returned
replay_rc=0, passed the requiredvllm:metrics-prefix gate, and uploaded nonempty request and server-metrics artifacts.Router behavior proved from live and uploaded logs
X-Session-IDextraction, explicit prefill and decode selections, HTTP 200 decode responses, and repeatedTwo-stage processing completed successfullyMetrics behavior
/metricsendpoint in this mode, so AIPerf reports 3/4 while the required backend metrics and finalvllm:gate passFailure-driven correction
The preceding exact-head attempt proved topology, Router, dataset, and metrics plumbing but all three workers later OOMed during long-context warmup at
gpu-memory-utilization: 0.90. The failures were vLLM CUDA allocation failures, not Router failures. Reducing the reservation to 0.85 leaves roughly 9 GiB more activation workspace per GPU while still providing about 47 GiB KV per DEP4 rank and 86 GiB per DEP8 rank. The successful workflow above validates that correction under the full fast profile.Static validation
git diff --checkpassThis validation branch is stacked on the existing Router integration work while keeping the three GB200 evidence points isolated and reproducible.
Final Router-readiness validation
Current InferenceX head:
66ea40cc7060dfa84b492db6f24fe9c18f8cde5bPinned srt-slurm head:
b4bf6eb0abefa16c5108b339dabb903214a4c806Readiness correction
The official Router considers a pool ready after any one unique backend host is healthy, while
/workerscan list all DP-expanded URLs before every registered base API is usable. The earlier srt-slurm parser counted registered URLs without requiring every underlying base, which allowed a 2P/2D benchmark to start while one advertised P/D host was still unavailable.The srt-slurm fix keeps this behavior behind the vLLM Router adapter: after the expected expanded
/workerscount passes, it derives every unique direct/healthURL from the logical bases advertised to Router and requires HTTP 200 from all of them. Other frontend readiness semantics are unchanged. Focused tests pass 54/54; the full local suite passes 942 tests, with five unrelated pre-existing platform/path/mock failures; exact-head srt-slurm CI is green.Disaggregated 2P/2D evidence
Job 93681770486 succeeded on Slurm 22971. Router expanded to 8P + 8D; the all-base barrier held until all four P/D APIs were healthy; the realized AIPerf command contained all four explicit metrics URLs and captured 16 backend engine series. The profile completed 518/518 successful with 0 errors, passed the required
vllm:metric gate, and repeated session IDs retained stable affinity independently in prefill and decode pools. Six artifacts, including Router and all server logs plus raw/aggregate AgentX results, are present.The parent workflow 31460081245 is red only because its workflow definition came from an unrelated
mainrevision whose missingEVALSassignment causedfromJson('')at graph construction. Every materialized workload/result job succeeded; this is not a Router or benchmark failure.Aggregate multinode DEP8 evidence
Workflow 31497389903 and job 93798572870 are green on exact tested InferenceX head
08f55e5c7d83a86ee4474c3db29cf5c818924716; Slurm 22973 completed0:0./healthendpoints before AIPerf started.vllm:gate, and successfully collected Router plus both explicit backend metrics endpoints.10.30.1.165:6100and engines 4-7 from10.30.1.39:6100.X-Session-IDmappings across 46 sessions, 34 repeated sessions, zero affinity changes, and routed traffic on all eight ranks.gb200-nv: 1/1.Head
66ea40cc7only synchronizes newer upstream content and resolves the append-only changelog comparison after that exact run; it does not change the validated Router runtime/configuration from08f55e5c7.Current-head official sweep
The official InferenceX sweep on current integration head
66ea40cc7060dfa84b492db6f24fe9c18f8cde5bis fully green: workflow 31502477920 completed withsuccess.COMPLETED 0:0COMPLETED 0:0COMPLETED 0:0COMPLETED 0:0Every performance point passed the required
vllm:metrics gate. Live logs confirm that the Router expanded the expected DP ranks and the new direct barrier required all advertised base APIs before each benchmark. Every eval completed all 1,319 GSM8K requests witheval_exit=0; collect-evals also succeeded. The workflow published 22 nonexpired artifacts (about 115 MB), including raw/aggregate performance results, all evaluation outputs, and Router plus server logs for every topology.Note
Medium Risk
Large additive benchmark and Slurm-launcher surface area on production clusters; failures affect CI cost and result trust, but changes are mostly config and orchestration rather than core inference code.
Overview
Adds first-class
sgl-routerandvllm-routerruntimes to CI priority, cluster launchers, and AgentX metric backends, alongside new multi-node srt-slurm recipes and master sweep entries for GB200 DeepSeek-R1 (aggregate DEP4/DEP8, disagg 1P/2D and 2P/2D) plus B200/H200 DSV4 and GLM-5.2 agentic paths.AgentX harness: optional
AIPERF_MAX_CONTEXT_LENGTH(with validation) for replay caps; agentic runs no longer blindly inherit workflowMAX_MODEL_LEN.agentic_srt.shdrains on worker vLLM/SGLang queue metrics and only polls a Dynamo frontend gauge when applicable.Launch plumbing: exact
SRT_SLURM_REPOSITORY/SRT_SLURM_REFcheckouts with in-repo recipe overlays; router dependency setup scripts (released wheels plus a fork-built SGLang router wheel for P/D fix);wait_for_slurm_job_successso native-router jobs fail the workflow on non-zero Slurm exit.Sweep classification now treats
sgl-router/vllm-routeras distinct framework criteria.Reviewed by Cursor Bugbot for commit 66ea40c. Bugbot is set up for automated code reviews on this repo. Configure here.