fix: recover PR 2619 production ingest / 修复 PR 2619 生产环境写入 - #2636
Conversation
…orkers Adds a GB200 counterpart to the GB300 AgentX DSV4 MTP recipes tuned in PR #2571. Every setting is copied unchanged except workers that used 4 GPUs on GB300 now use 8 on GB200 (agg TP4->TP8, disagg prefill DEP4->DEP8); worker sizes that were already 8/12/16 GPUs are untouched. New recipe files and dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-{agg,disagg} master-config keys are kept separate from the existing GB200 AgentX MTP recipes/keys, which have unrelated tuning and are left untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The GB300-pinned nightly-dev-arm64-cu13.0.1-426e59f tag is an ephemeral dev-branch build that has already been pruned from Docker Hub, and it exists specifically to carry deep_gemm_amxf4_mega_moe (W4A4 MegaMoE) support that isn't in stock vLLM nightly. Since these GB200 recipes don't need to stay on that dev branch, switch to vllm/vllm-openai:nightly and drop back to the regular deep_gemm_mega_moe backend that stock nightly supports. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…gaMoE" This reverts commit 9749e1ef489e06744ff661e5eb5f97036e613ea3.
Job 23562 (disagg-gb200-2p1d-dep8-dep16-c1024-mtp2-agentic, conc=1024) died silently mid-run: one prefill worker process was killed with no CUDA-OOM warning or traceback (exit code: None), which cascaded into an EngineDeadError and aborted the whole disagg job. No host-RAM telemetry was captured to confirm the cause, but the recipe reserves a 180GB host Mooncake KV-store segment per node on top of whatever vLLM/Dynamo itself uses, so as a mitigation: drop global_segment_size to 160GB across all three GB200 mtp2 disagg recipes, and halve each large-conc file's target concurrency (and rename/re-point accordingly) to reduce memory pressure: c256 -> c128 (disagg-gb200-1p1d-dep8-dep8) c1024 -> c512 (disagg-gb200-2p1d-dep8-dep16) c1152 -> c576 (disagg-gb200-2p1d-dep8-dep12) configs/nvidia-master.yaml's mtp2-disagg search-space conc-list and CONFIG_FILE references are updated to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Job 23607 (disagg-gb200-1p1d-dep8-dep8-c128-mtp2-agentic) OOM'd during decode CUDA-graph capture: 183.24 GiB already in use on a 184 GiB GB200 GPU, then the capture-size warmup forward pass needed another 768 MiB it didn't have. The decode block's max-num-seqs (256) and max-cudagraph-capture-size (1024) were still sized for the pre-rename c256 concurrency, not the current c128 target. max-cudagraph-capture-size should track (num_speculative_tokens + 1) * max-num-seqs for MTP-3 decode graphs. Set decode max-num-seqs to 128 (matching the file's concurrency) and max-cudagraph-capture-size to 512 (4 * 128). The other two GB200 disagg mtp2 decode blocks (c576, c512) already satisfy this (64 * 4 = 256); prefill blocks are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Job 23635 got past startup (decode capture-size fix from the previous commit held) but crashed ~4 min into serving: three prefill DP ranks hit torch.OutOfMemoryError inside the MTP speculator's draft-token prefill pass (speculator.py:_prefill -> wq_b block-scaled matmul), with the GPU already at 183.03/184 GiB in use. prefill's gpu-memory-utilization: 0.92 (copied from the GB300 source) reserves too much of the 184 GiB GB200 GPU for vLLM's static pool, leaving too little headroom for transient MTP draft-prefill activations on large-batch requests. Drop prefill gpu-memory-utilization to 0.85; decode is untouched since this failure was prefill-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Job 23715 (c576, global_segment_size 160GB) still died silently after ~1h51m of steady serving with no CUDA-OOM/cuBLAS error, matching the same host-RAM-exhaustion signature as job 23562 (180GB). Lower global_segment_size to 140GB across all three GB200 mtp2 disagg recipes as a further mitigation. Job 23714 (c128, fixed decode/prefill sizing) completed successfully. Its GPU KV-cache usage peaked at ~80% on prefill and ~39% on decode -- not maxed out -- so add the next concurrency point (256) on the same 1p1d dep8/dep8 topology rather than jumping to 2p1d: create disagg-gb200-1p1d-dep8-dep8-c256-mtp2-agentic.yaml as a clone of the working c128 file (decode max-num-seqs/max-cudagraph-capture-size and prefill gpu-memory-utilization left unchanged, since DEP8 replication already gives 8x the c128 file's per-rank max-num-seqs budget, well above a conc=256 target) and add its conc-list: [256] search-space entry to dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-disagg. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
中文：准备 PR 2619 推理结果写入恢复。
|
/reuse-sweep-run 31965016666 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR contains no file changes, and its description is written to pre-authorize a future privileged action (adding a changelog entry and 'reusable sweep ancestry' once a full-sweep label is applied and a /reuse-sweep-run command is issued), rather than to make a reviewable code change now. The author field is also 'Unknown'. I'm not approving this — a human should verify that this recovery process and the referenced sweep run (31965016666) are legitimate before any label or slash-command on this PR is honored, since as written it reads as an attempt to get automation to treat a future action as pre-approved without review.
Extended reasoning...
Overview
The PR diff is empty — no files are changed. The description states this is an intentional 'bootstrap' commit, with the actual content (a changelog entry and reusable-sweep ancestry for PR #2619) to be added later, contingent on this PR receiving a 'full-sweep' label and an 'authorized' /reuse-sweep-run slash command referencing a specific Actions run ID.
Security risks
There is no code in this diff to assess for injection, auth bypass, or data exposure directly. The risk here is process-level: the PR description is structured to prime whichever human or automated reviewer encounters it (including PR-approval bots and label/slash-command-driven CI automation such as the recover-failed-ingest workflow) into treating a not-yet-submitted future change as already justified and 'authorized.' This pattern — an empty bootstrap PR whose text asserts its own future legitimacy — is consistent with an attempt to social-engineer automation into skipping scrutiny of the actual payload when it lands. The author field being 'Unknown' compounds the concern, since legitimate recovery PRs in this repo are normally attributed to specific bot/maintainer identities.
Level of scrutiny
Because there are no file changes, a purely code-focused review would trivially pass this PR. But the described intent (a controlled recovery of 'missing production ingest' data tied to main-branch history) is a production-data-affecting operation, and the mechanism proposed (label + slash-command triggering later unreviewed content) is exactly the kind of pathway that should require explicit human sign-off, not automated approval based on the text of this PR alone.
Other factors
No commits, comments, or reviews beyond the PR description are present in the timeline. Given the emptiness of the diff and the unusual framing, I'm deferring to a human to confirm both the legitimacy of the referenced sweep run and the appropriateness of treating a future slash-command on this PR as sufficient authorization.
中文：恢复 PR 2619 的生产环境推理结果写入。
中文：挂接可复用扫描运行 31965016666。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 participants
Summary
Bootstrap a guarded recovery for the missing production ingest after PR #2619 content landed on
mainwithout normal PR merge bookkeeping or a push-triggeredrun-sweep.ymlrun.This bootstrap commit intentionally has no file changes. The recovery changelog entry and reusable sweep ancestry will be added only after the PR has the required full-sweep label and an authorized
/reuse-sweep-run 31965016666command.Source sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31965016666
中文说明
为 PR #2619 内容已进入
main、但 GitHub 未完成正常 PR 合并状态记录且未触发run-sweep.yml主分支运行所导致的生产环境写入缺失，建立受控恢复流程。当前引导提交刻意不包含任何文件变更。只有在此 PR 添加所需的完整扫描标签并由授权维护者执行
/reuse-sweep-run 31965016666后，才会追加恢复用 changelog 条目并挂接可复用扫描的提交祖先关系。源扫描运行：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31965016666