[Klaud Cold] Add H100 DeepSeek V4.1 Flash AgentX / 新增 H100 配方 - #2974
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it modifies a shared SLURM launcher (runners/launch_h100-dgxc-slurm.sh) that other H100 recipes depend on and introduces a new memory-capped benchmark script, a human look would still be worthwhile.
What was reviewed: the launcher's new framework-tagged/untagged script resolution and its /ix vs /workspace mount branching (covered by runners/test_dsv41flash_h100.py); the new dsv41flash_fp4_h100_vllm_mtp.sh indexer-buffer math and max-num-seqs scaling; the perf-changelog.yaml entry and configs/nvidia-master.yaml addition. Also checked: the new select_available_server_port() probe-then-bind race in benchmark_lib.sh (real TOCTOU window, but narrow and same pattern likely acceptable for this use), and the hardcoded MODEL_PREFIX == dsv41flash mount check versus sibling scripts' self-detecting INFMAX_CONTAINER_WORKSPACE fallback (works for this recipe but is less robust than the sibling pattern).
Extended reasoning...
Overview
This PR adds an H100 arm for DeepSeek-V4.1-Flash agentic serving: a new benchmark script (benchmarks/single_node/agentic/dsv41flash_fp4_h100_vllm_mtp.sh) with memory caps (--max-num-batched-tokens 4096, scaled --max-num-seqs, --gpu-memory-utilization 0.92), a config entry in configs/nvidia-master.yaml, a changelog entry, docs, and — described by the author as worth reviewing on its own — a fix to runners/launch_h100-dgxc-slurm.sh so it resolves framework-tagged script names (mirroring the H200 launcher) and mounts DeepSeek-V4.1-Flash's AgentX runtime dirs at /ix instead of /workspace. benchmark_lib.sh gains a new select_available_server_port() helper, and tests are added in runners/test_dsv41flash_h100.py and utils/evals/test_run_eval_dispatch.py.
Security risks
None identified. This is internal ML benchmarking infrastructure (SLURM job launching, vLLM server flags, config/YAML) with no auth, crypto, or externally-facing surface. No secrets or credentials are introduced or handled differently.
Level of scrutiny
Moderate-to-high is warranted despite the absence of confirmed bugs: the launcher change affects script resolution for all recipes on the h100-dgxc cluster (not just the new one), and getting the fallback logic wrong could silently run the wrong script for existing recipes. The PR does add direct test coverage for both the new framework-tagged path and the pre-framework qwen3.5 fallback path, which mitigates this. The perf-changelog.yaml/config changes are low-risk, mechanical additions following existing patterns.
Other factors
Three candidate issues were investigated by the automated bug hunt and ruled out rather than reported as findings: a pr-link: TBD changelog validation concern, the select_available_server_port() TOCTOU race, and the hardcoded /ix mount check diverging from the sibling scripts' self-detecting pattern. None of these were escalated as bugs, and my own reading of the diff and test files did not surface anything beyond what those investigations already covered. Given the shared-launcher blast radius and the presence of a real (if narrow) race condition that was judged acceptable rather than fixed, I think a brief human confirmation is worthwhile even though no bug was found.
This review covers commit 2e038e1, which is no longer the latest commit on this pull request; later commits are not covered by it.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34489122340 |
Follow-up to #2963, which dropped this arm after it OOMed at concurrency 1. The blocker is the sparse attention indexer's [max-num-batched-tokens, max-model-len] buffer, 16 GiB at 1M context with the shared script's 8192 batched tokens. Give H100 its own script that caps batched tokens at 4096, sets max-num-seqs from the trajectory concurrency instead of vLLM's default 1024, and enables expandable_segments. Also teach the h100 launcher to resolve framework-tagged script names, which it never could. 承接 #2963（该 PR 在此分支于并发 1 OOM 后将其移除）。瓶颈是稀疏注意力 indexer 的 [max-num-batched-tokens, max-model-len] 缓冲区，在 1M 上下文下按共享脚本的 8192 batched tokens 计为 16 GiB。为 H100 提供独立脚本：将 batched tokens 限制为 4096、 依轨迹并发设置 max-num-seqs（而非 vLLM 默认的 1024）、启用 expandable_segments。 同时让 h100 launcher 能够解析带 framework 的脚本名，此前它完全无法解析。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
补充 perf-changelog 的 pr-link。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The concurrency-1 agentx-fast run came up healthy and reported 13.47 GiB KV per GPU, 7,022,899 tokens, and 6.70x maximum concurrency at 1M tokens per request. Trim the list from 1-8 to 1-4 so a trajectory replaying near full context cannot drive the batch into preemption. 并发 1 的 agentx-fast 运行健康启动，报告每 GPU 13.47 GiB KV、7,022,899 tokens、 在每请求 1M tokens 下最大并发 6.70x。据此将并发列表从 1-8 收窄为 1-4，避免接近 满上下文的轨迹把批次推入抢占。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DSv4.1 Flash 吞吐测试使用 thinking 开启、五个草稿 token 对应的黄金合成 AL 3.51，关闭自适应验证；准确率评测保留真实验证。同步中英文文档并追加性能变更记录。
8cc14ae to
9164c5b
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34494707037 |
perf-changelog.yaml is append-only, and the rebase onto main dropped PR #2969's dsv4-fp4-gb300-dynamo-sglang-agentic-agg entry, which failed check-changelog. Rebuild the file as main's content plus this branch's own appended entries. perf-changelog.yaml 为只可追加文件，而向 main 的 rebase 丢掉了 PR #2969 的 dsv4-fp4-gb300-dynamo-sglang-agentic-agg 条目，导致 check-changelog 失败。 将该文件重建为 main 的内容加上本分支自身追加的条目。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
同步最新 main，保留性能变更日志历史字节，并在末尾保留本 PR 的变更记录。
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34504981128 |
Resolve conflicts with the merged H200, GB300, B200, B300, and GB200 arms: keep every master-config entry, keep main's shared and H200 sections ahead of this branch's H100 section, and rewrite the H200 section's "no H100 arm" paragraph to point at the H100 arm this branch adds. Changelog entries append at the tail. 将 main 合并入 H100 DSv4.1 Flash AgentX 分支，解决与已合入的 H200、GB300、B200、 B300 和 GB200 分支的冲突：保留全部 master 配置条目，main 的共享小节与 H200 小节置于 本分支 H100 小节之前，并将 H200 小节中「没有 H100 分支」的段落改为指向本分支新增的 H100 分支；changelog 条目追加到文件末尾。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/reuse-sweep-run 34504981128 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8e1e980. Configure here.
| SERVER_LOG="$RESULT_DIR/server.log" | ||
| export VLLM_ENGINE_READY_TIMEOUT_S=3600 | ||
| export VLLM_USE_RUST_FRONTEND=1 | ||
| export VLLM_USE_V2_MODEL_RUNNER=1 |
There was a problem hiding this comment.
H100 enables non-default V2 runner
Medium Severity
The H100 script exports VLLM_USE_V2_MODEL_RUNNER=1, but the shared Flash recipe it copies leaves the official default runner. Other Flash SKUs already dropped the V2 runner, and the H100 notes never list this as an 80 GB workaround, so H100 serving and throughput would diverge from the rest of the family.
Reviewed by Cursor Bugbot for commit 8e1e980. Configure here.
3 participants
Add H100 TP8 DSv4.1 Flash AgentX at concurrency 1–4, using the full
semianalysis_cc_traces_weka_062126corpus. Five-token DSpark throughput uses golden AL 3.51 (thinking on), with adaptive verification disabled; accuracy evals retain real block verification. Engram stays in host DRAM through UVA; KV stays on GPU. Retains the H100 memory limits and measured concurrency cap of 4. Validation: Bash, exact-key matrix, changelog, launcher/eval tests, and both serving-mode command checks pass. GPU validation pending.新增 H100 TP8 DSv4.1 Flash AgentX，并发 1–4，使用完整
semianalysis_cc_traces_weka_062126语料。五 token DSpark 吞吐测试采用 thinking 开启时的黄金 AL 3.51，并关闭自适应验证；准确率 eval 保留真实块验证。Engram 通过 UVA 访问主机 DRAM，KV 保留在 GPU。保留 H100 显存限制及基于实测的并发上限 4。验证：Bash、精确配置矩阵、changelog、launcher/eval 测试及两种服务模式的命令检查通过；GPU 验证待完成。Note
Medium Risk
Changes H100 Slurm launcher script resolution and mount paths for all recipes, and introduces memory-tuned serving flags that affect whether 1M-context agentic runs OOM on 80 GB GPUs.
Overview
Adds an H100 AgentX arm for DeepSeek-V4.1-Flash (
dsv41flash-fp4-h100-vllm-agentic-dspark): TP8, fp4 vLLM, five-token DSpark, and agentic-coding sweeps at concurrency 1–4 (capped from measured ~6.7× max at 1M tokens/request).Because 80 GB H100 cannot use the shared 1M-context flags (16 GiB sparse-attention indexer buffer at 8192 batched tokens), the PR introduces a dedicated
dsv41flash_fp4_h100_vllm_mtp.shwith--max-num-batched-tokens 4096, explicit--max-num-seqs,gpu-memory-utilization 0.92, andexpandable_segments, while keeping full 1M context and the uncapped trace corpus. Throughput uses golden synthetic AL 3.51; evals keep real block rejection.launch_h100-dgxc-slurm.shnow resolves_h100_<framework>[_mtp].sh(with untagged fallback) and mounts the repo at/ixfordsv41flashso AgentX does not write under/workspace. Docs (EN/ZH), perf-changelog, andtest_dsv41flash_h100.pycover the recipe, launcher mounts, and indexer VRAM budget.Reviewed by Cursor Bugbot for commit 8e1e980. Bugbot is set up for automated code reviews on this repo. Configure here.