Skip to content

[Klaud Cold] Update kimik3-fp4-h200-vllm-agentic vLLM image to v0.29.0 (digest-pinned) / 将 kimik3-fp4-h200-vllm-agentic 的 vLLM 镜像更新为 v0.29.0（按摘要固定） - #2919

Closed
Klaud-Cold wants to merge 1 commit into
mainfrom
klaud/auto-236d5cc046c58966-0a6aa48a38ae2b4c
Closed

[Klaud Cold] Update kimik3-fp4-h200-vllm-agentic vLLM image to v0.29.0 (digest-pinned) / 将 kimik3-fp4-h200-vllm-agentic 的 vLLM 镜像更新为 v0.29.0（按摘要固定）#2919
Klaud-Cold wants to merge 1 commit into
mainfrom
klaud/auto-236d5cc046c58966-0a6aa48a38ae2b4c

Conversation

@Klaud-Cold

@Klaud-Cold Klaud-Cold commented Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Current status: Stopped. Confirmed infrastructure blocker: the new image's container squash is not staged on cluster:h200-dgxc and the shared H200 launcher does not import vLLM-lane images, so every job of the initial targeted run failed inside srtctl seconds after its Slurm job started, before any container or model was loaded. The run was cancelled and all 80 jobs are terminal. Repairs used: 0/5 (no in-scope repair exists: the fix is a cluster staging step or a change to the shared launcher runners/launch_h200-dgxc-slurm.sh, both outside this PR's edit scope). PR closed as draft with no sweep label; the remote branch is deleted so the candidate can be retried once the squash exists.
Next step (manual, outside Klaud Cold scope): stage /data/gharunners/containers/vllm_vllm-openai_v0.29.0_sha256_c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1.sqsh on the H200 DGXC cluster (enroot import -o <path> docker://vllm/vllm-openai@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1), or extend the launcher's enroot import block (currently gated on MODEL_PREFIX == "glm5.2" at runners/launch_h200-dgxc-slurm.sh:192) to the vllm/kimik3 lane; then let the auto-sweep reselect this candidate.

Candidate 236d5cc046c58966-0a6aa48a38ae2b4c, family configs/nvidia-master.yaml:kimik3-fp4-h200-vllm-agentic (Kimi-K3 FP4, H200, vLLM, DSpark speculative decoding reported as mtp, aggregated, AgentX agentic-coding traces). Runner cluster:h200-dgxc resolves in configs/runners.yaml to h200-dgxc-slurm_0..13 only, so the single telemetry target is h200; check-capacity --cluster h200 exited 0 before edits (12:27Z), before branch creation (12:34Z) and is re-run before every dispatch. Review reasons: release-string-mismatch, agentx-age. Green benchmarks do not prove that the repository's global checks pass.

Change

  • configs/nvidia-master.yaml kimik3-fp4-h200-vllm-agentic.image: vllm/vllm-openai:kimi-k3 -> vllm/vllm-openai:v0.29.0@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1.
  • The three referenced, unshared recipes benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-h200-{tp16dp2ep32-latency,tp8dp4ep32-balanced,tp8dp4ep32-vllm-simple}-agentic.yaml: model.container and identity.container.image set to the same string; identity.frameworks.vllm 0.1.dev19262+gb6bbf29dd.d20260727 -> 0.29.0. Nothing else changed (model, precision, TP/DP/EP topology, DSpark config with golden AL 2.51, marlin MoE backend, FLASHMLA, KV offload, commands, resources, eval selection).
  • Old image: a locally built fork image (OCI labels build.commit=unknown, build.pipeline=local, local/vllm-openai:dev), vLLM 0.1.dev19262+gb6bbf29dd built 2026-07-27, CUDA 13.0.1; commit b6bbf29dd does not exist in vllm-project/vllm. The mutable tag was last pushed 2026-07-27T15:17Z (manifest list sha256:e90e2603…).
  • New image: vLLM release v0.29.0 (GitHub release published 2026-09-09T08:54Z, tag commit 98dff2a81d747d1dba01a47f939f48c3526d4206), Docker Hub tag pushed 2026-09-09T06:06Z, manifest-list digest sha256:c2914767… (linux/amd64 image sha256:082ca6f0…), CUDA_VERSION=13.0.2, TORCH_CUDA_ARCH_LIST includes 9.0 (H200). Same CUDA 13.0 line as the old image, so the driver pairing on cluster:h200-dgxc is unchanged. Digest-pinned in the same tag@sha256 style as the existing lmsysorg/sglang:v0.5.14-cu130@sha256:… entries.
  • Compatibility review against the v0.29.0 source tree (not runtime proof): KimiK3ForConditionalGeneration and K3DSparkModel are registered (vllm/models/kimi_k3/nvidia/{model,dspark_mla}.py); kimi_k3 tool-call and reasoning parsers exist (vllm/tool_parsers/kimi_k3_tool_parser.py, vllm/reasoning/kimi_k3_reasoning_parser.py); speculative-config fields method=dspark, draft_sample_method, rejection_sample_method=synthetic, synthetic_acceptance_length are accepted; moe-backend marlin is a valid MoEBackend and the checkpoint's compressed-tensors mxfp4-pack-quantized MoE method falls back to MarlinExperts off SM100; FLASHMLA supports compute capability 9; load-format fastsafetensors, --no-enable-flashinfer-autotune, --enable-prompt-tokens-details, --language-model-only, fuse_allreduce_rms and SimpleCPUOffloadConnector are present; VLLM_USE_V2_MODEL_RUNNER is honoured. VLLM_ROUTED_DOWN_PROJ_STREAM_TOKEN_THRESHOLD is not read anywhere in v0.29.0 (fork-only env var; left in place, harmless).
  • Local checks: generate_sweep_configs.py test-config --config-files configs/nvidia-master.yaml --config-keys kimik3-fp4-h200-vllm-agentic yields the same 35 rows as the base SHA 9847f875 with only image differing (node-count 4 everywhere, run-eval on all 35 rows with kimi-vendor / kimi_tool_call_schema); utils/matrix_logic pytest 297 passed; the three recipes parse and model.container == identity.container.image == master image.
  • Known risk (confirmed by the targeted run, see Initial attempt): runners/launch_h200-dgxc-slurm.sh maps the master image to /data/gharunners/containers/<image with [/:@#] -> _>.sqsh and only runs enroot import for the GLM-5.2 lane, so the vLLM lane depends on that squash already existing on the cluster. That launcher is shared code and outside this PR's scope.

Baseline (published 2026-08-07)

  • Queries: GET /api/v1/workflow-info?date=2026-08-07&benchmarkType=agentic_traces and GET /api/v1/benchmarks?model=Kimi-K3&date=2026-08-07&exact=true (53 rows; 35 match hardware h200, framework vllm, model kimik3, precision fp4, spec_method=mtp, disagg=false, benchmark_type=agentic_traces, image vllm/vllm-openai:kimi-k3, is_multinode=true, 32 GPUs). GET /api/v1/evaluations has no row for this identity (the family's eval is the kimi-vendor tool-call schema smoke, which is not published as a task score): published evals N/A.
  • Producer for all 35 points: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30781313910/attempts/3 (PR sweep for [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353 on branch agent/kimik3-h200-agentx, head SHA 114c1bd140ba75e082100ad11f34e3cf0adf9e3d, started 2026-08-07T06:58:13Z; the image at that SHA was vllm/vllm-openai:kimi-k3). Changelog entry: config key kimik3-fp4-h200-vllm-agentic, PR [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353, base b5b459da6e06b8941f00b777e6090adf0ef515d7, head 56218493432eddad9505722bae739c33ac60f257. Curve snapshot id 2267 is a database id, not a producer id. Frozen for all attempts; the old image is never dispatched.
  • Recipe mapping per point: TP16xDP2 EP32 = agg-h200-tp16dp2ep32-latency-agentic (conc 1-12), TP8xDP4 EP32 = agg-h200-tp8dp4ep32-balanced-agentic (conc 1-16), TP8xDP4 EP32 + vllm-simple DRAM offload = agg-h200-tp8dp4ep32-vllm-simple-agentic (conc 8-32). Dataset: AgentX traces semianalysis_cc_traces_weka_062126, one-hour profile.
Topology Conc Total tput (tok/s) Total tput/GPU (tok/s) Output tput/GPU (tok/s) Mean TPOT (ms) Median TTFT (s) Mean TTFT (s) Requests Row id
TP16xDP2 EP32 1 2163.9 67.62 0.151 184.1 3.82 3.83 23 438866
TP16xDP2 EP32 2 1817.0 56.78 0.260 181.2 2.78 3.95 33 438864
TP16xDP2 EP32 3 2380.4 74.39 0.565 201.1 4.74 9.33 61 438869
TP16xDP2 EP32 4 2467.6 77.11 0.445 245.7 3.60 19.38 85 438873
TP16xDP2 EP32 5 2942.2 91.94 0.547 217.0 3.40 40.18 52 438895
TP16xDP2 EP32 6 3359.1 104.97 0.483 258.1 6.92 53.20 74 438871
TP16xDP2 EP32 7 3887.1 121.47 0.693 258.4 4.73 54.47 112 438890
TP16xDP2 EP32 8 4038.5 126.20 0.531 292.3 17.21 68.95 110 438872
TP16xDP2 EP32 10 4049.1 126.53 0.546 300.3 74.35 118.36 113 438877
TP16xDP2 EP32 12 2522.9 78.84 0.364 359.0 387.63 437.44 63 438874
TP8xDP4 EP32 1 2235.9 69.87 0.156 175.3 4.66 8.81 23 438893
TP8xDP4 EP32 2 1818.8 56.84 0.260 181.9 4.47 8.16 33 438896
TP8xDP4 EP32 3 2409.8 75.31 0.566 190.6 4.69 9.28 62 438888
TP8xDP4 EP32 4 2804.0 87.62 0.502 193.5 3.04 7.94 91 438867
TP8xDP4 EP32 5 2921.6 91.30 0.553 214.1 7.40 30.01 59 438887
TP8xDP4 EP32 6 3584.0 112.00 0.485 260.8 7.11 37.07 76 438879
TP8xDP4 EP32 7 4057.6 126.80 0.751 284.1 4.06 21.13 116 438894
TP8xDP4 EP32 8 4551.5 142.23 0.729 270.3 3.59 18.14 131 438875
TP8xDP4 EP32 10 4957.3 154.91 0.738 294.8 4.44 21.11 162 438880
TP8xDP4 EP32 12 7083.0 221.34 0.750 432.1 8.80 43.94 223 438891
TP8xDP4 EP32 14 6010.9 187.84 0.714 448.2 22.13 53.57 177 438863
TP8xDP4 EP32 16 5431.2 169.73 0.710 463.9 28.43 68.38 165 438870
TP8xDP4 EP32 + vllm-simple DRAM offload 8 4415.9 138.00 0.681 290.2 4.73 13.84 128 438868
TP8xDP4 EP32 + vllm-simple DRAM offload 10 5245.4 163.92 0.887 295.3 3.83 9.92 172 438885
TP8xDP4 EP32 + vllm-simple DRAM offload 12 6687.8 208.99 0.729 441.1 9.63 27.23 201 438878
TP8xDP4 EP32 + vllm-simple DRAM offload 14 6032.6 188.52 0.709 459.0 11.66 45.04 174 438882
TP8xDP4 EP32 + vllm-simple DRAM offload 16 5094.6 159.21 0.657 512.8 10.84 59.64 156 438865
TP8xDP4 EP32 + vllm-simple DRAM offload 18 5649.3 176.54 0.729 479.8 21.72 49.77 154 438862
TP8xDP4 EP32 + vllm-simple DRAM offload 20 6032.4 188.51 0.640 568.0 37.62 94.53 170 438892
TP8xDP4 EP32 + vllm-simple DRAM offload 22 5998.0 187.44 0.647 616.0 58.65 115.65 179 438881
TP8xDP4 EP32 + vllm-simple DRAM offload 24 6234.3 194.82 0.716 657.0 79.24 149.87 212 438884
TP8xDP4 EP32 + vllm-simple DRAM offload 26 8442.0 263.81 1.054 512.4 84.11 124.95 288 438883
TP8xDP4 EP32 + vllm-simple DRAM offload 28 7398.3 231.20 0.900 610.2 74.73 132.66 281 438876
TP8xDP4 EP32 + vllm-simple DRAM offload 30 7158.3 223.70 0.963 625.2 136.42 174.34 260 438886
TP8xDP4 EP32 + vllm-simple DRAM offload 32 5768.3 180.26 0.763 643.3 202.64 299.94 215 438889

Initial attempt

  • Image / commit: vllm/vllm-openai:v0.29.0@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1 @ 9cef3907040fc32e43c3c332adbb8121be651c2a.
  • Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34352381748 (e2e-tests.yml on main, inputs.ref=9cef3907040fc32e43c3c332adbb8121be651c2a, generate-cli-command="test-config --config-files configs/nvidia-master.yaml --config-keys kimik3-fp4-h200-vllm-agentic", fail-fast=true, klaud-run=true, test-name=klaud-34349769668-236d5cc046c58966-0a6aa48a38ae2b4c; capacity gate exit 0 at 12:40:04Z; dispatched 2026-09-09T12:40:21Z). No sweep label; PR stays draft.
  • Outcome: failed and cancelled (run status completed/cancelled at 12:53Z; jobs: 16 failure, 54 cancelled, 6 skipped, 4 success for the non-GPU setup/collector jobs). Each failed job's srtctl apply submitted a 4-node Slurm job (for example job 82455) whose orchestrator aborted about 15 s later; no container was started and no model weights were loaded.
  • Benchmark / eval results: N/A (every job failed before the server started; no agg_bmk.json, no eval artifacts; the uploaded multinode_server_logs_* artifacts are ~2 KB stubs).
  • Per-point deltas vs baseline: N/A (no measurement).
  • Diagnosis (confirmed from the job logs of 102468725474, 102468726738 and three further failed jobs, identical error): the launcher resolved the master image to /data/gharunners/containers/vllm_vllm-openai_v0.29.0_sha256_c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1.sqsh and wrote it into srtslurm.yaml containers:; the Slurm-side orchestrator then aborted in src/srtctl/core/runtime.py:275 from_config with FileNotFoundError: Container image path does not exist: …vllm_vllm-openai_v0.29.0_sha256_….sqsh (✗ Sweep failed (exit code: 1)). Cause: runners/launch_h200-dgxc-slurm.sh only runs enroot import when MODEL_PREFIX == "glm5.2" (line 192); the vllm/kimik3 lane (line 189) expects the squash to already exist. The current vllm/vllm-openai:kimi-k3 squash was imported by the generic import block that existed when [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353 merged and was later restricted to GLM-5.2 in Refresh GLM-5.2 FP8 H200 AgentX 2P2D with MTP #2529. This is a cluster-staging / shared-launcher gap, not an image incompatibility: the v0.29.0 image itself was never started, so its runtime compatibility with this recipe remains unverified (the static source review above is favourable).
  • Repair: none attempted. The only fixes are (a) staging the squash on the cluster or (b) changing the shared launcher, both outside the permitted edit scope (family image and its unshared recipe YAMLs only). Same deterministic failure on every job, so the run was cancelled rather than allowed to churn through the remaining 54 jobs.
  • Next step: stop and release the candidate (see status).

Final full sweep

  • N/A. Not reached: targeted validation did not pass, so no perf-changelog.yaml entry was appended, the PR was never marked ready and full-sweep-enabled was never applied. No run-sweep.yml run exists for head 9cef3907040fc32e43c3c332adbb8121be651c2a.

Disposition

  • Stop reason: confirmed infrastructure blocker (container squash for the new image not staged on cluster:h200-dgxc; shared launcher does not import it). Repairs used 0/5. Owned runs: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34352381748 cancelled and confirmed terminal (80/80 jobs completed). PR closed as draft with no sweep labels; remote branch klaud/auto-236d5cc046c58966-0a6aa48a38ae2b4c deleted so a later auto-sweep can retry this candidate once the squash is staged (or the launcher imports vLLM images). A retry before that will fail the same way within seconds of each Slurm allocation.

当前状态： 已停止。已确认基础设施阻塞：新镜像的容器 squash 文件未在 cluster:h200-dgxc 上预置，且共享的 H200 启动器不会为 vLLM 通道导入镜像，因此初次定向运行的每个作业都在其 Slurm 作业启动后数秒内于 srtctl 中失败，未启动任何容器、未加载任何模型。该运行已取消，全部 80 个作业均已终止。已用修复次数：0/5（范围内不存在可行修复：需要在集群上预置 squash，或修改共享启动器 runners/launch_h200-dgxc-slurm.sh，两者都超出本 PR 的编辑范围）。PR 以草稿状态关闭且无 sweep 标签；远程分支已删除，以便 squash 就位后候选可被重新选中。
下一步（人工，超出 Klaud Cold 范围）： 在 H200 DGXC 集群上预置 /data/gharunners/containers/vllm_vllm-openai_v0.29.0_sha256_c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1.sqsh（enroot import -o <路径> docker://vllm/vllm-openai@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1），或将启动器中目前仅对 MODEL_PREFIX == "glm5.2" 生效的 enroot import 块（runners/launch_h200-dgxc-slurm.sh:192）扩展到 vllm/kimik3 通道；随后让自动巡检重新选中该候选。

候选 236d5cc046c58966-0a6aa48a38ae2b4c，系列 configs/nvidia-master.yaml:kimik3-fp4-h200-vllm-agentic（Kimi-K3 FP4、H200、vLLM、以 mtp 标记的 DSpark 投机解码、聚合部署、AgentX agentic-coding 轨迹）。运行器 cluster:h200-dgxc 在 configs/runners.yaml 中仅解析到 h200-dgxc-slurm_0..13，因此唯一遥测目标为 h200；check-capacity --cluster h200 在编辑前（12:27Z）与建分支前（12:34Z）均返回 0，且每次派发前都会重新运行。评审原因：release-string-mismatch、agentx-age。基准通过不能证明仓库全局检查通过。

变更

  • configs/nvidia-master.yaml 中 kimik3-fp4-h200-vllm-agentic.image：vllm/vllm-openai:kimi-k3 -> vllm/vllm-openai:v0.29.0@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1。
  • 三个被引用且未共享的配方 benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-h200-{tp16dp2ep32-latency,tp8dp4ep32-balanced,tp8dp4ep32-vllm-simple}-agentic.yaml：model.container 与 identity.container.image 设为同一字符串；identity.frameworks.vllm 由 0.1.dev19262+gb6bbf29dd.d20260727 改为 0.29.0。其他均未改动（模型、精度、TP/DP/EP 拓扑、golden AL 2.51 的 DSpark 配置、marlin MoE 后端、FLASHMLA、KV 卸载、命令、资源、评测选择）。
  • 旧镜像：本地构建的分支镜像（OCI 标签 build.commit=unknown、build.pipeline=local、local/vllm-openai:dev），vLLM 0.1.dev19262+gb6bbf29dd，构建于 2026-07-27，CUDA 13.0.1；提交 b6bbf29dd 不存在于 vllm-project/vllm。该可变标签最后推送于 2026-07-27T15:17Z（manifest list sha256:e90e2603…）。
  • 新镜像：vLLM 发布版 v0.29.0（GitHub release 发布于 2026-09-09T08:54Z，标签提交 98dff2a81d747d1dba01a47f939f48c3526d4206），Docker Hub 标签推送于 2026-09-09T06:06Z，manifest-list 摘要 sha256:c2914767…（linux/amd64 镜像 sha256:082ca6f0…），CUDA_VERSION=13.0.2，TORCH_CUDA_ARCH_LIST 含 9.0（H200）。与旧镜像同为 CUDA 13.0 系列，cluster:h200-dgxc 的驱动配套不变。采用与现有 lmsysorg/sglang:v0.5.14-cu130@sha256:… 条目相同的 tag@sha256 摘要固定方式。
  • 基于 v0.29.0 源码树的兼容性审查（非运行时证明）：已注册 KimiK3ForConditionalGeneration 与 K3DSparkModel（vllm/models/kimi_k3/nvidia/{model,dspark_mla}.py）；存在 kimi_k3 工具调用与推理解析器（vllm/tool_parsers/kimi_k3_tool_parser.py、vllm/reasoning/kimi_k3_reasoning_parser.py）；speculative-config 的 method=dspark、draft_sample_method、rejection_sample_method=synthetic、synthetic_acceptance_length 均被接受；moe-backend marlin 是合法的 MoEBackend，且该 checkpoint 的 compressed-tensors mxfp4-pack-quantized MoE 方法在非 SM100 设备上回退到 MarlinExperts；FLASHMLA 支持计算能力 9；load-format fastsafetensors、--no-enable-flashinfer-autotune、--enable-prompt-tokens-details、--language-model-only、fuse_allreduce_rms 与 SimpleCPUOffloadConnector 均存在；VLLM_USE_V2_MODEL_RUNNER 被识别。VLLM_ROUTED_DOWN_PROJ_STREAM_TOKEN_THRESHOLD 在 v0.29.0 中无任何读取（仅分支使用的环境变量；保留原样，无害）。
  • 本地检查：generate_sweep_configs.py test-config --config-files configs/nvidia-master.yaml --config-keys kimik3-fp4-h200-vllm-agentic 与基线 SHA 9847f875 生成相同的 35 行，仅 image 不同（节点数均为 4，35 行均 run-eval，评测为 kimi-vendor / kimi_tool_call_schema）；utils/matrix_logic pytest 297 项通过；三个配方可解析且 model.container == identity.container.image == 主配置镜像。
  • 已知风险（已由定向运行确认，见「初次尝试」）：runners/launch_h200-dgxc-slurm.sh 将主镜像映射到 /data/gharunners/containers/<镜像名中 [/:@#] 替换为 _>.sqsh，且仅对 GLM-5.2 通道执行 enroot import，因此 vLLM 通道依赖该 squash 文件已存在于集群。该启动器为共享代码，不在本 PR 范围内。

基线（发布日期 2026-08-07）

  • 查询：GET /api/v1/workflow-info?date=2026-08-07&benchmarkType=agentic_traces 与 GET /api/v1/benchmarks?model=Kimi-K3&date=2026-08-07&exact=true（53 行；35 行匹配硬件 h200、框架 vllm、模型 kimik3、精度 fp4、spec_method=mtp、disagg=false、benchmark_type=agentic_traces、镜像 vllm/vllm-openai:kimi-k3、is_multinode=true、32 GPU）。GET /api/v1/evaluations 中没有该身份的记录（本系列评测为 kimi-vendor 工具调用 schema 冒烟测试，不作为任务分数发布）：已发布评测 N/A。
  • 全部 35 个点的生产运行：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30781313910/attempts/3（PR [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353 在分支 agent/kimik3-h200-agentx 上的 PR sweep，head SHA 114c1bd140ba75e082100ad11f34e3cf0adf9e3d，开始于 2026-08-07T06:58:13Z；该 SHA 处的镜像为 vllm/vllm-openai:kimi-k3）。changelog 条目：配置键 kimik3-fp4-h200-vllm-agentic，PR [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353，base b5b459da6e06b8941f00b777e6090adf0ef515d7，head 56218493432eddad9505722bae739c33ac60f257。曲线快照 id 2267 是数据库 id，不是生产运行 id。所有尝试均冻结此基线；旧镜像绝不派发。
  • 逐点配方映射：TP16xDP2 EP32 = agg-h200-tp16dp2ep32-latency-agentic（并发 1-12），TP8xDP4 EP32 = agg-h200-tp8dp4ep32-balanced-agentic（并发 1-16），TP8xDP4 EP32（vllm-simple DRAM 卸载） = agg-h200-tp8dp4ep32-vllm-simple-agentic（并发 8-32）。数据集：AgentX 轨迹 semianalysis_cc_traces_weka_062126，一小时 profile。
拓扑 并发 总吞吐 (tok/s) 总吞吐/GPU (tok/s) 输出吞吐/GPU (tok/s) 平均 TPOT (ms) 中位 TTFT (s) 平均 TTFT (s) 请求数 行 id
TP16xDP2 EP32 1 2163.9 67.62 0.151 184.1 3.82 3.83 23 438866
TP16xDP2 EP32 2 1817.0 56.78 0.260 181.2 2.78 3.95 33 438864
TP16xDP2 EP32 3 2380.4 74.39 0.565 201.1 4.74 9.33 61 438869
TP16xDP2 EP32 4 2467.6 77.11 0.445 245.7 3.60 19.38 85 438873
TP16xDP2 EP32 5 2942.2 91.94 0.547 217.0 3.40 40.18 52 438895
TP16xDP2 EP32 6 3359.1 104.97 0.483 258.1 6.92 53.20 74 438871
TP16xDP2 EP32 7 3887.1 121.47 0.693 258.4 4.73 54.47 112 438890
TP16xDP2 EP32 8 4038.5 126.20 0.531 292.3 17.21 68.95 110 438872
TP16xDP2 EP32 10 4049.1 126.53 0.546 300.3 74.35 118.36 113 438877
TP16xDP2 EP32 12 2522.9 78.84 0.364 359.0 387.63 437.44 63 438874
TP8xDP4 EP32 1 2235.9 69.87 0.156 175.3 4.66 8.81 23 438893
TP8xDP4 EP32 2 1818.8 56.84 0.260 181.9 4.47 8.16 33 438896
TP8xDP4 EP32 3 2409.8 75.31 0.566 190.6 4.69 9.28 62 438888
TP8xDP4 EP32 4 2804.0 87.62 0.502 193.5 3.04 7.94 91 438867
TP8xDP4 EP32 5 2921.6 91.30 0.553 214.1 7.40 30.01 59 438887
TP8xDP4 EP32 6 3584.0 112.00 0.485 260.8 7.11 37.07 76 438879
TP8xDP4 EP32 7 4057.6 126.80 0.751 284.1 4.06 21.13 116 438894
TP8xDP4 EP32 8 4551.5 142.23 0.729 270.3 3.59 18.14 131 438875
TP8xDP4 EP32 10 4957.3 154.91 0.738 294.8 4.44 21.11 162 438880
TP8xDP4 EP32 12 7083.0 221.34 0.750 432.1 8.80 43.94 223 438891
TP8xDP4 EP32 14 6010.9 187.84 0.714 448.2 22.13 53.57 177 438863
TP8xDP4 EP32 16 5431.2 169.73 0.710 463.9 28.43 68.38 165 438870
TP8xDP4 EP32（vllm-simple DRAM 卸载） 8 4415.9 138.00 0.681 290.2 4.73 13.84 128 438868
TP8xDP4 EP32（vllm-simple DRAM 卸载） 10 5245.4 163.92 0.887 295.3 3.83 9.92 172 438885
TP8xDP4 EP32（vllm-simple DRAM 卸载） 12 6687.8 208.99 0.729 441.1 9.63 27.23 201 438878
TP8xDP4 EP32（vllm-simple DRAM 卸载） 14 6032.6 188.52 0.709 459.0 11.66 45.04 174 438882
TP8xDP4 EP32（vllm-simple DRAM 卸载） 16 5094.6 159.21 0.657 512.8 10.84 59.64 156 438865
TP8xDP4 EP32（vllm-simple DRAM 卸载） 18 5649.3 176.54 0.729 479.8 21.72 49.77 154 438862
TP8xDP4 EP32（vllm-simple DRAM 卸载） 20 6032.4 188.51 0.640 568.0 37.62 94.53 170 438892
TP8xDP4 EP32（vllm-simple DRAM 卸载） 22 5998.0 187.44 0.647 616.0 58.65 115.65 179 438881
TP8xDP4 EP32（vllm-simple DRAM 卸载） 24 6234.3 194.82 0.716 657.0 79.24 149.87 212 438884
TP8xDP4 EP32（vllm-simple DRAM 卸载） 26 8442.0 263.81 1.054 512.4 84.11 124.95 288 438883
TP8xDP4 EP32（vllm-simple DRAM 卸载） 28 7398.3 231.20 0.900 610.2 74.73 132.66 281 438876
TP8xDP4 EP32（vllm-simple DRAM 卸载） 30 7158.3 223.70 0.963 625.2 136.42 174.34 260 438886
TP8xDP4 EP32（vllm-simple DRAM 卸载） 32 5768.3 180.26 0.763 643.3 202.64 299.94 215 438889

初次尝试

  • 镜像 / 提交：vllm/vllm-openai:v0.29.0@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1 @ 9cef3907040fc32e43c3c332adbb8121be651c2a。
  • 运行：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34352381748（`main` 上的 e2e-tests.yml，inputs.ref=9cef3907040fc32e43c3c332adbb8121be651c2a，generate-cli-command="test-config --config-files configs/nvidia-master.yaml --config-keys kimik3-fp4-h200-vllm-agentic"，fail-fast=true，klaud-run=true，test-name=klaud-34349769668-236d5cc046c58966-0a6aa48a38ae2b4c；容量门禁 12:40:04Z 返回 0；派发于 2026-09-09T12:40:21Z）。未加 sweep 标签；PR 保持草稿。
  • 结果：失败并已取消（运行状态于 12:53Z 变为 completed/cancelled；作业：16 个失败、54 个取消、6 个跳过、4 个成功——后者为不占用 GPU 的准备/收集作业）。每个失败作业的 srtctl apply 都提交了一个 4 节点 Slurm 作业（例如作业 82455），其编排器约 15 秒后中止；未启动容器，未加载模型权重。
  • 基准 / 评测结果：N/A（所有作业均在服务启动前失败；无 agg_bmk.json、无评测产物；已上传的 multinode_server_logs_* 产物仅为约 2 KB 的空壳）。
  • 逐点相对基线差异：N/A（无测量）。
  • 诊断（由作业 102468725474、102468726738 及另外三个失败作业的日志确认，错误完全一致）：启动器将主镜像解析为 /data/gharunners/containers/vllm_vllm-openai_v0.29.0_sha256_c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1.sqsh 并写入 srtslurm.yaml 的 containers:；随后 Slurm 侧编排器在 src/srtctl/core/runtime.py:275 from_config 抛出 FileNotFoundError: Container image path does not exist: …vllm_vllm-openai_v0.29.0_sha256_….sqsh（✗ Sweep failed (exit code: 1)）。原因：runners/launch_h200-dgxc-slurm.sh 仅在 MODEL_PREFIX == "glm5.2" 时执行 enroot import（第 192 行）；vllm/kimik3 通道（第 189 行）假定 squash 已存在。现有的 vllm/vllm-openai:kimi-k3 squash 是 [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353 合并时通用导入块导入的，该导入块随后在 Refresh GLM-5.2 FP8 H200 AgentX 2P2D with MTP #2529 中被限制为仅 GLM-5.2。这是集群预置 / 共享启动器的缺口，而非镜像不兼容：v0.29.0 镜像从未被启动，其与本配方的运行时兼容性仍未验证（上文的静态源码审查结果是正面的）。
  • 修复：未尝试。唯一的修复方式是 (a) 在集群上预置 squash 或 (b) 修改共享启动器，二者均超出允许的编辑范围（仅限系列镜像及其未共享的配方 YAML）。每个作业都是同样的确定性失败，因此取消运行而不是让剩余 54 个作业继续空转。
  • 下一步：停止并释放候选（见状态）。

最终全量扫描

  • N/A。未到达：定向验证未通过，因此未追加 perf-changelog.yaml 条目，PR 未转为 ready，也未添加 full-sweep-enabled 标签。head 9cef3907040fc32e43c3c332adbb8121be651c2a 不存在任何 run-sweep.yml 运行。

处置

  • 停止原因：已确认基础设施阻塞（新镜像的容器 squash 未在 cluster:h200-dgxc 上预置；共享启动器不会导入它）。已用修复次数 0/5。自有运行：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34352381748 已取消并确认终止（80/80 个作业已完成）。PR 以草稿状态关闭且无 sweep 标签；远程分支 klaud/auto-236d5cc046c58966-0a6aa48a38ae2b4c 已删除，以便 squash 就位（或启动器支持导入 vLLM 镜像）后自动巡检可重试该候选。在此之前的重试会在每次 Slurm 分配后数秒内以同样方式失败。

🤖 Generated with Claude Code

@Klaud-Cold @claude
feat(kimik3-h200): refresh vLLM image to v0.29.0 (digest-pinned)
9cef390 
Move kimik3-fp4-h200-vllm-agentic from the mutable fork build
vllm/vllm-openai:kimi-k3 (vLLM 0.1.dev19262+gb6bbf29dd, 2026-07-27) to the
upstream release vllm/vllm-openai:v0.29.0 pinned by manifest-list digest.
model.container, identity.container.image and identity.frameworks.vllm in the
three referenced H200 srt-slurm recipes are updated to match the master image.
Model, precision, topology, DSpark settings, workloads, commands and resources
are unchanged.

将 kimik3-fp4-h200-vllm-agentic 的镜像从可变的分支构建 vllm/vllm-openai:kimi-k3
（vLLM 0.1.dev19262+gb6bbf29dd，2026-07-27）更新为按 manifest-list 摘要固定的上游
发布版本 vllm/vllm-openai:v0.29.0。三个被引用的 H200 srt-slurm 配方中的
model.container、identity.container.image 与 identity.frameworks.vllm 已同步更新为
主配置镜像。模型、精度、拓扑、DSpark 设置、负载、命令与资源均保持不变。

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@Klaud-Cold

Klaud-Cold commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Dispatch — initial targeted run started. Status: draft PR with the image refresh (vllm/vllm-openai:kimi-k3 -> vllm/vllm-openai:v0.29.0@sha256:c2914767…, commit 9cef390) is open with no sweep label. Capacity gate for telemetry cluster h200 exited 0 at 12:40:04Z. Action: dispatched e2e-tests.yml from main for the full family (35 AgentX points, 35 kimi-vendor evals, 4 H200 nodes each) with fail-fast=true and klaud-run=true: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34352381748. Next: monitor to completion; on any failure, inspect the first server-side error. Repairs used: 0/5.

派发 — 初次定向运行已启动。 状态：包含镜像刷新（vllm/vllm-openai:kimi-k3 -> vllm/vllm-openai:v0.29.0@sha256:c2914767…，提交 9cef390）的草稿 PR 已打开，未加 sweep 标签。遥测集群 h200 的容量门禁于 12:40:04Z 返回 0。操作：从 main 派发覆盖整个系列的 e2e-tests.yml（35 个 AgentX 点、35 个 kimi-vendor 评测，每点 4 个 H200 节点），fail-fast=true、klaud-run=true：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34352381748。下一步：监控至结束；若失败则检查首个服务端错误。已用修复次数：0/5。

@Klaud-Cold

Klaud-Cold commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Termination — confirmed infrastructure blocker, run cancelled, candidate released. Status: the initial targeted run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34352381748 failed deterministically and was cancelled at 12:53Z; all 80 jobs are terminal (16 failed, 54 cancelled, 6 skipped, 4 non-GPU setup jobs succeeded). Confirmed finding (identical in every inspected failed job, e.g. 102468725474 and 102468726738): each srtctl apply submitted a 4-node Slurm job that aborted about 10 s later in srtctl/core/runtime.py from_config with FileNotFoundError: Container image path does not exist: /data/gharunners/containers/vllm_vllm-openai_v0.29.0_sha256_c2914767….sqsh; no container started and no weights were loaded. Cause: runners/launch_h200-dgxc-slurm.sh only runs enroot import for MODEL_PREFIX == "glm5.2" (line 192), so the vllm/kimik3 lane needs its squash pre-staged; the new image's squash does not exist on the cluster. This is a staging/shared-launcher gap, not evidence of image incompatibility (the v0.29.0 image never started). No in-scope repair exists (the launcher is shared code); repairs used 0/5. Action: PR body finalized, no sweep label, PR closed as draft, remote branch klaud/auto-236d5cc046c58966-0a6aa48a38ae2b4c deleted so the candidate can be retried once /data/gharunners/containers/vllm_vllm-openai_v0.29.0_sha256_c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1.sqsh is staged or the launcher imports vLLM images. Commit: 9cef390.

终止 — 已确认基础设施阻塞，运行已取消，候选已释放。 状态：初次定向运行 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34352381748 以确定性方式失败，并于 12:53Z 取消；全部 80 个作业均已终止（16 个失败、54 个取消、6 个跳过、4 个不占用 GPU 的准备作业成功）。已确认的发现（在每个检查过的失败作业中完全一致，例如 102468725474 与 102468726738）：每次 srtctl apply 提交的 4 节点 Slurm 作业约 10 秒后在 srtctl/core/runtime.py from_config 中止，报 FileNotFoundError: Container image path does not exist: /data/gharunners/containers/vllm_vllm-openai_v0.29.0_sha256_c2914767….sqsh；未启动容器，未加载权重。原因：runners/launch_h200-dgxc-slurm.sh 仅对 MODEL_PREFIX == "glm5.2" 执行 enroot import（第 192 行），因此 vllm/kimik3 通道需要预置 squash；新镜像的 squash 在集群上不存在。这是预置 / 共享启动器的缺口，而非镜像不兼容的证据（v0.29.0 镜像从未启动）。范围内不存在可行修复（启动器为共享代码）；已用修复次数 0/5。操作：PR 正文已定稿，无 sweep 标签，PR 以草稿状态关闭，远程分支 klaud/auto-236d5cc046c58966-0a6aa48a38ae2b4c 已删除，以便在 /data/gharunners/containers/vllm_vllm-openai_v0.29.0_sha256_c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1.sqsh 预置完成或启动器支持导入 vLLM 镜像后重试该候选。提交：9cef3907040fc32e43c3c332adbb8121be651c2a。

@Klaud-Cold

Klaud-Cold commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: confirmed infrastructure blocker (new image squash not staged on cluster:h200-dgxc; shared launcher imports only GLM-5.2 images). Branch deleted so the candidate can be retried once staged. / 关闭：已确认基础设施阻塞（新镜像 squash 未在 cluster:h200-dgxc 预置；共享启动器仅导入 GLM-5.2 镜像）。已删除分支，以便预置完成后重试该候选。

@Klaud-Cold Klaud-Cold closed this Sep 9, 2026
@Klaud-Cold
Klaud-Cold deleted the klaud/auto-236d5cc046c58966-0a6aa48a38ae2b4c branch September 9, 2026 12:57
@Klaud-Cold Klaud-Cold mentioned this pull request Sep 10, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Ankur-singh Ankur-singh Awaiting requested review from Ankur-singh Ankur-singh will be requested when the pull request is marked ready for review Ankur-singh is a code owner
@kedarpotdar-nv kedarpotdar-nv Awaiting requested review from kedarpotdar-nv kedarpotdar-nv will be requested when the pull request is marked ready for review kedarpotdar-nv is a code owner

Assignees

No one assigned

Labels

None yet

Projects

InferenceMAX Board
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Klaud-Cold