Skip to content

[Klaud Cold] Update qwen3.5-fp4-gb300-dynamo-sglang SGLang image to v0.5.19-cu130 / 将 qwen3.5-fp4-gb300-dynamo-sglang 的 SGLang 镜像更新至 v0.5.19-cu130 - #2909

Closed
Klaud-Cold wants to merge 3 commits into
mainfrom
klaud/auto-233c0f70c5dbad2a-3d9c146c5e9ad2d6
Closed

[Klaud Cold] Update qwen3.5-fp4-gb300-dynamo-sglang SGLang image to v0.5.19-cu130 / 将 qwen3.5-fp4-gb300-dynamo-sglang 的 SGLang 镜像更新至 v0.5.19-cu130#2909
Klaud-Cold wants to merge 3 commits into
mainfrom
klaud/auto-233c0f70c5dbad2a-3d9c146c5e9ad2d6

Conversation

@Klaud-Cold

@Klaud-Cold Klaud-Cold commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Status / 状态

Current status: Stopped — final full sweep failed on a confirmed CI-infrastructure blocker that is outside this PR's edit scope. The image refresh itself works: Repair 1/5's targeted run passed 4/4 benchmark points and 4/4 gsm8k evals with +8.3% to +15.6% throughput per GPU versus the 2026-08-10 baseline. In the labeled sweep (https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946) the 1P1D benchmark completed but result processing crashed with OSError: [Errno 36] File name too long because the shared multinode template appends _recipe-<fingerprint> to the result filename, pushing this family's 1P1D (conc1x4x8x16x32x64x256) aggregate name to 256 bytes (NAME_MAX is 255); the 5P1D and 6P1D benchmark jobs failed when pyxis could not pull the nginx frontend image from Docker Hub (anonymous pull, error 23). No in-scope change can shorten the generated filename, so the sweep cannot pass for this family until the shared template/result-processing code is fixed. Disposition: sweep run cancelled, label removed, PR returned to draft and closed, remote branch deleted so the candidate can be retried after the shared fix lands. Repairs used: 1 of 5.

Family: configs/nvidia-master.yaml:qwen3.5-fp4-gb300-dynamo-sglang (Qwen3.5-397B-A17B NVFP4-V2, GB300, dynamo-sglang, disaggregated, 8k1k, no speculative decoding). Target cluster: gb300-nv (runner gb300gb300-nv_0/1/2 in configs/runners.yaml).

Change: image: lmsysorg/sglang:v0.5.14-cu130lmsysorg/sglang:v0.5.19-cu130; recipes' dynamo.version and the family's router.version 1.1.01.5.0.dev20260908 (Repair 1, see below) (Docker Hub manifest digest sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9, pushed 2026-09-04, linux/arm64 present; release v0.5.19 published 2026-09-05). The four srt-slurm recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/8k1k/disagg/stp/ keep container: "dynamo-sglang", which runners/launch_gb300-nv.sh maps to the master image's squash file, so no recipe edit is needed. Matrix generation at base and head differs only in image (4 points, node counts 2/9/10/11, evals enabled). Every recipe server flag exists in v0.5.19 (--mamba-scheduler-strategy remains a deprecated alias of --mamba-radix-cache-strategy, as in v0.5.14).

Green targeted benchmarks do not prove that global PR checks pass.

Baseline (published 2026-08-10)

Point (P/D topology, kv) conc tput/GPU (tok/s) median TTFT (s) median TPOT (s) median E2E (s)
1P TP4 / 1D TP4, mooncake 1 242.87 0.381 0.00417 4.27
1P TP4 / 1D TP4, mooncake 4 806.75 0.418 0.00486 5.00
1P TP4 / 1D TP4, mooncake 8 1344.60 0.724 0.00567 5.98
1P TP4 / 1D TP4, mooncake 16 2292.35 0.745 0.00672 6.93
1P TP4 / 1D TP4, mooncake 32 3629.11 0.889 0.00849 8.72
1P TP4 / 1D TP4, mooncake 64 5745.02 1.034 0.01077 10.99
1P TP4 / 1D TP4, mooncake 256 9091.45 15.103 0.01383 28.01
5P DEP4 / 1D DEP16, nixl 2048 15626.02 12.200 0.01688 27.87
6P DEP4 / 1D DEP16, mooncake 5120 17301.32 40.608 0.01900 57.85
7P DEP4 / 1D DEP16, mooncake 5120 18370.36 27.722 0.02154 47.60
  • Published evals (gsm8k, n=1319, same producer run): 1P/1D conc 256 em_strict 0.9659 / em_flexible 0.9606; 5P1D conc 2048 0.9674 / 0.9560; 6P1D conc 5120 0.9666 / 0.9530; 7P1D conc 5120 0.9682 / 0.9530.

Initial attempt

  • Image/commit: lmsysorg/sglang:v0.5.19-cu130, commit ebde675f2fa4519650be9c0349d12c9d75694bfa.
  • Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34303559979 (dispatched 2026-09-09T02:31:33Z from main, inputs.ref=ebde675f2fa4519650be9c0349d12c9d75694bfa, fail-fast=true, klaud-run=true). Outcome: failure at 02:43 UTC.
  • Changes: master image bump only.
  • Benchmark/eval results: N/A (1P1D benchmark and 1P1D eval jobs failed in "Launch multi-node job script" before serving; fail-fast cancelled the 5P1D/6P1D/7P1D benchmark and eval jobs). Empty results_bmk/eval_results_all artifacts.
  • Deltas vs baseline: N/A (no measurements).
  • Diagnosis (first server error, prefill worker on srtctl job 29041): ModuleNotFoundError: No module named 'sglang.srt.server_args_config_parser' raised from dynamo/sglang/args.py after "Dynamo 1.1.0 installed". sglang v0.5.19 moved that module to sglang.srt.utils.server_args_config_parser (Move server args config parser under utils sgl-project/sglang#36681; the old path exists in v0.5.14–v0.5.18). Dynamo v1.1.0 through v1.4.2 (and release/1.4.2) still import the old path; the fix (fix(sglang): follow config parser module move ai-dynamo/dynamo#14054, #14234) is only on main and in the 1.5.0.dev* PyPI nightlies; backport PR #14064 was closed. The v0.5.19 squash image itself imported fine.
  • Next step: Repair 1 — pin a Dynamo build that carries the compat shim.

Repair 1/5

  • Image/commit: lmsysorg/sglang:v0.5.19-cu130, commit 30fbdb6a38ad300d4618dcdfefa467ef0fd69a86.
  • Run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34304865256 (dispatched 2026-09-09T02:51:26Z from main, inputs.ref=30fbdb6a38ad300d4618dcdfefa467ef0fd69a86, fail-fast=true, klaud-run=true).
  • Changes: dynamo.version "1.1.0""1.5.0.dev20260908" in the four referenced recipes (8k1k_stp_lowlat_0/maxtpt_0/maxtpt_1/maxtpt_2.yaml); router.version "1.1.0""1.5.0.dev20260908" in the family's master entry so published router metadata matches the installed Dynamo. The srt-slurm producer installs ai-dynamo-runtime==V ai-dynamo==V from PyPI without the sglang extra, so the container's sglang stays v0.5.19. The ai_dynamo-1.5.0.dev20260908 wheel contains dynamo/sglang/_compat.py importing sglang.srt.utils.server_args_config_parser; every other sglang.srt module it imports exists in the v0.5.19 tree (multimodal encode-server imports are try/except-guarded). ai-dynamo-runtime ships a cp310-abi3 manylinux_2_28_aarch64 wheel.
  • Outcome: success (run completed 2026-09-09 05:27 UTC). All 4 multi-node benchmark jobs and all 4 eval-only jobs succeeded; results_bmk/agg_bmk.json has 10 rows, every row image: lmsysorg/sglang:v0.5.19-cu130; eval_results_all/agg_eval_all.json has 4 entries.
  • Eval results (gsm8k, n=1319, strict / flexible; baseline in parentheses): 1P1D conc 256 0.9697 / 0.9621 (0.9659 / 0.9606); 5P1D conc 2048 0.9659 / 0.9598 (0.9674 / 0.9560); 6P1D conc 5120 0.9636 / 0.9530 (0.9666 / 0.9530); 7P1D conc 5120 0.9659 / 0.9575 (0.9682 / 0.9530). All within ±0.5 pt of baseline (SE ≈ 0.005).
  • Deltas vs baseline (per point, new vs published 2026-08-10; same topology, concurrency and 8k1k dataset; throughput = tput_per_gpu):
conc tput/GPU new vs base (Δ) median TTFT new vs base (Δ) median TPOT new vs base (Δ) median E2E new vs base (Δ)
1 279.61 vs 242.87 (+15.1%) 0.150 vs 0.381 s (−60.6%) 3.84 vs 4.17 ms (−8.1%) 3.63 vs 4.27 s (−15.1%)
4 916.30 vs 806.75 (+13.6%) 0.183 vs 0.418 s (−56.2%) 4.57 vs 4.86 ms (−6.0%) 4.40 vs 5.00 s (−12.0%)
8 1554.42 vs 1344.60 (+15.6%) 0.195 vs 0.724 s (−73.1%) 5.31 vs 5.67 ms (−6.3%) 5.19 vs 5.98 s (−13.2%)
16 2595.83 vs 2292.35 (+13.2%) 0.230 vs 0.745 s (−69.2%) 6.30 vs 6.72 ms (−6.3%) 6.15 vs 6.93 s (−11.2%)
32 4049.07 vs 3629.11 (+11.6%) 0.333 vs 0.889 s (−62.5%) 8.07 vs 8.49 ms (−5.0%) 7.84 vs 8.72 s (−10.1%)
64 6347.02 vs 5745.02 (+10.5%) 0.407 vs 1.034 s (−60.6%) 10.34 vs 10.77 ms (−4.1%) 9.96 vs 10.99 s (−9.4%)
256 9847.69 vs 9091.45 (+8.3%) 13.694 vs 15.103 s (−9.3%) 13.48 vs 13.83 ms (−2.5%) 26.09 vs 28.01 s (−6.8%)

Wide-EP points (prefill DEP4 workers / 1 decode DEP16 worker):

Point conc tput/GPU new vs base (Δ) median TTFT new vs base (Δ) median TPOT new vs base (Δ) median E2E new vs base (Δ)
5P1D, nixl 2048 17654.45 vs 15626.02 (+13.0%) 9.474 vs 12.200 s (−22.3%) 17.07 vs 16.88 ms (+1.1%, regression) 25.39 vs 27.87 s (−8.9%)
6P1D, mooncake 5120 19273.44 vs 17301.32 (+11.4%) 35.541 vs 40.608 s (−12.5%) 18.67 vs 19.00 ms (−1.7%) 52.98 vs 57.85 s (−8.4%)
7P1D, mooncake 5120 20282.71 vs 18370.36 (+10.4%) 24.590 vs 27.722 s (−11.3%) 22.07 vs 21.54 ms (+2.5%, regression) 45.09 vs 47.60 s (−5.3%)

All 10 points matched a baseline point; none excluded. Single-run measurements, no repeat; the two TPOT regressions are small and reported, not gated.

  • Next step: append the perf-changelog.yaml entry and start the final full sweep.

Final full sweep

  • Image/commit: lmsysorg/sglang:v0.5.19-cu130, head 278090e154fbbca8b653b284c51c56b40add9534 (adds the perf-changelog.yaml entry; validator utils/validate_perf_changelog.py --base-ref origin/main --head-ref HEAD passed locally, prior bytes preserved).
  • Transition: capacity check on gb300-nv passed at 05:28 UTC; PR marked ready; full-sweep-enabled applied as the sole sweep label.
  • Runs on the exact head: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946 (active; triggered by the full-sweep-enabled label event; check-changelog passed, setup running). https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315043758 and https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044003 (ready-for-review events) were cancelled by the sweep concurrency group before any benchmark job ran.
  • Result: failure (run cancelled by Klaud Cold after the failures below made success impossible; job states at cancellation are listed under "Terminal states").
  • Job outcomes: 7P1D c5120 benchmark success (05:54–06:35 UTC); 1P1D eval and 7P1D eval success; 1P1D c1–256 benchmark failure (06:04 UTC); 5P1D c2048 benchmark failure (06:44 UTC); 6P1D c5120 benchmark failure (06:54 UTC); 5P1D eval was in progress and 6P1D eval queued when the run was cancelled.
  • Diagnosis 1 (1P1D benchmark, deterministic, out of scope): the srtctl sweep itself finished ("Benchmark completed successfully", 7/7 concurrencies, all requests completed, power telemetry publication_valid: true). The Process result step then failed in utils/process_result.py with OSError: [Errno 36] File name too long: 'agg_qwen3.5_8k1k_fp4_dynamo-sglang_prefill-tp4-…-nw1_decode-tp4-…-nw1_disagg-true_spec-none_conc1x4x8x16x32x64x256_gb300-nv_00_recipe-0190dba4907b78b3_sa-bench_isl_8192_osl_1024_conc16_gpus_8_ctx_4_gen_4.json' (256 bytes; the matching power_validation_… name is 269 bytes; Linux NAME_MAX is 255). benchmark-multinode-tmpl.yml appends _recipe-${RECIPE_FINGERPRINT:0:16} to RESULT_FILENAME whenever run-sweep.yml supplies a recipe fingerprint (since feat(ci): support additive append-only curve updates #2613, 2026-08-14); the manual e2e-tests.yml path passes no fingerprint, which is why the identical point produced a 232-byte name and passed in the targeted run. The published baseline predates the fingerprint suffix and no sweep has published this family since, so every labeled sweep of this family's 1P1D point will hit this regardless of image. Fixing it requires shared template/result-processing changes, which this PR must not touch.
  • Diagnosis 2 (5P1D and 6P1D benchmarks, infrastructure flake, out of scope): both jobs died 6–7 minutes after submission when the nginx frontend container failed to start: pyxis: importing docker image: nginx … Authenticating with user: <anonymous> … Fetching image manifest list … pyxis: child … failed with error code: 23 … failed to import docker image. The recipes' frontend.nginx_container: nginx is pulled from Docker Hub by pyxis on the head node; the same pull succeeded for the 1P1D and 7P1D sweep jobs and for all four jobs of the targeted run, so this is a registry/network failure, not an image or recipe defect.
  • Next step: none within scope. Recommended follow-up for maintainers: shorten the generated multinode result filename (e.g. truncate/hash the conc… list or the _recipe- suffix) or move the nginx frontend to the pre-imported squash alias, then let the planner retry this candidate.
  • Terminal states (confirmed 2026-09-09 07:2x UTC): sweep run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946cancelled (jobs: 1P1D/5P1D/6P1D benchmark failure, 7P1D benchmark success, 1P1D/5P1D/7P1D eval success, 6P1D eval cancelled; no unfinished jobs). Owned e2e runs: 34303559979 failure (initial attempt), 34304865256 success (Repair 1); both completed with no unfinished jobs.

状态

当前状态: 已停止 —— 最终全量 sweep 因已确认的 CI 基础设施阻塞而失败,该问题超出本 PR 的编辑范围。 镜像更新本身可用:修复 1/5 的定向运行通过了 4/4 个基准点与 4/4 个 gsm8k 评测,每 GPU 吞吐相对 2026-08-10 基线提升 +8.3% 至 +15.6%。在带标签的 sweep(https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946)中,1P1D 基准已跑完,但结果处理因 OSError: [Errno 36] File name too long 崩溃:共享的多节点模板会在结果文件名后追加 _recipe-<fingerprint>,使该配置族 1P1D(conc1x4x8x16x32x64x256)的聚合文件名达到 256 字节(NAME_MAX 为 255);5P1D 与 6P1D 基准作业则因 pyxis 无法从 Docker Hub 拉取 nginx 前端镜像(匿名拉取,错误码 23)而失败。范围内没有任何改动能缩短生成的文件名,因此在共享模板/结果处理代码修复之前,该配置族的 sweep 无法通过。处置:取消 sweep 运行、移除标签、PR 转回草稿并关闭、删除远程分支,以便共享修复落地后可重试该候选。已用修复次数:1/5。

配置族:configs/nvidia-master.yaml:qwen3.5-fp4-gb300-dynamo-sglang(Qwen3.5-397B-A17B NVFP4-V2,GB300,dynamo-sglang,分离式,8k1k,无投机解码)。目标集群:gb300-nvconfigs/runners.yaml 中 runner gb300gb300-nv_0/1/2)。

变更:image: lmsysorg/sglang:v0.5.14-cu130lmsysorg/sglang:v0.5.19-cu130;配方的 dynamo.version 与配置族的 router.version 1.1.01.5.0.dev20260908(修复 1,见下文)(Docker Hub 清单摘要 sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9,2026-09-04 推送,包含 linux/arm64;v0.5.19 发布于 2026-09-05)。benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/8k1k/disagg/stp/ 下的四个 srt-slurm 配方继续使用 container: "dynamo-sglang" 别名,runners/launch_gb300-nv.sh 会将其映射到主配置镜像的 squash 文件,因此无需修改配方。基线与头部提交生成的矩阵仅 image 字段不同(4 个点,节点数 2/9/10/11,启用评测)。配方中的所有服务端参数在 v0.5.19 中均存在(--mamba-scheduler-strategy 与 v0.5.14 一样仍是 --mamba-radix-cache-strategy 的弃用别名)。

定向基准测试通过并不代表全局 PR 检查通过。

基线(发布日期 2026-08-10)

点(P/D 拓扑,kv) 并发 每 GPU 吞吐 (tok/s) 中位 TTFT (s) 中位 TPOT (s) 中位 E2E (s)
1P TP4 / 1D TP4,mooncake 1 242.87 0.381 0.00417 4.27
1P TP4 / 1D TP4,mooncake 4 806.75 0.418 0.00486 5.00
1P TP4 / 1D TP4,mooncake 8 1344.60 0.724 0.00567 5.98
1P TP4 / 1D TP4,mooncake 16 2292.35 0.745 0.00672 6.93
1P TP4 / 1D TP4,mooncake 32 3629.11 0.889 0.00849 8.72
1P TP4 / 1D TP4,mooncake 64 5745.02 1.034 0.01077 10.99
1P TP4 / 1D TP4,mooncake 256 9091.45 15.103 0.01383 28.01
5P DEP4 / 1D DEP16,nixl 2048 15626.02 12.200 0.01688 27.87
6P DEP4 / 1D DEP16,mooncake 5120 17301.32 40.608 0.01900 57.85
7P DEP4 / 1D DEP16,mooncake 5120 18370.36 27.722 0.02154 47.60
  • 已发布评测(gsm8k,n=1319,同一生产运行):1P/1D 并发 256 em_strict 0.9659 / em_flexible 0.9606;5P1D 并发 2048 0.9674 / 0.9560;6P1D 并发 5120 0.9666 / 0.9530;7P1D 并发 5120 0.9682 / 0.9530。

初始尝试

  • 镜像/提交:lmsysorg/sglang:v0.5.19-cu130,提交 ebde675f2fa4519650be9c0349d12c9d75694bfa
  • 运行:https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34303559979(2026-09-09T02:31:33Zmain 派发,inputs.ref=ebde675f2fa4519650be9c0349d12c9d75694bfafail-fast=trueklaud-run=true)。结果:失败(02:43 UTC)。
  • 变更:仅更新主配置镜像。
  • 基准/评测结果:N/A(1P1D 基准与 1P1D 评测作业在“Launch multi-node job script”步骤、服务启动前失败;fail-fast 取消了 5P1D/6P1D/7P1D 的基准与评测作业)。results_bmk/eval_results_all 产物为空。
  • 相对基线的差异:N/A(无测量结果)。
  • 诊断(首个服务端错误,srtctl 作业 29041 的 prefill worker):在“Dynamo 1.1.0 installed”之后,dynamo/sglang/args.py 抛出 ModuleNotFoundError: No module named 'sglang.srt.server_args_config_parser'。sglang v0.5.19 已将该模块移至 sglang.srt.utils.server_args_config_parserMove server args config parser under utils sgl-project/sglang#36681;旧路径存在于 v0.5.14–v0.5.18)。Dynamo v1.1.0 至 v1.4.2(及 release/1.4.2)仍导入旧路径;修复(fix(sglang): follow config parser module move ai-dynamo/dynamo#14054、#14234)仅在 main 分支及 PyPI 的 1.5.0.dev* 每日构建中;回移 PR #14064 已关闭。v0.5.19 squash 镜像本身可正常导入。
  • 下一步:修复 1 —— 固定包含兼容层的 Dynamo 构建。

修复 1/5

  • 镜像/提交:lmsysorg/sglang:v0.5.19-cu130,提交 30fbdb6a38ad300d4618dcdfefa467ef0fd69a86
  • 运行:https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34304865256(2026-09-09T02:51:26Zmain 派发,inputs.ref=30fbdb6a38ad300d4618dcdfefa467ef0fd69a86fail-fast=trueklaud-run=true)。
  • 变更:四个被引用配方(8k1k_stp_lowlat_0/maxtpt_0/maxtpt_1/maxtpt_2.yaml)的 dynamo.version "1.1.0""1.5.0.dev20260908";配置族主配置条目的 router.version "1.1.0""1.5.0.dev20260908",使发布的路由器元数据与实际安装的 Dynamo 一致。srt-slurm 生产端从 PyPI 安装 ai-dynamo-runtime==V ai-dynamo==V(不带 sglang extra),容器内 sglang 保持 v0.5.19。ai_dynamo-1.5.0.dev20260908 wheel 包含导入 sglang.srt.utils.server_args_config_parserdynamo/sglang/_compat.py;其导入的其他所有 sglang.srt 模块均存在于 v0.5.19 源码树中(多模态 encode-server 导入有 try/except 保护)。ai-dynamo-runtime 提供 cp310-abi3 manylinux_2_28_aarch64 wheel。
  • 结果:成功(运行于 2026-09-09 05:27 UTC 完成)。全部 4 个多节点基准作业与 4 个仅评测作业均成功;results_bmk/agg_bmk.json 共 10 行,每行 image: lmsysorg/sglang:v0.5.19-cu130eval_results_all/agg_eval_all.json 共 4 条。
  • 评测结果(gsm8k,n=1319,strict / flexible;括号内为基线):1P1D 并发 256 0.9697 / 0.9621(0.9659 / 0.9606);5P1D 并发 2048 0.9659 / 0.9598(0.9674 / 0.9560);6P1D 并发 5120 0.9636 / 0.9530(0.9666 / 0.9530);7P1D 并发 5120 0.9659 / 0.9575(0.9682 / 0.9530)。均在基线 ±0.5 个百分点内(标准误约 0.005)。
  • 相对基线的差异(逐点,新值 vs 2026-08-10 发布值;拓扑、并发与 8k1k 数据集相同;吞吐为 tput_per_gpu):
并发 每 GPU 吞吐 新 vs 基线(Δ) 中位 TTFT 新 vs 基线(Δ) 中位 TPOT 新 vs 基线(Δ) 中位 E2E 新 vs 基线(Δ)
1 279.61 vs 242.87(+15.1%) 0.150 vs 0.381 s(−60.6%) 3.84 vs 4.17 ms(−8.1%) 3.63 vs 4.27 s(−15.1%)
4 916.30 vs 806.75(+13.6%) 0.183 vs 0.418 s(−56.2%) 4.57 vs 4.86 ms(−6.0%) 4.40 vs 5.00 s(−12.0%)
8 1554.42 vs 1344.60(+15.6%) 0.195 vs 0.724 s(−73.1%) 5.31 vs 5.67 ms(−6.3%) 5.19 vs 5.98 s(−13.2%)
16 2595.83 vs 2292.35(+13.2%) 0.230 vs 0.745 s(−69.2%) 6.30 vs 6.72 ms(−6.3%) 6.15 vs 6.93 s(−11.2%)
32 4049.07 vs 3629.11(+11.6%) 0.333 vs 0.889 s(−62.5%) 8.07 vs 8.49 ms(−5.0%) 7.84 vs 8.72 s(−10.1%)
64 6347.02 vs 5745.02(+10.5%) 0.407 vs 1.034 s(−60.6%) 10.34 vs 10.77 ms(−4.1%) 9.96 vs 10.99 s(−9.4%)
256 9847.69 vs 9091.45(+8.3%) 13.694 vs 15.103 s(−9.3%) 13.48 vs 13.83 ms(−2.5%) 26.09 vs 28.01 s(−6.8%)

宽 EP 点(prefill DEP4 worker / 1 个 decode DEP16 worker):

并发 每 GPU 吞吐 新 vs 基线(Δ) 中位 TTFT 新 vs 基线(Δ) 中位 TPOT 新 vs 基线(Δ) 中位 E2E 新 vs 基线(Δ)
5P1D,nixl 2048 17654.45 vs 15626.02(+13.0%) 9.474 vs 12.200 s(−22.3%) 17.07 vs 16.88 ms(+1.1%,回退) 25.39 vs 27.87 s(−8.9%)
6P1D,mooncake 5120 19273.44 vs 17301.32(+11.4%) 35.541 vs 40.608 s(−12.5%) 18.67 vs 19.00 ms(−1.7%) 52.98 vs 57.85 s(−8.4%)
7P1D,mooncake 5120 20282.71 vs 18370.36(+10.4%) 24.590 vs 27.722 s(−11.3%) 22.07 vs 21.54 ms(+2.5%,回退) 45.09 vs 47.60 s(−5.3%)

全部 10 个点均匹配到基线点;无排除项。均为单次测量、未重复;两处 TPOT 轻微回退仅作报告,不设门限。

  • 下一步:追加 perf-changelog.yaml 条目并启动最终全量 sweep。

最终全量 sweep

  • 镜像/提交:lmsysorg/sglang:v0.5.19-cu130,头部 278090e154fbbca8b653b284c51c56b40add9534(新增 perf-changelog.yaml 条目;本地 utils/validate_perf_changelog.py --base-ref origin/main --head-ref HEAD 校验通过,既有字节完整保留)。
  • 转换:05:28 UTC gb300-nv 容量检查通过;PR 标记为 ready;添加 full-sweep-enabled 作为唯一 sweep 标签。
  • 精确头部上的运行:https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946(活动运行;由 full-sweep-enabled 标签事件触发;check-changelog 已通过,setup 运行中)。https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315043758https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044003(ready-for-review 事件)在任何基准作业运行前即被 sweep 并发组取消。
  • 结果:失败(在下列失败使成功不再可能后,由 Klaud Cold 取消运行;取消时的作业状态见“终态”)。
  • 作业结果:7P1D c5120 基准成功(05:54–06:35 UTC);1P1D 与 7P1D 评测成功;1P1D c1–256 基准失败(06:04 UTC);5P1D c2048 基准失败(06:44 UTC);6P1D c5120 基准失败(06:54 UTC);取消时 5P1D 评测正在运行、6P1D 评测排队中。
  • 诊断 1(1P1D 基准,确定性,超出范围):srtctl sweep 本身已完成(“Benchmark completed successfully”,7/7 个并发、所有请求完成,功耗遥测 publication_valid: true)。随后 Process result 步骤在 utils/process_result.py 中因 OSError: [Errno 36] File name too long: 'agg_qwen3.5_8k1k_fp4_dynamo-sglang_prefill-tp4-…-nw1_decode-tp4-…-nw1_disagg-true_spec-none_conc1x4x8x16x32x64x256_gb300-nv_00_recipe-0190dba4907b78b3_sa-bench_isl_8192_osl_1024_conc16_gpus_8_ctx_4_gen_4.json' 失败(256 字节;对应的 power_validation_… 文件名为 269 字节;Linux NAME_MAX 为 255)。只要 run-sweep.yml 提供配方指纹(自 feat(ci): support additive append-only curve updates #2613,2026-08-14 起),benchmark-multinode-tmpl.yml 就会在 RESULT_FILENAME 后追加 _recipe-${RECIPE_FINGERPRINT:0:16};手动 e2e-tests.yml 路径不传指纹,因此同一点在定向运行中生成 232 字节的文件名并通过。已发布基线早于指纹后缀的引入,此后该配置族没有任何 sweep 发布过数据,因此无论镜像如何,该配置族 1P1D 点在任何带标签的 sweep 中都会触发此问题。修复需要改动共享模板/结果处理代码,本 PR 不得触碰。
  • 诊断 2(5P1D 与 6P1D 基准,基础设施抖动,超出范围):两个作业在提交后 6–7 分钟因 nginx 前端容器无法启动而终止:pyxis: importing docker image: nginx … Authenticating with user: <anonymous> … Fetching image manifest list … pyxis: child … failed with error code: 23 … failed to import docker image。配方的 frontend.nginx_container: nginx 由头节点上的 pyxis 从 Docker Hub 拉取;同样的拉取在 1P1D、7P1D sweep 作业及定向运行的全部四个作业中均成功,因此这是镜像仓库/网络故障,而非镜像或配方缺陷。
  • 下一步:范围内无可行操作。建议维护者:缩短生成的多节点结果文件名(例如截断/哈希 conc… 列表或 _recipe- 后缀),或将 nginx 前端改用预导入的 squash 别名,然后让规划器重试该候选。
  • 终态(2026-09-09 07:2x UTC 确认):sweep 运行 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946已取消(作业:1P1D/5P1D/6P1D 基准失败,7P1D 基准成功,1P1D/5P1D/7P1D 评测成功,6P1D 评测已取消;无未完成作业)。自有 e2e 运行:34303559979 失败(初始尝试)、34304865256 成功(修复 1);均已完成且无未完成作业。

🤖 Generated with Claude Code

Update qwen3.5-fp4-gb300-dynamo-sglang from lmsysorg/sglang:v0.5.14-cu130
to lmsysorg/sglang:v0.5.19-cu130 (Docker Hub manifest digest
sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9,
linux/arm64 present). The four srt-slurm recipes keep the
`container: "dynamo-sglang"` alias, which runners/launch_gb300-nv.sh maps
to the master image's squash file, so model, topology, workloads and
recipe references are unchanged.

将 qwen3.5-fp4-gb300-dynamo-sglang 的镜像从 lmsysorg/sglang:v0.5.14-cu130
更新至 lmsysorg/sglang:v0.5.19-cu130(Docker Hub 清单摘要
sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9,
包含 linux/arm64)。四个 srt-slurm 配方继续使用 `container: "dynamo-sglang"`
别名,runners/launch_gb300-nv.sh 会将其映射到主配置镜像的 squash 文件,
因此模型、拓扑、负载和配方引用均保持不变。

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Dispatch. Draft PR opened with the single-line image bump (lmsysorg/sglang:v0.5.14-cu130lmsysorg/sglang:v0.5.19-cu130, commit ebde675f2fa4519650be9c0349d12c9d75694bfa). Capacity check on gb300-nv passed before the edit, before the push and before dispatch. Initial targeted run dispatched from main with test-config --config-files configs/nvidia-master.yaml --config-keys qwen3.5-fp4-gb300-dynamo-sglang, fail-fast=true, klaud-run=true: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34303559979 — 4 benchmark jobs (1P1D TP4 c1–256, 5P1D c2048, 6P1D c5120, 7P1D c5120) plus 4 default gsm8k eval jobs are queued behind human work.

Next step: monitor to completion and compare with the published 2026-08-10 baseline (run 29473335540).


派发。 已创建仅修改一行镜像的草稿 PR(lmsysorg/sglang:v0.5.14-cu130lmsysorg/sglang:v0.5.19-cu130,提交 ebde675f2fa4519650be9c0349d12c9d75694bfa)。gb300-nv 的容量检查在编辑前、推送前和派发前均通过。初始定向运行已从 main 派发,参数 test-config --config-files configs/nvidia-master.yaml --config-keys qwen3.5-fp4-gb300-dynamo-sglangfail-fast=trueklaud-run=truehttps://github.com/SemiAnalysisAI/InferenceX/actions/runs/34303559979 —— 4 个基准作业(1P1D TP4 c1–256、5P1D c2048、6P1D c5120、7P1D c5120)及 4 个默认 gsm8k 评测作业已排队,优先级低于人工作业。

下一步:监控至完成,并与 2026-08-10 发布的基线(运行 29473335540)对比。

Dynamo 1.1.0 (installed at runtime from the recipes' dynamo.version) imports
sglang.srt.server_args_config_parser, which sglang v0.5.19 moved to
sglang.srt.utils.server_args_config_parser (sgl-project/sglang#36681). The
prefill worker exited with ModuleNotFoundError before serving. No stable
Dynamo release carries the fix (ai-dynamo/dynamo#14054, #14234 are on main;
the release/1.4.2 backport #14064 was closed). The ai-dynamo
1.5.0.dev20260908 wheel ships dynamo/sglang/_compat.py with the new import
path, so pin it in the four qwen3.5 GB300 FP4 8k1k recipes and mirror it in
the family's router metadata.

Dynamo 1.1.0(由配方的 dynamo.version 在运行时安装)导入
sglang.srt.server_args_config_parser,而 sglang v0.5.19 已将其移至
sglang.srt.utils.server_args_config_parser(sgl-project/sglang#36681),
导致 prefill worker 在启动前因 ModuleNotFoundError 退出。目前没有稳定版
Dynamo 包含该修复(ai-dynamo/dynamo#14054、#14234 仅在 main 分支;
release/1.4.2 的回移 #14064 已关闭)。ai-dynamo 1.5.0.dev20260908 wheel
自带使用新导入路径的 dynamo/sglang/_compat.py,因此在四个 qwen3.5 GB300
FP4 8k1k 配方中固定该版本,并同步更新该配置族的路由器元数据。

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Initial attempt failed → Repair 1/5 dispatched.


初始尝试失败 → 已派发修复 1/5。

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Repair 1/5 progress. Run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34304865256 is past the previous failure point: the Dynamo 1.5.0.dev20260908 pin fixed the startup import error.

  • Confirmed: 1P1D TP4/TP4 benchmark job succeeded with all 7 points; versus the published 2026-08-10 baseline, throughput per GPU is +8.3% (conc 256) to +15.6% (conc 8), median TTFT is 9–73% lower and median TPOT 2.5–8.1% lower at every point (full table in the PR body). gsm8k evals passed: 1P1D conc 256 strict 0.9697 / flexible 0.9621 (baseline 0.9659 / 0.9606); 7P1D conc 5120 strict 0.9659 / flexible 0.9575 (baseline 0.9682 / 0.9530).
  • In progress: 7P1D c5120 benchmark (started 03:16 UTC); 5P1D c2048 and 6P1D c5120 benchmark + eval jobs are queued behind human work.
  • Next step: wait for the remaining jobs, then compare the wide-EP points. Green targeted benchmarks do not prove global PR checks pass.

修复 1/5 进展。 运行 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34304865256 已越过先前的失败点:固定 Dynamo 1.5.0.dev20260908 修复了启动时的导入错误。

  • 已确认: 1P1D TP4/TP4 基准作业成功,7 个点全部完成;相对 2026-08-10 发布的基线,每 GPU 吞吐提升 +8.3%(并发 256)至 +15.6%(并发 8),各点中位 TTFT 降低 9–73%,中位 TPOT 降低 2.5–8.1%(完整表格见 PR 正文)。gsm8k 评测通过:1P1D 并发 256 strict 0.9697 / flexible 0.9621(基线 0.9659 / 0.9606);7P1D 并发 5120 strict 0.9659 / flexible 0.9575(基线 0.9682 / 0.9530)。
  • 进行中: 7P1D c5120 基准(03:16 UTC 开始);5P1D c2048 与 6P1D c5120 的基准 + 评测作业在人工作业之后排队。
  • 下一步: 等待其余作业完成,再对比宽 EP 点。定向基准测试通过并不代表全局 PR 检查通过。

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Repair 1/5 progress: 3 of 4 benchmark points and 3 of 4 evals passed. Run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34304865256.

  • Confirmed: 7P1D c5120 benchmark succeeded: tput/GPU 20282.71 vs 18370.36 (+10.4%), median TTFT 24.59 vs 27.72 s (−11.3%), median TPOT 22.07 vs 21.54 ms (+2.5%, slight regression), median E2E 45.09 vs 47.60 s (−5.3%). 5P1D c2048 (nixl) succeeded: tput/GPU 17654.45 vs 15626.02 (+13.0%), median TTFT 9.47 vs 12.20 s (−22.3%), median TPOT 17.07 vs 16.88 ms (+1.1%), median E2E 25.39 vs 27.87 s (−8.9%). 5P1D gsm8k eval (conc 2048): strict 0.9659 / flexible 0.9598 (baseline 0.9674 / 0.9560).
  • In progress: 6P1D c5120 eval (started 04:29 UTC); 6P1D c5120 benchmark still queued.
  • Next step: wait for the 6P1D pair, then finalize Repair 1 results and, if green, append the perf-changelog.yaml entry and start the final full sweep.

修复 1/5 进展:4 个基准点中的 3 个、4 个评测中的 3 个已通过。 运行 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34304865256。

  • 已确认: 7P1D c5120 基准成功:每 GPU 吞吐 20282.71 vs 18370.36(+10.4%),中位 TTFT 24.59 vs 27.72 s(−11.3%),中位 TPOT 22.07 vs 21.54 ms(+2.5%,轻微回退),中位 E2E 45.09 vs 47.60 s(−5.3%)。5P1D c2048(nixl)成功:每 GPU 吞吐 17654.45 vs 15626.02(+13.0%),中位 TTFT 9.47 vs 12.20 s(−22.3%),中位 TPOT 17.07 vs 16.88 ms(+1.1%),中位 E2E 25.39 vs 27.87 s(−8.9%)。5P1D gsm8k 评测(并发 2048):strict 0.9659 / flexible 0.9598(基线 0.9674 / 0.9560)。
  • 进行中: 6P1D c5120 评测(04:29 UTC 开始);6P1D c5120 基准仍在排队。
  • 下一步: 等待 6P1D 两个作业完成,随后汇总修复 1 结果;若全部通过,则追加 perf-changelog.yaml 条目并启动最终全量 sweep。

Append the perf-changelog.yaml entry for qwen3.5-fp4-gb300-dynamo-sglang
(image lmsysorg/sglang:v0.5.19-cu130, Dynamo 1.5.0.dev20260908) with the
PR link, preserving all prior bytes.

为 qwen3.5-fp4-gb300-dynamo-sglang(镜像 lmsysorg/sglang:v0.5.19-cu130,
Dynamo 1.5.0.dev20260908)追加 perf-changelog.yaml 条目并附 PR 链接,
保留所有既有字节。

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Klaud-Cold
Klaud-Cold marked this pull request as ready for review September 9, 2026 05:28
@Klaud-Cold
Klaud-Cold requested a review from a team September 9, 2026 05:28
@claude

claude Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Claude finished @Klaud-Cold's task in 1m 1s —— View job


Review

LGTM - no blocking issues found

Verified: configs/nvidia-master.yaml change comes with a perf-changelog.yaml entry appended at the end of the file; the new image lmsysorg/sglang:v0.5.19-cu130 is a public Docker Hub reference; all four recipes referenced by the family's four search-space points (8k1k_stp_lowlat_0/maxtpt_0/maxtpt_1/maxtpt_2.yaml) carry the matching Dynamo 1.5.0.dev20260908 pin with no stray 1.1.0 left under gb300-fp4; and the router.version metadata matches the installed Dynamo.


@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Repair 1/5 passed → final full sweep started.


修复 1/5 通过 → 最终全量 sweep 已启动。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, straightforward config bump. Verified the dynamo.version/router.version pin to 1.5.0.dev20260908 is applied consistently across all four sibling recipes referenced by the qwen3.5-fp4-gb300-dynamo-sglang family in configs/nvidia-master.yaml (lowlat_0, maxtpt_0/1/2 — no sibling recipe missed), the image bump to v0.5.19-cu130 is only in the master config as intended (recipes use container: dynamo-sglang), and the perf-changelog.yaml entry is appended at the tail without disturbing prior entries.

Extended reasoning...

Overview

This PR bumps the Qwen3.5-397B-A17B FP4 GB300 disaggregated Dynamo-SGLang recipe family's SGLang image from v0.5.14-cu130 to v0.5.19-cu130, and in lockstep pins dynamo.version/router.version from 1.1.0 to the dev build 1.5.0.dev20260908 across the four 8k1k STP recipe YAMLs and the corresponding configs/nvidia-master.yaml entry, plus an appended perf-changelog.yaml entry referencing PR #2909.

Security risks

None. This is a pure configuration/version-pin change to internal benchmark recipe YAML and a changelog file — no code paths, auth, crypto, or permission logic are touched.

Level of scrutiny

Low. The change is mechanical and self-contained: a version bump plus a compatibility pin, fully confirmed by diffing the actual files. I cross-checked that the four recipe files touched are exactly the four referenced by configs/nvidia-master.yaml's qwen3.5-fp4-gb300-dynamo-sglang scenarios (lowlat_0, maxtpt_0/1/2), so no sibling recipe in the family was missed, and that the image bump only needed to land in the master config (recipes reference container: dynamo-sglang, not the raw image tag).

Other factors

The bug hunter reported no findings, and the changelog entry is correctly appended at the file's tail in the established format used by sibling entries (matching the pattern of other version-bump entries, including omission of scenario-type for fixed-seq-len-only families). No CODEOWNERS restriction applies to these plain config/data files based on repository conventions for this recipe area, and the PR conversation timeline shows no outstanding CHANGES_REQUESTED or unaddressed third-party objections.

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Final full sweep in progress (correction: the active run-sweep.yml run on head 278090e154fbbca8b653b284c51c56b40add9534 is https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946, created by the full-sweep-enabled label event; the two ready-for-review runs on the same SHA were cancelled by the sweep concurrency group before any benchmark ran).

  • Status: check-changelog and setup passed; the sweep fanned out the 4 multi-node 8k1k benchmark jobs and 4 eval jobs (no single-node canary candidate). 1P1D and 7P1D gsm8k evals already succeeded; 1P1D and 7P1D benchmarks running; 5P1D/6P1D queued.
  • Next step: wait for completion with reusable artifacts.

最终全量 sweep 进行中(更正:头部 278090e154fbbca8b653b284c51c56b40add9534 上的活动 run-sweep.yml 运行为 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946,由 full-sweep-enabled 标签事件创建;同一 SHA 上的两个 ready-for-review 运行在任何基准作业运行前即被 sweep 并发组取消)。

  • 状态: check-changelogsetup 已通过;sweep 已展开 4 个多节点 8k1k 基准作业与 4 个评测作业(无单节点 canary 候选)。1P1D 与 7P1D 的 gsm8k 评测已成功;1P1D 与 7P1D 基准运行中;5P1D/6P1D 排队中。
  • 下一步: 等待完成并产出可复用产物。

@Klaud-Cold

Klaud-Cold commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

CuTe DSL linear-attention backend switch / 切换 CuTe DSL 线性注意力后端

Working on @adibarra's request to switch the GDN linear-attention kernels to the CuTe DSL backend in the four qwen3.5/gb300-fp4/8k1k/disagg/stp/ recipes.

正在处理 @adibarra 的请求:将四个 qwen3.5/gb300-fp4/8k1k/disagg/stp/ 配方的 GDN 线性注意力内核切换到 CuTe DSL 后端。

Todo

  • Gather context (AGENTS.md, docs, recipes, current sweep status)
  • Verify linear-attn-backend: cutedsl is valid in sglang v0.5.19 (server_args + kernel SM guards)
  • Wait for the in-flight full sweep on 278090e to finish (clean v0.5.19-only datapoint)
  • Push the recipe change (4 recipes, prefill + decode blocks) as a new commit
  • Dispatch targeted run on the same 10 points and monitor to completion
  • Compare vs the v0.5.19 numbers from run 34304865256 (tput/GPU, TTFT, TPOT, E2E + 4 gsm8k evals)
  • If regression/startup failure: fallback repairs (linear-attn-decode-backend: cutedsl, then revert)
  • If win: update perf-changelog.yaml entry and re-run the full sweep

View job run · Branch

@Klaud-Cold

Klaud-Cold commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Claude finished @adibarra's task in 13m 37s —— View job


Linear-attention backend A/B/C on v0.5.19 / v0.5.19 上的线性注意力后端 A/B/C 对比

Working on @adibarra's A/B/C request on the four qwen3.5/gb300-fp4/8k1k/disagg/stp/ recipes. / 正在处理 @adibarra 的 A/B/C 请求。

Todo

  • Gather context; cancel superseded C-only worker (run 34317442100)
  • Verify variant flags in sglang v0.5.19 (explicit linear-attn-prefill-backend: flashinfer bypasses the 8192-chunk auto-gate; CuTe DSL GDN prefill needs SM100+, satisfied on GB300)
  • Prepare variant commits locally — B: 7b234dcc4 (adds FlashInfer prefill, 8 sites), C: 8c20e76e1 (replaces with linear-attn-backend: "cutedsl"); matrix generation validated at both (4 points, nodes 2/9/10/11)
  • Diagnose 1P1D failure in the in-flight full sweep (see below) and commit a fix (773633d8a, local)
  • Wait for the in-flight full sweep on 278090e to finish (cannot succeed — see below; waiting for capacity/instruction compliance)
  • Push variants, dispatch targeted 10-point runs for B and C, monitor
  • Compare A vs B vs C per decision rule; B' fallback if B fails at chunked-prefill 65536
  • Apply winner, update perf-changelog.yaml, re-run full sweep on final head

⚠️ Blocking finding: the full sweep on 278090e cannot succeed / 阻塞性发现:278090e 上的全量 sweep 无法成功

The 1P1D benchmark job of run 34315044946 failed at 06:04 UTC in "Process result", after all 7 benchmark points ran fine (05:32→06:03): utils/process_result.py crashed with OSError: [Errno 36] File name too long writing agg_qwen3.5_8k1k_..._conc1x4x8x16x32x64x256_gb300-nv_00_recipe-0190dba4907b78b3_sa-bench_..._conc16_....json256 bytes, one over the 255-byte filesystem limit. Root cause: PR sweeps (unlike targeted e2e runs) inject recipe-fingerprint via utils/process_changelog.py:542, and benchmark-multinode-tmpl.yml appends _recipe-<16 hex> to RESULT_FILENAME; this family's 1P1D point (7-value conc list + long disagg topology string) yields a 252-char result filename that only overflows once the agg_ prefix is added. That's why Repair 1's targeted run passed but every PR sweep of this family deterministically fails this point — a pre-existing main bug this family is the first to trip. The 7P1D benchmark is still running and 5P1D/6P1D are queued (no fail-fast on labeled sweeps); their artifacts should still be produced, but the run will conclude failure.

Fix committed locally (773633d8a): truncate the filename fingerprint suffix from 16 to 8 hex chars in benchmark-multinode-tmpl.yml and benchmark-tmpl.yml (agg name drops to 248). Only the filename changes — the full fingerprint still reaches result rows via the RECIPE_FINGERPRINT env var, and nothing parses the fingerprint back out of file/artifact names (verified across collect-results, reuse tooling, and process_result.py). Since PR sweeps run workflows from the merge ref, carrying this fix on this branch unblocks the final sweep without waiting for a separate PR to main.

运行 34315044946 的 1P1D 基准作业在全部 7 个点正常完成后,于“Process result”步骤因 OSError: [Errno 36] File name too long 失败:PR sweep(不同于定向 e2e 运行)会注入 recipe-fingerprint,模板将 _recipe-<16 位十六进制> 追加到 RESULT_FILENAME,该配置族 1P1D 点的结果文件名为 252 字符,加上 agg_ 前缀后达到 256 字节,超过 255 字节的文件系统上限。这是 main 上已存在的 bug,本配置族是首个触发者;该配置族的每次 PR sweep 都会必然失败于此点。已在本地提交修复(773633d8a):将文件名中的指纹后缀从 16 位截短为 8 位(完整指纹仍通过环境变量写入结果行)。PR sweep 使用合并引用的工作流,因此在本分支携带该修复即可解锁最终 sweep。

Next / 下一步

Waiting for run 34315044946 to reach a terminal state (7P1D running; 5P1D/6P1D queued), then: convert PR to draft → push B/C/fix commits → dispatch targeted e2e runs on 7b234dcc4 (B) and 8c20e76e1 (C) with fail-fast=true, klaud-run=true → compare against control A (Repair-1 numbers from run 34304865256). / 等待 sweep 终止后:转草稿 → 推送 → 在两个 SHA 上派发定向运行并与控制组 A 对比。


· Branch

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@Klaud-Cold

Copy link
Copy Markdown
Collaborator Author

Termination: final full sweep failed on an out-of-scope CI blocker; closing this PR.

  • Confirmed finding 1 (deterministic, shared code): in the labeled sweep https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946 the 1P1D benchmark completed all 7 concurrencies, but Process result crashed with OSError: [Errno 36] File name too longbenchmark-multinode-tmpl.yml appends _recipe-<16-char fingerprint> to RESULT_FILENAME in sweeps (since feat(ci): support additive append-only curve updates #2613), making this family's 1P1D aggregate filename 256 bytes (power_validation_… 269 bytes) against NAME_MAX 255. The same point produced a 232-byte name and passed in the fingerprint-less targeted run. No image or recipe change can shorten it, and no sweep has published this family since the suffix landed, so every labeled sweep of this family will fail here until the shared template/result-processing code is fixed.
  • Confirmed finding 2 (infrastructure flake): the 5P1D and 6P1D sweep benchmarks failed at startup because pyxis could not import the nginx frontend image from Docker Hub (anonymous pull, child … failed with error code: 23); the same pull succeeded in the other six jobs.
  • What works: the refresh itself is validated by the targeted run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34304865256 — 4/4 points, 4/4 evals, throughput per GPU +8.3% to +15.6% (details in the PR body).
  • Actions: sweep run cancelled and confirmed terminal (all owned runs finished: 34303559979 failure, 34304865256 success, 34315044946 cancelled); removing full-sweep-enabled, returning to draft, closing the PR and deleting the branch klaud/auto-233c0f70c5dbad2a-3d9c146c5e9ad2d6 so the candidate can be retried once the shared filename issue is fixed. Repairs used: 1 of 5. Suggested maintainer follow-up: shorten the generated multinode result filename (truncate/hash the conc… list or the _recipe- suffix) and consider serving nginx from the pre-imported squash alias.

终止:最终全量 sweep 因超出范围的 CI 阻塞而失败;关闭本 PR。

  • 已确认发现 1(确定性,共享代码): 在带标签的 sweep https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34315044946 中,1P1D 基准完成了全部 7 个并发,但 Process resultOSError: [Errno 36] File name too long 崩溃——benchmark-multinode-tmpl.yml 在 sweep 中会为 RESULT_FILENAME 追加 _recipe-<16 位指纹>(自 feat(ci): support additive append-only curve updates #2613 起),使该配置族 1P1D 的聚合文件名达到 256 字节(power_validation_… 为 269 字节),超过 NAME_MAX 255。同一点在不带指纹的定向运行中生成 232 字节文件名并通过。任何镜像或配方改动都无法缩短该文件名,且自该后缀引入以来该配置族没有任何 sweep 发布过数据,因此在共享模板/结果处理代码修复之前,该配置族的每次带标签 sweep 都会在此失败。
  • 已确认发现 2(基础设施抖动): 5P1D 与 6P1D sweep 基准在启动时失败,原因是 pyxis 无法从 Docker Hub 导入 nginx 前端镜像(匿名拉取,child … failed with error code: 23);同样的拉取在其余六个作业中均成功。
  • 可用部分: 镜像更新本身已由定向运行 https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34304865256 验证——4/4 个点、4/4 个评测,每 GPU 吞吐 +8.3% 至 +15.6%(详见 PR 正文)。
  • 操作: 已取消 sweep 运行并确认终态(所有自有运行均已结束:34303559979 失败、34304865256 成功、34315044946 已取消);移除 full-sweep-enabled、转回草稿、关闭 PR 并删除分支 klaud/auto-233c0f70c5dbad2a-3d9c146c5e9ad2d6,以便共享文件名问题修复后重试该候选。已用修复次数:1/5。建议维护者跟进:缩短生成的多节点结果文件名(截断/哈希 conc… 列表或 _recipe- 后缀),并考虑让 nginx 使用预导入的 squash 别名。

@Klaud-Cold
Klaud-Cold marked this pull request as draft September 9, 2026 07:17
@Klaud-Cold Klaud-Cold closed this Sep 9, 2026
@Klaud-Cold
Klaud-Cold deleted the klaud/auto-233c0f70c5dbad2a-3d9c146c5e9ad2d6 branch September 9, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant