feat(runners): add the B300 DSXE cluster and retire the B300 NV launcher / 添加 B300 DSXE 集群并下线 B300 NV 启动脚本 - #2826
Conversation
Registers the B300 DSXE Slurm cluster: a launcher, its 18 runner labels, and
its hardware facts. Removes launch_b300-nv.sh, whose fleet is decommissioned
and whose runner labels were already dropped from runners.yaml.
Both the multi-node and single-node paths now share one enroot import helper
instead of open-coding the import twice. The helper is defined above the
IS_MULTINODE branch so it is in scope for both, and it:
- serializes concurrent importers on a per-image flock, so only the first
caller pulls and the rest reuse the result rather than failing on an
existing -o target;
- verifies the squash file and re-imports it when it is missing or invalid;
- bounds the Slurm step with --time, so a lost step fails the job instead of
leaving the client waiting indefinitely;
- probes the shared path before allocating, so a warm cache costs no Slurm
allocation at all.
The import runs on a compute node because enroot builds the squashfs over an
overlay mount, which the shared filesystem cannot back.
Also repoints the workflow path filter, comments and docs that named the
removed launcher, and drops the SALLOC_EXCLUDE default, which listed nodes
belonging to the retired fleet.
configs/nvidia-master.yaml still pins 13 configs to cluster:b300-nv, so
full-sweep matrix generation stays broken exactly as it is on main today.
Resolving those entries is left to a separate change.
注册 B300 DSXE Slurm 集群：新增启动脚本、18 个 runner 标签及其硬件信息。
删除 launch_b300-nv.sh，该集群已下线，其 runner 标签此前已从 runners.yaml 中移除。
多节点与单节点路径现在共用同一个 enroot 导入函数，不再各自实现一遍。该函数定义在
IS_MULTINODE 分支之前，因此两条路径都能调用，并且：
- 使用按镜像的 flock 串行化并发导入，只有第一个调用方真正拉取镜像，其余调用方复用
结果，而不会因为 -o 目标已存在而失败；
- 校验 squash 文件，缺失或损坏时重新导入；
- 使用 --time 限制 Slurm step，step 丢失时任务直接失败，而不会让客户端无限等待；
- 在申请资源前先探测共享路径，因此缓存命中时完全不占用 Slurm 资源。
导入在计算节点上执行，因为 enroot 需要在 overlay 挂载之上构建 squashfs，而共享文件
系统无法支撑该挂载。
同时更新了引用已删除启动脚本的工作流路径过滤器、注释和文档，并移除 SALLOC_EXCLUDE
默认值，其中列出的节点属于已下线的集群。
configs/nvidia-master.yaml 中仍有 13 个配置指向 cluster:b300-nv，因此 full-sweep
矩阵生成与当前 main 分支一样仍然失败。这些条目留待单独的改动处理。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 48e7728. Configure here.
| rm -f \"$sqsh\" | ||
| enroot import -o \"$sqsh\" \"docker://$image_ref\" | ||
| unsquashfs -l \"$sqsh\" > /dev/null | ||
| " || { echo "Error: enroot import failed for $image_ref -> $sqsh" >&2; exit 1; } |
There was a problem hiding this comment.
Import lock timeout too short
Medium Severity
Concurrent first-time imports can abort waiters while the pull is still running. The srun step allows up to 120 minutes, but flock gives waiters only 3600 seconds. A sweep's leaves all call import_squash_image on the same image, so a 60–120 minute enroot pull makes those waiters fail even though the importer would finish inside the step limit.
Reviewed by Cursor Bugbot for commit 48e7728. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
This automated review ran and did not find any bugs, but I'd still like a human to look given the scope of this change.
What was reviewed: the new 515-line runners/launch_b300-dsxe.sh replacing the deleted launch_b300-nv.sh, the new cluster:b300-dsxe runner label/hardware entries in configs/runners.yaml, and the workflow/README/CONFIGS.md references updated to match. I checked whether the cluster:b300-nv label existed in runners.yaml before this PR (it did not — configs/nvidia-master.yaml's ~13 runner: cluster:b300-nv entries were already dangling pre-PR, so this isn't a regression introduced here) and confirmed .github/workflows/test-process-result.yml's path trigger was updated to the new script name.
Extended reasoning...
Overview
This PR retires runners/launch_b300-nv.sh (590 lines, deleted) and introduces runners/launch_b300-dsxe.sh (515 lines, new) for a renamed B300 cluster, adds a new cluster:b300-dsxe label/hardware block to configs/runners.yaml, and touches three GitHub workflow files, two benchmark shell scripts, and two docs files to reference the new launcher/cluster name.
Security risks
None identified. The changes are CI/build infrastructure (runner selection, Slurm launch scripts, docs) with no new handling of secrets, auth, or untrusted input beyond what already existed in the retired script.
Level of scrutiny
I verified one specific concern myself: whether the new cluster:b300-dsxe label in runners.yaml was actually a rename that orphaned configs/nvidia-master.yaml entries pointing at cluster:b300-nv. Diffing against the pre-PR runners.yaml shows cluster:b300-nv was never defined there — nvidia-master.yaml's ~13 references to it were already dangling before this PR, so that particular drift is pre-existing rather than introduced by this change. Several other candidates flagged by the bug-hunting sub-agents (dropped default_time_limit in the generated srtslurm.yaml, a hardcoded srt-slurm version tag replacing per-model pins, and unconditional --no-preflight on multi-node jobs) were investigated and ultimately not reported as confirmed bugs, but they represent real behavioral/design differences from the retired script that a human familiar with the B300 fleet's operational history should confirm are intentional.
Other factors
This is a large, security-review-appropriate-for-infra change (full launcher rewrite plus cluster rename across configs and workflows) with real operational consequences if a launch parameter regressed silently, so even with zero confirmed bugs I think a human pass adds value here.
This review covers commit 48e7728, which is no longer the latest commit on this pull request; later commits are not covered by it.
Extend the TP8 x DCP8 Mooncake sweep with concurrency 10, 12 and 14. They land in the recipe's conc 9-16 band and draft with DSpark level 3 at the committed golden AL 3.00. Repoint the runner from the retired cluster:b300-nv (#2826) to cluster:b300-dsxe so the sweep can schedule. 为 kimik3-fp4-b300-vllm-agentic-dspark 的 TP8 x DCP8 Mooncake 扫描新增并发 10、12、14。这些点落在 recipe 现有的 conc 9-16 区间，以 DSpark level 3 和已提交的 golden AL 3.00 起草。同时将 runner 从已在 #2826 退役的 cluster:b300-nv 改为 cluster:b300-dsxe，使扫描可以调度。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d in #2826 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
launch_b300-nv.sh was retired on main (#2826) in favor of the B300 DSXE cluster, so PR #2654's runner move to cluster:b300-dsxe needs the GLM-5.2 AgentX special-casing carried over: the pinned NVIDIA/srt-slurm#313 fork checkout (scoped to just the glm5.2/b300-fp4 recipes), and the UCX_TLS-unset wrapper around srtctl apply for dynamo-sglang. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 participants
Summary / 概述
Registers the B300 DSXE Slurm cluster and removes the launcher for the decommissioned B300 NV fleet.
runners.yamllost the B300 runner labels when that fleet was retired, butrunners/launch_b300-nv.shand a number of references to it stayed behind. This adds the replacement cluster and finishes the removal.注册 B300 DSXE Slurm 集群，并删除已下线的 B300 NV 集群的启动脚本。
该集群下线时，
runners.yaml中的 B300 runner 标签已被移除，但runners/launch_b300-nv.sh及若干对它的引用仍然保留。本 PR 新增替代集群并完成清理。What changed / 改动内容
runners/launch_b300-dsxe.sh— new launcher for the DSXE cluster.runners/launch_b300-nv.sh— removed.configs/runners.yaml—b300SKU group,cluster:b300-dsxelabels (18 runners), and the cluster's hardware facts.test-process-result.ymlpath filter, plus comments and docs that named the removed launcher.benchmark-tmpl.yml— dropped theSALLOC_EXCLUDEdefault, which listed nodes belonging to the retired fleet and can never match a node on this cluster.runners/launch_b300-dsxe.sh— DSXE 集群的新启动脚本。runners/launch_b300-nv.sh— 已删除。configs/runners.yaml—b300SKU 分组、cluster:b300-dsxe标签（18 个 runner）以及该集群的硬件信息。test-process-result.yml的路径过滤器，以及提到已删除启动脚本的注释和文档。benchmark-tmpl.yml— 移除SALLOC_EXCLUDE默认值，其中列出的节点属于已下线集群，在本集群上永远不会匹配。Shared enroot import / 共用的 enroot 导入逻辑
The multi-node and single-node paths used to open-code the container import separately. They now share one helper, defined above the
IS_MULTINODEbranch so it is in scope for both.Every leaf of a sweep imports the same image concurrently, so the helper:
flock, so only the first caller pulls and the rest reuse the result instead of failing on an existing-otarget;--time, so a lost step fails the job rather than leaving the client waiting indefinitely;The import runs on a compute node rather than the submit host: enroot builds the squashfs over an overlay mount, which the shared filesystem cannot back.
多节点与单节点路径此前各自实现了一遍容器导入逻辑，现在共用同一个函数。该函数定义在
IS_MULTINODE分支之前，因此两条路径都能调用。同一次 sweep 的各个作业会并发导入同一镜像，因此该函数：
flock串行化导入，只有第一个调用方真正拉取镜像，其余调用方复用结果，而不会因为-o目标已存在而失败；--time限制 Slurm step，step 丢失时任务直接失败，而不会让客户端无限等待；导入在计算节点而非提交节点上执行：enroot 需要在 overlay 挂载之上构建 squashfs，而共享文件系统无法支撑该挂载。
Known follow-up / 已知的后续工作
configs/nvidia-master.yamlstill pins 13 configs tocluster:b300-nv. Full-sweep matrix generation therefore stays broken exactly as it is onmaintoday — this PR does not introduce that failure, and it moves the error from an unresolvableb300SKU group to the remainingcluster:b300-nventries. Resolving them (repoint or remove) is a separate decision and a separate change.configs/nvidia-master.yaml中仍有 13 个配置指向cluster:b300-nv，因此 full-sweep 矩阵生成与当前main分支一样仍然失败 —— 该问题并非本 PR 引入，本 PR 只是把错误从无法解析的b300SKU 分组前移到了剩余的cluster:b300-nv条目。如何处理这些条目（改指向或删除）是另一个决策，将在单独的改动中完成。Validation / 验证
bash -non the new launcher and every edited shell script; YAML parses for all edited workflows andrunners.yaml.utils/matrix_logicsuite: 246 passed.runners.yamlresolves: 18 entries in both label lists, hardware entry present.srun/flock/enroot/unsquashfs: four concurrent callers produce exactly one pull and four successes; a warm cache allocates nothing; a corrupted squash file is re-imported; a failing import aborts the launcher.A single-node smoke run is still worth doing before merge — that is the path whose execution context changed, and no local test covers a real
enroot import.bash -n；所有修改过的工作流和runners.yaml均可正常解析。utils/matrix_logic测试套件：246 项通过。runners.yaml可正确解析：两个标签列表各 18 项，硬件条目齐全。srun/flock/enroot/unsquashfs验证导入函数：四个并发调用只产生一次拉取且全部成功；缓存命中时不占用任何资源；损坏的 squash 文件会被重新导入；导入失败时启动脚本会中止。合并前仍建议做一次单节点冒烟测试 —— 这条路径的执行环境发生了变化，且本地测试无法覆盖真实的
enroot import。Note
Medium Risk
Changes core CI launch paths for all B300 jobs (Slurm allocation, enroot import, model paths, and srt-slurm behavior); misconfiguration would fail or mis-schedule benchmarks rather than affecting app security.
Overview
Migrates B300 benchmarking from the retired NV fleet to the new DSXE Slurm cluster (
dsxe-sa-b300-prd0), replacingrunners/launch_b300-nv.shwithrunners/launch_b300-dsxe.shand wiring CI/docs to the new runner prefix.configs/runners.yamlnow lists 18b300-dsxe_*runners under both theb300SKU andcluster:b300-dsxe, with ~3.98M MiB host DRAM (up from the old NV hardware metadata). Workflows and docs (speedbench-al.yml,test-process-result.yml, agentic README,CONFIGS.md,RUNNER_SETUP.md) point atlaunch_b300-dsxe.shandcluster:b300-dsxe.benchmark-tmpl.ymldrops the defaultSALLOC_EXCLUDEfor broken NV nodes.The new launcher centralizes cluster facts (squash dir under the runner user,
/scratch/modelsstaging, alias-drivensrtslurm.yamlfor multinode) and unifies enroot import inimport_squash_image(flock, compute-nodesrun, cache probe, time-bounded steps). Multinode path pins NVIDIA/srt-slurm v1.0.87 (or the power fork when dcgm-power is enabled) and always uses--no-preflightbecause weights are node-local. Single-node path mounts only the model root that holdsMODEL_PATHand still supports optionalSALLOC_EXCLUDE.Not in this PR:
configs/nvidia-master.yamlmay still referencecluster:b300-nv, so full-sweep matrix generation can remain broken until those entries are repointed.Reviewed by Cursor Bugbot for commit 2611e8f. Bugbot is set up for automated code reviews on this repo. Configure here.