refactor: standardize srt-slurm v2.2.1 and migrate recipes

[cquil11]

Standardize NVIDIA Slurm launchers on srt-slurm v2.2.1, migrate the owned recipes, and organize them under consistent model-first paths. Recipe PR #3103 has been integrated into this branch.

• Pin NVIDIA srt-slurm at v2.2.1 (984180e5b8755aef85e9995048b5a16cb5336bce) through utils/srt-slurm; preserve the existing AIPerf submodule.
• Use shared job-local runtime setup and provenance stamping. TileRT remains one manual fork exception at 6bc3f306bdafa1edfb5dded2fcda8f1ccede1bde, with no separate submodule.
• Remove per-model checkout chains and obsolete runtime patches. Use native post-eval dispatch with explicit inputs, schema-2 node counting and synthetic injection, and validated power CSV parsing.
• Migrate all 716 recipes to schema 2 and organize them as <model-prefix>/<engine>/<gpu>-<precision>/<workload>/<recipe>.yaml. Update active/deprecated master references, launcher path rules, workflow filters, and English/Chinese documentation together.
• Preserve original dedicated-service placement and include the four upstream EPLB expert-assignment assets. Job-local staging resolves all 34 EPLB references across 17 recipes.

Validation: 263 focused launcher, injector, power and matrix tests passed during implementation. Recipe organization preserves all 618 generated multinode rows after path substitution and all 45 expanded variants across six override bundles. The restored service blocks match fresh migration of their originals. Three independent reviewers assessed #3103 before its integration into this development branch; their approval of that integration does not establish production readiness.

Known power limitations retained for PowerX maintainer follow-up: the pinned upstream runtime rejects 18 NVIDIA recipes with dedicated-infrastructure power telemetry (697 of 715 NVIDIA files pass validation). Multinode AgentX power also requires measurement-window metadata that the pin does not provide. TileRT passed its fork validator. Hardware validation has not run. Preexisting internal recipe-name inconsistencies and worker-only scheduling estimates remain outside this migration's scope.

No performance changelog entry or new srt-slurm test workflow is included, as requested.

Release upgrade validation: 263 focused tests pass with v2.2.1; all 618 generated multinode matrix entries are unchanged. Eleven SGLang recipes now explicitly select sglang-router to preserve their original Model Gateway behavior after the upstream frontend rename. The same 18 dedicated-infrastructure telemetry validation failures remain. The user explicitly requested merging with the known power limitations and will notify the PowerX maintainer. No hardware sweep was run.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

Claude finished @cquil11's task in 8m 51s —— View job

---

Review of PR #3086

• Gather context: diff stats, changed non-recipe files
• Review launcher / runtime changes (runners/launch_*.sh, slurm_utils.sh)
• Review matrix, power, and injector changes
• Verify perf-changelog.yaml is appended at the end and covers master config edits
• Sample migrated recipes for correctness
• Run available tests (blocked: pytest, python3, network access require approval in this sandbox)
• Post inline findings and summary

Summary

No blocking issues found. Master config edits are covered by a perf-changelog.yaml entry appended at the tail, and all 85 config keys I spot-checked exist. Launcher consolidation into setup_srt_slurm preserves the power-stamp semantics (power-producer-sha.txt only for USES_DCGM_POWER / USES_KIMIK3_POWER), the dcgm_exporter telemetry detection matches all 119 power recipes, every top-level worker role carries a nodes field (1336/1336) so the schema-2 node counting cannot emit nodes:0, and the role-scoped acceptance injectors target block-style env in all recipes that set SYNTHETIC_ACCEPTANCE=true. Base-wrapped override recipes (b200-fp4/8k1k.yaml, glm5.2-agentx.yaml, etc.) carry no telemetry, so the power concurrency injector is never applied to them.

One 🟡 warning posted inline: the eval-override test does not validate the --set paths (post_eval.command, post_eval.passthrough_env, frontend.placement.node=head) against the pinned schema, so a field-name mismatch would only fail on hardware. I could not fetch the upstream srt-slurm sources from this sandbox to confirm them independently, so the pending eval sweep is the real gate here.

Not run locally due to sandbox restrictions: the pytest suites and the srt-slurm clone the new test-srt-slurm.yml workflow performs. CI results for that workflow should be treated as the authoritative check.

---

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline finding, I double-checked the ruled-out concern that the schema-2 role-env injector (runners/synthetic_injectors/_roles.py) only matches roles: anchored at column 0 — relevant because the six base:-wrapped multi-variant recipes this PR carries forward (e.g. b200-fp4/8k1k.yaml, glm5-mtp.yaml x2, glm5.2-agentx.yaml x2) nest roles: under base:/override_* at indent 2, not column 0. I confirmed in configs/nvidia-master.yaml that none of the matrix scenarios referencing those six files via their override_*/zip_override_* selectors currently set SYNTHETIC_ACCEPTANCE=true (MTP variants there route through separately-resolved standalone recipe files instead), so the injector gap is real but latent, not exercised by today's matrix.

Extended reasoning...

This is the narrow findings-present exception: a confirmed CI-trigger-path finding is already posted inline, so my role here is limited to noting what else was checked and ruled out. I independently re-verified one of the two previously ruled-out candidates (the _roles.py column-0 roles: anchor) against the actual recipe files this PR adds/keeps and against configs/nvidia-master.yaml, since the fragment I was given implied it might affect real files. It does affect the YAML structure of six specific recipes, but none of the matrix entries that select MTP variants from those particular files currently enable SYNTHETIC_ACCEPTANCE, so the gap does not fire under the present configuration — consistent with the prior rule-out, now corroborated with a concrete code-path trace rather than just restated.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f47df86. Configure here.}