Port GLM-5.1 TileRT AgentX to native srt-slurm - #2750
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
cquil11
added
the
agentx-fast
|
Claude finished @cquil11's task in 6m 0s —— View job Review of PR #2750
SummaryFound 1 blocking issue (inline comment on Everything else checked out: the deleted agentic wrapper has no remaining references; the agentic config's single MTP point routes cleanly through the pinned TileRT srt-slurm fork with |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ $FRAMEWORK != "dynamo-vllm" ]] && | ||
| [[ $MODEL_PREFIX != "dsv4" || $PRECISION != "fp4" || $FRAMEWORK != "dynamo-sglang" || $SPEC_DECODING != "mtp" ]]; then | ||
| [[ $MODEL_PREFIX != "dsv4" || $PRECISION != "fp4" || $FRAMEWORK != "dynamo-sglang" || $SPEC_DECODING != "mtp" ]] && | ||
| [[ $MODEL_PREFIX != "glm5.1" || $PRECISION != "fp8" || $FRAMEWORK != "tilert" || $SPEC_DECODING != "mtp" ]]; then |
There was a problem hiding this comment.
🔴 BLOCKING: This gate also captures the existing non-agentic glm5.1-fp8-b200-tilert fixed-seq-len points (1k1k and 8k1k in configs/nvidia-master.yaml:9754), which are also MODEL_PREFIX=glm5.1 / PRECISION=fp8 / FRAMEWORK=tilert / SPEC_DECODING=mtp. Before this PR they fell through to the compat launcher (which routes them to benchmarks/multi_node/glm5.1_fp8_b200_tilert-disagg.sh); now they continue down the native srt-slurm path.
Why it matters: Those two existing sweep points will hard-fail:
- They set no
CONFIG_FILEin additional-settings, so the run dies at theCONFIG_FILE is not setcheck (line 307). - The recipe-clone chain's TileRT branch (line 111) requires
IS_AGENTIC == 1, so they'd clone the default NVIDIA srt-slurm with kimi-k2.6 recipes instead of the TileRT fork. - With
FRAMEWORK == tilertbutIS_AGENTIC != 1,DEFAULT_MOUNTS_BLOCK(line 245–251) appends indented mount entries to an empty string, emitting mapping lines without adefault_mounts:parent key — malformedsrtslurm.yaml.
Fix: Restrict the srt-slurm diversion to the agentic point so fixed-seq-len keeps its compat route:
| [[ $MODEL_PREFIX != "glm5.1" || $PRECISION != "fp8" || $FRAMEWORK != "tilert" || $SPEC_DECODING != "mtp" ]]; then | |
| [[ $MODEL_PREFIX != "glm5.1" || $PRECISION != "fp8" || $FRAMEWORK != "tilert" || $SPEC_DECODING != "mtp" || "${IS_AGENTIC:-0}" != "1" ]]; then |
(Alternatively, also port the fixed-seq-len config to srt-slurm recipes in this PR — but as-is those two points break.)
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33031731488 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| export MODEL_PATH="${MODEL_PATH:-$NSCALE_MODEL_ROOT/Kimi-K3}" | ||
| export SRT_SLURM_MODEL_PREFIX="kimik3" | ||
| elif [[ $MODEL_PREFIX == "glm5.1" && $PRECISION == "fp8" && $FRAMEWORK == "tilert" ]]; then | ||
| export SRT_SLURM_MODEL_PREFIX="glm5.1-fp8" |
There was a problem hiding this comment.
Pre-staged GLM-5.1 path unused
High Severity
The glm5.1 TileRT branch exports SRT_SLURM_MODEL_PREFIX but never sets MODEL_PATH, so model_paths is empty. The recipe uses hf:zai-org/GLM-5.1-FP8 instead of that alias, so prefill does not load /scratch/models/GLM-5.1-FP8. Unlike other Nscale models, glm5.1 also omits --no-preflight, so apply can fail on the login node.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 63170fb. Configure here.
| UCX_MEMTYPE_CACHE: "n" | ||
| UCX_MEMTYPE_REG_WHOLE: "n" | ||
| decode_environment: | ||
| <<: *common_environment |
There was a problem hiding this comment.
Nscale UCX devices not configured
Medium Severity
Prefill and decode UCX env sets UCX_MEMTYPE_CACHE and UCX_MEMTYPE_REG_WHOLE but not UCX_NET_DEVICES. The compat TileRT path on this cluster pins the eight RoCE HCAs (mlx5_0 through mlx5_7) for NIXL. Without that pin, KV transfer can bind the wrong NICs and fail or fall back off RDMA.
Reviewed by Cursor Bugbot for commit 63170fb. Configure here.
| fi | ||
|
|
||
| SRTCTL_ROOT="${GITHUB_WORKSPACE}/${SRT_REPO_DIR}" | ||
| echo "Creating srtslurm.yaml configuration..." |
There was a problem hiding this comment.
DCGM import URI is double-transformed
High Severity
import_squash now runs every image through enroot_uri_for_image, but the DCGM path still pre-rewrites nvcr.io/ to nvcr.io# first. The helper then treats nvcr.io#nvidia as the registry and emits an invalid Enroot URI, so power-enabled B200 nscale launches fail when the exporter squash is not already cached.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 01ac482. Configure here.
| elif [[ $MODEL_PREFIX == "glm5.1" && $PRECISION == "fp8" && $FRAMEWORK == "tilert" ]]; then | ||
| export SRT_SLURM_MODEL_PREFIX="glm5.1-fp8" | ||
| else | ||
| run_compat_launcher | ||
| fi | ||
|
|
||
| if [[ $FRAMEWORK != "dynamo-vllm" ]] && | ||
| [[ $MODEL_PREFIX != "dsv4" || $PRECISION != "fp4" || $FRAMEWORK != "dynamo-sglang" || $SPEC_DECODING != "mtp" ]]; then | ||
| [[ $MODEL_PREFIX != "dsv4" || $PRECISION != "fp4" || $FRAMEWORK != "dynamo-sglang" || $SPEC_DECODING != "mtp" ]] && | ||
| [[ $MODEL_PREFIX != "glm5.1" || $PRECISION != "fp8" || $FRAMEWORK != "tilert" || $SPEC_DECODING != "mtp" ]]; then | ||
| run_compat_launcher | ||
| fi |
There was a problem hiding this comment.
🔴 New glm5.1+fp8+tilert compat-bypass condition (lines 51-52, 59) is not scoped to IS_AGENTIC, so it also captures the pre-existing non-agentic glm5.1-fp8-b200-tilert recipe, which previously worked via run_compat_launcher's legacy tilert exec path.
Extended reasoning...
The non-agentic glm5.1-fp8-b200-tilert master-config entry (configs/nvidia-master.yaml, ~line 9754) sets spec-decoding: mtp on all its scenarios and has no CONFIG_FILE additional-setting. After this diff it no longer falls through to run_compat_launcher (line 51-59 match), but the srt-slurm clone dispatch at line 111 only fires when IS_AGENTIC==1, so this recipe instead falls into the generic else clone branch (line 144, clones vanilla NVIDIA/srt-slurm and copies the unrelated kimi-k2.6 recipe) and then unconditionally hits if [[ -z "$CONFIG_FILE" ]]; then echo Error: CONFIG_FILE is not set...; exit 1; fi (line 307-311). A benchmark that ran successfully before this PR now fails every time with 'CONFIG_FILE is not set', a user-visible regression this diff must avoid by scoping the new elif at line 51-52 (or the line 59 clause) to IS_AGENTIC, or by giving the non-agentic recipe its own CONFIG_FILE/clone branch.
Verification: normal. The new glm5.1+fp8+tilert branches are not scoped to IS_AGENTIC, so they capture the pre-existing NON-agentic recipe glm5.1-fp8-b200-tilert (configs/nvidia-master.yaml:9754) and break it. That recipe: framework: tilert, precision: fp8, model-prefix: glm5.1, spec-decoding: "mtp" on both fixed-seq-len scenarios (so IS_AGENTIC=0), and no CONFIG_FILE additional-setting. Merged trace…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33032282475 |
01ac482 to
666d9bd
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33037207390 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33037692277 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33037692277 |
539f4c7 to
c3a5b76
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33050263338 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33056748068 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33070434180 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33072516720 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 4 total unresolved issues (including 3 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ce74d8a. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33074180129 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33075765704 |
cquil11
removed
the
agentx-fast
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33079289298 |
1 participant
Summary
Validation
Note
Medium Risk
Touches Slurm launcher orchestration, dual-container Enroot imports, and AgentX context/metrics wiring; misconfiguration could break runs or skew trace filtering, but scope is limited to one B200 TileRT agentic recipe.
Overview
Moves B200 GLM-5.1 FP8 TileRT AgentX MTP 1P1D off the manual
submit.sh/ salloc path and onto native srt-slurm, deletingglm5.1_fp8_b200_tilert-disagg.shand wiringnvidia-master.yamlto a new recipe atrecipes/tilert/glm5.1/b200-fp8/agentic/disagg-1p1d-tp8-mtp.yaml.The recipe defines heterogeneous vLLM prefill + TileRT decode (NIXL, MTP, 202,752 context), TileRT router frontend, and a custom benchmark that runs
agentic_srt.shwithAIPERF_MAX_CONTEXT_LENGTH=202752,AIPERF_REQUIRED_SERVER_METRIC_PREFIX=vllm:(prefill Prometheus only), and Weka trace filtering aligned to that limit.agentic_srt.shnow optionally setsMAX_MODEL_LENfromAIPERF_MAX_CONTEXT_LENGTHafterbenchmark_libclears inherited limits.launch_b200-nscale-slurm.shadds a TileRT agentic branch: pins SemiAnalysisAI/srt-slurm atd1e6c97…, copies TileRT recipes, squashes prefill + decode images (including ghcr.io viaenroot_uri_for_image), and mounts shared TileRT weight cache plus workspace.Reviewed by Cursor Bugbot for commit 5084f22. Bugbot is set up for automated code reviews on this repo. Configure here.