[ROCm][DistInf] Enable vLLM DI CI with buildkite/slurm - #47030
Conversation
|
This PR requires #46482 to be merged first. |
|
This pull request has merge conflicts that must be resolved before it can be |
Match the upstream/reference vLLM disagg CI (vllm-project/vllm#47030), which runs 1P1D TP8 on DeepSeek-V3 (standard fp8, no quant remap). The MiniMax-M3-MXFP8 default crashed at VllmConfig with "Missing 'quant_algo'" because the mxfp8->modelopt_mxfp8 remap forces a non-ModelOpt checkpoint down the ModelOpt path. MXFP8 enablement is deferred; run the reference model to exercise the disagg path itself. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the aiter-side injection (aiter_vllm_quant_patch.py / sitecustomize.py / aiter_vllm_cli.py heredocs + the vllm_disagg.sh rewrite that swapped `vllm serve` for `python3 aiter_vllm_cli.py serve`). The wrapper called main() at module top level with no `if __name__ == "__main__":` guard, so vLLM's spawn-based engine core re-imported it and re-ran main() during bootstrap -> "An attempt has been made to start a new process before the current process has finished its bootstrapping phase". This broke every model that reaches engine spawn (DeepSeek-V3 job 2068); MiniMax only masked it by failing earlier at quant config. Match the reference disagg CI (vllm-project/vllm#47030), which runs the guarded `vllm serve` console entrypoint directly with no wrapper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* CI: add vLLM disagg Spur smoke workflow * CI: fix vLLM disagg workflow actionlint * CI: add actionlint runner labels * CI: schedule vLLM DI smoke nightly * change runner * CI: fix vLLM disagg Spur smoke submission * CI: move vLLM disagg smoke to idle nodes * CI: run vLLM disagg smoke on fabric 13 14 * CI: pin vLLM disagg smoke to fabric-15,16 Move the 2-node vLLM disagg smoke off fabric-13,14 (used by the ATOM smoke) to fabric-15,16. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: point vLLM disagg log/scripts stage to a writable dir The Slurm job runs as the xinhuang runner user and cannot write to /data/csrikris/logs/ (another user's dir) -> "Permission denied" on the disagg log, so the job failed on launch. Point DISAGG_SCRIPTS_STAGE at /data/xinhuang/vllm-disagg-logs/ which the runner owns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: stage vLLM disagg scripts and pass DISAGG_SCRIPTS_DIR run_xPyD_disagg.slurm bind-mounts DISAGG_SCRIPTS_DIR into the container and runs vllm_disagg.sh from it; unset, it falls back to SLURM_SUBMIT_DIR=/tmp on Spur, so the container got an empty mount and failed with "vllm_disagg.sh: No such file or directory" (rc=127). Copy the checked-out .buildkite/amd-disagg scripts into the compute-visible DISAGG_SCRIPTS_STAGE and pass DISAGG_SCRIPTS_DIR pointing at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: fix vLLM disagg model name and request one task per node - MODEL_NAME mini-max-fp8 -> MiniMax-M3-MXFP8 (the actual entry in the vLLM branch's .buildkite/amd-disagg/models.yaml; the old name failed with "model mini-max-fp8 not found in models.yaml"). - Add --nodes/--ntasks-per-node=1 to sbatch so both nodes get a Slurm task; the default NumTasks=1 left the second node without a task, so rank 1 never launched and produced no logs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: fix vLLM disagg Spur launch * CI: pass MiniMax MXFP8 quantization to vLLM * CI: fix vLLM disagg workflow syntax * Revert "CI: fix vLLM disagg workflow syntax" This reverts commit d0b1e51. * Revert "CI: pass MiniMax MXFP8 quantization to vLLM" This reverts commit bd922bd. * CI: adapt vLLM image for MiniMax MXFP8 * CI: align vLLM disagg MiniMax launch * CI: move vLLM disagg smoke to idle nodes * CI: wrap vLLM CLI for MiniMax MXFP8 * CI: default vLLM disagg smoke to DeepSeek-V3 Match the upstream/reference vLLM disagg CI (vllm-project/vllm#47030), which runs 1P1D TP8 on DeepSeek-V3 (standard fp8, no quant remap). The MiniMax-M3-MXFP8 default crashed at VllmConfig with "Missing 'quant_algo'" because the mxfp8->modelopt_mxfp8 remap forces a non-ModelOpt checkpoint down the ModelOpt path. MXFP8 enablement is deferred; run the reference model to exercise the disagg path itself. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: run vLLM disagg smoke via `vllm serve` directly (drop CI wrapper) Remove the aiter-side injection (aiter_vllm_quant_patch.py / sitecustomize.py / aiter_vllm_cli.py heredocs + the vllm_disagg.sh rewrite that swapped `vllm serve` for `python3 aiter_vllm_cli.py serve`). The wrapper called main() at module top level with no `if __name__ == "__main__":` guard, so vLLM's spawn-based engine core re-imported it and re-ran main() during bootstrap -> "An attempt has been made to start a new process before the current process has finished its bootstrapping phase". This broke every model that reaches engine spawn (DeepSeek-V3 job 2068); MiniMax only masked it by failing earlier at quant config. Match the reference disagg CI (vllm-project/vllm#47030), which runs the guarded `vllm serve` console entrypoint directly with no wrapper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: pin vLLM disagg smoke to idle nodes fabric-9,10 fabric-2,3 were occupied; retarget the default SPUR_NODELIST to a currently-idle pair so the smoke job starts without queueing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: pin vLLM disagg smoke to fabric-12,14 fabric-9,10 consistently hit Slurm NODE_FAIL/JobLaunchFailure (job never launches, 0s). Probe jobs confirmed fabric-12,14 launch cleanly (the `srun` 127 in the probe is expected: Spur compute nodes have no srun on PATH, and run_xPyD_disagg.slurm takes its no-srun Spur path via docker). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: pin vLLM disagg smoke back to proven nodes fabric-2,3 The real job requests `#SBATCH --gres=gpu:8 --exclusive`; fabric-9,10,12,14 NODE_FAIL/JobLaunchFailure at the 8-GPU exclusive prolog (node-level GPU health, not our code) while trivial probes without gpu:8 still launch there. fabric-2,3 are the only pair proven to launch the full disagg job (earlier jobs 2041/2068 loaded the model there); both idle now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: submit disagg job bare (no sbatch CLI resource flags) for Spur Spur NODE_FAILs (JobLaunchFailure, 0s) any job carrying sbatch CLI resource flags; our monitor then reads sacct State=COMPLETED and reports a false green. Every CI job failed this way regardless of node, while bare-submitted jobs from others ran fine. Match the reference submitter run-slurm-disagg-test.sh: pass NO CLI resource flags, rely on the #SBATCH directives in run_xPyD_disagg.slurm, and let Slurm pick free nodes. Drops the --nodelist/--nodes/--ntasks pinning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: surface real disagg result (gsm8k + exit code) in job summary The GitHub step summary only showed sbatch output + a raw sacct table, so the actual test outcome was invisible on the run page. Read the disagg PD log on shared NFS (${DISAGG_SCRIPTS_STAGE}/vllm-disagg-pd-<jobid>.log) and add to the summary: model, real State/ExitCode/Elapsed, the `[vllm_disagg] PASS/FAIL: exact_match=... >= threshold` verdict, and the gsm8k results table. Also flag the Spur NODE_FAIL case (ExitCode -1:0 / 0s elapsed) so a launch failure is not mistaken for a pass. Display only; the pass/fail gate is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: register aiter-di-ci-spur-login-vllm runner label for actionlint The vLLM disagg smoke job runs on the self-hosted runner label aiter-di-ci-spur-login-vllm, which was missing from .github/actionlint.yaml, so the Actionlint check failed with "label ... is unknown". Add it to the self-hosted-runner labels list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: re-run Actionlint when its own config (.github/actionlint.yaml) changes Actionlint only triggered on .github/workflows/** paths, so updates to the actionlint runner-label allowlist (.github/actionlint.yaml) never re-ran the check. Add .github/actionlint.yaml to the pull_request/push path filters. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: restrict vLLM disagg smoke to fabric-15,16 via in-script #SBATCH --nodelist Pin the disagg job to a fixed node pair within our 13-16 pool. Spur NODE_FAILs jobs carrying sbatch CLI resource flags, but honors in-script #SBATCH directives (proven by ATOM pd_submit.sh, which writes #SBATCH --nodelist into the submitted script + bare sbatch). So inject #SBATCH --nodelist=${SPUR_NODELIST} into the bare-submitted script instead of passing --nodelist on the CLI, and default SPUR_NODELIST to fabric-15,16. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: move vLLM disagg smoke to fabric-13,15 (disjoint from ATOM CI 14,16) The ATOM DI nightly pins DeepSeek to fabric-14,16; pinning vLLM to 15,16 collided on fabric-16 and queued behind it. Use fabric-13,15 so the two CIs run on disjoint node pairs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * CI: run vLLM disagg smoke on nightly schedule only (drop pull_request) Keep only the nightly `schedule` (30 18 * * *, 2:30 AM Beijing, offset from the ATOM DI nightly at 16:00 UTC) and `workflow_dispatch` for manual runs. Remove the pull_request trigger so the disagg smoke no longer runs per-PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: JiaoliangYu <jiaolyu@amd.com> Co-authored-by: JiaoliangYu <JiaoliangYu@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JiaoliangYu <Jiaoliang.Yu@amd.com>
|
@tjtanaa @functionstackx @chunfangamd The unnecessary patches are removed and code is cleaned up and added more tests that we wish to add. The only change that needs to be done is removal of moriio_toy_proxy.py which requires PR #47764 need to be merged. |
|
@dllehr-amd @khluu @chunfangamd @functionstackx @tjtanaa https://buildkite.com/vllm/amd-distributed-inference-ci/builds/17/canvas For vLLM Router we need this PR to be merged; but can go indepedently. |
|
@lcskrishna can you fix the commit sign offs? The DCO check is failing on that? Looks like 28 of the commits aren't valid? |
a42850c to
2681825
Compare
|
@dllehr-amd Fixed the DCO part. |
hi @lcskrishna is the plan to merge this first with just toy_proxy enabled and then after the vllm router fix happens to then enable full vllm nightly DI CI with vllm-router? |
| set -o pipefail # propagate docker run exit code through the tee pipe | ||
| echo "Rank $rank on $(hostname)" | ||
| docker rm -f "$DOCKER_CONT_NAME" 2>/dev/null || true | ||
| docker ps -q | xargs --no-run-if-empty docker stop; |
There was a problem hiding this comment.
This seems a little dangerous in a CI environment no? Can you clarify a bit why we are unilaterally stopping all containers?
There was a problem hiding this comment.
@dllehr-amd The DI CI is running independent of the current CI pipelines we have. In this case, we are stopping all the containers so to avoid any OOM issues while performing a model load while running.
hi @lcskrishna it seems like #45043 has merged now, do u wanna add it to this PR or follow up PR? i created this issue here to track wideEP #51056 |
hi @lcskrishna here is the follow up issue to track this #51057 |
@functionstackx please note the tests contain both vllm-router and toy proxy. Btw please note: vLLM Router PR is merged. |
gotcha thank you @lcskrishna so it seems like the only thing left that does PR is missing is wideEP decode? |
Yes wideEP will be seperately enabled and not under this PR at the moment. |
Enable the Buildkite CI steps that exercise vLLM disaggregated inference
P/D with the MoRI-IO KV connector on ROCm AMD devices. The suite brings up
a full prefill/decode topology, health-gates every server, and then runs
the GSM8k accuracy workload.
Two modes are supported:
- WIDE_EP_MODE=0: general P/D disaggregation with independent TP8 servers
- WIDE_EP_MODE=1: data parallelism + expert parallelism (DP8/EP8/TP1)
Files added under .buildkite/amd-disagg:
- vllm_disagg.sh launcher for the proxy, prefill and decode
servers; runs the accuracy / benchmark modes
- cluster.sh single-sourced cluster config (ports, topology,
RDMA/NIC and MoRI env) that varies per cluster
- run-slurm-disagg-test.sh foreground submitter invoked by the CI step
- run_xPyD_disagg.slurm SLURM job script, one container per node
- models.yaml per-model flags and parallelism settings
- pipeline-disagg.yaml Buildkite step matrix for the AMD DI tests
GSM8k results (exact_match):
- 1P1D TP8 0.947
- 1P1D DP8 + EP 0.953
- 2P2D DP16 + EP 0.932
Co-authored-by: Sheral Kumar <shekumar@amd.com>
Co-authored-by: avininjamay8 <Avinash.Paul@amd.com>
Co-authored-by: tej <37236721+itej89@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
b55ceb2 to
3e90fd2
Compare
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
| set -o pipefail # propagate docker run exit code through the tee pipe | ||
| echo "Rank $rank on $(hostname)" | ||
| local -a stale=() | ||
| mapfile -t stale < <(docker ps -aq --filter name=vllm_disagg_ --filter name=vllm_router_ 2>/dev/null || true) |
There was a problem hiding this comment.
Ok I like this .. I think this protects at least anyone else that is running workloads from getting their dockers killed, right @dllehr-amd ?
dllehr-amd
left a comment
There was a problem hiding this comment.
Thanks for addressing the comments @lcskrishna!!
|
/ci run |
|
✅ Triggered Buildkite CI #83185 for commit |
…47030) Signed-off-by: lcskrishna <lollachaitanya@gmail.com> Co-authored-by: Sheral Kumar <shekumar@amd.com> Co-authored-by: avininjamay8 <Avinash.Paul@amd.com> Co-authored-by: tej <37236721+itej89@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Purpose
This PR enables the Buildkite CI scirpts that exercises vLLM Disaggregated inference P/D with MoRI IO KV connector on ROCm AMD devices. This brings up a full PD topology health-gates on everyserver, runs the GSM8k accuracy.
This PR supports with two modes WIDE_EP_MODE=0 / 1 to support general PD disaggregation wtih TP8 indepdent servers (When WIDE_EP_MODE=0) and DP + Expert Parallelism (when WIDE_EP_MODE=1)
The pipeline is getting executed in the AMD Distributed Inference CI and the queue name to use is amd_mi350_ainic.
Files added:
The PD Disagg step are present in .buildkite/amd-disagg folder :
Test Plan
This PR currently supports two modes with 1P1D:
Test Result
GSM8k results for the above tests:
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.