Skip to content

[ROCm][DistInf] Enable vLLM DI CI with buildkite/slurm - #47030

Merged
dllehr-amd merged 4 commits into
vllm-project:mainfrom
lcskrishna:csrikris-vllm-ci
Aug 10, 2026
Merged

dllehr-amd merged 4 commits into
vllm-project:mainfrom
lcskrishna:csrikris-vllm-ci

Conversation

@lcskrishna

@lcskrishna lcskrishna commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Purpose

This PR enables the Buildkite CI scirpts that exercises vLLM Disaggregated inference P/D with MoRI IO KV connector on ROCm AMD devices. This brings up a full PD topology health-gates on everyserver, runs the GSM8k accuracy.
This PR supports with two modes WIDE_EP_MODE=0 / 1 to support general PD disaggregation wtih TP8 indepdent servers (When WIDE_EP_MODE=0) and DP + Expert Parallelism (when WIDE_EP_MODE=1)

The pipeline is getting executed in the AMD Distributed Inference CI and the queue name to use is amd_mi350_ainic.

Files added:

The PD Disagg step are present in .buildkite/amd-disagg folder :

  • vllm_disagg.sh - Launcher script which launches - proxy, prefill, decode and runs the accuracy / bench modes.
  • cluster.sh - Single sourced configs like PORTS, Topology, RDMA/NIC & MoRI Env variable which can vary from cluster to cluster.
  • run-slurm-disagg-test.sh - This is foreground submitter (the single command that CI step runs).
  • pipeline.disagg.yaml - The buildkite step snippet for AMD DI test matrix.

Test Plan

This PR currently supports two modes with 1P1D:

  • ROCm PD Disagg 1P1D TP8
  • ROCm PD Disagg 1P1D DP + Expert Parallelim (DP8/EP8/TP1)

Test Result

GSM8k results for the above tests:

  • 1P1D TP8 → 0.947
  • 1P1D DP8 EP (198364) → 0.953
  • 2P2D DP16 EP (198381) → 0.932

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added ci/build rocm Related to AMD ROCm labels Jun 29, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Jun 29, 2026
@lcskrishna

Copy link
Copy Markdown
Contributor Author

@lcskrishna

Copy link
Copy Markdown
Contributor Author

This PR requires #46482 to be merged first.

Comment thread .buildkite/amd-disagg/moriio_toy_proxy_server.py Outdated
Comment thread .buildkite/amd-disagg/run-slurm-disagg-test.sh Outdated
Comment thread .buildkite/amd-disagg/apply_39276_rebased.py Outdated
Comment thread .buildkite/amd-disagg/apply_moriio_2pd_patches.sh Outdated
Comment thread .buildkite/amd-disagg/moriio_toy_proxy_server.py Outdated
Comment thread .buildkite/amd-disagg/run_xPyD_disagg.slurm Outdated
Comment thread .buildkite/amd-disagg/moriio_toy_proxy_server.py Outdated
Comment thread .buildkite/amd-disagg/moriio_toy_proxy_server.py Outdated
Comment thread .buildkite/amd-disagg/moriio_toy_proxy_server.py Outdated
@mergify

mergify Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lcskrishna.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jul 15, 2026
JiaoliangYu pushed a commit to gyohuangxin/aiter that referenced this pull request Jul 15, 2026
Match the upstream/reference vLLM disagg CI (vllm-project/vllm#47030),
which runs 1P1D TP8 on DeepSeek-V3 (standard fp8, no quant remap). The
MiniMax-M3-MXFP8 default crashed at VllmConfig with "Missing 'quant_algo'"
because the mxfp8->modelopt_mxfp8 remap forces a non-ModelOpt checkpoint
down the ModelOpt path. MXFP8 enablement is deferred; run the reference
model to exercise the disagg path itself.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
JiaoliangYu pushed a commit to gyohuangxin/aiter that referenced this pull request Jul 15, 2026
Remove the aiter-side injection (aiter_vllm_quant_patch.py / sitecustomize.py
/ aiter_vllm_cli.py heredocs + the vllm_disagg.sh rewrite that swapped
`vllm serve` for `python3 aiter_vllm_cli.py serve`). The wrapper called
main() at module top level with no `if __name__ == "__main__":` guard, so
vLLM's spawn-based engine core re-imported it and re-ran main() during
bootstrap -> "An attempt has been made to start a new process before the
current process has finished its bootstrapping phase". This broke every
model that reaches engine spawn (DeepSeek-V3 job 2068); MiniMax only
masked it by failing earlier at quant config.

Match the reference disagg CI (vllm-project/vllm#47030), which runs the
guarded `vllm serve` console entrypoint directly with no wrapper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gyohuangxin added a commit to ROCm/aiter that referenced this pull request Jul 16, 2026
* CI: add vLLM disagg Spur smoke workflow

* CI: fix vLLM disagg workflow actionlint

* CI: add actionlint runner labels

* CI: schedule vLLM DI smoke nightly

* change runner

* CI: fix vLLM disagg Spur smoke submission

* CI: move vLLM disagg smoke to idle nodes

* CI: run vLLM disagg smoke on fabric 13 14

* CI: pin vLLM disagg smoke to fabric-15,16

Move the 2-node vLLM disagg smoke off fabric-13,14 (used by the ATOM smoke)
to fabric-15,16.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: point vLLM disagg log/scripts stage to a writable dir

The Slurm job runs as the xinhuang runner user and cannot write to
/data/csrikris/logs/ (another user's dir) -> "Permission denied" on the
disagg log, so the job failed on launch. Point DISAGG_SCRIPTS_STAGE at
/data/xinhuang/vllm-disagg-logs/ which the runner owns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: stage vLLM disagg scripts and pass DISAGG_SCRIPTS_DIR

run_xPyD_disagg.slurm bind-mounts DISAGG_SCRIPTS_DIR into the container and runs
vllm_disagg.sh from it; unset, it falls back to SLURM_SUBMIT_DIR=/tmp on Spur, so
the container got an empty mount and failed with
"vllm_disagg.sh: No such file or directory" (rc=127). Copy the checked-out
.buildkite/amd-disagg scripts into the compute-visible DISAGG_SCRIPTS_STAGE and
pass DISAGG_SCRIPTS_DIR pointing at it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: fix vLLM disagg model name and request one task per node

- MODEL_NAME mini-max-fp8 -> MiniMax-M3-MXFP8 (the actual entry in the vLLM
  branch's .buildkite/amd-disagg/models.yaml; the old name failed with
  "model mini-max-fp8 not found in models.yaml").
- Add --nodes/--ntasks-per-node=1 to sbatch so both nodes get a Slurm task;
  the default NumTasks=1 left the second node without a task, so rank 1 never
  launched and produced no logs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: fix vLLM disagg Spur launch

* CI: pass MiniMax MXFP8 quantization to vLLM

* CI: fix vLLM disagg workflow syntax

* Revert "CI: fix vLLM disagg workflow syntax"

This reverts commit d0b1e51.

* Revert "CI: pass MiniMax MXFP8 quantization to vLLM"

This reverts commit bd922bd.

* CI: adapt vLLM image for MiniMax MXFP8

* CI: align vLLM disagg MiniMax launch

* CI: move vLLM disagg smoke to idle nodes

* CI: wrap vLLM CLI for MiniMax MXFP8

* CI: default vLLM disagg smoke to DeepSeek-V3

Match the upstream/reference vLLM disagg CI (vllm-project/vllm#47030),
which runs 1P1D TP8 on DeepSeek-V3 (standard fp8, no quant remap). The
MiniMax-M3-MXFP8 default crashed at VllmConfig with "Missing 'quant_algo'"
because the mxfp8->modelopt_mxfp8 remap forces a non-ModelOpt checkpoint
down the ModelOpt path. MXFP8 enablement is deferred; run the reference
model to exercise the disagg path itself.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: run vLLM disagg smoke via `vllm serve` directly (drop CI wrapper)

Remove the aiter-side injection (aiter_vllm_quant_patch.py / sitecustomize.py
/ aiter_vllm_cli.py heredocs + the vllm_disagg.sh rewrite that swapped
`vllm serve` for `python3 aiter_vllm_cli.py serve`). The wrapper called
main() at module top level with no `if __name__ == "__main__":` guard, so
vLLM's spawn-based engine core re-imported it and re-ran main() during
bootstrap -> "An attempt has been made to start a new process before the
current process has finished its bootstrapping phase". This broke every
model that reaches engine spawn (DeepSeek-V3 job 2068); MiniMax only
masked it by failing earlier at quant config.

Match the reference disagg CI (vllm-project/vllm#47030), which runs the
guarded `vllm serve` console entrypoint directly with no wrapper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: pin vLLM disagg smoke to idle nodes fabric-9,10

fabric-2,3 were occupied; retarget the default SPUR_NODELIST to a
currently-idle pair so the smoke job starts without queueing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: pin vLLM disagg smoke to fabric-12,14

fabric-9,10 consistently hit Slurm NODE_FAIL/JobLaunchFailure (job never
launches, 0s). Probe jobs confirmed fabric-12,14 launch cleanly (the
`srun` 127 in the probe is expected: Spur compute nodes have no srun on
PATH, and run_xPyD_disagg.slurm takes its no-srun Spur path via docker).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: pin vLLM disagg smoke back to proven nodes fabric-2,3

The real job requests `#SBATCH --gres=gpu:8 --exclusive`; fabric-9,10,12,14
NODE_FAIL/JobLaunchFailure at the 8-GPU exclusive prolog (node-level GPU
health, not our code) while trivial probes without gpu:8 still launch there.
fabric-2,3 are the only pair proven to launch the full disagg job (earlier
jobs 2041/2068 loaded the model there); both idle now.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: submit disagg job bare (no sbatch CLI resource flags) for Spur

Spur NODE_FAILs (JobLaunchFailure, 0s) any job carrying sbatch CLI
resource flags; our monitor then reads sacct State=COMPLETED and reports
a false green. Every CI job failed this way regardless of node, while
bare-submitted jobs from others ran fine. Match the reference submitter
run-slurm-disagg-test.sh: pass NO CLI resource flags, rely on the
#SBATCH directives in run_xPyD_disagg.slurm, and let Slurm pick free
nodes. Drops the --nodelist/--nodes/--ntasks pinning.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: surface real disagg result (gsm8k + exit code) in job summary

The GitHub step summary only showed sbatch output + a raw sacct table, so
the actual test outcome was invisible on the run page. Read the disagg PD
log on shared NFS (${DISAGG_SCRIPTS_STAGE}/vllm-disagg-pd-<jobid>.log) and
add to the summary: model, real State/ExitCode/Elapsed, the
`[vllm_disagg] PASS/FAIL: exact_match=... >= threshold` verdict, and the
gsm8k results table. Also flag the Spur NODE_FAIL case (ExitCode -1:0 /
0s elapsed) so a launch failure is not mistaken for a pass. Display only;
the pass/fail gate is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: register aiter-di-ci-spur-login-vllm runner label for actionlint

The vLLM disagg smoke job runs on the self-hosted runner label
aiter-di-ci-spur-login-vllm, which was missing from .github/actionlint.yaml,
so the Actionlint check failed with "label ... is unknown". Add it to the
self-hosted-runner labels list.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: re-run Actionlint when its own config (.github/actionlint.yaml) changes

Actionlint only triggered on .github/workflows/** paths, so updates to the
actionlint runner-label allowlist (.github/actionlint.yaml) never re-ran the
check. Add .github/actionlint.yaml to the pull_request/push path filters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: restrict vLLM disagg smoke to fabric-15,16 via in-script #SBATCH --nodelist

Pin the disagg job to a fixed node pair within our 13-16 pool. Spur
NODE_FAILs jobs carrying sbatch CLI resource flags, but honors in-script
#SBATCH directives (proven by ATOM pd_submit.sh, which writes
#SBATCH --nodelist into the submitted script + bare sbatch). So inject
#SBATCH --nodelist=${SPUR_NODELIST} into the bare-submitted script instead
of passing --nodelist on the CLI, and default SPUR_NODELIST to
fabric-15,16.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: move vLLM disagg smoke to fabric-13,15 (disjoint from ATOM CI 14,16)

The ATOM DI nightly pins DeepSeek to fabric-14,16; pinning vLLM to 15,16
collided on fabric-16 and queued behind it. Use fabric-13,15 so the two
CIs run on disjoint node pairs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* CI: run vLLM disagg smoke on nightly schedule only (drop pull_request)

Keep only the nightly `schedule` (30 18 * * *, 2:30 AM Beijing, offset
from the ATOM DI nightly at 16:00 UTC) and `workflow_dispatch` for manual
runs. Remove the pull_request trigger so the disagg smoke no longer runs
per-PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>
Co-authored-by: JiaoliangYu <JiaoliangYu@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: JiaoliangYu <Jiaoliang.Yu@amd.com>
@mergify mergify Bot removed the needs-rebase label Jul 16, 2026
@lcskrishna

Copy link
Copy Markdown
Contributor Author

@tjtanaa @functionstackx @chunfangamd The unnecessary patches are removed and code is cleaned up and added more tests that we wish to add. The only change that needs to be done is removal of moriio_toy_proxy.py which requires PR #47764 need to be merged.

@lcskrishna

Copy link
Copy Markdown
Contributor Author

@dllehr-amd @khluu @chunfangamd @functionstackx @tjtanaa
New pipeline & queue has been created and all the initial tests pass successfully on vLLM nightly.

https://buildkite.com/vllm/amd-distributed-inference-ci/builds/17/canvas

For vLLM Router we need this PR to be merged; but can go indepedently.
vllm-project/router#192

@dllehr-amd

Copy link
Copy Markdown
Collaborator

@lcskrishna can you fix the commit sign offs? The DCO check is failing on that? Looks like 28 of the commits aren't valid?

@lcskrishna

Copy link
Copy Markdown
Contributor Author

@dllehr-amd Fixed the DCO part.

@functionstackx

Copy link
Copy Markdown
Contributor

For vLLM Router we need this PR to be merged; but can go indepedently. vllm-project/router#192

hi @lcskrishna is the plan to merge this first with just toy_proxy enabled and then after the vllm router fix happens to then enable full vllm nightly DI CI with vllm-router?

@dllehr-amd dllehr-amd added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 4, 2026
set -o pipefail # propagate docker run exit code through the tee pipe
echo "Rank $rank on $(hostname)"
docker rm -f "$DOCKER_CONT_NAME" 2>/dev/null || true
docker ps -q | xargs --no-run-if-empty docker stop;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a little dangerous in a CI environment no? Can you clarify a bit why we are unilaterally stopping all containers?

@lcskrishna lcskrishna Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dllehr-amd The DI CI is running independent of the current CI pipelines we have. In this case, we are stopping all the containers so to avoid any OOM issues while performing a model load while running.

@functionstackx

Copy link
Copy Markdown
Contributor

unnecessary patches are removed and code is cleaned up

hi @lcskrishna it seems like #45043 has merged now, do u wanna add it to this PR or follow up PR?

i created this issue here to track wideEP #51056

@functionstackx

Copy link
Copy Markdown
Contributor

For vLLM Router we need this PR to be merged; but can go indepedently. vllm-project/router#192

hi @lcskrishna is the plan to merge this first with just toy_proxy enabled and then after the vllm router fix happens to then enable full vllm nightly DI CI with vllm-router?

hi @lcskrishna here is the follow up issue to track this #51057

@lcskrishna

Copy link
Copy Markdown
Contributor Author

For vLLM Router we need this PR to be merged; but can go indepedently. vllm-project/router#192

hi @lcskrishna is the plan to merge this first with just toy_proxy enabled and then after the vllm router fix happens to then enable full vllm nightly DI CI with vllm-router?

@functionstackx please note the tests contain both vllm-router and toy proxy. Btw please note: vLLM Router PR is merged.
vllm-project/router#192

@functionstackx

Copy link
Copy Markdown
Contributor

For vLLM Router we need this PR to be merged; but can go indepedently. vllm-project/router#192

hi @lcskrishna is the plan to merge this first with just toy_proxy enabled and then after the vllm router fix happens to then enable full vllm nightly DI CI with vllm-router?

@functionstackx please note the tests contain both vllm-router and toy proxy. Btw please note: vLLM Router PR is merged. vllm-project/router#192

gotcha thank you @lcskrishna so it seems like the only thing left that does PR is missing is wideEP decode?

@lcskrishna

Copy link
Copy Markdown
Contributor Author

For vLLM Router we need this PR to be merged; but can go indepedently. vllm-project/router#192

hi @lcskrishna is the plan to merge this first with just toy_proxy enabled and then after the vllm router fix happens to then enable full vllm nightly DI CI with vllm-router?

@functionstackx please note the tests contain both vllm-router and toy proxy. Btw please note: vLLM Router PR is merged. vllm-project/router#192

gotcha thank you @lcskrishna so it seems like the only thing left that does PR is missing is wideEP decode?

Yes wideEP will be seperately enabled and not under this PR at the moment.

Enable the Buildkite CI steps that exercise vLLM disaggregated inference
P/D with the MoRI-IO KV connector on ROCm AMD devices. The suite brings up
a full prefill/decode topology, health-gates every server, and then runs
the GSM8k accuracy workload.

Two modes are supported:
  - WIDE_EP_MODE=0: general P/D disaggregation with independent TP8 servers
  - WIDE_EP_MODE=1: data parallelism + expert parallelism (DP8/EP8/TP1)

Files added under .buildkite/amd-disagg:
  - vllm_disagg.sh            launcher for the proxy, prefill and decode
                              servers; runs the accuracy / benchmark modes
  - cluster.sh                single-sourced cluster config (ports, topology,
                              RDMA/NIC and MoRI env) that varies per cluster
  - run-slurm-disagg-test.sh  foreground submitter invoked by the CI step
  - run_xPyD_disagg.slurm     SLURM job script, one container per node
  - models.yaml               per-model flags and parallelism settings
  - pipeline-disagg.yaml      Buildkite step matrix for the AMD DI tests

GSM8k results (exact_match):
  - 1P1D TP8         0.947
  - 1P1D DP8 + EP    0.953
  - 2P2D DP16 + EP   0.932

Co-authored-by: Sheral Kumar <shekumar@amd.com>
Co-authored-by: avininjamay8 <Avinash.Paul@amd.com>
Co-authored-by: tej <37236721+itej89@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
@lcskrishna
lcskrishna requested a review from dllehr-amd August 7, 2026 10:51
Comment thread .buildkite/test-amd.yaml Outdated
set -o pipefail # propagate docker run exit code through the tee pipe
echo "Rank $rank on $(hostname)"
local -a stale=()
mapfile -t stale < <(docker ps -aq --filter name=vllm_disagg_ --filter name=vllm_router_ 2>/dev/null || true)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I like this .. I think this protects at least anyone else that is running workloads from getting their dockers killed, right @dllehr-amd ?

@dllehr-amd dllehr-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the comments @lcskrishna!!

@dllehr-amd
dllehr-amd enabled auto-merge (squash) August 10, 2026 17:47
@dllehr-amd

Copy link
Copy Markdown
Collaborator

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #83185 for commit dc6887263ec6.

@dllehr-amd
dllehr-amd merged commit 1482d2e into vllm-project:main Aug 10, 2026
18 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Aug 10, 2026
zyp2014 pushed a commit to zyp2014/vllm that referenced this pull request Aug 21, 2026
…47030)

Signed-off-by: lcskrishna <lollachaitanya@gmail.com>
Co-authored-by: Sheral Kumar <shekumar@amd.com>
Co-authored-by: avininjamay8 <Avinash.Paul@amd.com>
Co-authored-by: tej <37236721+itej89@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants