Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
0a857ad
fix(e2e): re-point HunyuanImage3 offline test to text_to_image.py
chethanuk Sep 14, 2026
92715f3
[CI][ROCm] Add non-blocking engine and model executor GPU coverage (#…
andyluo7 Sep 15, 2026
4a8ac29
[Frontend] Add MiniMax H3 video upscale workflow (WF-07) (#7472)
xuexueligao Sep 15, 2026
a3dee6f
[Lingbot World] Bound image condition storage and extend temporal RoP…
wtz2333 Sep 15, 2026
952022c
[Diffusion][Perf] Add Qwen-Image QK RoPE Triton path (#5931)
dongbo910220 Sep 15, 2026
7893475
[Tests] Test image edit input limits in tiny model framework (#7434)
NickCao Sep 15, 2026
c17126c
[Model] Add Gepard-1.0 OpenAI /v1/audio/speech serving (#7499)
mjZhaoElaine Sep 15, 2026
6040bb9
[Perf][Cosmos3] Add SeaCache support for Cosmos3 (#6922)
yzhautouskay Sep 15, 2026
bbf77cc
[CI][ROCm] Add non-blocking entrypoint GPU coverage (#7398)
andyluo7 Sep 15, 2026
65693b3
[Bugfix][Core] Keep sample-rate snapshots bounded in delta output (#7…
LOGO127 Sep 15, 2026
f420cfc
[NPU][CosyVoice3] Add CPU STFT fallback (#7520)
peterDengcx Sep 16, 2026
5527f3f
[CI][Bugfix] Update Qwen3-TTS CI Validation Message (#7276)
FayeSpica Sep 16, 2026
45a3e77
[Bugfix] Route Music3 float32 attention to SDPA (#7354)
liuyao0322 Sep 16, 2026
a1a234a
[Voxtral] Raise BadRequest for ref_audio (#7012)
clodaghwalsh17 Sep 16, 2026
c197cd2
[Bugfix] Fix Qwen3-Omni streaming crops in mixed-length batches (#7340)
dshah1333 Sep 16, 2026
2185c4e
[Bugfix] Tolerate a bounded TTS tail in the audio-text similarity gat…
tlysanhuo Sep 16, 2026
3eb5c3b
[Bugfix][Wan2.2] Apply UniPC flow shift once (#7576)
HAAZZZEEEE Sep 16, 2026
3319f6c
[CI/Build][Hardware][Ascend] Add torchmetrics to the NPU CI image (#7…
GodHu777777 Sep 16, 2026
7bceb1a
[Bugfix] Align video frame consumption reporting with prompt sampling…
psv666 Sep 16, 2026
5dbae2b
[Bugfix] Per-key accumulation strategy for Qwen3-TTS codec frame outp…
linyueqian Sep 16, 2026
275720b
[Bugfix] Keep async video jobs queued until inference and abort on de…
ZhengWG Sep 16, 2026
a2138b8
[Feature][Qwen3-TTS] Make Code2Wav honor the stage dtype (#6059)
gxxx-hum Sep 16, 2026
21d86ec
[Diffusion] MAGI-2: regional compile coverage (#7174)
cr-gao Sep 16, 2026
5e90241
[Bugfix] Skip generic text-only warmup for DreamZero (#7548)
tzhouam Sep 16, 2026
9005d78
[LingBot World] Reuse the text encode for a prompt already encoded (#…
linzhenpl07 Sep 16, 2026
99ff4f3
[Core][Frontend] Unified Full-duplex Framework (#7413)
chickeyton Sep 16, 2026
507cb1d
[Attention] [P1] Keep H3 VSA policy in the model and share sparse pri…
lishunyang12 Sep 16, 2026
5f7f62e
[Bugfix][MiniCPM-o] Stabilize code2wav CFM CUDA graph caching with re…
y-null Sep 16, 2026
e78a5d0
[Frontend] Add MiniMax-H3 references and Ref2VA workflow (#7483)
LinzeShi Sep 16, 2026
d554e6b
[feat] Extend disaggregated MiniMax-H3 stage with VAE encoders (#6939)
asukaqaq-s Sep 16, 2026
afa837f
[Frontend] Add MiniMax H3 ComfyUI text-to-video workflow (WF-01) (#7423)
zhuhu00 Sep 16, 2026
249e84d
[Boogu-Image] Fuse QKV/FFN projections + switch to diffusion RMSNorm …
Bounty-hunter Sep 16, 2026
217a6af
[Diffusion] Fuse Qwen-Image-Edit select01 modulation for CUDA (#5921)
dongbo910220 Sep 16, 2026
e6cebf8
[Bugfix][Omni-DuplexEval] Load local Hugging Face dataset layouts in …
RyanYun09 Sep 16, 2026
fa639b8
[Bugfix] Fail a duplex session that outgrows max_model_len instead of…
twu3202 Sep 16, 2026
538eddd
[Perf][CosyVoice3] Bounded-window incremental HiFT vocoder streaming …
timzsu Sep 16, 2026
1053f7b
[Bugfix] Preserve benchmark metric sample counts in results (#7624)
psv666 Sep 16, 2026
81b5be2
[Bugfix][Qwen-Image] Restore RotaryEmbedding CUDA RoPE for Diffusers …
NumberWan Sep 16, 2026
3d75254
[CI][MiniCPM-o] Lower the Daily-Omni accuracy gate to 0.77 (#7657)
y-null Sep 16, 2026
2a68471
[Bugfix] Reject failed diffusion benchmark warmups (#7092)
yuweih205 Sep 16, 2026
d4dba0d
[Misc] Remove dead deploy keys from personaplex.yaml (#7454)
THUqliu Sep 16, 2026
4392af5
[Bugfix][Qwen3-Omni] Include deferred residual in captured Thinker st…
dshah1333 Sep 16, 2026
38c7eee
[BugFix][NPU] Fix Fish Speech S2 Pro NPU support (#7546)
KrystalRay Sep 16, 2026
c97b5fc
[Bugfix][Frontend] Route synthetic aborts through final stage (#7006)
CarrotSwordsman Sep 16, 2026
04c55d0
Preserve audio boundary embeddings in Qwen2.5-Omni interleaved input …
nodeeeeee Sep 16, 2026
a8434a6
[BugFix][TTS] Fix MOSS Realtime serving and generation (#5661)
akshatvishu Sep 16, 2026
5aadcbb
[Bugfix] Fix Breeze-TTS-2 repeated silence and audio EOS handling
linyueqian Sep 16, 2026
63032c9
[Frontend] Add ComfyUI Generate Music node with initial MiniMax Music…
FayeSpica Sep 16, 2026
c32aaeb
[Bugfix] Fix Qwen3-TTS word timestamps with async chunking (#7544)
Bezdarnost Sep 16, 2026
c8b3a17
[Frontend] Allow explicit turn-based serving (#7675)
LinzeShi Sep 17, 2026
d81777f
[Doc] Scope Spark FP8 guidance to the validated revision (#7536)
bojiang-li Sep 17, 2026
72cc012
[CI][bugfix] Fix MiniMax H3 FP8 quality test remote-code loading (#7617)
david6666666 Sep 17, 2026
4fab34c
[Bugfix] Update ComfyUI serving-mode mock after helper rename (#7694)
LinzeShi Sep 17, 2026
becfefd
[Hardware][Ascend] Add Wan2.2 MXFP4 UOS, Smooth and W4A8 fallback (#7…
HAAZZZEEEE Sep 17, 2026
f1a6e7c
[Core] Retain structured configs through runtime startup (#6849)
maithilijoshi20 Sep 17, 2026
a769d93
[Bugfix][NPU] Fix talker_mtp ACL graph capture missing force_uniform_…
Wallbreazzz Sep 17, 2026
4200768
[Frontend] Scale Omni serving across API processes (#6923)
Sy0307 Sep 17, 2026
2241525
[Model][PersonaPlex] Add paced multi-session validation (#7428)
LOGO127 Sep 17, 2026
8c3d340
[CI] Fix Structured Structured Diffusion Config Wan Test (#7707)
alex-jw-brooks Sep 17, 2026
cb439f3
[Diffusion][Model] Add direct FastH3 8-Step V2 checkpoint loading (#7…
david6666666 Sep 17, 2026
825ab20
[Model][PersonaPlex] add per-row RingKV offsets and active rows (#7670)
MrDongsls Sep 17, 2026
e3be42e
[Core] Give the OpenAI Realtime wire codec a reusable home (RFC #6592…
psv666 Sep 17, 2026
f5e4f5f
[Bugfix][MiniCPM-o] Size the duplex HD slice reservation from the fra…
twu3202 Sep 17, 2026
bf74544
[Voxtral] Follow up to raise BadRequest for ref_audio (#7649)
clodaghwalsh17 Sep 17, 2026
873e9bf
[Doc][NPU] Refresh MiniMax-H3 Ascend recipes for 0.28-era configurati…
brandneway Sep 17, 2026
d0755b5
[Model] Add π0.5 (Pi0.5) VLA model support (#6950)
chenchaoxu7575 Sep 17, 2026
5b22f55
fix(minicpmo45): deadline-align native duplex silence continuation (#…
Tiagosf00 Sep 17, 2026
c61b302
[Qwen3-Omni] Remove redundant text embedding in DeepStack path (#7573)
howard-shan Sep 17, 2026
eb1abba
[Feature][TTS] MOSS-TTS: place the reference-audio encoder on the cod…
gcanlin Sep 17, 2026
882bf8b
[Frontend] Add a shared realtime web UI for MiniCPM-o and Qwen3-Omni …
amy-why-3459 Sep 17, 2026
2ab5d17
[4/N] Unify worker-side MP4 encoding and wire Wan into it (#7048)
specture724 Sep 17, 2026
97912de
[Kernel] Fuse LongCat paired Q/K RoPE (#7500)
dongbo910220 Sep 17, 2026
9cdc57d
[Bugfix][MAGI-2] Fix layerwise offload OOM by pointing offload block …
BeatSeat Sep 17, 2026
a6570b6
[Diffusion] Fuse ERNIE-Image Q/K RoPE for eager execution (#7502)
dongbo910220 Sep 17, 2026
b81f2c5
[Misc] treewide: remove shm_threshold_bytes from deploy config (#7522)
NickCao Sep 17, 2026
77d8de7
[XPU][HunyuanImage3] Fix HunyuanImage-3.0 text-to-image on XPU (#7674)
Joshna-Medisetty Sep 17, 2026
6bda549
[Bugfix] Fix Parallel State Initialization For MoE + Diffusion (#7676)
alex-jw-brooks Sep 17, 2026
f8197e1
[5/N] Schedule chunk transfers as their own stage (#7406)
specture724 Sep 18, 2026
242c3cd
[Hardware][Ascend] Integrate MindIE dense and sparse quantized attent…
De-cs Sep 18, 2026
1c7476e
[CI/Build][ROCm] Stabilize shared AMD test lanes (#7706)
andyluo7 Sep 18, 2026
9fbc36a
[AR-Diffusion] Bound KV metadata for long rollouts (#7498)
wtz2333 Sep 18, 2026
5d3e6dc
[Bugfix][MiniMax-H3] Use Hopper-safe modulation precision (#7693)
david6666666 Sep 18, 2026
823af92
[Bugfix][Ascend NPU][Diffusion] Load over-wide unquantized fallback w…
brandneway Sep 18, 2026
eb20759
[Test][Ascend] Share HunyuanVideo-1.5 E2E across GPU and NPU (#7547)
zengchuang-hw Sep 18, 2026
aa2fada
[Perf][Diffusion] Activate NPU/GPU matmul for LoRA delta computation …
holykie Sep 18, 2026
6df62d5
[CI/Build][MiniCPM-o] Fix the Nightly tests (#7758)
chickeyton Sep 18, 2026
3084b5d
[BugFix][TTS] Restore MOSS v1 codec streaming (#6420)
akshatvishu Sep 18, 2026
4b0c187
[Bugfix][Core] Fix CLI parallel flags losing to nested deploy paralle…
zwhzzz0821 Sep 18, 2026
0a5ece4
[CI/Build] Retarget skipped HiDream and MammothModa2 E2E tests to cur…
yenuo26 Sep 18, 2026
d4ffde1
[CI][Qwen3-Omni] Compare structured-config sampling defaults against …
linyueqian Sep 18, 2026
0570f7a
[CI][Qwen3-Omni] Keep the deploy-YAML literal in the structured-confi…
linyueqian Sep 18, 2026
fbb4e63
[Bugfix][Core] Apply Realtime item truncation once per command (#7800)
NolenLiang Sep 18, 2026
ca3ccfc
[XPU][Bugfix] Make AR-Diffusion KV preallocation device-agnostic (#7777)
Joshna-Medisetty Sep 18, 2026
7b4fc7c
[Model] Migrate MammothModa2 DiT to the shared diffusion runtime (#7134)
Levius-Fubuki Sep 18, 2026
4c7a98c
[XPU][CI] Drop the USTC PyPI mirror from the XPU image build (#7808)
Joshna-Medisetty Sep 18, 2026
232dbc8
[CI/Build] Restrict ERNIE fused RoPE tests to NVIDIA CUDA (#7771)
andyluo7 Sep 19, 2026
ea84055
[Refactor][Diffusion] Resolve distributed layerwise topology in the p…
specture724 Sep 19, 2026
470ec60
[Core] Refactor omni prefix cache into Manager/Controller (#6654)
ZhengWG Sep 19, 2026
f90c267
[Bugfix] Use monotonic clocks for test server startup timeouts (#7628)
psv666 Sep 19, 2026
5875356
Merge branch 'main' into fix/issue-7495-bug-nightly-ci-failed-testse2…
Gaohan123 Sep 19, 2026
698f716
[Refactor] Reject unknown diffusion config fields (#5172)
TaffyOfficial Sep 19, 2026
3571018
[Bugfix] Preserve HunyuanImage3 DiT-only offline accuracy contract
Gaohan123 Sep 19, 2026
8c0d9dd
Merge remote-tracking branch 'upstream/main' into fix/pr-7525-review
Gaohan123 Sep 19, 2026
a21b939
[Bugfix] Drop obsolete mode from shared text-to-image config
Gaohan123 Sep 19, 2026
415005b
[Model] Add circlestone-labs/Anima (#4083)
akshatvishu Sep 19, 2026
f9f53f2
[Bugfix]Include Reference Audio Cache Salt for HiggsAudio v2 (#7826)
sphinxkkkbc Sep 19, 2026
23d8c83
[Bugfix][Qwen-Image] Keep AutoRound W4A16 on CPU when offloading; Fix…
NumberWan Sep 19, 2026
491025e
Merge branch 'main' into fix/issue-7495-bug-nightly-ci-failed-testse2…
Gaohan123 Sep 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
16 changes: 14 additions & 2 deletions .buildkite/amd/test-amd-merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,34 @@ steps:
steps:
- label: "Simple · Model Executor Test · Shard %N/%t"
agent_pool: mi300_1
parallelism: 2
parallelism: 3
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: Blocking
timeout_in_minutes: 90
commands:
- export VLLM_ROCM_USE_AITER=0
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
# that is not supported on CPU
- "pytest -sv tests/model_executor -m 'core_model and cpu' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
- "pytest -sv tests/model_executor -m 'core_model and cpu and not omni' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"

- label: "Simple · Omni Processor Test"
agent_pool: mi300_1
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: Blocking
timeout_in_minutes: 60
commands:
- export VLLM_ROCM_USE_AITER=0
- "pytest -sv tests/model_executor/models/test_omni_processing.py -m 'core_model and cpu and omni'"

- label: "Simple · Diffusion Test · Shard %N/%t"
agent_pool: mi300_1
parallelism: 4
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: Blocking
timeout_in_minutes: 45
commands:
- export VLLM_ROCM_USE_AITER=0
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
Expand Down
146 changes: 142 additions & 4 deletions .buildkite/amd/test-amd-ready.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,26 +9,39 @@ steps:
steps:
- label: "Simple · Model Executor Test · Shard %N/%t"
agent_pool: mi300_1
parallelism: 2
parallelism: 3
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: Blocking
timeout_in_minutes: 90
commands:
- export VLLM_ROCM_USE_AITER=0
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
# that is not supported on CPU
- "pytest -sv tests/model_executor -m 'core_model and cpu' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
- "pytest -sv tests/model_executor -m 'core_model and cpu and not omni' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"

- label: "Simple · Diffusion Test"
- label: "Simple · Omni Processor Test"
agent_pool: mi300_1
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: Blocking
timeout_in_minutes: 60
commands:
- export VLLM_ROCM_USE_AITER=0
- "pytest -sv tests/model_executor/models/test_omni_processing.py -m 'core_model and cpu and omni'"

- label: "Simple · Diffusion Test · Shard %N/%t"
agent_pool: mi300_1
parallelism: 4
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: Blocking
timeout_in_minutes: 45
commands:
- export VLLM_ROCM_USE_AITER=0
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
# that is not supported on CPU
- "pytest -sv tests/diffusion -m 'core_model and cpu' --ignore=tests/diffusion/cache/test_teacache_extractors.py"
- "pytest -sv tests/diffusion -m 'core_model and cpu' --ignore=tests/diffusion/cache/test_teacache_extractors.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"

- label: "Simple · Engine&Entrypoints Test"
agent_pool: mi300_1
Expand Down Expand Up @@ -69,6 +82,27 @@ steps:

- group: ":card_index_dividers: Diffusion Test"
steps:
# Keep real FlashAttention execution out of the CPU-marked Simple lane.
# AITER may need to build several kernels on a cold worker, so run one
# pytest worker with job-local build directories and a bounded timeout.
- label: "ROCm · FlashAttention/AITER GPU Coverage"
agent_pool: mi300_1
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: NonBlocking
timeout_in_minutes: 60
commands:
- export VLLM_ROCM_USE_AITER=1
- export AITER_JIT_DIR="/tmp/vllm-omni-aiter-$$BUILDKITE_JOB_ID"
- export TORCH_EXTENSIONS_DIR="/tmp/vllm-omni-torch-extensions-$$BUILDKITE_JOB_ID"
- mkdir -p "$$AITER_JIT_DIR" "$$TORCH_EXTENSIONS_DIR"
- >-
timeout --signal=TERM --kill-after=2m 55m
pytest -s -v -n 1
tests/diffusion/attention/
-m 'core_model and rocm and MI325 and cards_1'
--run-level "core_model"

- label: "Diffusion · Batch Test"
agent_pool: mi300_1
depends_on: amd-build
Expand Down Expand Up @@ -160,6 +194,39 @@ steps:
# - export VLLM_WORKER_MULTIPROC_METHOD=spawn
# - timeout 20m pytest -s -v tests/e2e/offline_inference/test_stable_audio_expansion.py -m "advanced_model and diffusion and L4" --run-level advanced_model

- label: "ROCm · Engine & Model Executor GPU Coverage (R2-01)"
agent_pool: mi300_1
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: NonBlocking
timeout_in_minutes: 20
artifact_paths:
- "artifacts/rocm-r2-01/**/*"
commands:
# Native AMD tests run from /app/vllm-omni, while Buildkite uploads
# artifacts relative to its checkout directory.
- export R2_01_ARTIFACT_DIR="$$BUILDKITE_BUILD_CHECKOUT_PATH/artifacts/rocm-r2-01"
- mkdir -p "$$R2_01_ARTIFACT_DIR"
# The CUDA ready job also covers tests/worker/test_batched_omni_output.py.
# Keep the initial ROCm burn-in scoped to the requested directories.
# Verbose pytest output and JUnit retain node IDs without a second
# collection-only pass over the same GPU-backed test tree.
# The CUDA/L4-only CosyVoice3 ragged-flow parity test cold-JITs multiple
# AITER kernels on ROCm and is covered by the CUDA ready lane.
- >-
pytest -s -v -ra
tests/engine/ tests/model_executor/
-m 'core_model and cuda and not
(cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)'
-k 'not test_code2wav_streaming_batch_matches_ragged_flow_numerics'
--run-level "core_model"
--durations=0
--junitxml="$$R2_01_ARTIFACT_DIR/pytest.xml"
2>&1 | tee "$$R2_01_ARTIFACT_DIR/pytest.log"
- >-
tail -n 100 "$$R2_01_ARTIFACT_DIR/pytest.log" >
"$$R2_01_ARTIFACT_DIR/pytest-summary.txt"

- label: "Engine Test"
agent_pool: mi300_1
depends_on: amd-build
Expand All @@ -172,6 +239,71 @@ steps:
pytest -s -v tests/engine/test_async_omni_engine_abort.py
'

- label: "ROCm · Entrypoints GPU Coverage (R2-02)"
agent_pool: mi300_1
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: NonBlocking
timeout_in_minutes: 30
env:
VLLM_WORKER_MULTIPROC_METHOD: spawn
artifact_paths:
- "artifacts/rocm-r2-02/**/*"
commands:
# Native AMD tests run from /app/vllm-omni, while Buildkite uploads
# artifacts relative to its checkout directory.
- export R2_02_ARTIFACT_DIR="$$BUILDKITE_BUILD_CHECKOUT_PATH/artifacts/rocm-r2-02"
- mkdir -p "$$R2_02_ARTIFACT_DIR"
- >-
ps -eo pid=,ppid=,lstart=,stat=,comm= >
"$$R2_02_ARTIFACT_DIR/processes-before.txt"
# Verbose pytest output and JUnit retain selected node IDs without a
# second collection-only pass over the same GPU-backed test tree.
# The inner timeout leaves five minutes for teardown verification and
# artifact generation before the Buildkite step timeout.
- >-
timeout --signal=TERM --kill-after=1m 25m
pytest -s -v -ra
tests/entrypoints/
-m 'core_model and cuda and not
(cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)'
--run-level "core_model"
--durations=0
--junitxml="$$R2_02_ARTIFACT_DIR/pytest.xml"
2>&1 | tee "$$R2_02_ARTIFACT_DIR/pytest.log"
- >-
tail -n 100 "$$R2_02_ARTIFACT_DIR/pytest.log" >
"$$R2_02_ARTIFACT_DIR/pytest-summary.txt"
- |
sleep 5
ps -eo pid=,ppid=,lstart=,stat=,comm= > "$$R2_02_ARTIFACT_DIR/processes-after.txt"
# Compare PID plus process start time so PID reuse cannot hide a leak.
# Double dollars preserve awk field references through Buildkite interpolation.
awk '
NR == FNR {
if ($$9 != "ps") {
baseline[$$1 "|" $$3 "|" $$4 "|" $$5 "|" $$6 "|" $$7] = 1
}
next
}
$$9 != "ps" {
identity = $$1 "|" $$3 "|" $$4 "|" $$5 "|" $$6 "|" $$7
if (!(identity in baseline)) {
print
leaked = 1
}
}
END {
if (leaked) {
print "status=FAIL leaked_processes_detected=1"
exit 1
}
print "status=PASS leaked_processes=0"
}
' "$$R2_02_ARTIFACT_DIR/processes-before.txt" \
"$$R2_02_ARTIFACT_DIR/processes-after.txt" \
2>&1 | tee "$$R2_02_ARTIFACT_DIR/process-cleanup.txt"

# - label: "Omni Model Test Qwen3-Omni"
# agent_pool: mi300_2
# depends_on: amd-build
Expand Down Expand Up @@ -201,6 +333,12 @@ steps:
depends_on: amd-build
mirror_hardwares: [amdproduction]
grade: Blocking
# A ROCm hardware hang aborts the interpreter with 134. Let Buildkite
# create one fresh pod instead of retrying on the poisoned GPU process.
retry:
automatic:
- exit_status: 134
limit: 1
commands:
- pytest -s -v tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py -m "slow" --run-level "core_model"

Expand Down
16 changes: 16 additions & 0 deletions .buildkite/amd/test-template-amd-omni.j2
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,14 @@ steps:
{% for path in step.artifact_paths %}
- "{{ path }}"
{% endfor %}
{% endif %}
{% if step.retry %}
retry:
automatic:
{% for retry_rule in step.retry.automatic %}
- exit_status: {{ retry_rule.exit_status }}
limit: {{ retry_rule.limit }}
{% endfor %}
{% endif %}
{% if step.timeout_in_minutes %}
timeout_in_minutes: {{ step.timeout_in_minutes }}
Expand Down Expand Up @@ -224,6 +232,14 @@ steps:
{% for path in step.artifact_paths %}
- "{{ path }}"
{% endfor %}
{% endif %}
{% if step.retry %}
retry:
automatic:
{% for retry_rule in step.retry.automatic %}
- exit_status: {{ retry_rule.exit_status }}
limit: {{ retry_rule.limit }}
{% endfor %}
{% endif %}
{% if step.timeout_in_minutes %}
timeout_in_minutes: {{ step.timeout_in_minutes }}
Expand Down
13 changes: 12 additions & 1 deletion .buildkite/common/ci_source_file_dependencies.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,14 @@ x-minicpmo-4-5: &minicpmo_4_5
x-minicpmo-4-5-duplex: &minicpmo_4_5_duplex
- vllm_omni/deploy/minicpmo_4_5_duplex.yaml
- vllm_omni/engine/duplex/
- vllm_omni/engine/duplex_omni_engine.py
- vllm_omni/engine/duplex_orchestrator.py
- vllm_omni/engine/omni_engine_base.py
- vllm_omni/entrypoints/duplex/
- vllm_omni/entrypoints/duplex_omni.py
- vllm_omni/entrypoints/async_omni_base.py
- vllm_omni/protocol/
- vllm_omni/clients/
- vllm_omni/entrypoints/duplex_request_client.py
- vllm_omni/outputs/duplex.py
- vllm_omni/model_executor/duplex_sampling.py
- vllm_omni/experimental/fullduplex/
Expand Down Expand Up @@ -95,6 +100,7 @@ x-hunyuan-image3: &hunyuan_image3

x-minimax-h3: &minimax_h3
- vllm_omni/diffusion/models/minimax_h3/
- vllm_omni/diffusion/layers/indexed_modulation.py

x-hunyuan-video: &hunyuan_video
- vllm_omni/diffusion/models/hunyuan_video/
Expand Down Expand Up @@ -249,8 +255,13 @@ source_file_dependencies:
- tests/assets/minicpmo_4_5/
- tests/clients/
- tests/engine/duplex/
- tests/engine/test_duplex_omni_engine.py
- tests/engine/test_duplex_orchestrator.py
- tests/entrypoints/duplex/
- tests/entrypoints/test_duplex_omni.py
- tests/entrypoints/openai_api/test_duplex_api_server.py
- tests/model_executor/models/minicpmo_4_5/duplex/
- tests/protocol/

omni_minicpmo_4_5_duplex_perf:
- *minicpmo_4_5
Expand Down
4 changes: 3 additions & 1 deletion .buildkite/cuda/test-merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,9 @@ steps:
# 0.26 startup regression is understood and fixed.
timeout_in_minutes: 90
commands:
- pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and H100" --run-level "advanced_model"
# Turn-based server VAD over /v1/realtime is deferred to RFC #7181 PR 5
# (see the P0 thread on #7413); the test stays in the tree and runs again there.
- pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and H100" --run-level "advanced_model" --deselect tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py::TestQwen3OmniRealtimeWebSocket::test_server_vad_multi_turn_without_client_commit
mirror_hardwares: h100_2

- label: "Entrypoint Test with L4"
Expand Down
11 changes: 11 additions & 0 deletions .buildkite/cuda/test-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,17 @@ steps:
-m "full_model and diffusion and H100 and B200 and cards_1"
--run-level "full_model"

- label: ":robot_face: Robot Policy OpenPI · π0.5 · H100 · Single-GPU"
timeout_in_minutes: 120
commands:
- pip install --no-deps "openpi-client @ git+https://github.com/Physical-Intelligence/openpi.git@215abfb217dbac7d5f1273282331b9b1866c0479#subdirectory=packages/openpi-client"
- >-
pytest -sv
tests/e2e/online_serving/test_pi05_expansion.py
-m "full_model and diffusion and H100 and cards_1"
--run-level "full_model"
mirror_hardwares: h100_1

- label: ":full_moon: Diffusion X2I(&A&T) · Function Test with H100 · 2-GPU"
source_file_dependencies: diffusion_qwen_image_function
timeout_in_minutes: 120
Expand Down
6 changes: 6 additions & 0 deletions .buildkite/cuda/test-ready.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,12 @@ steps:
depends_on: upload-ready-pipeline
steps:

- label: "Diffusion · FlashAttention Test"
timeout_in_minutes: 20
commands:
- timeout 15m pytest -s -v tests/diffusion/attention/ -m 'core_model and cuda and L4 and cards_1' --run-level "core_model"
mirror_hardwares: l4_1

- label: "Diffusion · Offloader Test"
commands:
- timeout 40m pytest -s -v tests/diffusion/offloader -m "core_model and cuda" --run-level "core_model"
Expand Down
3 changes: 2 additions & 1 deletion .buildkite/cuda/test-weekly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,13 +191,14 @@ steps:
# from downgrading transformers; see
# examples/offline_inference/text_to_speech/README.md.
- label: "TTS · Gepard-1.0 · L4"
timeout_in_minutes: 60
timeout_in_minutes: 90
commands:
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
- printf '%s\n' "transformers>=5.5.3" "huggingface-hub>=1.0" > /tmp/gepard-overrides.txt
- uv pip install --python "$(python3 -c 'import sys; print(sys.executable)')" "nemo_toolkit[tts]==2.7.3" --overrides /tmp/gepard-overrides.txt
- >-
pytest -s -v tests/e2e/offline_inference/test_gepard_tts.py
tests/e2e/online_serving/test_gepard_tts.py
-m "tts and cuda"
--run-level "full_model"
mirror_hardwares: l4_1
Expand Down
31 changes: 31 additions & 0 deletions .buildkite/npu/test-npu-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,37 @@ steps:
build.env("NIGHTLY") == "1" ||
build.pull_request.labels includes "nightly-test"
steps:
- label: ":full_moon: Diffusion X2V · HunyuanVideo-1.5 Function Test"
source_file_dependencies: diffusion_hunyuan_video_function
timeout_in_minutes: 180
mirror_hardwares: a2b3_npu_4
env:
VLLM_WORKER_MULTIPROC_METHOD: spawn
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
HF_TOKEN: "${HF_TOKEN}"
HF_HUB_OFFLINE: "1"
commands:
- pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m "npu and A2" --run-level "full_model"
- label: ":full_moon: Diffusion X2V · HunyuanVideo-1.5 Perf Test"
source_file_dependencies: diffusion_hunyuan_video_perf
key: nightly-diffusion-x2v-performance-hunyuanvideo15
timeout_in_minutes: 180
mirror_hardwares: a3_npu_2
env:
VLLM_WORKER_MULTIPROC_METHOD: spawn
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
HF_TOKEN: "${HF_TOKEN}"
HF_HUB_OFFLINE: "1"
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
commands:
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
- |
set +e
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "npu and A3" --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json
EXIT=$$?
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
exit $$EXIT
- label: ":full_moon: Diffusion X2V · Wan 2.2 Function Test"
source_file_dependencies: diffusion_wan22_function
mirror_hardwares: a2b3_npu_4
Expand Down
Loading