Skip to content

[CI/Build][ROCm] Run Hunyuan reliability without expert parallel - #9428

Merged
andyluo7 merged 2 commits into
vllm-project:mainfrom
hacZ17:ci/rocm-hunyuan-reliability-ep-off
Oct 28, 2026
Merged

andyluo7 merged 2 commits into
vllm-project:mainfrom
hacZ17:ci/rocm-hunyuan-reliability-ep-off

Conversation

@hacZ17

@hacZ17 hacZ17 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • ROCm HunyuanImage process-kill coverage uses hunyuan_image3_dit_rocm_reliability.yaml, which keeps TP=4 and sets enable_expert_parallel: false.
  • CUDA reliability still uses hunyuan_image3_dit.yaml with expert parallel on.
  • A separate non-blocking weekly job runs one fault-free 512x512 request on that original expert-parallel deploy. The nightly pixel-accuracy job is unchanged.

Why this is not a duplicate

No open PR splits the HunyuanImage reliability deploy this way. This does not change the Qwen OOM recovery server set in #9425.

Contributor intent

  1. What outcome do you want to achieve? Stop the weekly HunyuanImage process-kill matrix from failing in the healthy warmup, while keeping the full-model expert-parallel failure visible.
  2. What is the highest-leverage variable for that outcome? Expert parallel on the TP=4 DiT deploy. The weekly cases died in warmup before any process was killed.
  3. How will this work move that variable? Point ROCm fault injection at an overlay with expert parallel off, and add one non-blocking weekly baseline that still serves one request with expert parallel on.
  4. Whose customer or user needs does this address, and what are those needs? AMD weekly CI owners need process-kill coverage to test fault handling, and they still need a lane that shows whether the expert-parallel runtime can finish a full-model request.

AI assistance disclosure

Cursor, using Grok 4.7, drafted the deploy overlay, the weekly baseline job, and this description. The human contributor directed this containment for the case where only expert-parallel-on fails. The four-case vLLM 0.31/0.32 comparison was not rerun on this machine. Local contract tests passed. The MI300 jobs have not been run at this head.

Test plan

  • .venv/bin/python -m pytest tests/buildkite/test_amd_pipeline.py::test_hunyuan_rocm_reliability_disables_expert_parallel tests/buildkite/test_amd_pipeline.py::test_weekly_reliability_is_distinct_from_nightly tests/buildkite/test_amd_suite_selection.py::test_debug_weekly_selects_four_unconditional_jobs -p no:shard --override-ini='addopts=' — 3 passed
  • Changed-file pre-commit hooks passed, including Buildkite schema, Ruff, mypy 3.10, and test marks
  • On the exact head, run the weekly Hunyuan reliability job and the expert-parallel baseline on MI300

Base: c84638f3208f0fffd872200adfc5860103a6244b
Head: f9efc69a0415547ce9396dd094ab8cf2804309ee


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-26T17:40:13.028960Z
We couldn’t complete this review. Comment @codex review to try again.
f9efc69 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@vllm-omni-review-bot

Copy link
Copy Markdown

This PR appears to belong to: docs/design/module/vllm_omni_config.md, docs/design/module/entrypoints.md.

Module owners: @alex-jw-brooks @lishunyang12 @NickCao

Routing: @alex-jw-brooks via module of the changed files, CODEOWNERS; @lishunyang12 via module of the changed files, CODEOWNERS; @NickCao via module of the changed files, CODEOWNERS

@hacZ17, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

@andyluo7
andyluo7 force-pushed the ci/rocm-hunyuan-reliability-ep-off branch from f9efc69 to 5fe8c90 Compare October 28, 2026 00:25
@andyluo7 andyluo7 added ready label to trigger buildkite CI weekly-test label to trigger buildkite weekly test CI amd-test Used to trigger AMD CI separately. and removed ready label to trigger buildkite CI labels Oct 28, 2026
@andyluo7
andyluo7 force-pushed the ci/rocm-hunyuan-reliability-ep-off branch from 5fe8c90 to 88e4c4b Compare October 28, 2026 00:47
@andyluo7 andyluo7 added weekly-test label to trigger buildkite weekly test CI amd-test Used to trigger AMD CI separately. ready label to trigger buildkite CI and removed weekly-test label to trigger buildkite weekly test CI amd-test Used to trigger AMD CI separately. ready label to trigger buildkite CI labels Oct 28, 2026
@andyluo7
andyluo7 force-pushed the ci/rocm-hunyuan-reliability-ep-off branch from 88e4c4b to b7f7994 Compare October 28, 2026 01:28
@andyluo7 andyluo7 added ready label to trigger buildkite CI weekly-test label to trigger buildkite weekly test CI and removed weekly-test label to trigger buildkite weekly test CI amd-test Used to trigger AMD CI separately. ready label to trigger buildkite CI labels Oct 28, 2026
@andyluo7 andyluo7 added the amd-test Used to trigger AMD CI separately. label Oct 28, 2026
haic0 and others added 2 commits October 27, 2026 21:10
The weekly process-kill matrix was dying in the healthy TP=4 expert-parallel
warmup before any fault was injected. Keep that path as one non-blocking
baseline, and point ROCm fault injection at an overlay with expert parallel off.

Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Co-authored-by: Grok 4.7 <noreply@x.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: lm10-arg <347490427+lm10-arg@users.noreply.github.com>
@andyluo7 andyluo7 removed weekly-test label to trigger buildkite weekly test CI amd-test Used to trigger AMD CI separately. labels Oct 28, 2026
@andyluo7
andyluo7 force-pushed the ci/rocm-hunyuan-reliability-ep-off branch from b7f7994 to a311af7 Compare October 28, 2026 02:16
@andyluo7 andyluo7 added ready label to trigger buildkite CI weekly-test label to trigger buildkite weekly test CI amd-test Used to trigger AMD CI separately. and removed ready label to trigger buildkite CI labels Oct 28, 2026
@andyluo7

Copy link
Copy Markdown
Collaborator

Exact-head qualification for a311af7 (rebased onto main 4c51b23):

  • CUDA READY #18345 passed with all 18 jobs terminal and no nonzero exit.
  • AMD #14888 targeted Hunyuan coverage passed:
    • Reliability job 01a1804c-bc2b-4294-acf4-94c464c81c56: 16 passed; the JUnit guard enforces zero skipped tests; four artifacts uploaded.
    • TP=4 expert-parallel startup job 01a1804c-bc32-4e2c-98ca-1c33b3fd7770: /health returned HTTP 200, 1 passed; the JUnit guard enforces zero skipped tests; four artifacts uploaded.
  • Changed-file pre-commit and Buildkite pipeline validation passed locally.
  • Full local pytest was not runnable on this Mac because Python 3.14 lacks torch; the matching-head CUDA and ROCm jobs above provide the runtime evidence.

The rebase preserved #9435's AMD pipeline additions and resolved the shared test constant without changing #9428's behavior. No actionable findings remain.

@andyluo7 andyluo7 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at exact head a311af7 after review, required CUDA READY qualification, and targeted AMD Hunyuan reliability plus expert-parallel startup qualification.

@andyluo7
andyluo7 merged commit 506eaae into vllm-project:main Oct 28, 2026
8 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd-test Used to trigger AMD CI separately. weekly-test label to trigger buildkite weekly test CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants