Skip to content

[Bugfix][CI/Build][ROCm] Fix AuK FP8 and diffusion CI contracts - #8643

Merged
andyluo7 merged 11 commits into
vllm-project:mainfrom
haic0:ci/rocm-stabilize-diffusion-shards-20261008
Oct 8, 2026
Merged

andyluo7 merged 11 commits into
vllm-project:mainfrom
haic0:ci/rocm-stabilize-diffusion-shards-20261008

Conversation

@haic0

@haic0 haic0 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fix AMD CI contracts and guard AuK's NVIDIA-only FP8 execution on ROCm:

  • Move JoyImage GPU tests out of the CPU-marked module into an explicit CUDA+ROCm one-GPU suite using the serving inference contract (torch.inference_mode()). Keep TORCH_SDPA comparisons and platform-default ROCm FLASH_ATTN/AITER assertions.
  • Own the complete default-AITER JoyImage suite in one nonblocking AMD nightly job with isolated caches, JUnit, durations, and provisional 90-minute job / 80-minute command budgets. AMD READY/MERGE do not duplicate it; CUDA selection still includes it.
  • Compare all image outputs and valid text outputs across padded backends, retaining tolerance and adding padded-text perturbation invariance.
  • Run the MiniMax-H3 VDN solve-rule reference on ROCm GPU for HIP wheels, with an explicit no-GPU skip; non-ROCm remains CPU.
  • Separate MiniCPM-o NVIDIA graph-capture expectations from ROCm eager fallback while retaining shared-arena coverage.
  • Reject HIP in AuK's shared fp8_supported() predicate before querying NVIDIA capability. Production uses its existing model-dtype fallback on ROCm; the ten NVIDIA-only FP8 GPU cases skip there. CPU regressions retain Ada/Hopper acceptance and unsupported-device boundaries. FP8 arithmetic and NVIDIA numerical/graph coverage are unchanged.
  • Use five blocking AMD diffusion CPU shards in both READY and MERGE with 60-minute headroom, --durations=50, fail-closed pytest, and unique checkout-relative JUnit artifacts.

Root cause and current-main diagnosis

AMD build #13504 ran at current-main SHA 1e0c4f3d54a91c30de97aa332e772dfe7338f52c and had four deterministic failed jobs:

  • Model Executor shards 2/3 exposed three MiniCPM-o test-contract cases: test_startup_precapture_uses_shared_arena_and_freezes_admission, test_large_batch_and_frozen_wrapper_stay_eager, and test_failed_capture_blocks_even_eager_retry. On HIP, production correctly chose eager fallback before those CPU tests reached mocked NVIDIA-capture expectations.
  • Diffusion shard 3 failed test_delta_factors_apply_the_vdn_solve_rule because the AMD CPU wheel lacks LAPACK for float64 Cholesky. The test-only ROCm path now keeps all operands and references on GPU; production is unchanged.
  • Diffusion shard 4 routed JoyImage GPU work through the CPU shard. Dense AITER compilation took 1,446.6s, masked varlen compilation took 280.9s, test_transformer_shape_and_masked_forward failed under the non-serving gradient contract, and the 45-minute job timed out before pytest printed its failure summary.

Earlier #13484/#13499 runs exposed the same JoyImage inference-contract and cold-AITER budget issues. The JoyImage/MiniMax/MiniCPM fixes do not broaden skips, relax tolerance, restore CPU LAPACK, or change their production routing. The AuK follow-up separately rejects the unsupported HIP FP8 path.

AuK FP8 follow-up

The supplied AMD #13507 Model Test log reports ten AuK FP8 cases failing with HIPBLAS_STATUS_NOT_SUPPORTED; the following Mammoth SDPA command passed all 40 cases. Upstream #8442 (e623cef98) introduced the AuK code/tests and the rebase inherited them unchanged. ROCm exposes torch.cuda, so a CUDA device/capability check alone admitted an unsupported E4M3 torch._scaled_mm path.

The fix rejects torch.version.hip is not None before querying NVIDIA capability. Existing production fallback and test skips use the same predicate. No broad file exclusion, numerical-tolerance change, FP8 arithmetic change, or new CI job is introduced.

Local validation of this follow-up:

  • An isolated harness executes the exact predicate and seven exact CPU test cases with mocked torch: the pre-fix HIP case fails; after the guard, 7 pass. This is logic evidence, not full pytest or device qualification.
  • All 13 pre-existing functions and their decorators in the AuK test module are AST-identical; all ten FP8 GPU cases remain intact.
  • Changed-file pre-commit and manual mypy-3.12 pass (the repository mypy wrapper excludes production model trees); Python 3.10 syntax checks and git diff --check pass.
  • Full AuK pytest and AMD/NVIDIA hardware execution were not run locally: this shell lacks PyTorch, vLLM and the model test dependencies.

Reproduction and qualification commands from repository root, using a matching vLLM/vLLM-Omni environment and dev dependencies (including pytest-mock); no model download is needed:

# CPU regression suite, on CPU or ROCm
pytest -sv tests/diffusion/models/auk/test_auk_fp8_linear.py \
  -m 'core_model and cpu' --run-level core_model

# Existing GPU cases: 10 should skip on HIP; all 10 should execute on NVIDIA L4/H100
pytest -sv tests/diffusion/models/auk/test_auk_fp8_linear.py \
  -m 'core_model and cuda and cards_1' --run-level core_model

Exact revision

Base: 1e0c4f3d54a91c30de97aa332e772dfe7338f52c
Head: 4998aa272cbe82984eb11ae54d28d3a26d1d74ce

The first ten intentional commits were true-rebased onto base 1e0c4f3d at head 46318fb4. git range-diff matched all ten commits, every commit retains Signed-off-by:, AI attribution is preserved, and the aggregate binary patch is byte-identical before/after rebase (f3a40bd5d77a40785fac23234361561edde3ee85f345ef773315d76174d5aba9). No merge commit was introduced. One additional signed-off AuK guard/regression commit, 4998aa272, is a direct child of 46318fb4; those ten rebased commits remain untouched.

Pre-rebase AMD evidence

AMD build #13506 is exact-head evidence for pre-rebase head 38fafcd021b61b23a073fce2b9cdaac2b69acafd, not qualification for the new head:

  • Both READY and MERGE five-shard diffusion sets passed. Every shard ran nonzero tests, printed durations, generated pytest-{0..4}-of-5.xml, and uploaded JUnit.
  • MiniMax's solve-rule case passed in both sets.
  • All affected MiniCPM-o CPU cases passed in READY and MERGE; the dedicated ROCm eager-fallback case passed in R2-01 before that broader nonblocking job timed out elsewhere.
  • Dedicated nightly JoyImage ran all 10 default-AITER cases: 10 passed in 27m53s with slowest-test durations and uploaded artifacts/joy-image-gpu/pytest.xml.
  • One diffusion shard printed a leaked shared_memory warning after its successful pytest summary; artifact upload then succeeded. It is observational/noncausal follow-up evidence, not a test failure.

Rebased-head validation

  • .venv/bin/python -m pytest tests/buildkite/test_amd_pipeline.py tests/buildkite/test_amd_bootstrap.py tests/buildkite/test_amd_suite_selection.py tests/buildkite/test_rocm_dockerfile.py -v -> 64 passed.
  • Focused MiniMax/MiniCPM/JoyImage CPU regressions -> 7 passed. JoyImage GPU module -> 10 tests collected.
  • Five-shard collection -> unsharded 8,033; shard counts [1540, 1627, 1653, 1596, 1617]; all nonzero and pairwise disjoint, with union exactly equal to unsharded.
  • MiniJinja READY/MERGE/nightly renders parsed successfully: no JoyImage job in READY/MERGE, exactly one nightly JoyImage job, and one five-way CPU shard step in each blocking suite.
  • Changed-file pre-commit passed, including Ruff, mypy 3.10, test marks, SPDX, forbidden imports, torch.cuda, and Buildkite schema. Manual mypy-3.12, git diff --check, DCO/trailer audit, and diff-preservation checks passed.
  • Duplicate searches found no separate open/merged PR owning this combined MiniCPM/MiniMax/JoyImage AMD fix.

A broader local mixed CPU/GPU invocation is not claimed as passing: this host has mismatched vLLM 0.31.0 / vLLM-Omni 0.30.1 and reports UnspecifiedOmniPlatform, so platform-construction cases failed locally. The valid ROCm runtime evidence is #13506, and the rebased head still needs fresh AMD qualification.

Merge qualification still required

This PR is open and ready for review, but not merge-ready. Under the revised validation plan, require fresh final-head AMD READY and MERGE execution whose Buildkite commit equals 4998aa272cbe82984eb11ae54d28d3a26d1d74ce, including both five-shard sets, MiniMax solve-rule execution and affected MiniCPM cases. Require the AuK CPU regressions to pass on ROCm, the ten unsupported FP8 GPU cases to skip on HIP, and retained NVIDIA FP8 numerical/graph cases to execute successfully. Hosted checks, CUDA/NPU checks, and human approval also remain required.

Reuse #13506's ten-case JoyImage default-AITER pass with readable JUnit/durations as verified equivalent-rebase evidence: the original ten-commit series and its aggregate patch match across the rebase, its eight test/CI files remain identical, and the new base did not change JoyImage, diffusion attention, CI routing or dependency/image pins. This is not a claim that the final SHA itself ran the JoyImage GPU lane. The subsequent AuK-only commit does not modify JoyImage, shared attention, CI routing or dependency/image pins, so the documented evidence-reuse scope remains unchanged. A full fresh AMD nightly is not required solely for this byte-identical rebase. Rerun only targeted JoyImage coverage if relevant code/runtime dependencies or actual ROCm/AITER image provenance change, or a maintainer requests exact-head hardware evidence.

The nightly-test label was removed, ready was refreshed once with merge-test preserved, and previous-head READY/MERGE qualification was registered as AMD #13507 / CUDA #16916 / Intel #9789 / NPU #8635. Those builds cover the previous head, not the AuK follow-up. No healthy builds were canceled or queued GitHub Actions runs restarted.

The only production behavior change is rejecting AuK's unsupported HIP FP8 opt-in and using its existing model-dtype fallback. NVIDIA FP8 arithmetic, kernels and numerical/graph assertions are unchanged; no new accuracy or performance claims are made. Timeout budgets remain temporary headroom while cold-runtime evidence accumulates.

AI assistance: Cursor agents helped investigate CI, preserve/rebase the commit series, run validation, and draft this description. OpenAI Codex drafted the AuK HIP guard and CPU regressions, ran isolated logic/static checks, and prepared this description update. The human contributor remains responsible for reviewing every changed line and validating final behavior.

@haic0

haic0 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Exact-head self-review for 46318fb4aa372d66a4dccd891ad022eabafbaf3e on base 1e0c4f3d54a91c30de97aa332e772dfe7338f52c:

  • True-rebased all ten intentional commits: 0 behind / 10 ahead, no merge commit. The 10/10 range-diff matches, every commit remains signed, AI attribution is preserved, and the aggregate patch is byte-identical (f3a40bd5d77a40785fac23234361561edde3ee85f345ef773315d76174d5aba9).
  • Reviewed all eight changed files and rendered semantics. Production code is untouched. JoyImage GPU ownership is nightly on AMD and remains selected by CUDA; READY/MERGE keep identical five-shard CPU contracts. MiniMax uses co-located ROCm tensors, and MiniCPM-o separates NVIDIA capture from ROCm eager fallback.
  • Rebased-head checks passed: 64 AMD contract/bootstrap/selection/schema tests; 7 focused MiniMax/MiniCPM/JoyImage CPU regressions; 10 JoyImage GPU tests collected; five-shard collection [1540, 1627, 1653, 1596, 1617] is nonzero/disjoint and unions to all 8,033 selected tests; all three MiniJinja renders; changed-file pre-commit; manual mypy 3.12; diff/DCO audit.
  • The broader mixed local invocation is not claimed as passing because this host reports a mismatched vLLM/vLLM-Omni install and UnspecifiedOmniPlatform. Fresh exact-head AMD is therefore still required.
  • AMD #13506 at pre-rebase head 38fafcd021b61b23a073fce2b9cdaac2b69acafd passed both five-shard sets, MiniMax, all affected MiniCPM cases, and the 10-case nightly JoyImage job with JUnit/durations. Its one post-summary shared-memory warning was noncausal. It is pre-rebase evidence only.

No additional author-fixable finding is known. Fresh AMD READY/MERGE/nightly for 46318fb4aa372d66a4dccd891ad022eabafbaf3e, hosted/CUDA/NPU checks, and human approval remain blocking.

@vllm-omni-review-bot

Copy link
Copy Markdown

This PR appears to belong to: docs/design/module/diffusion/index.md.

Module owners: @david6666666 @Isotr0py @princepride

Routing: @david6666666 via module named in the PR description, model owner; @Isotr0py via module named in the PR description; @princepride via module named in the PR description

@haic0, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

@andyluo7
andyluo7 force-pushed the ci/rocm-stabilize-diffusion-shards-20261008 branch from 75b9f3a to 2c08263 Compare October 8, 2026 17:11
@andyluo7 andyluo7 changed the title [CI/Build][ROCm] Stabilize AMD diffusion CPU shards [CI/Build][ROCm] Fix JoyImage inference and MiniCPM platform contracts Oct 8, 2026
@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Updated-head implementation review for 7554bd4d8df40bb6bfaab861427bb2098f55c0d3:

  • Separate JoyImage and MiniCPM-o commits extend the existing MiniMax/sharding repair; all seven changed files are tests or CI.
  • JoyImage GPU functions no longer inherit CPU marks. Serving inference mode is local to the GPU tests, explicit SDPA checks remain explicit, and default-backend masked/unmasked checks require FLASH_ATTN on ROCm. Identical blocking AMD READY/MERGE jobs and existing CUDA model selection retain coverage.
  • MiniCPM-o CPU capture mocks explicitly simulate non-HIP eligibility. Only NVIDIA exact-shape capture expectations are marked nvidia_only; ROCm-compatible shared-arena tests and production HIP/NPU guards are retained. ROCm fallback tests assert no NVIDIA capture and correct eager continuation outputs/cache.
  • The MiniMax ROCm GPU workaround preserves float64 reference math; it avoids, but does not restore, CPU LAPACK.
  • JUnit paths now resolve under the Buildkite checkout rather than the native image's /app working directory.
  • Changed-file pre-commit, 11 generic pipeline tests, exact READY/MERGE rendering, and diff checks passed. A broader local diagnostic run had 165 passes and three missing-installation/PyTorch environment failures. Model-runtime qualification is not claimed from this Mac.

Requesting exact-head READY/MERGE CI. Human approval and terminal inspectable AMD/CUDA results remain required; no merge is requested before those gates pass.

@andyluo7 andyluo7 added ready label to trigger buildkite CI merge-test label to trigger buildkite merge test CI labels Oct 8, 2026
@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

@vllm-omni-review-bot Please review the updated test/CI contracts at 7554bd4, especially JoyImage default-backend coverage, MiniCPM-o NVIDIA versus ROCm expectations, and JUnit checkout paths. Human approval and exact-head CI qualification remain required.

@vllm-omni-review-bot

Copy link
Copy Markdown
Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Exact-head CI checkpoint for 7554bd4d8df40bb6bfaab861427bb2098f55c0d3:

  • AMD #13492 rendered both READY/MERGE: two JoyImage blocking GPU jobs, two sets of five diffusion CPU shards, and both MiniCPM CPU shard selections. Image build passed; the changed one-GPU target jobs are reserved and have no runtime verdict yet.
  • CUDA #16901, Intel #9775, and NPU #8619 reference the same head.
  • Docs and DCO passed. Required pre-commit/CUDA and hardware qualification remain pending. Human review approval is still absent; automated review requested.
  • One unrelated AMD job failed: Diffusion Sequence Parallelism shard 1/2. Two riverclouds/qwen_image_random correctness cases hit Hugging Face HTTP 429. That test file is unchanged from the PR base. Retry metadata permits a retry; no retry has been made and the rest of the build remains intact.

A signed-in Buildkite operator should retry only that HTTP-429 job after rate limiting clears. Please retain healthy jobs and changed-target evidence; do not restart the full build or treat aggregate red as a JoyImage/MiniCPM regression.

@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Follow-up at current head 7776e11: CUDA #16901 exposed a relative-import collection failure in the new JoyImage GPU file. Fixed with an absolute tests.diffusion.models.joy_image helper import and an import-resolution regression. Twelve focused pipeline/import tests and changed-file pre-commit pass. The old 7554bd4 builds are diagnostic history, not current-head qualification. No production behavior or tolerances changed. @vllm-omni-review-bot Please review the current head; human approval and exact-head CI remain required.

@haic0

haic0 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Review evidence updated for rebased head 46318fb4aa372d66a4dccd891ad022eabafbaf3e on base 1e0c4f3d54a91c30de97aa332e772dfe7338f52c:

  • Current-main AMD #13504 at 1e0c4f3d54a91c30de97aa332e772dfe7338f52c had four deterministic failed jobs: three MiniCPM-o graph test-contract cases across Model Executor shards 2/3; MiniMax float64 CPU Cholesky without LAPACK in Diffusion shard 3; and JoyImage GPU/AITER work misrouted through Diffusion shard 4, where dense/varlen cold compilation plus the non-serving gradient contract produced a failure and 45-minute timeout.
  • Pre-rebase exact-head AMD #13506 at 38fafcd021b61b23a073fce2b9cdaac2b69acafd passed READY and MERGE five-shard sets with nonzero execution, durations, unique JUnit and successful artifact uploads; MiniMax passed twice; all affected MiniCPM-o cases passed; and nightly JoyImage passed all 10 default-AITER cases in 27m53s with JUnit/durations.
  • One #13506 shard emitted a leaked-shared_memory warning only after its successful pytest summary; JUnit upload completed afterward. Treat it as observational/noncausal cleanup evidence.
  • The true rebase preserved all ten commits and the byte-identical functional patch. Local post-rebase contract/render/shard/pre-commit/mypy/diff/DCO checks pass.

#13506 is explicitly pre-rebase evidence. It does not qualify 46318fb4aa372d66a4dccd891ad022eabafbaf3e; fresh exact-head AMD READY, MERGE, and nightly jobs plus hosted/CUDA/NPU checks and human approval remain required. No labels, triggers, reviewer requests, or merge actions were changed in this update.

@andyluo7 andyluo7 added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 8, 2026
@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Current-head checkpoint: preserved the author-added MiniMax ROCm no-GPU guard and fast-forwarded to d99864a. GPU-backed AMD CI still runs the float64 solve-rule assertion. Refreshed READY once with MERGE already present to register qualification at this head; older runs are diagnostic history. Please hold the branch steady while exact-head CI completes. Human approval remains required.

@haic0
haic0 force-pushed the ci/rocm-stabilize-diffusion-shards-20261008 branch from d99864a to 504bebc Compare October 8, 2026 17:33
@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

@yenuo26 @congw729 @NickCao Could you review the rebased current head 504bebca82d91c39ff9d04e0e9da10898bd9e713 on base 2f5db3999c4134afcf45697f8d9817676c3aeb15 when available?

All seven intentional commits remain in this one PR; the diff is limited to seven test/CI files, with no production-code changes. Please focus on:

  • JoyImage GPU test ownership and serving inference-mode contract, including explicit SDPA checks and masked/unmasked platform-default attention parity. Both AMD READY/MERGE lanes are blocking, and CUDA model coverage remains selected.
  • MiniMax's co-located float64 ROCm GPU solve/reference math and explicit HIP no-GPU skip. This avoids the CPU LAPACK gap; it does not restore CPU LAPACK. Native AMD CI must execute the solve-rule test on a visible GPU rather than accept a skip.
  • MiniCPM-o's NVIDIA-only exact-shape capture expectations versus ROCm eager output/cache fallback, while retaining ROCm-compatible shared-arena coverage and existing production guards.
  • Identical AMD READY/MERGE five-shard CPU coverage, checkout-relative JUnit artifacts, temporary timeout headroom, and preserved markers/exclusions/fail-closed pytest behavior.

Current-head pre-commit, docs, and DCO have passed. Exact-head Buildkite qualification is being requested now; pre-rebase builds are diagnostic history, not qualification for this head. The author's focused-test results are recorded in the existing self-review comment. Human approval and inspectable AMD READY/MERGE plus CUDA hardware results remain required before merge.

Please keep the branch steady while qualification runs unless a concrete failure or review finding requires a change. This is a review request, not a request to merge before those gates pass.

@andyluo7 andyluo7 added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 8, 2026
@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Published two follow-up commits on current head 8a66574f5d42444657f33531cad87d177e6592d5, preserving the author's rebased seven-commit history:

  • 922f5b74727b69a362e1a57a1af833e237e2913f: JoyImage default-backend parity now compares all image tokens and valid text tokens. FlashAttention removes padded self-attention queries whereas the SDPA reference computes them; padded query outputs are not part of JoyImage's consumed output contract. Existing tolerance and backend assertions are unchanged. Perturbing padded text must leave all image/valid-text outputs unchanged, so the narrowed comparison cannot hide a missing key mask.
  • 8a66574f5d42444657f33531cad87d177e6592d5: AMD READY's broad diffusion-model command excludes the JoyImage GPU file, preserving the two dedicated blocking READY/MERGE lanes and existing CUDA selection. The new routing regression failed before the exclusion and passed afterward. Timeout, resources, blocking grade, existing exclusions, and marker expression in the broad lane are unchanged.

Validation: all 13 focused pipeline/import tests passed using the dependency-light --noconftest invocation; all changed-file pre-commit hooks passed, including Ruff, mypy, hardware markers and Buildkite schema; exact READY/MERGE template rendering and git diff --check passed. This Mac has no PyTorch, so these are static/routing checks, not GPU qualification.

Additional qualification blocker observed after the earlier triage: both dedicated JoyImage jobs on pre-fix AMD #13499 ended with exit 124. The dense AITER module built successfully in 1370.8s / 1416.9s; seven test cases then passed, but the masked varlen module started with too little time left in the 25-minute command budget. These two commits do not alter that dedicated-lane budget. A bounded timeout/prebuilt-kernel strategy still needs agreement before AMD can be claimed qualified; padding parity was not reached in those AMD jobs.

No production code, tolerance, LAPACK workaround, MiniCPM contract, existing job cancellation, or PR merge was changed. GPU CI and human approval remain pending.

@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Published one additional commit to existing #8643 (no new PR): 38fafcd021b61b23a073fce2b9cdaac2b69acafd, [CI/Build][ROCm] Move JoyImage default-AITER coverage to nightly.

  • Removed the dedicated JoyImage GPU jobs from AMD READY and MERGE; retained READY's broad-model exclusion.
  • Added exactly one explicit nonblocking AMD nightly JoyImage lane on mi300_1, retaining the full GPU suite, default AITER/inference-mode contract, isolated build caches, and JUnit artifacts. The job/command budgets are provisionally 90m/80m; full masked-kernel cold compilation has not yet been measured.
  • JoyImage CPU units, CUDA coverage, GPU assertions/tolerances, MiniMax/LAPACK, and MiniCPM changes are unchanged.
  • Routing regressions now enforce READY/MERGE absence and unique nonblocking nightly ownership, including backend, marker, cache, budget and artifact contracts. The new checks failed against the old routing before the move; all 15 focused pipeline tests pass afterward.
  • All changed-file pre-commit hooks and diff checks pass. MiniJinja rendering confirms no dedicated JoyImage GPU jobs in READY/MERGE and exactly one nightly job with soft_fail: true; the blocking diffusion CPU shards remain unchanged.

Qualification is still pending: obtain one passing AMD JoyImage run on this exact final head before merging #8643. CUDA success does not qualify ROCm/AITER, and previous-head timeout jobs did not reach the masked/default-backend parity cases. This moves cold compilation off the routine AMD READY/MERGE critical path; it does not establish that compilation is faster or that the provisional timeout is sufficient. Revisit blocking AMD coverage once exact-module prebuilds or measured cold runtimes make its budget reliable.

The push was a normal fast-forward, and GitHub/fork branch both confirm the new SHA. No CI labels were cycled, builds canceled, or full nightly runs manually started. Existing ready/merge-test labels do not select the new nightly lane; a targeted qualification run still needs to be arranged.

@andyluo7 andyluo7 added the nightly-test label to trigger buildkite nightly test CI label Oct 8, 2026
@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Triggered final-head CI for 38fafcd021b61b23a073fce2b9cdaac2b69acafd by adding nightly-test, preserving ready and merge-test. No ready pulse was needed, no builds were canceled, and the already queued GitHub Actions runs were not restarted.

Exact SHA independently verified from each Buildkite build JSON:

AMD bootstrap logs confirm selection of all three suites. Live job records show exactly one JoyImage GPU job, 01a11cd0-c747-4612-a90b-5ef7f9f37725, and ten diffusion CPU shards (five in READY and five in MERGE). Uploaded JoyImage configuration retains the default-AITER command, isolated caches, JUnit artifacts, 90m job / 80m command budgets, and soft_fail: true. It is waiting on the build dependency; this is registration, not a passing runtime result.

Final-head docs and DCO have passed. GitHub pre-commit and Python 3.11/3.12 wheels are queued. AMD/CUDA/NPU validation is pending. Human review is still required; existing reviewers are NickCao, yenuo26, and congw729, with no submitted reviews or unresolved inline threads at this check.

Before merge, inspect the exact-head changed-target results, including all ten AMD JoyImage cases and JUnit, rather than relying on the nonblocking nightly aggregate. Reuse successful results and retry only actionable transient failures. No code/base/head changes, review approval, or merge were performed for this trigger.

haic0 and others added 4 commits October 8, 2026 15:36
Co-authored-by: GPT-5.6 Sol <noreply@cursor.com>
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
…lback

Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
andyluo7 and others added 6 commits October 8, 2026 15:36
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: GPT-5.6 Sol <noreply@cursor.com>
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
FlashAttention removes padded self-attention queries whereas SDPA computes them. Compare all image tokens and only valid text tokens without changing tolerance or default-backend assertions. Perturb padded text to verify it cannot influence valid outputs.

Signed-off-by: andyluo7 <andy.luo@amd.com>
Exclude JoyImage from the broad AMD READY model command so its cold AITER compilation is owned by the blocking READY/MERGE JoyImage jobs. Add a regression test preserving the broad selector, timeout, exclusions and blocking grade, and assert CUDA still includes JoyImage.

Signed-off-by: andyluo7 <andy.luo@amd.com>
Remove duplicate dedicated JoyImage GPU lanes from AMD READY and MERGE while retaining the READY broad-model exclusion. Run the complete default-AITER inference suite once in nonblocking nightly coverage with isolated build caches, JUnit artifacts, and provisional 90m job / 80m command budgets.

Add regressions for READY/MERGE absence and unique nightly ownership, preserving CUDA selection, backend assertions and existing CPU coverage. Exact-head AMD runtime qualification remains required before merging this attention-test change.

Signed-off-by: andyluo7 <andy.luo@amd.com>
@haic0
haic0 force-pushed the ci/rocm-stabilize-diffusion-shards-20261008 branch from 38fafcd to 46318fb Compare October 8, 2026 19:39
@andyluo7 andyluo7 added ready label to trigger buildkite CI and removed nightly-test label to trigger buildkite nightly test CI ready label to trigger buildkite CI labels Oct 8, 2026
@andyluo7

andyluo7 commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Proceeding with the revised qualification plan at 46318fb4aa372d66a4dccd891ad022eabafbaf3e on base 1e0c4f3d54a91c30de97aa332e772dfe7338f52c:

  • Removed nightly-test, preserving merge-test, and refreshed ready once. Fresh builds independently confirm this head: AMD #13507, CUDA #16916, Intel #9789, and NPU #8635. No full nightly was requested, healthy builds canceled, or queued hosted checks restarted.
  • Reuse AMD #13506's JoyImage lane: all 10 default-AITER cases passed in 27m53s with JUnit uploaded. Independently rechecked 10/10 unchanged range-diff, byte-identical aggregate patch, identical eight changed files, and no new-base changes to JoyImage/attention/CI or dependency/image pins. This is equivalent-rebase evidence, not a claim that the new SHA itself ran this GPU lane.
  • Fresh AMD READY/MERGE and CUDA/hosted checks still need to complete. The PR description now states this revised gate, rather than requiring an unnecessary full-nightly repetition solely for the equivalent rebase. If actual ROCm/AITER runtime provenance changes or relevant code changes, reevaluate targeted JoyImage qualification.

@NickCao @yenuo26 @congw729 Please review the current test/CI patch and the documented evidence reuse when available. Human approval remains required. Please keep the branch steady during qualification unless a concrete failure or review finding requires a change. This is not a request to merge before the remaining gates pass.

ROCm exposes GPUs through the CUDA namespace, so the Ada/Hopper capability check could admit HIP devices and fail in torch._scaled_mm with HIPBLAS_STATUS_NOT_SUPPORTED. Reject HIP before checking NVIDIA capability so the existing AuK pipeline keeps its model-dtype fallback and GPU tests skip unsupported execution.

Add CPU regressions for HIP rejection without a capability query, validated Ada/Hopper acceptance, unsupported architecture boundaries, and unavailable CUDA. Preserve all existing FP8 numerical and graph tests.

AI-assisted-by: OpenAI Codex
Signed-off-by: andyluo7 <andy.luo@amd.com>
@andyluo7 andyluo7 changed the title [CI/Build][ROCm] Fix JoyImage inference and MiniCPM platform contracts [Bugfix][CI/Build][ROCm] Fix AuK FP8 and diffusion CI contracts Oct 8, 2026
@andyluo7 andyluo7 added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 8, 2026

@andyluo7 andyluo7 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andyluo7
andyluo7 merged commit f9b97d2 into vllm-project:main Oct 8, 2026
9 checks passed
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Oct 14, 2026
…-project#8643)

Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: GPT-5.6 Sol <noreply@cursor.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-test label to trigger buildkite merge test CI ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants