[lm10-arg]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Fix the CI regressions introduced by the vLLM 0.33 upgrade and restore an approval-ready exact-head READY result.

2. What is the highest-leverage variable for that outcome?

Deterministic exact-head AMD/ROCm READY CI.

3. How will this work move that variable?

It fixes four established compatibility regressions: repository-local tools imports being shadowed by vLLM; normal fused-vs-separate GEMM rounding being treated as bitwise inequality; unconditional CUDA-only Triton rounding in MOSS; and TheRock OpenBLAS directories being absent from AITER artifact validation. Focused L1 regressions cover the deterministic import and library-discovery paths, while AMD READY validates the ROCm/PyTorch behavior at the submitted head.

4. Whose customer or user needs does this address, and what are those needs?

This addresses customers and users affected by regressions from the vLLM 0.33 upgrade. They need stable AMD/ROCm CI.

Purpose

PR #9648 upgraded vLLM-Omni to vLLM 0.33. Its direct parent passed AMD build #15302 with 42/42 jobs, while the upgrade exposed three blocking failures across AMD builds #15305, #15307, #15310, and #15313. Exact-head AMD READY #15317 then exposed a fourth compatibility regression in its NonBlocking R2 coverage lane; the first portable-binding retry in #15322 confirmed that HIP lacks mul_rn and requires a compile-time backend split:

• The vLLM source tree provides a regular top-level tools package. Because Omni's tools/ was only a namespace package, imports such as tools.pre_commit, tools.minimax_h3, and tools.sheetsage2_transcribe resolved to vLLM instead of this repository.
• PyTorch 2.13.0+rocm10.0.0 can produce normal floating-point rounding differences between separate and fused gate/up CPU GEMMs. The test required bitwise equality even though its contract is numerical equivalence.
• MOSS local attention bound round-to-nearest multiplication through the removed CUDA-specific Triton libdevice path. The vLLM 0.33 portable binding maps to HIP on ROCm, where mul_rn is unavailable, so this operation must be CUDA-only while ROCm uses normal multiplication.
• The vLLM 0.33 ROCm image installs librocm-openblas.so.* under TheRock _rocm_sdk_* packages. AITER artifact validation searched only torch/lib and /opt/rocm/lib{,64}, reported librocm-openblas.so.0 => not found, rejected the compatible artifact, and cold-compiled it for roughly 1,540 seconds in build #15313.

This change:

• uses the vLLM 0.33 portable Triton tldevice binding and a constexpr backend split so explicit round-to-nearest multiplication is CUDA-only;
• makes repository-local tools a regular package and adds a regression that proves it wins over a later dependency package;
• uses torch.testing.assert_close for the gate/up numerical parity assertions while retaining exact QKV assertions; and
• adds sorted, unique parent directories of TheRock OpenBLAS libraries to the AITER loader path, with a focused regression test; and\n- refreshes the ROCm R2-01 exact collection guard from 1,416 to 1,420, matching the post-upgrade selection observed in AMD READY #15323 (all 502 shard tests passed; only the stale guard returned 1).

No public API, supported configuration, default, or active design contract changes. Documentation and examples are unchanged because this is limited to CI/developer tooling and test semantics.

AI assistance: Codex was used to investigate the CI failures, implement the scoped fixes, add regression tests, run local validation, and draft this description. Actual results and the remaining hardware validation gap are listed below.

Test Plan

vLLM Version: 0.33.0 ROCm CI image

vLLM-Omni Commit: 4e7f3c9a3fa54ad0a4baedc59db4858ba8979e8d

Local L1 commands:

pytest -q -o addopts='' --confcutdir=tests/buildkite tests/buildkite/test_aiter_mha_cache.py
pytest -q -o addopts='' --confcutdir=tests/tools tests/tools/test_tools_package.py
PYTHONPATH=. pytest -q -o addopts='' --confcutdir=tests/helpers/tests tests/helpers/tests/test_check_test_marks.py

CI-like L1 command in a complete repository environment:

cd tests
pytest -s -v -m "core_model and cpu"

Focused Qwen regression command:

pytest -q tests/model_executor/models/qwen3_tts/test_code_predictor_dtype.py::TestCodePredictorFusedProjections::test_fused_gate_up_matches_separate

READY acceptance criteria:

• repository-local tools imports collect successfully;
• the focused Qwen fused gate/up test passes in the vLLM 0.33 ROCm environment;
• MOSS local-attention kernels exclude CUDA-only mul_rn on ROCm while retaining it on CUDA;
• AITER consumers restore the compatible artifact without librocm-openblas.so.0 => not found or a fallback cold compile;
• all applicable exact-head READY gates pass. An Intel CI job that is only waiting for infrastructure/runner capacity is non-blocking; an Intel source or test failure is not waived.

Test Result

• tests/buildkite/test_aiter_mha_cache.py: 33 passed
• tests/tools/test_tools_package.py: 1 passed
• tests/helpers/tests/test_check_test_marks.py: 49 passed
• pre-commit across all six changed files: passed all applicable hooks, including Ruff, mypy, test marks, SPDX, forbidden imports, and torch.cuda checks
• python3 -m py_compile <six changed Python files>: passed
• git diff --check origin/main...HEAD: passed\n- focused R2-01 pipeline contracts: 4 passed\n- full Buildkite contract files: 240 passed, 7 skipped; one additional test could not import torch because the local Python 3.14 environment does not have it installed\n- pre-commit on the three collection-guard files: passed all applicable hooks, including Buildkite pipeline validation

The MOSS backend split passed py_compile and all applicable pre-commit hooks.

The Qwen regression was not run locally because this Mac does not have the repository Torch/vLLM environment and is using Python 3.14, outside the project's supported Python 3.11-3.13 range. Exact-head READY evidence is recorded below.

\n## Exact-head READY Result\n\n- CUDA #18795: passed on 4e7f3c9a3fa54ad0a4baedc59db4858ba8979e8d; 19/19 jobs finished, with zero nonzero exits, soft failures, or retries.\n- AMD/ROCm #15332: passed on the same exact head; 48/48 jobs finished, with zero nonzero exits, soft failures, or retries.\n- R2-01 collection guard verified exactly 1,420 selected tests; all three shards exited 0. Shard 1 reported 502 passed, 3,188 deselected.\n

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-11-04T14:22:17.349165Z We couldn’t complete this review. Comment @codex review to try again.	80276c5	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to be related to model: MinimaxH3.

Model owners: @david6666666 @yenuo26 @congw729

Routing: @david6666666 via model owner; @yenuo26 via CI owner, CODEOWNERS; @congw729 via CODEOWNERS

@lm10-arg, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 4e7f3c9a3fa5 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[amy-why-3459]

Thanks for tracking these down. The four fixes are small and well-scoped; a few code-level notes:

1. AITER loader path (aiter_mha_cache.py) – see inline: the glob result is unordered and can produce duplicate parent dirs. Minor, but worth tightening since this list is used to build the loader path that decides cache acceptance.

2. MOSS libdevice = tldevice – the kernels rely on libdevice.mul_rn (round-to-nearest is the whole point of that line) and libdevice.exp. Please make sure the exact-head AMD READY run actually executes the MOSS local-attention tests that previously failed to compile (the 19 tests mentioned), so we know mul_rn resolves on the HIP libdevice and not only that the module imports. On CUDA this should be behavior-preserving since tldevice dispatches to the same CUDA libdevice.

3. test_fused_gate_up_matches_separate – switching to torch.testing.assert_close makes sense for fused vs separate GEMMs. The default tolerances depend on dtype (e.g. bf16 rtol≈1.6e-2), which is fairly loose; consider passing explicit rtol/atol matching the dtype used in the test so the contract is visible and a real regression in the fused weight packing can't slip through under the defaults.

4. tools/__init__.py – making tools a regular package looks right, and since [tool.setuptools.packages.find] only includes vllm_omni* it won't get shipped as a top-level tools package in the wheel. The regression test covers the repo-root-first ordering, which matches how CI runs. Just double-check nothing in CI relied on namespace-package merging of tools across multiple path entries.

No blocking issues from my side on the code itself; the PR body already notes that exact-head AMD READY is still needed.

[amy-why-3459]

Incremental look at 349fbf1 (gate the MOSS round-to-nearest multiply to CUDA):

• Gating libdevice.mul_rn behind a CUDA_RN_OPS constexpr resolves the HIP concern from my previous pass, and importing tldevice as libdevice directly is cleaner than the module-level alias.
• On non-CUDA platforms the fallback v * current_probability is a plain multiply that feeds straight into the following tl.fma chain, so the compiler is free to contract it the same way the original comment warns about for SM89. That means fused vs. unfused bitwise parity is no longer pinned on ROCm. If the MOSS parity tests (fused/unfused or assert_close with zero tolerance) run on the AMD lane, it would be worth either confirming they still pass there or relaxing/skipping the bitwise check for HIP with a short comment, so a future flaky failure isn't misread as a kernel regression.

[amy-why-3459]

Incremental look at 13a4c72 (deterministic TheRock library discovery):

• Collecting the OpenBLAS parent directories into a set and sorting them removes both the duplicate entries (one per SONAME file) and the filesystem-order dependence, and the updated test covers both cases. Looks good to me.
• Minor note: the resulting order between multiple _rocm_sdk_* packages is now plain lexicographic (_rocm_sdk_libraries before _rocm_sdk_runtime). That's fine for determinism. If one package's copy should take precedence when both ship librocm-openblas, a short comment saying the alphabetical order is intentional (or an explicit priority) would make that clear to the next reader.

[andyluo7]

Approved after exact-head READY qualification. CUDA #18795 passed 19/19 jobs and AMD/ROCm #15332 passed 48/48 jobs on 4e7f3c9, with zero nonzero exits, soft failures, or retries. R2-01 verified the exact 1,420-node collection and all three shards exited 0.