[tzhouam]

vLLM 0.33 moves runtime hooks and serving APIs that Omni overrides, and changes the CUDA/Torch dependency set. This PR restores those integration points and repairs failures exposed by validating the existing workloads against the final release.

Contributor intent

1. Outcome: Align Omni with final vLLM 0.33.0, including compatible images and installation instructions.
2. Baseline: Omni main 41ef6bdbc9321e785943b41e583735f48d70edfe; candidate d8397156120955c4e24141519356bbc8977243c4; vLLM 0.32.0 5df63a72 → final 0.33.0 4ba787ec9959e566d3b587a796dff07673324da5.
3. Scope: Upstream interface adaptations, upgrade-validation fixes, and the tests/documentation needed to verify them.
4. Users: Existing Omni model and serving users upgrading to this runtime.

Why these changes are needed

Area	Why the rebase needs this change
Scheduler and cache lifecycle	vLLM 0.33 moved completed-KV publication and stopped-session enqueueing into update_from_output, which Omni overrides. Restore those steps in the Omni paths, publish only materialized KV, drain deferred HiSparse loads at the completed-step fence, and respect the new final-token truncation boundary. Otherwise prefixes are not cached, stopped streams can remain unscheduled, or session append hits the upstream assertion.
Worker and sampler integration	Adapt MRv2 graph/PCP/encoder/draft dispatch, the new two-dimensional staged-write grid, async KV-connector salvage, and Talker's has_topp_only_rows argument. These prevent wrong graph selection, incomplete writes, lost connector results and callback/masking errors. Register custom-sampler kernels during startup so their first use is covered by warmup.
Serving and errors	update moved CLI imports, renderer/derenderer and chat-template initialization, requested weight-version metadata, and profiler session options. Preserve upstream RequestError messages/codes through scheduler, output processor and frontend, including errors issued before execution; avoid duplicate duplex error delivery.
Profiling	The 0.33 profiling API forwards per-session profiler_kwargs; Omni overrides must accept and reset these options. Validate override names, restore state if construction fails, and rebuild accumulating sessions so an invalid request or a reused acc_events=True profiler does not contaminate a later session.
Multimodal/KV planning	use upstream's new global-group planning API and preserve Bagel img2img prefix attention through a narrow host-side alias to image. Without the alias, the new shared prefix helper misses those bidirectional spans. Original modality/cache identities remain intact.
CUDA installation	the final wheel moves to Torch 2.14/Pillow 13. Resolve its declared NCCL dependency instead of inheriting the image's optional EFA override; repair only the pinned image's verified orphan telemetry exporter while retaining LMCache and the telemetry packages; rebuild unchanged LMCache 0.5.6 against Torch 2.14 because its supplied extension references the old Torch ABI. Normal dependency resolution, uv pip check and native imports remain required.
Gradio/dev extras	existing VoxCPM 2.0.3 requires Gradio <7, while Gradio 6.31 requires Pillow <13 and vLLM requires Pillow >=13. A hash-pinned source backport includes the actual Pillow-13 PNG compatibility change and updated bound, with licenses and reproducible build inputs. This keeps existing demo/dev extras installable without dropping features or bypassing dependency checks.
Numerical compatibility discovered during validation	retain existing rounding contracts for AdaLN, Magi GEMM additions, MOSS attention and MiniMax RMS reduction; strengthen Wan's fastpath rejection probes. Qwen Image's conservative eager BF16 CUDA SDPA path uses reference normalization/rotation and separate text GEMMs; its scheduler timesteps use the pipeline device. These address observed parity failures; they do not establish that an upstream kernel is generally incorrect. The private, hashed cuDNN 9.25.1 target addresses the original Qwen-Image 2.1 pixel gate while preserving Torch's declared Python dependency metadata.
Z-Image online quantization	Online FP8 rewrites lm_head.weight storage. Aliasing that storage with the embedding table makes vocabulary lookup go out of bounds. When quantization is configured, leave the unused head detached; prompt encoding consumes hidden states rather than head logits.
MiniCPM	stop discarding a sampled token when it crosses the soft character target. Cut on an already accepted complete prefix, allowing UTF-8 continuation; retain hard token/request caps. Scope the existing C-operator/QK-fusion profile to CUDA duplex Stage 0, preserve BF16 head dtype, and carry KernelConfig through the typed stage-config path. Turn profiles reset the inherited kernel defaults. This improves the tested local case, but does not resolve the H100 recognition failure below.
Simple CPU test compatibility	The scheduler now requires a typed grammar_compile_error_reqs dictionary; initialize it in the existing mock fixtures rather than weakening production validation. Isolate the real router test's SageMaker root-logging side effect, measure SHM wakeups after connector startup, and keep synthetic Event workers independent of coverage-exit I/O. Publish synthetic JPEGs through a complete in-memory encoding and atomic cache replacement so parallel collectors cannot open a truncated file; decode cached pixels and payload from the same byte snapshot, and regenerate incomplete cache entries. Preserve the original warning, timing, exit-code and selected-case assertions.
Platform/build/test integration	Align platform release inputs and cache keys; include the Torch-stable-ABI FlashAttention Hub variant and upstream gfx90a AITER guard. Ready duplex tests use real weights because the live client waits for learned turn terminators. Update active installation commands and affected design contracts; retain historical release examples and explicitly document the unchanged MUSA source-pin boundary.

The rebase adopts main's CPU collection fix #9785: its atomic overlay publisher, stable SHM parameter IDs and 36-line publication regression are unchanged from main. The equivalent local publisher/IDs and three local regression cases were removed from this diff and archived locally. Main's combined Engine&Entrypoints&Diffusion CPU lane retains its 25-minute budget; its complete current-head CI result is pending. Main's L4 thinker KV-budget fix is also retained.

Full-nightly comparison (same frozen main baseline): main control at 41ef6bdbc9321e785943b41e583735f48d70edfe and candidate at d8397156120955c4e24141519356bbc8977243c4. The new candidate result is pending; the main control already has failures, including MiniCPM and Qwen-Image latency. Main CPU Coverage passed. The one-off candidate and reused main control retain NIGHTLY=1, RUN_HUNYUAN_IMAGE3_PERF=1, all 16 performance lanes and EMAIL_DISTRIBUTION=0. Four-hour refreshes, daily CI and background automation remain disabled.

Scope and validation

Against the comparison merge base, the diff is 201 files, +3,787 / -660 lines:

Category	Added / removed lines
Runtime and configuration	+963 / -306
Tests	+1,887 / -138
Docs, recipes and examples	+506 / -176
Build, dependencies and CI	+431 / -40

Temporary recognition traces, Conv1 precision experiments, FP32-head experiments and unrelated feature work are excluded. Existing main MRv2 DCP/KV gaps are not claimed fixed.

Current candidate d8397156120955c4e24141519356bbc8977243c4, native vLLM 0.33 / Torch 2.14:

• Full CI CPU Coverage selection, xdist -n 4 --dist loadgroup, with an independent cold temporary cache: 34,468 passed, 92 skipped, 1 xfailed, 2 failed and 1 setup error. An L20X CUDA device was visible because existing CPU-marked Helios/Kandinsky initialization uses CUDA; this is not a no-GPU qualification. All eight new image tests passed within this full selection.
• The full local run is not green: the chunked-video MP4 byte comparison failed, and CPU communication plus a Kandinsky NCCL setup reported EADDRINUSE. The video and CPU communication nodes passed one rerun with their original assertions, four workers and coverage. The Kandinsky rerun hid CUDA and therefore could not initialize NCCL; a subsequent normal GPU reservation timed out. That node remains unverified locally, pending the new CI. All three nodes passed parent CI 3273, and this follow-up changes none of their source.
• Eight synthetic-image cache regressions: the old code produced 7 failures / 1 pass, including the observed PIL.UnidentifiedImageError; the fix passed all 8. They cover cold/hot concurrent reads, incomplete-cache repair, failed publication preserving the old image, and matching cached pixels/payload bytes.
• Helpers plus H3 module collection under coverage and four workers: 354 passed.
• Applicable pre-commit checks passed for both files in this follow-up, including mypy, Ruff, test markers and SPDX. Independent review found no blocking issue. The earlier 199-path pre-commit check and scheduler/logging/runtime results belong to parent 3a803b6c4; the follow-up above changes only the synthetic-image helper and its tests.

Build 3273 passed 34,443 CPU tests but failed during collection while opening a shared synthetic JPEG. Main 3272's corresponding CPU Coverage job passed. The image helper is unchanged between their comparison base and the old candidate: the failure does not establish a Pillow/vLLM kernel regression. Its direct final-path write exposes partial JPEGs to parallel collectors; that race is reproduced locally. The CI log did not preserve the failed file or writer identity.

Fresh candidate full-nightly correctness and performance results remain pending at the links above; the focused local passes and disclosed full-suite result do not establish full model or performance acceptance.

Unresolved MiniCPM quality: H100 CI 3267/3268 at the earlier candidate revisions reproduced identical 10/12 scene coverage (missing 11/12), although a local L20X case passed. Similar vLLM 0.32 reports already exist, so this is not claimed to have first appeared in 0.33. It remains tracked in issue #9828. Further MiniCPM investigation has stopped for this PR; no H100 quality fix or all-green full-CI claim is made.

AI assistance: Codex prepared these scoped compatibility changes, validation and review follow-ups under the maintainer's instructions.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-11-03T03:11:59.796891Z We couldn’t complete this review. Comment @codex review to try again.	d575a59	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[tzhouam]

@amy-why-3459 Following up on your de6fb7d6 review:

The config and fallback changes are in 9ec70209. The 12-case regression resolves fresh CUDA deploys through typed configuration and OmniEngineArgs. It checks the duplex profile, complete kernel/compilation defaults for all three turn variants, and global/stage-0 CLI precedence. The UTF-8 fallback now documents its hard token/request bound, decode exceptions emit DEBUG tracebacks, and the test-comment spacing is fixed. The focused config/sampling run passed 436 tests, 2 skipped.

Update: the previously pending before/after measurement completed before the investigation was stopped. On one reserved L20X GPU, two synchronized sessions × eight responses per session used identical inputs/reference audio and temperature 0, excluding the first two responses each. Each group yielded 432 Stage0 ITL samples and 558 output tokens, with identical public transcripts. Before619 → current9ec: median 5.508 → 5.445 ms, mean 5.586 → 6.016 ms, p95 6.081 → 6.335 ms. This single ordered pair has mixed results and does not prove regression-free behavior or an established regression. Isolated sampler CPU time and decode-call counts were not exposed by the original driver and were not captured.

H100 recognition remained 10/12 scenes in both builds 3267 and 3268; the local L20X result does not qualify it. Further MiniCPM investigation has been stopped at the maintainer's request. Temporary diagnostic work was archived locally and removed from the working tree; no trace commit or new diagnostic CI was published.

[tzhouam]

@amy-why-3459 Update on your 9ec70209 review: all three suggestions are implemented in e6fa9f1e.

• The Dockerfile states that the immutable base-image digest must move with VLLM_VERSION.
• The NPU test patches device_name and device_type on the existing platform object, preserving the rest of its interface and every original graph-mode assertion.
• The shared helper is public as add_stored_gemm_outputs; the MAGI imports and callers are updated without changing arithmetic.

The related existing checks passed 81 tests, including the complete 51-case MiniCPM pipeline test file. The original aggregate Model Executor selection also passed on a normally reserved local L20X GPU: 8,960 passed, 8 skipped, retaining all 8,968 selected cases. Applicable pre-commit hooks passed. These local results do not claim physical NPU qualification.

[amy-why-3459]

Incremental look at e6fa9f1e only ("Harden vLLM 0.33 compatibility adapters"), since my last pass on 9ec70209. My earlier points are unaffected by this commit.

1. _finish_error_requests now raises TypeError on the scheduler step path. If grammar_compile_error_reqs is ever a set or list (e.g. an upstream type change in a later vLLM bump), this raises inside the engine-core loop and fails every in-flight request, not just the ones with grammar errors. The new test pins that behaviour. Since the container type is a property of the installed vLLM rather than of a request, it might be safer to check it once (at scheduler init or in the existing version-compat check) and keep the per-step path non-fatal, or at least make sure the raised error surfaces clearly in the engine-core logs.

2. set_profiler_kwargs (minor). signature(torch.profiler.profile).bind(**kwargs) validates names but not types, so a wrongly typed value still fails later in the constructor. That path is covered by the restore-on-failure test, so this is fine; just noting that the docs' "checks override names" wording is accurate and shouldn't be read as full validation. The scalar-only reuse check matches the docs (lists, enums and callables always rebuild).

3. Dockerfile CPU preload smoke. any(path in maps for path in LD_PRELOAD.split() if "libcudnn.so.9" in path) compares the literal LD_PRELOAD entry against /proc/self/maps, which lists resolved paths. If the preload entry ever points at a symlink (e.g. libcudnn.so.9 -> libcudnn.so.9.x.y), this assertion fails even though the library is loaded. Comparing os.path.realpath(path) would make it robust.

4. Nit: the _add_stored_gemm_outputs -> add_stored_gemm_outputs rename looks complete (I couldn't find other callers of the old name), and the new test_reference_precision.py CPU/CUDA tests nicely exercise the real upstream QKVParallelLinear path instead of a mock.

The profiler-kwargs hardening (mapping check, unknown-name rejection, restore on construction failure, warning on active overrides) looks good and is well covered by the new tests.

[tzhouam]

On the latest e6fa9f1e review:

• cuDNN preload path: Fixed in 22a6b97b by comparing os.path.realpath(path) with /proc/self/maps. A real symlink preload reproduced failure of the literal-path check and passed the resolved-path check plus the actual Torch CPU smoke. The existing runtime test file passed 144 tests, and applicable Dockerfile pre-commit hooks passed.
• Grammar container mismatch: The guard remains on the step path; it has not been moved to initialization. The selected vLLM 0.33 source initializes the container as dict[str, RequestError]. I verified the suggested logging alternative: the engine-core exception handler logs EngineCore encountered a fatal error. with the full exception, sends engine-dead, and re-raises. An incompatible future container type is therefore an explicit engine compatibility failure, not a silently lost request. Moving validation to initialization remains a possible future refinement; this check does not claim compatibility with an untested later pin.
• Profiler: Agreed: signature binding checks names, while constructor failures are rolled back; no claim of complete value-type validation. The documentation keeps that distinction.

The prior three follow-ups are also implemented: digest/version synchronization comment, preserved platform object in the NPU fixture, and public GEMM helper. Their original reply has been updated with the code and validation results. MiniCPM investigation has been stopped at the maintainer's request; no temporary tracing was published or new diagnostic CI launched.

[amy-why-3459]

Incremental review of 22a6b97b ("Resolve cuDNN preload paths in CPU smoke check"), compared with e6fa9f1e, which I reviewed earlier.

Comparing os.path.realpath(path) against /proc/self/maps fixes the symlink case from my last review, since the loader records the resolved file path in the maps.

One minor edge case: os.environ["LD_PRELOAD"].split() only splits on whitespace, but the dynamic loader also accepts colon-separated entries. If a later layer or CI step ever sets LD_PRELOAD=/a.so:/opt/.../libcudnn.so.9, the combined string goes through realpath as a single path and the assertion fails even though cuDNN is loaded. Splitting on both (for example re.split(r"[:\s]+", ...)) would make the check independent of how the variable is composed.

The explanation of the grammar container guard (failing explicitly through the engine-core fatal-error path rather than dropping requests) and of the profiler binding scope makes sense to me. I have nothing further on the rest of this commit.

[amy-why-3459]

Incremental look at 3a803b6c only ("Fix Simple test fixtures and parallel isolation"), since my last pass at 22a6b97b. The rest of the rebased stack wasn't re-reviewed here. This increment only touches tests.

• grammar_compile_error_reqs = {} on the MagicMock schedulers: This makes sense. The vLLM 0.33 update_from_output path now reads that attribute, and an auto-created MagicMock attribute doesn't behave like the real dict there. It's now copied into 13 hand-built mocks across 8 files, though, so the next upstream attribute will need the same sweep. A small shared helper in a test util (for example make_mock_scheduler(**overrides) that pre-sets the attributes update_from_output reads) would keep these in sync. Fine as a follow-up.
• test_sibling_stage_processes_share_wakeups: Replacing the time.sleep(0.05) with the sender_ready + start barrier, and moving cleanup into finally, fixes the ordering race and the process leak on failure. One nit: if process.join(5) still times out after kill(), process.close() raises ValueError, which would hide the original assertion error. Guarding the close with if not process.is_alive() would keep the real failure visible.
• os._exit(0) in the event-only worker and the dead process: This is a sensible way to keep inherited coverage flushing out of the 0.2s shutdown budget. Since the test only checks shutdown timing and liveness, skipping the interpreter teardown doesn't hide anything it asserts on.
• Root logger level restore in test_serving_speech.py: Using setLevel in a finalizer (instead of assigning root.level) is right, because it also clears the logger cache that later caplog-based tests depend on.

Nothing blocking in this increment.

[amy-why-3459]

Looked at the new commit d8397156 ("Publish synthetic image cache atomically") only. The tests/helpers/media.py change looks correct: the cached path decodes the same bytes it returns, an undecodable cache falls back to regenerating, and the write goes to a unique temp file with os.replace, with the temp file removed in finally. io, os, uuid, Image and logger are all imported at module level, so removing the local PILImage import is safe.

A few small notes:

1. Overlap with #9834. This overlaps with #9834, which makes the same synthetic-image cache change in the same helper. If both are meant to land, the second one will probably conflict in generate_synthetic_image and in the new tests/helpers/tests/test_synthetic_image.py. It may help to say in the description which PR carries it.
2. np_array on first versus later calls. When the image is freshly generated, np_array is still built from the in-memory PIL image, but base64 holds the lossy JPEG bytes. On a cache hit, np_array is decoded from the JPEG. So the same (width, height) returns slightly different pixels depending on whether the cache was warm. This behaviour already existed, but since test_cached_pixels_and_payload_use_the_same_generation now pins pixel/payload consistency for the cached path, decoding image_bytes on the fresh path too (or noting the difference in the docstring) would make both paths behave the same.
3. Truncated-JPEG detection depends on global Pillow state. The except OSError fallback relies on image.load() raising on a truncated file. If anything in the test process sets ImageFile.LOAD_TRUNCATED_IMAGES = True, a half-written cache would be accepted silently. That's unlikely now that writes are atomic, so it's only worth a comment.

Only 3a803b6c..d8397156 was reviewed; I did not run the tests.

[Gaohan123]

LGTM. Remaining issue can be solved in follow-up PR