[Rebase] Rebase to vLLM 0.29.0

[tzhouam]

Purpose

Adapt vLLM-Omni to the vLLM 0.29 release API, pinned to 98dff2a81d747d1dba01a47f939f48c3526d4206 in the CUDA CI image. Preserve Omni's multimodal, staged-generation, and diffusion behavior while importing or inheriting upstream implementations wherever their contracts match.

Tests

No rebase introduced errors:

Change inventory and reasons

This inventory is measured at PR head 1a7c6e51f0239b7bd759909a53294e1b6b135079, against merge base 087ac46318c42d1fa1e4de13a8053f94683898bd, using git diff --numstat 087ac463...1a7c6e51f. It matches GitHub: 116 files, +3,091 / −1,665 = 4,756 changed lines; net +1,426 lines.

The table covers 100% of changed lines, exceeding 95%. Each file is assigned once to its primary aspect; counts include associated tests, comments, and blank lines, not just executable code. Multi-purpose files are not split into subjective hunk-level counts.

Aspect	Files	Added	Deleted	Changed lines	Reason for the changes / reuse approach
Other model processors and weight loading	18	530	221	751	Migrate Bagel, CosyVoice3, GLM-TTS, OmniVoice, StepAudio2, Voxtral and smaller model integrations to active token/media processing APIs. Preserve model-specific image pathways, voice-cloning prompts and stage outputs. Import upstream AutoWeightsLoader/WeightsMapper, remove obsolete and pass-through hooks, and tokenize OmniVoice prompts only once. Includes shared API/loader and OmniVoice regression tests.
Hunyuan backbone and integration	3	643	28	671	vLLM removed the native Hunyuan classes that Omni's image model subclasses. Retain a 625-line compatibility backbone for the existing integration, using upstream execution layers; omit unused parent weight-loading methods and remove inactive processor hooks. A full Transformers-backend replacement is not implemented or validated in this PR.
Renderer and shared processor plumbing	6	374	267	641	The engine now processes inputs through renderers, so the old preprocessor customization was inactive. Construct a subclass of the selected upstream renderer and extend only singleton routing for no-media processor kwargs and Omni metadata. Preserve concrete-renderer overrides and tokenizer-less stages. Share common custom-processor plumbing and test sync/async routing and construction.
GLM image processor migration	2	103	459	562	Replace obsolete text-processing hooks with the active token/media path. Preserve target-shape generation suffixes, source-image placeholders, processed grids and image-to-image behavior. Most deletions remove inactive or duplicated legacy processing; regression tests cover target prompt formatting.
KV layout and attention migration	17	461	43	504	vLLM replaced per-spec block-stride flags and shared_by tensors with explicit physical layouts and strides. Resolve/propagate diffusion layouts outside vLLM's normal engine-core path, validate backend compatibility and construct correctly addressed tensors. Import KVCacheLayout, KVCacheTensor and compute_layout_strides; retain Omni-specific integration.
Qwen2.5/Qwen3 Omni migration	8	126	243	369	Move existing video pre-sampling and per-video audio-mask adjustments to the active hook. Upstream Qwen bypasses generic pre/post hooks, so a narrow wrapper is still needed. Qwen3 inherits this wrapper and upstream Qwen3 audio processing; remove duplicate prompt-update code. Adapt weight mapping and related model APIs.
MiniCPM migration	2	102	126	228	Adapt media processing, token placeholders and output fields to the new API while retaining long-audio chunking and Omni behavior. Delegate prompt batching to upstream MiniCPMVMultiModalProcessor, filtering the Omni-only use_tts kwarg.
Ming migration	2	77	85	162	Move model-specific prompt rewriting out of the removed processing hook; use token-ID placeholder targets and the new media-processing signature. Use upstream weight loading with stage exclusions.
Runtime, scheduler, platform and worker compatibility	12	150	10	160	Preserve the v1 model-runner contract, initialize the new shutdown timeout, adapt sampler state/structured-output handling and embedding-related calls, keep frontend-only logging flags out of stage validation, and expose platform-detection errors. Includes scheduler/config/platform regressions.
Rotary embedding compatibility	4	109	17	126	Adapt rotary broadcasting and preserve Qwen-Image complex-FP32 frequency precision until rotation, avoiding an observed accuracy regression. Includes dtype, layout and reference-comparison tests.
Serving and API compatibility	19	66	59	125	Follow upstream serving/protocol/launcher module moves and callback/video-decoding interfaces; remove superseded imports and update serving fixtures. These are mostly small compatibility edits across callers.
Docker runtime installation	1	104	20	124	Install the exact pinned vLLM wheel over the configured v0.28 base image and align its runtime dependencies/native-library loading. This is the current pinned-image strategy, not a claim that a matching official release image remains unavailable.
MiniMax H3 migration	1	40	44	84	Preserve H3's segmented presentation using processed image/video grids under the new API. Keep model-specific placeholder discovery and media reprocessing where cache misses alone cannot reconstruct the complete presentation.
Buildkite HF-token handling	3	60	12	72	Preserve Kubernetes-secret credentials when scheduled-build environment values would override HF_TOKEN; restore the conventional variable from a dedicated alias. This is an independent CI reliability fix carried by the branch, with tests.
Other diffusion test compatibility	11	42	22	64	Update fixtures/mocks for changed upstream signatures and dependency interfaces, including SANA GEMM dispatch and CPU-safe distributed/attention setup.
Image accuracy test compatibility	3	51	1	52	Match the Qwen-Image FlashAttention availability probe to the Diffusers API and adjust dependency-sensitive accuracy-test setup without lowering thresholds.
Audio accuracy test corrections	3	45	3	48	Measure dots.tts PCM quality at its native 48 kHz rate, with assertion regressions. This corrects the measurement rather than lowering the HNR threshold.
Quantization test compatibility	1	8	5	13	Adapt the ModelOpt NVFP4 NaN-clamp test to the updated upstream interface.
Total / coverage	116	3,091	1,665	4,756 (100%)	All changed files accounted for once.

Why the diff is still large

• Tests contribute +1,101 / −163 = 1,264 changed lines (26.6%), already included in the categories above, not additional to them.
• Deletions count toward review size. GLM and preprocessing remove code that already exists on the base branch; deleting it reduces maintenance but still enlarges the visible deletion count.
• The largest retained copy is Hunyuan's 625-line backbone. The whole diff is not copied upstream code: it also includes API adaptations, tests, build configuration and removal of old implementations.
• Independent CI and accuracy-test fixes are explicitly identified above rather than presented as mandatory model-API changes.

Why model-specific processor changes remain

We do not need a duplicate processor implementation for every model. Generic parsing, caching, loading and compatible model processing should come from upstream. Local overrides retain only behavior absent from upstream: stage-specific inputs, generation prompt formatting, voice-cloning conventions, image pathways or per-video audio settings.

For example, Qwen's old _call_hf_processor customization is moved to _apply_hf_processor_main because the old hook is no longer invoked. The override prepares Omni kwargs, delegates actual processing to super(), and restores the per-video mask. Qwen3 inherits that implementation rather than copying it. Upstream Qwen's custom implementation does not call the generic _preprocess_hf_mm_data/_postprocess_hf_mm_data hooks, preventing a smaller hook-only override at the pinned version.

The branch also removes obsolete _hf_processor_applies_updates/_call_hf_processor overrides, Voxtral's pass-through override, and Qwen3's duplicate prompt-update override. The renderer now uses actual upstream-class inheritance rather than a proxy with a manually maintained method allow-list.

Validation and limitations

Evidence is commit-scoped; earlier results are not certification of the current head.

• At cleanup commit ebd306a38, focused CPU suites passed 453 tests, 7 skipped, including renderer/input routing, model processing, stage initialization, weight mapping, GLM formatting and Hunyuan diffusion tests. Processor imports, Ruff and whitespace checks passed. GPU/end-to-end CI was not rerun for that cleanup.
• Subsequent commit records report focused validation: Voxtral tests (62); Qwen3/Qwen2.5 tests (20 + 81); OmniVoice tests (106) plus shared-processor/TTS tests (162); renderer-head input tests (32), stage-init tests (55) and entrypoint tests (548). These are separate reported runs and must not be summed as unique full-suite coverage. The latest renderer commit also records pre-commit, including mypy, and import checks.
• Historical Buildkite 3028, on 325217e0a and the earlier vLLM pin, passed the investigated original rebase-only cases: VoiceDesign checkpoint, SANA cache-backend setup, MiniMax I2VA accuracy, Qwen3 speaker output, Qwen-Image accuracy and dots.tts PCM measurement. This evidence does not validate the current head or pin. Fixes already present in base 087ac463 are not counted as new PR changes in the inventory.
• At this description refresh, GitHub reports Buildkite release 2834 and documentation as pending, CodeQL queued, and DCO action required for missing sign-off on 736c2f53a. GitHub also reports merge conflicts. These statuses can change; consult live checks before merging.
• Accuracy thresholds are unchanged. Full current-head GPU/end-to-end validation remains outstanding; this description does not claim all checks are green or that the PR is merge-ready.

Remaining reuse opportunities (not implemented here)

• Replace the local Hunyuan MLP with upstream LlamaMLP after parameter/loading/parallelism equivalence tests.
• Delegate diffusion layout selection to upstream resolve_kv_cache_layout while preserving backend validation and explicitly testing environment/connector policy differences.
• Investigate constructing Omni's FP32-router Hunyuan MoE block directly instead of creating and replacing the compatibility block.
• A full Hunyuan Transformers-backend migration requires independent model-parity validation; there is no native Hunyuan class available to simply import from the pinned vLLM version.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/model_integration.md.

Module owners: @tzhouam @gcanlin

@tzhouam, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 0ae9bdd8c2ee produced:

• Priority: high. Prompt maintainer attention is suggested.

• Quality: low. Mechanical checks suggest considering rework or declining this PR before review effort is spent.

Quality evidence:

• Q3: touches 14 modules with no linked issue
• Q6: 120 changed files

These are automated triage suggestions only — the final decision belongs to the maintainers.

[Gaohan123]

Please also modify the installation docs

[Gaohan123]

Remove this

[Gaohan123]

LGTM. Thanks