Dev/rebase 0.14.0

[tzhouam]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

This PR partially rebase to vllm v0.14.0 and supporting online/offline for Qwen 3 Omni.

Test Plan Online

Follow Readme

Test Result Online

audio_0.wav

Chat completion output from text: Based on the provided audio and images, here is an analysis of your questions:

### 1. What is recited in the audio?

The speaker recites the first verse of the classic English nursery rhyme "Mary Had a Little Lamb":

> "Mary had a little lamb, its fleece was white as snow; and everywhere that Mary went, the lamb was sure to go."

The speaker also mentions that these were the "first words I spoke in the original phonograph," indicating they are likely recounting a historical or personal anecdote about early sound recording.

---

### 2. What is the content of this image?

The image shows the Tokyo Skytree, a prominent telecommunications and observation tower in Tokyo, Japan. It is viewed from a low angle through the pink blossoms of cherry trees (sakura) in full bloom against a clear blue sky. This scene captures a beautiful springtime view, often associated with the Japanese tradition of *hanami* (flower viewing).

---

### 3. Why is this video funny?

The humor in the video stems from the contrast between the subject's appearance and their actions.

*   **Appearance:** The child is wearing large, thick-rimmed glasses that look comically oversized for their face.
*   **Action:** Despite the silly appearance, the child is engaged in a very serious and focused activity—reading a book. They turn the pages carefully and appear completely absorbed in the book.

The juxtaposition of a toddler dressed like a scholarly adult, intently reading, creates a charming and humorous effect.
Audio saved to audio_0.wav

Test Plan Offline

python3 end2end.py -q use_image

Test Result Offline

output_0_1a2bd30d-910c-4ad5-81ed-f19b9571836a.wav

INFO 01-16 05:15:09 [omni.py:295] [Orchestrator] Stage-1 reported ready
INFO 01-16 05:15:09 [omni.py:321] [Orchestrator] All stages initialized successfully
Adding requests:   0%|                                                                                                                                 | 0/1 [00:00<?, ?it/sThe image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'interleaved', 'mrope_section'}
Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'interleaved', 'mrope_section'}
(Worker pid=752775) [Stage-1] INFO 01-16 05:15:26 [mrope.py:452] Multimodal token idx changed!
(Worker pid=752775) [Stage-1] WARNING 01-16 05:15:26 [qwen3_omni_moe_code_predictor_mtp.py:228] Using sdpa attention backend (flash_attention_2 not available or failed)
(Worker pid=752772) [Stage-2] INFO 01-16 05:16:20 [mrope.py:452] Multimodal token idx changed!
INFO 01-16 05:16:21 [log_utils.py:550] {'type': 'request_level_metrics',
INFO 01-16 05:16:21 [log_utils.py:550]  'request_id': '0_1a2bd30d-910c-4ad5-81ed-f19b9571836a',
INFO 01-16 05:16:21 [log_utils.py:550]  'e2e_time_ms': 71268.56875419617,
INFO 01-16 05:16:21 [log_utils.py:550]  'e2e_tpt': 23.064261732749568,
INFO 01-16 05:16:21 [log_utils.py:550]  'e2e_total_tokens': 3090,
INFO 01-16 05:16:21 [log_utils.py:550]  'transfers_total_time_ms': 159.29198265075684,
INFO 01-16 05:16:21 [log_utils.py:550]  'transfers_total_bytes': 56000530,
INFO 01-16 05:16:21 [log_utils.py:550]  'stages': {0: {'stage_gen_time_ms': 14935.698509216309,
INFO 01-16 05:16:21 [log_utils.py:550]                 'num_tokens_out': 223,
INFO 01-16 05:16:21 [log_utils.py:550]                 'num_tokens_in': 2093},
INFO 01-16 05:16:21 [log_utils.py:550]             1: {'stage_gen_time_ms': 54997.80225753784, 'num_tokens_out': 774},
INFO 01-16 05:16:21 [log_utils.py:550]             2: {'stage_gen_time_ms': 273.1940746307373, 'num_tokens_out': 0}}}
Processed prompts: 100%|█████████████████████████████████████████████████████████████████| 1/1 [01:11<00:00, 71.27s/req, est. speed stage-2 tok/s: 43.36, avg e2e_lat: 0.0ms]
INFO 01-16 05:16:21 [omni.py:782] [Summary] {'e2e_requests': 1,██████████████████████████| 1/1 [01:11<00:00, 71.27s/req, est. speed stage-2 tok/s: 43.36, avg e2e_lat: 0.0ms]
INFO 01-16 05:16:21 [omni.py:782]  'e2e_total_time_ms': 71269.86789703369,
INFO 01-16 05:16:21 [omni.py:782]  'e2e_sum_time_ms': 71268.56875419617,
INFO 01-16 05:16:21 [omni.py:782]  'e2e_total_tokens': 3090,
INFO 01-16 05:16:21 [omni.py:782]  'e2e_avg_time_per_request_ms': 71268.56875419617,
INFO 01-16 05:16:21 [omni.py:782]  'e2e_avg_tokens_per_s': 43.3571215756745,
INFO 01-16 05:16:21 [omni.py:782]  'wall_time_ms': 71269.86789703369,
INFO 01-16 05:16:21 [omni.py:782]  'final_stage_id': {'0_1a2bd30d-910c-4ad5-81ed-f19b9571836a': 2},
INFO 01-16 05:16:21 [omni.py:782]  'stages': [{'stage_id': 0,
INFO 01-16 05:16:21 [omni.py:782]              'requests': 1,
INFO 01-16 05:16:21 [omni.py:782]              'tokens': 2316,
INFO 01-16 05:16:21 [omni.py:782]              'total_time_ms': 15139.074325561523,
INFO 01-16 05:16:21 [omni.py:782]              'avg_time_per_request_ms': 15139.074325561523,
INFO 01-16 05:16:21 [omni.py:782]              'avg_tokens_per_s': 152.98161236249146},
INFO 01-16 05:16:21 [omni.py:782]             {'stage_id': 1,
INFO 01-16 05:16:21 [omni.py:782]              'requests': 1,
INFO 01-16 05:16:21 [omni.py:782]              'tokens': 774,
INFO 01-16 05:16:21 [omni.py:782]              'total_time_ms': 55089.155197143555,
INFO 01-16 05:16:21 [omni.py:782]              'avg_time_per_request_ms': 55089.155197143555,
INFO 01-16 05:16:21 [omni.py:782]              'avg_tokens_per_s': 14.049952249769351},
INFO 01-16 05:16:21 [omni.py:782]             {'stage_id': 2,
INFO 01-16 05:16:21 [omni.py:782]              'requests': 1,
INFO 01-16 05:16:21 [omni.py:782]              'tokens': 0,
INFO 01-16 05:16:21 [omni.py:782]              'total_time_ms': 299.47686195373535,
INFO 01-16 05:16:21 [omni.py:782]              'avg_time_per_request_ms': 299.47686195373535,
INFO 01-16 05:16:21 [omni.py:782]              'avg_tokens_per_s': 0.0}],
INFO 01-16 05:16:21 [omni.py:782]  'transfers': [{'from_stage': 0,
INFO 01-16 05:16:21 [omni.py:782]                 'to_stage': 1,
INFO 01-16 05:16:21 [omni.py:782]                 'samples': 1,
INFO 01-16 05:16:21 [omni.py:782]                 'total_bytes': 49699758,
INFO 01-16 05:16:21 [omni.py:782]                 'total_time_ms': 69.85020637512207,
INFO 01-16 05:16:21 [omni.py:782]                 'tx_mbps': 5692.153031943067,
INFO 01-16 05:16:21 [omni.py:782]                 'rx_samples': 1,
INFO 01-16 05:16:21 [omni.py:782]                 'rx_total_bytes': 49699758,
INFO 01-16 05:16:21 [omni.py:782]                 'rx_total_time_ms': 65.75202941894531,
INFO 01-16 05:16:21 [omni.py:782]                 'rx_mbps': 6046.932201387521,
INFO 01-16 05:16:21 [omni.py:782]                 'total_samples': 1,
INFO 01-16 05:16:21 [omni.py:782]                 'total_transfer_time_ms': 138.01193237304688,
INFO 01-16 05:16:21 [omni.py:782]                 'total_mbps': 2880.8962903677825},
INFO 01-16 05:16:21 [omni.py:782]                {'from_stage': 1,
INFO 01-16 05:16:21 [omni.py:782]                 'to_stage': 2,
INFO 01-16 05:16:21 [omni.py:782]                 'samples': 1,
INFO 01-16 05:16:21 [omni.py:782]                 'total_bytes': 6300772,
INFO 01-16 05:16:21 [omni.py:782]                 'total_time_ms': 7.778406143188477,
INFO 01-16 05:16:21 [omni.py:782]                 'tx_mbps': 6480.270517134222,
INFO 01-16 05:16:21 [omni.py:782]                 'rx_samples': 1,
INFO 01-16 05:16:21 [omni.py:782]                 'rx_total_bytes': 6300772,
INFO 01-16 05:16:21 [omni.py:782]                 'rx_total_time_ms': 11.675119400024414,
INFO 01-16 05:16:21 [omni.py:782]                 'rx_mbps': 4317.401327809512,
INFO 01-16 05:16:21 [omni.py:782]                 'total_samples': 1,
INFO 01-16 05:16:21 [omni.py:782]                 'total_transfer_time_ms': 21.28005027770996,
INFO 01-16 05:16:21 [omni.py:782]                 'total_mbps': 2368.705681715355}]}
[Stage-1] INFO 01-16 05:16:21 [omni_stage.py:675] Received shutdown signal
[Stage-2] INFO 01-16 05:16:21 [omni_stage.py:675] Received shutdown signal
[Stage-0] INFO 01-16 05:16:21 [omni_stage.py:675] Received shutdown signal
Request ID: 0_1a2bd30d-910c-4ad5-81ed-f19b9571836a, Text saved to output_audio/0_1a2bd30d-910c-4ad5-81ed-f19b9571836a.txt
Request ID: 0_1a2bd30d-910c-4ad5-81ed-f19b9571836a, Saved audio to output_audio/output_0_1a2bd30d-910c-4ad5-81ed-f19b9571836a.wav
(Worker_TP1 pid=747414) (Worker pid=752772) [Stage-0] INFO 01-16 05:16:38 [multiproc_executor.py:707] Parent process exited, terminating worker
(Worker pid=752775) (Worker_TP0 pid=747413) [Stage-1] INFO 01-16 05:16:38 [multiproc_executor.py:707] Parent process exited, terminating worker
[Stage-2] INFO 01-16 05:16:38 [multiproc_executor.py:707] Parent process exited, terminating worker
[Stage-0] INFO 01-16 05:16:38 [multiproc_executor.py:707] Parent process exited, terminating worker

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 236f73e419

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Align EngineCoreOutput with routed_experts field

OmniARScheduler now always passes routed_experts into EngineCoreOutput. In vllm_omni, EngineCoreOutput is patched to OmniEngineCoreOutput (see vllm_omni/patch.py), which does not define a routed_experts field (vllm_omni/engine/__init__.py). msgspec.Struct will raise TypeError: __init__() got an unexpected keyword argument 'routed_experts', so any request that produces output will crash even when routed_experts is None. Either add the field to OmniEngineCoreOutput or guard this keyword by engine type/feature flag.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Reject auto tool choice when no tool parser

With tool_parsing_unavailable true, the new logic only errors for tool_choice="auto" when enable_auto_tools is false; if enable_auto_tools is true but tool_parser is still None (HF tokenizers, not Harmony/Mistral), the request is accepted. Downstream parsing is gated on tool_parser being set, so auto tool calls are never extracted and the tool choice is silently ignored. This is user-visible for requests with tools but no --tool-call-parser. Consider restoring the requirement that auto needs a parser or explicitly rejecting when tool_parsing_unavailable and tool_choice is not none.

Useful? React with 👍 / 👎.

[hsliuustc0106]

review from gemini according to logs:

Based on the logs you provided, your system is running a multi-stage multimodal pipeline (likely Qwen2-VL or a similar Omni-model architecture) using an Orchestrator.

There are three main areas for optimization: Environment/Hardware, Configuration, and Logging/Observability.

---

1. Critical Performance Bottlenecks

Fix Attention Backend (High Impact)

Your logs show a major warning:

Using sdpa attention backend (flash_attention_2 not available or failed)

• Problem: SDPA is significantly slower and more memory-intensive than Flash Attention 2, especially for high-resolution images or long sequences.

• Optimization: Install flash-attn. Ensure your CUDA version and GPU (Ampere or newer, e.g., A100, H100, RTX 30/40 series) support it. This will drastically reduce the stage_gen_time_ms.

Eliminate RoPE Parameter Errors

Unrecognized keys in rope_parameters ... {'interleaved', 'mrope_section'}

• Problem: This indicates a version mismatch between your transformers library and the model configuration. The model is falling back to default Rotary Positional Embeddings, which can degrade the quality of multimodal spatial understanding.

• Optimization: Update your environment: pip install -U transformers accelerate.

Stage-1 Latency

Looking at the summary:

• Stage-0: 152.9 tok/s (Fast)

• Stage-1: 14.0 tok/s (Slow bottleneck)

• Optimization: Stage-1 is where your heavy lifting happens. Since you are using Worker pid and multiproc_executor, consider rebalancing your GPU resources. If Stage-1 is your main LLM/Vision-Language block, it needs more compute or a more aggressive quantization (e.g., bitsandbytes 4-bit or AWQ).

---

2. Infrastructure & Data Transfer

Fast vs. Slow Image Processor

The log mentions: The image processor ... is now loaded as a fast processor by default.

• Optimization: If you are seeing CPU spikes during the "Adding requests" phase, explicitly set use_fast=True in your AutoProcessor loading code to suppress the warning and ensure you aren't hitting the "breaking change" fallback.

Inter-Stage Transfer (NCCL/Shared Memory)

• Status: Your transfer speed is ~2800 MB/s to 6000 MB/s.

• Optimization: For a single-node setup, ensure you are using Shared Memory (shm) or NCCL for P2P transfers between workers. If these workers are on different GPUs, ensure NVLink is active.

---

3. Log Cleanup & Readability

The current logs are "noisy" because JSON metrics are being printed line-by-line via log_utils.py:550.

• Single-Line JSON: Configure your logger to output the metrics dictionary as a single minimized JSON string. This makes it much easier for log aggregators (like ELK or Datadog) to parse.

• Disable TQDM in Production: The progress bars (0%|...|) create massive amounts of "empty" line noise in file-based logs. Use disable=True in your tqdm instances when not sys.stdout.isatty().

---

Summary Table of Optimization Tasks

Category | Action Item | Expected Result -- | -- | -- Compute | Install flash-attn | 2x - 3x speedup in Stage-1 Software | Update transformers | Fix RoPE errors & improve accuracy Pipeline | Set use_fast=True for Processor | Avoid initialization overhead IO | Move logs to single-line JSON | Reduce log volume by 80% Model | Apply AWQ/GPTQ Quantization | Increase Stage-1 tok/s

[tzhouam]

Inter-Stage Transfer (NCCL/Shared Memory)
Status: Your transfer speed is ~2800 MB/s to 6000 MB/s.

Optimization: For a single-node setup, ensure you are using Shared Memory (shm) or NCCL for P2P transfers between workers. If these workers are on different GPUs, ensure NVLink is active.

@natureofnature Are we going to develop NCCL based Connectors to further reduce the data transfer overhead?

[tzhouam]

Eliminate RoPE Parameter Errors
Unrecognized keys in rope_parameters ... {'interleaved', 'mrope_section'}

Problem: This indicates a version mismatch between your transformers library and the model configuration. The model is falling back to default Rotary Positional Embeddings, which can degrade the quality of multimodal spatial understanding.

Optimization: Update your environment: pip install -U transformers accelerate.

Will deal with this later.

[tzhouam]

3. Log Cleanup & Readability
   The current logs are "noisy" because JSON metrics are being printed line-by-line via log_utils.py:550.

Single-Line JSON: Configure your logger to output the metrics dictionary as a single minimized JSON string. This makes it much easier for log aggregators (like ELK or Datadog) to parse.

Disable TQDM in Production: The progress bars (0%|...|) create massive amounts of "empty" line noise in file-based logs. Use disable=True in your tqdm instances when not sys.stdout.isatty().

Please pay attention to this. @Bounty-hunter
From my view, we can leave it until the rebase finished.

[natureofnature]

Inter-Stage Transfer (NCCL/Shared Memory)
Status: Your transfer speed is ~2800 MB/s to 6000 MB/s.

Optimization: For a single-node setup, ensure you are using Shared Memory (shm) or NCCL for P2P transfers between workers. If these workers are on different GPUs, ensure NVLink is active.

@natureofnature Are we going to develop NCCL based Connectors to further reduce the data transfer overhead?

Will implement a version.

[hsliuustc0106]

Inter-Stage Transfer (NCCL/Shared Memory)
Status: Your transfer speed is ~2800 MB/s to 6000 MB/s.

Optimization: For a single-node setup, ensure you are using Shared Memory (shm) or NCCL for P2P transfers between workers. If these workers are on different GPUs, ensure NVLink is active.

@natureofnature Are we going to develop NCCL based Connectors to further reduce the data transfer overhead?

Will implement a version.

we can do it later, maybe in v0.15.0