[AlessioNetti]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

We want to minimize orchestrator host overhead for full duplex models and increase maximum real-time concurrency accordingly. Our changes are targeted at small models such as Hibiki zero, where even a single-GPU deployment is sufficient for most scenarios and host overhead is more critical.

2. What is the highest-leverage variable for that outcome?

It looks like the full duplex orchestrator thread is a real bottleneck, especially for models such as Hibiki zero. This is mainly due to the fact that the orchestrator runs in a serial asyncio loop, and makes use of thread pool offloading for several tasks (e.g., waveform conversion). This leads to noticeable GIL contention and saturation at high concurrency regimes.

3. How will this work move that variable?

By avoiding thread offloading for certain operations (e.g., when waveform conversion is a no-op), we can materially reduce GIL contention and handoff overhead, and improve the real-time factor at high concurrency.

4. Whose customer or user needs does this address, and what are those needs?

The main need we are addressing is to minimize cost of deployment, i.e., allow as many concurrent streams as possible on the same GPU (or same set of GPUs), and ideally reach a regime where host overhead is no longer a concern.

Purpose

We use a custom vllm-omni implementation of the Moshi/Hibiki architecture (as described in PR 8554) that is integrated as an out-of-tree full duplex plugin.

At high concurrency regimes, we noticed via pyspy profiling that the orchestrator thread is easily saturated, which will lead to latency falling behind real-time. It appears that this performance hit is coming from GIL contention with the main frontend thread, as well as from the handoffs associated with the thread pool used for offloading.

This PR attacks this bottleneck and implements two small optimizations that significantly improve latency at no cost:

1. In DuplexSessionRunner._on_append_audio(), waveform conversion in convert_input_audio_with_rate() is offloaded to a thread pool even when it is a no-op (i.e., when audio is already pcm_f32le). We now skip this round-trip entirely when audio is already in the right format;
2. DuplexOmniEngine.submit_command_async() always performs thread offloading to call the blocking Janus queue's put() method. This can be avoided if the queue has capacity - we now first attempt a non-blocking put() in the main thread, and proceed with thread offloading only if the queue is full. Note that the queue's default size is 256.

The two optimizations above lead to a measurable impact on latency, with a ~0.1 improvement on the real-time factor that allows us to reach concurrency 64 when producing ~61s of output audio. We observe a larger improvement at concurrency 96, where latency remains however slower than real-time.

At a lower level, we observed a ~41% reduction in voluntary context switches in the main process, and a large reduction in pre-processing median times (i.e., _on_append_audio()) at concurrency 96, from 53.89ms down to 0.003ms.

Test Plan

vLLM Version: 0.32.0

vLLM-Omni commit: b06d34f35

Tested branch commit: 40916cfdc

Prerequisites: Repository test dependencies installed. CPU-only tests; no GPU or model checkpoint required.

Added or strengthened coverage:

Test	Coverage
New: test_input_conversion_offload_only_for_encoded_formats	Four cases verify that float32 PCM bypasses conversion offload, while other formats retain it. All formats can subsequently commit and submit input successfully.
New: test_full_command_queue_offloads_wait_and_preserves_sequential_order	Verifies that a saturated queue waits through the executor, reuses the same message object between attempts and preserves sequential command order.
Updated: test_submit_command_is_one_way_on_the_request_queue	Verifies that uncontended submission enqueues the command without an executor hop or control RPC.
Updated: test_audio_admission_is_held_until_preprocessing_finishes	Four cases cover decoding/VAD and success/failure. Uses PCM16 to exercise conversion offload and verifies that pending-input accounting continues to enforce backpressure.

Run the added and updated coverage:

python -m pytest \
  tests/engine/test_duplex_omni_engine.py \
  tests/engine/duplex/test_session_runner.py \
  -k 'input_conversion_offload or full_command_queue or submit_command_is_one_way or audio_admission_is_held' \
  -m 'core_model and cpu' --run-level core_model \
  -o addopts='' -q --disable-warnings --tb=short

Test Result

• Focused CPU regression selection: 10 passed, 154 deselected, 18 warnings in 3.92s.
• All applicable non-mypy pre-commit hooks passed.
• Mypy hooks for Python 3.10–3.13 failed with the same single error, which comes from the unchanged base.
• The branch merged cleanly with upstream main at the time of checking.

Note

To avoid agent spam, vLLM-Omni limits contributors without write access to 5 open PRs. If this blocks well-intentioned critical work, please contact a committer to request bypass list access.
(anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-27T16:03:23.527094Z We couldn’t complete this review. Comment @codex review to try again.	40916cf	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/engine_orchestration.md, docs/design/module/observability.md.

Module owners: @fake0fan @tzhouam @lishunyang12

Routing: @fake0fan via module of the changed files, CODEOWNERS; @tzhouam via module of the changed files, CODEOWNERS; @lishunyang12 via module of the changed files

@AlessioNetti, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 40916cfdc254 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[hsliuustc0106]

@AlessioNetti — performance-evidence TODO for 40916cfdc254c96a320d5e2c08f57652e49fa646.

Source inspection supports the native-PCM bypass and uncontended queue fast path, but the supplied CPU test count does not verify the reported RTF, context-switch and preprocessing improvements.

Please attach reproducible matched A/B results for baseline b06d34f355c948cea6cdbe6fce1aed485cec7c4a versus this head: Hibiki checkpoint and external plugin revision, hardware/software, launch/benchmark commands, audio/chunk workload and concurrency 64/96, warmup/repetitions, raw metrics and variability. Include output completeness/parity and sustained audio cadence to rule out dropped work.

Acceptance: identical workload/configuration on both commits reproduces the claimed metrics with the correctness guard passing, or narrow the performance claims to the evidence available. This is an evidence request, not a demonstrated code defect. Review was source-only; no local or GPU tests were executed here.

[AlessioNetti]

Hi @hsliuustc0106 - thanks for the feedback. I'm unable to share the full Hibiki model implementation and much of the frontend code, but I will attach as much evidence as possible.

First, below you can find min, median and max per-request latency values from our internal benchmark. Each request feeds 60s of audio split into 160ms frames (which are internally further split into 80ms frames as expected by the Mimi encoder), and produces 60,960ms of output audio (750 + 12 80ms frames). We evaluated concurrency levels 64 and 96, issuing concurrency_level * 2 requests in total and using one warmup batch per concurrency level.

	Concurrency 64	Concurrency 96
Baseline	64,283ms / 66,132ms / 69,151ms	96,198ms / 99,594ms / 101,073ms
Optimized	60,573ms / 61,004ms / 62,302ms	89,767ms / 92,316ms / 92,984ms

The measurements were collected on a host equipped with an NVIDIA RTX Pro 6000 Blackwell GPU (CUDA 13.3), and an AMD EPYC 9454 48-Core CPU. The Baseline and Optimized versions correspond to the references in the PR's description.

I also included a self-contained synthetic CPU benchmark, which you can find as a gist here. This micro-benchmark mocks away any model-related work (including model-specific post-processing) but uses a real vllm-omni full-duplex orchestrator, and measures several key latency metrics. It can be called as in the following:

python benchmark_duplex_host.py --label baseline --output /tmp/baseline.json --enqueue-workers 32 --preprocess-workers 4 --frames-per-session 512 --warmup-frames 64 --cell-timeout-s 600

Note that by default the micro-benchmark will not pace input audio frames, but will send them in a single burst - to enable paced execution the --mode paced argument must be supplied.

Using the command above (again, in burst mode), I obtained the following measurements for the vllm-omni baseline at concurrency levels 64 and 96:

Measuring baseline: concurrency=64, format=pcm_f32le
  elapsed median=6086.05 ms, range=6032.94–6230.51 ms; all guards passed
  voluntary context switches median=436257
  preprocess_ms median=4.727307; enqueue_ms median=6.536482 (medians of per-repetition frame medians)
  executor submissions median: enqueue=32768, preprocess=32768
Measuring baseline: concurrency=96, format=pcm_f32le
  elapsed median=9014.01 ms, range=8907.89–9399.61 ms; all guards passed
  voluntary context switches median=634663
  preprocess_ms median=7.155678; enqueue_ms median=9.395981 (medians of per-repetition frame medians)
  executor submissions median: enqueue=49152, preprocess=49152

And for this PR's head:

Measuring opt: concurrency=64, format=pcm_f32le
  elapsed median=1703.91 ms, range=1670.31–1722.41 ms; all guards passed
  voluntary context switches median=71458
  preprocess_ms median=0.001892; enqueue_ms median=0.002744 (medians of per-repetition frame medians)
  executor submissions median: enqueue=1, preprocess=0
Measuring opt: concurrency=96, format=pcm_f32le
  elapsed median=2307.67 ms, range=2246.74–2341.00 ms; all guards passed
  voluntary context switches median=102967
  preprocess_ms median=0.001904; enqueue_ms median=0.001913 (medians of per-repetition frame medians)
  executor submissions median: enqueue=13, preprocess=0

You can observe the same stark reduction in voluntary context switches and pre-processing latency that I mentioned in the PR description. elapsed median describes per-repetition latency (i.e., concurrency requests), which in this specific benchmark also sees a strong reduction. Note that the benchmark also includes correctness checks for the ordering, content and format of the output PCM frames.

Please do let me know if you'd like me to push the micro-benchmark as well, and if these results address your concerns.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-zcode-glm53flash-100-c10-20261104.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on claude-code (mimo-v2.6-pro) under experiment fleet-strict-glm-mimo-5050-c2-20261105.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-glm-100-c4-await-qwen-20261105.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on claude-code (qwen3.8-max) under experiment fleet-strict-glm-qwen38max-5050-c4-20261105.

[vllm-omni-review-bot]

Omni ReviewBot review status

Review for commit 40916cfdc254c96a320d5e2c08f57652e49fa646 ended as blocked (step 'fetch' (pr.fetch_diff): cannot materialize PR #9491 at 40916cf — refusing to review the live checkout for a pinned request).

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on claude-code (qwen3.8-flash) under experiment fleet-strict-glm-qwen38flash-5050-c4-20261105.