[Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs by yinpeiqi · Pull Request #4527 · vllm-project/vllm-omni

yinpeiqi · 2026-06-18T07:09:54Z

PLEASE FILL IN THE PR DESCRIPTION HERE.

Purpose

Fixes #4469.

This PR does two things:

Separate inter-stage payloads from client-facing multimodal outputs
- Worker runners now partition flat per-request payloads into:
  - inter_stage_outputs: keys intended for stage-to-stage connector transport
  - multimodal_outputs: client-facing keys only
- Partitioning is driven by a small client-facing root-key whitelist in partition_flat_payload().
- OmniARScheduler forwards client-facing data through OmniEngineCoreOutput.multimodal_output, while chunk_transfer_adapter.save_async() now uses inter_stage_output only (no fallback to client-facing mm_output).
- This avoids connector traffic being polluted by client-only fields such as final audio/image outputs, and keeps inter-stage tensors/metadata on the connector path.
Add an optional Orchestrator load monitor
- New CLI flag: --enable-orch-monitor
- When enabled, the orchestrator aggregates 1s windows during the run and writes a single columnar JSON file at shutdown.
- Default output path: ./vllm_omni_orch_monitor_<MMDDHHMM>.json
- Optional path override via VLLM_OMNI_ORCH_MONITOR_PATH
- When disabled, a null monitor is used so there is no runtime overhead.
- Recorded metrics:
  - Orchestrator poll loop: duration_s, loop_idle, loop_active
  - Per stage replica: outputs_queue_size, inflight
- Shutdown also logs a short summary to the server log.

Performance

Benchmark setup:

Sweep request rate (RPS)
Fixed thinker / talker batch size: 32
Concurrency: unlimited

Before this fix

TTFT stays very high across the sweep. This matches the orchestrator bottleneck described in #4469: client-facing multimodal payloads were also flowing through the connector path, increasing orchestrator poll and forwarding overhead.

In this setup, the system is very sensitive to backlog. Once the orchestrator or stage queues start piling up, latency spikes quickly and the run becomes unstable — there is little headroom to recover. Results before this fix were also hard to reproduce: repeated runs at the same settings often diverged sharply, so the chart above should be read as directional rather than a single stable baseline (some run-to-run variance is expected).

The sweep shown here uses 200 requests. When we extend the same workload to 1000 requests, the old path almost always breaks down under accumulated backlog. We did not include those longer runs here because they frequently fail or produce unusable latency profiles.

After this fix

TTFT drops significantly, especially at higher load. Separating inter-stage and client-facing outputs removes unnecessary connector traffic and reduces orchestrator work per poll. Behavior is much more stable across repeated runs, and the same workload scales more predictably when request count increases.

Orchestrator monitor validation

With --enable-orch-monitor, post-run JSON and shutdown summary show the orchestrator is no longer the limiting factor:

Two follow-up runs under load: poll-loop activity stays healthy and the orchestrator does not become saturated again.

Per-replica outputs_queue_size remains low throughout the run. Metrics above were collected from the orchestrator monitor JSON/log output.

Test Plan

vLLM Version:

0.23.0

vLLM-Omni Commit:

a693ae67f6eec6df1d443581857a9c016850a0a2

Test Result

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

Resolve orchestrator shutdown conflict: keep main's single teardown path in run() finally; monitor flush remains there. Co-authored-by: Cursor <cursoragent@cursor.com>

yinpeiqi · 2026-06-18T07:13:14Z

@fake0fan @hsliuustc0106 @Gaohan123 @ZhengWG

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 314f3f269a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-06-18T07:14:00Z

+    {
+        "model_outputs",
+        "sr",
+        "audio",


Keep non-final audio outputs on the connector path

In the Voxtral TTS async pipeline, stage 0 (audio_generation) is non-final and its generator2tokenizer_async_chunk processor reads multimodal_output["audio"] to send acoustic-code chunks to stage 1. Classifying every top-level audio payload as client-facing makes partition_payload_list return no inter_stage_outputs for that stage; the scheduler now calls save_async only for inter_stage_output, so those chunks (and the finish marker path driven by them) never reach audio_tokenizer. This split needs to be stage/final-output aware, or avoid treating inter-stage audio keys as client-only.

Useful? React with 👍 / 👎.

Valid catch for Voxtral: stage-0 "audio" is inter-stage, but partition treats it as client mm.

I’d rather not fix this with root-key heuristics in mm_outputs.py. Proposal: each stage declares at model output time what is inter-stage vs client-facing (e.g. codes.* / hidden_states.* vs final model_outputs/audio), and the runner just forwards the two channels.

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

hsliuustc0106

Performance fix for orchestrator bottleneck: separates inter-stage from client outputs.

hsliuustc0106

Thanks for the performance work here. I found a couple of blocking regressions in the new inter-stage/client split, plus one monitor coverage issue.

Summary:

save_async still needs to run for terminal async-chunk markers even when the inter-stage payload is None.
The new inter_stage_outputs scheduler contract needs NPU producer parity or a compatibility fallback.
The orchestrator monitor should include replicas that register after startup.

hsliuustc0106 · 2026-06-18T19:01:11Z

+        except (AttributeError, NotImplementedError, RuntimeError, ValueError):
+            return 0
+
+    def replica_inflight_count(self, replica_id: int) -> int:


replica_inflight_count has no try/except, unlike its sibling replica_outputs_queue_size above. It's reached from _sample_replica_metrics -> _roll_window -> note_loop, which runs bare on the orchestration loop body (end of _orchestration_loop in orchestrator.py) -- outside the inner try/except. Since the monitor is meant to be safe to enable in production ("diagnostic only / no runtime overhead when disabled"), any future change to ReplicaInfo/StagePool shape would let --enable-orch-monitor kill the orchestration task. Worth wrapping the sampler call (or matching the sibling's defensive pattern) so observability can't take down the orchestrator.

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

hsliuustc0106

lgtm

hsliuustc0106 · 2026-06-28T06:02:29Z

check ci failures https://buildkite.com/vllm/vllm-omni/builds/11855/canvas

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

yinpeiqi · 2026-06-29T05:08:11Z

seems the ci do not triggered again, could you please re-add the tag? @hsliuustc0106

hsliuustc0106

lgtm

… split PR #4527 ("separates inter-stage from client outputs") broke the full-payload inter-stage handoff, starving the downstream stage of every multi-stage TTS/Omni pipeline: the consumer stage waited on connector input that never arrived, tripping the VLLM_OMNI_INPUT_WAIT_TIMEOUT_S safety net (FINISHED_ERROR) and yielding empty audio (transcript='') or request timeouts (Qwen3-TTS Base/CustomVoice exit 124, IndexTTS2, Qwen3-Omni, ...). Root cause (no-async-chunk / full-payload mode) -- VERIFIED: In gpu_ar_model_runner / gpu_generation_model_runner, #4527 changed the send-side accumulation to read from `pooler_inter` / `inter_stage_outputs`, the partition result that is None when async_chunk is False. But `should_accumulate_full_payload_output()` returns False when async_chunk is True. So the two conditions are mutually exclusive -- the accumulate block never ran, and the producer stage sent nothing. Fix: accumulate the full per-request payload (`pooler_output` / `per_req_payloads`), which is exactly the pre-#4527 behavior for this branch. Sibling fix (async-chunk mode): make `partition_flat_payload` non-lossy so the inter-stage fanout keeps every key (a downstream stage can't starve on a value that shares a root with a client-facing output, e.g. model_outputs); the client payload stays trimmed to `_CLIENT_MM_ROOT_KEYS` (the #4527 perf goal). Updates test_mm_outputs_partition.py to the non-lossy contract + regression test. Verification (H200, Qwen3-TTS-12Hz-0.6B-Base, offline no_async_chunk): - buggy code: stage-1 "timed out waiting for chunk/input ... FINISHED_ERROR" (the exact CI signature, reproduced with a shortened timeout) - with this fix: test_text_to_audio_001 PASSED, real audio produced (184s) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: tzhouam <tzhouam@connect.ust.hk>

Integrate upstream main 0899a1a (11 commits since 1b318d1) — orchestrator inter-stage/client output split (vllm-project#4527), diffusion request-level batching (vllm-project#4079), speech SSE default (vllm-project#4679), and assorted model/example migrations. Auto-merged with no conflicts. Signed-off-by: chickeyton <ngton2014@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…tage from client outputs (vllm-project#4527) Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com> Co-authored-by: Chenguang ZHENG <645327136@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk> Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>

…tage from client outputs (vllm-project#4527) Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com> Co-authored-by: Chenguang ZHENG <645327136@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>

yinpeiqi added 7 commits June 16, 2026 19:42

separate mm output and inter output

c932cca

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

revert redundant modify

dacc1cc

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

fix branch

1a8553e

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

update

4ef6319

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

update precommit

48528c6

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

update

6e2371d

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

update to monitor

314f3f2

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

yinpeiqi requested review from Gaohan123, gcanlin, hsliuustc0106, tzhouam, yenuo26 and ywang96 as code owners June 18, 2026 07:09

Merge origin/main into fix-orch.

89c955a

Resolve orchestrator shutdown conflict: keep main's single teardown path in run() finally; monitor flush remains there. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector Bot reviewed Jun 18, 2026

View reviewed changes

yinpeiqi changed the title ~~[Core] Separate inter-stage and client MM outputs; add orchestrator load monitor~~ [WIP] Separate inter-stage and client MM outputs; add orchestrator load monitor Jun 18, 2026

fix precommit

02af9a7

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

yinpeiqi force-pushed the fix-orch branch from 1c51b3c to 02af9a7 Compare June 18, 2026 07:42

update voxtral tts

1ed8b8e

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

yinpeiqi force-pushed the fix-orch branch from 064891b to 1ed8b8e Compare June 18, 2026 08:08

yinpeiqi requested review from ZeldaHuang, linyueqian, princepride and yuanheng-zhao as code owners June 18, 2026 08:08

Merge branch 'main' into fix-orch

366c5e0

hsliuustc0106 reviewed Jun 18, 2026

View reviewed changes

hsliuustc0106 added the core related to core module: cache, scheduler, engine, worker, modelrunner label Jun 18, 2026

hsliuustc0106 requested changes Jun 18, 2026

View reviewed changes

Comment thread vllm_omni/core/sched/omni_ar_scheduler.py Outdated

Comment thread vllm_omni/outputs.py

Comment thread vllm_omni/engine/orchestrator_monitor.py

hsliuustc0106 reviewed Jun 18, 2026

View reviewed changes

yinpeiqi and others added 4 commits June 25, 2026 13:30

update name

8ba09ff

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

add try catch

7062510

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

fix

63426df

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

Merge branch 'main' into fix-orch

87b2a45

hsliuustc0106 approved these changes Jun 28, 2026

View reviewed changes

hsliuustc0106 added ready label to trigger buildkite CI merge-test label to trigger buildkite merge test CI and removed ready label to trigger buildkite CI labels Jun 28, 2026

yinpeiqi and others added 2 commits June 28, 2026 16:35

fix ci

4601afb

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

Merge branch 'main' into fix-orch

17d8881

hsliuustc0106 added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI merge-test label to trigger buildkite merge test CI labels Jun 28, 2026

fix

5cec9c3

Signed-off-by: yinpeiqi <yinpeiqi809@gmail.com>

Merge branch 'main' into fix-orch

96ce61a

hsliuustc0106 added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Jun 29, 2026

hsliuustc0106 approved these changes Jun 29, 2026

View reviewed changes

hsliuustc0106 merged commit 504698d into vllm-project:main Jun 29, 2026
8 of 9 checks passed

wjinxu mentioned this pull request Jun 30, 2026

[Bug]: async_chunk: false multi-stage pipelines hang #4793

Closed

linyueqian mentioned this pull request Jul 8, 2026

Fix Voxtral TTS feedback and short ASR checks #4954

Merged

linyueqian mentioned this pull request Jul 24, 2026

[perf] Minicpm o4.5 optimization #5228

Merged

linyueqian mentioned this pull request Aug 11, 2026

[CI] Refresh Voxtral-4B-TTS perf baselines from a clean window #6032

Merged

vllm-omni-review-bot mentioned this pull request Oct 10, 2026

[Bugfix] Fix full-payload accumulation across preemption and aborts #7681

Open

Navigation Menu

Repository navigation

[Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs - #4527

[Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs#4527
hsliuustc0106 merged 33 commits into
vllm-project:mainfrom
yinpeiqi:fix-orch

yinpeiqi commented Jun 18, 2026

Uh oh!

yinpeiqi commented Jun 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Uh oh!

yinpeiqi Jun 18, 2026

Uh oh!

hsliuustc0106 left a comment

Uh oh!

hsliuustc0106 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hsliuustc0106 Jun 18, 2026

Uh oh!

hsliuustc0106 left a comment

Uh oh!

hsliuustc0106 commented Jun 28, 2026

Uh oh!

yinpeiqi commented Jun 29, 2026

Uh oh!

hsliuustc0106 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

yinpeiqi commented Jun 18, 2026

Purpose

Performance

Before this fix

After this fix

Orchestrator monitor validation

Test Plan

Test Result

Uh oh!

yinpeiqi commented Jun 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

yinpeiqi Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hsliuustc0106 Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 commented Jun 28, 2026

Uh oh!

yinpeiqi commented Jun 29, 2026

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants