[Refactor] Output Processor Phase 2: separate multimodal output channel (#1601)

[meghaagr13]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Add multimodal output field instead of pooling output to avoid collisions in two. Resovles Phase 2 of #1601

Test Plan

python -m pytest tests/e2e/offline_inference/test_cosyvoice3.py -v

Test Result

tests/e2e/.../test_cosyvoice3.py::...[cosyvoice3.yaml] PASSED
tests/e2e/.../test_cosyvoice3.py::...[cosyvoice3_async_chunk.yaml] PASSED
================== 2 passed, 18 warnings in 167.05s ==================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[hsliuustc0106]

PR #2744 - separate multimodal output channel (#1601)

OVERALL: 2 BLOCKERS FOUND
VERDICT: REQUEST_CHANGES

BLOCKER scan:

• Correctness: PASS
• Reliability/Safety: PASS
• Breaking Changes: PASS
• Test Coverage: needs tests (test plan present, results empty)
• Documentation: BLOCKING - PR body empty (template not filled)
• Security: PASS

BLOCKING ISSUES:

1. [Documentation] PR description empty - fill template with motivation, changes, test plan
2. [Test Coverage] Test result section empty - provide validation evidence

Fix DCO and pre-commit gate failures, fill description, and provide test results.

[hsliuustc0106]

BLOCKER scan:

This PR has multiple issues:

1. Merge conflict
2. Failing pre-commit checks
3. DCO sign-off required (ACTION_REQUIRED)

Please resolve all these issues before requesting review:

1. Rebase against main to resolve conflicts
2. Run pre-commit run --all-files and fix issues
3. Commit with --signoff or configure git to sign-off automatically

OVERALL: MULTIPLE ISSUES

VERDICT: REQUEST_CHANGES

[tzhouam]

Great, nice to see the phase 2 get started. If need any help, please inform me on Slack : )

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[meghaagr13]

@tzhouam pls take a look.

[tzhouam]

please have a look.
added a review from gpt.

[meghaagr13]

@tzhouam pls take a look

[tzhouam]

looks fine to me, but please try to solve the CI failures. You may refer to this report:

PR #2744 Buildkite CI Failure Debug Guide

PR: [Refactor] Output Processor Phase 2: separate multimodal output channel (#1601)
Failing commit: 4341d330800c15403953af84aeb977a645771f53 (branch meghaagr13:feature/1601-phase2-fresh)
Environment: vLLM 0.22.1rc1.dev263+g967c5c3bc (Dockerfile.ci wheel commit 967c5c3b)
Investigation date: 2026-06-10

---

1. Failure Overview

All three Buildkite pipelines failed, with 12 failed jobs in total:

Pipeline	Build	Failed Job	Root Cause
vllm-omni	#11139	Engine&Model Executor Test	Root cause B
		Omni · Qwen3-Omni Test	Root cause A
		Omni · Qwen3-Omni Realtime WebSocket	Root cause A
		Omni · VoxCPM2 Test	Root cause B
		TTS · VoxCPM2 Online Test	Root cause B
		TTS · Qwen3-TTS CustomVoice Test	Root cause A
		TTS · Qwen3-TTS Base Test	Root cause A
vllm-omni-amd-ci	#8560	mi325_1: Engine / Qwen3-TTS / Voxtral-TTS / CosyVoice3-TTS	A + B (same signatures)
vllm-omni-intel-ci	#5040	Intel: XPU Test	A (confirmed in logs)

Key takeaway: CUDA, ROCm, and XPU all show the same error signatures. This is not a platform issue — it is two code bugs introduced by the PR.

The visible symptoms are OmniEngineDeadError and openai.APIConnectionError: Connection error. Those are downstream effects of stage engine process crashes. The real root cause is in the StageEngineCoreProc encountered a fatal error traceback.

---

2. Root Cause A: OmniRequest Has No num_cached_tokens Attribute

Error Signature

File "/workdir/vllm_omni/core/sched/omni_generation_scheduler.py", line 547, in update_from_output
    num_cached = request.num_cached_tokens
AttributeError: 'OmniRequest' object has no attribute 'num_cached_tokens'. Did you mean: 'num_computed_tokens'?

Affected tests (generation stage — talker/code2wav crashes → API server connection failure):

• tests/e2e/online_serving/test_qwen3_omni.py (4 cases)
• tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py::test_streaming_audio_input_pcm_output_async_chunk_smoke
• tests/e2e/online_serving/test_qwen3_tts_customvoice.py / test_qwen3_tts_base.py
• AMD Qwen3-TTS / Voxtral-TTS / CosyVoice3-TTS E2E, Intel XPU Test

Analysis

On the PR branch, vllm_omni/core/sched/omni_generation_scheduler.py lines 547–550:

num_cached = request.num_cached_tokens
if num_cached < 0:
    logger.warning("Negative num_cached_tokens (%d) for request %s, clamping to 0", num_cached, req_id)
    num_cached = 0

Three problems:

1. num_cached is computed but never used — it was originally meant for the old OmniEngineCoreOutput(num_cached_tokens=...) field. That parameter was removed during upstream merges, leaving a dangling attribute access (dead code).
2. In the aligned vLLM 0.22 release, vllm.v1.request.Request no longer has a num_cached_tokens field (stats moved to prefill_stats; see prefill_stats=request.take_prefill_stats() on origin/main at the same location).
3. origin/main's omni_generation_scheduler.py has no num_cached code at all — this is merge-conflict residue from repeated upstream merges (the reviewer asked about this clamp in review item Add AR and Diffusion GPU Model Runners to vllm-omni (vLLM v1-Compatible) #4 on 5/19).

Fix

Delete the 4 dead lines (547–550) in omni_generation_scheduler.py on the PR branch. Then grep globally:

git grep -n "num_cached_tokens" -- vllm_omni/ tests/

Note: the PR also added num_cached_tokens to the _HashableRequest mock in tests/distributed/omni_connectors/test_chunk_transfer_adapter.py. If production code no longer reads it, clean up that mock field too.

---

3. Root Cause B: _build_multimodal_outputs Does Not Handle hidden_states_cpu=None

Error Signature

File "/workdir/vllm_omni/worker/gpu_ar_model_runner.py", line 1269, in sample_tokens
File "/workdir/vllm_omni/worker/gpu_ar_model_runner.py", line 912, in _build_multimodal_outputs
File "/workdir/vllm_omni/worker/gpu_ar_model_runner.py", line 884, in _resolve_req_hidden_states
    return hidden_states_cpu[start:end]
TypeError: 'NoneType' object is not subscriptable

Affected tests (AR stage crash):

• tests/engine/test_async_omni_engine_abort.py::test_abort (Qwen2.5-Omni thinker)
• tests/e2e/offline_inference/test_voxcpm2.py::test_voxcpm2_zero_shot_001 / test_voxcpm2_prefill_decode_mixed_batch_003
• tests/e2e/online_serving/test_voxcpm2_tts.py::test_text_to_audio_001
• Intel test_qwen2_5_omni_expansion.py, AMD Engine Test

Analysis

The PR added a dedicated multimodal channel builder _build_multimodal_outputs() (gpu_ar_model_runner.py ~line 867). It is called unconditionally at the end of sample_tokens() (~line 1270), except when engine_output_type == "text". However, hidden_states_cpu can legitimately be None in three paths (see assignment logic in sample_tokens() ~lines 1113–1133):

Path	Condition	Old code (pooler payload loop) handling
Sparse audio output	audio_sparse_output == True (VoxCPM2 and other TTS models)	Uses sparse_mm_index per request; does not slice hidden_states
Partial downstream requests	len(downstream_req_ids) != len(req_ids_output_copy) (Qwen2.5-Omni thinker mixed batch)	Uses per-request dict req_hidden_states_cpu
No hidden payload needed	needs_scheduled_hidden_payload == False	Does not build hidden at all

Meanwhile, _resolve_req_hidden_states() (~lines 870–885) does hidden_states_cpu[start:end] when combined_hidden_states is None (prefix cache disabled — CI uses enable_prefix_caching=False) → slicing None crashes.

The function signature declares hidden_states_cpu: torch.Tensor as non-optional, but call sites can pass None — the type annotation hides the bug.

Fix

Align _build_multimodal_outputs() with the old pooler-payload loop (~lines 1140–1260):

1. Pass in req_hidden_states_cpu (per-request dict), audio_sparse_output, sparse_mm_index, and downstream_req_id_set.
2. Loop over downstream_req_ids only, not all req_ids.
3. Hidden-state resolution priority: combined_hidden_states[rid] → req_hidden_states_cpu[rid] → hidden_states_cpu[start:end]. When none apply (sparse audio path), omit the "hidden" key and only include mm payload.
4. Change _resolve_req_hidden_states's hidden_states_cpu parameter to torch.Tensor | None and handle None explicitly.

---

4. Local Reproduction

4.1 Check Out PR Code (Without Polluting Your Working Branch)

cd /rebase/vllm-omni
git fetch origin pull/2744/head:refs/remotes/pr/2744   # already done
git worktree add /tmp/pr2744 pr/2744                   # isolate with worktree

4.2 Fastest Repro (No GPU Model, Pure Logic)

Root cause A can be verified in one line (any environment with aligned vLLM):

python3 -c "from vllm.v1.request import Request; import inspect; \
  print('num_cached_tokens' in inspect.getsource(Request.__init__))"   # False → field removed

4.3 GPU Repro (CI-Equivalent Image)

docker run --gpus all -it --rm -v /tmp/pr2744:/workdir -w /workdir \
  $(grep -oP 'FROM \K\S+' docker/Dockerfile.ci | head -1) bash

pip install -e . --no-deps

# Root cause B — crashes in ~2 minutes:
pytest tests/e2e/offline_inference/test_voxcpm2.py::test_voxcpm2_zero_shot_001 -v
pytest tests/engine/test_async_omni_engine_abort.py::test_abort -v

# Root cause A — requires Qwen3-TTS weights:
pytest tests/e2e/online_serving/test_qwen3_tts_base.py -v -k async_chunk

4.4 Verify the Fix

After applying fixes, run regression tests by impact area:

# Root cause A regression
pytest tests/e2e/online_serving/test_qwen3_tts_base.py tests/e2e/online_serving/test_qwen3_tts_customvoice.py -v
# Root cause B regression
pytest tests/e2e/offline_inference/test_voxcpm2.py tests/engine/test_async_omni_engine_abort.py -v
# PR unit tests should not regress
pytest tests/engine/test_ensure_tensor_values.py tests/engine/test_wire_multimodal_output.py tests/engine/test_output_processor.py -v

Once all pass, push and ask a maintainer to re-add the ready label to trigger Buildkite (that label was removed on 6/10).

---

5. How to Fetch Buildkite Logs Yourself

When CI fails, follow this workflow. Token is in /rebase/vllm-omni-rebase-agent/.env:

export BUILDKITE_API_TOKEN=<token>

# 1. Find build numbers for the three pipelines from the PR
gh pr view 2744 --repo vllm-project/vllm-omni --json statusCheckRollup

# 2. List failed jobs and their IDs
curl -s -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
  "https://api.buildkite.com/v2/organizations/vllm/pipelines/vllm-omni/builds/11139" \
  | python3 -c "import json,sys; [print(j['state'],j['name'],j['id']) \
      for j in json.load(sys.stdin)['jobs'] if j.get('state')=='failed']"

# 3. Download a single job log
curl -s -H "Authorization: Bearer $BUILDKITE_API_TOKEN" \
  "https://api.buildkite.com/v2/organizations/vllm/pipelines/vllm-omni/builds/11139/jobs/<JOB_ID>/log" \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['content'])" > job.log

# 4. Skip surface symptoms; go straight to the engine fatal traceback
grep -E "stage_engine_core_proc.py:188" job.log | tail -30

Tip: OmniEngineDeadError, APIConnectionError, and StagePool _poll_stage_raw failed are all downstream symptoms. The real root cause is always at the bottom of the StageEngineCoreProc encountered a fatal error stack trace.

---

6. Appendix: Failed Test → Root Cause Mapping

Test	Root Cause	Crashing Stage
test_async_omni_engine_abort.py::test_abort	B	Qwen2.5-Omni thinker (LLM_AR)
test_qwen3_omni.py::test_* (4 cases)	A	stage-2 code2wav (LLM_GENERATION)
test_qwen3_omni_realtime_websocket.py::*async_chunk_smoke	A	stage-2 code2wav
test_voxcpm2.py::test_voxcpm2_zero_shot_001 etc.	B	VoxCPM2 talker (LLM_AR, sparse audio output)
test_voxcpm2_tts.py::test_text_to_audio_001	B	Same as above (online)
test_qwen3_tts_base.py / test_qwen3_tts_customvoice.py	A	code2wav (LLM_GENERATION)
AMD mi325_1: *, Intel XPU Test	A + B	Same as above (cross-platform, same signatures)

[tzhouam]

@meghaagr13 Any help needed?