[leegangtoe]

Purpose

Fixes #7295.

Fix repeated multimodal requests when one stage-0 frontend serves multiple engine replicas.
The IPC sender cache omits tensors on a frontend hit. A request routed to a replica that has
never received those tensors then ends with finish_reason="error" and empty content, although
HTTP status is 200. With two replicas and round-robin routing, sending the same new audio three
times reproduces stop, error, stop.

Use vLLM's MultiModalProcessorOnlyCache on the shared frontend for replicated stage 0.
Select it at startup for distributed runtimes as well, including those initially holding zero
or one replica, because headless replicas can join later. This preserves cached processing
results while always supplying tensors to the selected receiver. The OpenAI chat path is
already rendered before engine routing, so raw multi_modal_uuids scoping alone cannot fix it.

Local single-replica deployments, disabled caches, existing processor-only caches, and
text-only renderers are unchanged. LRU/SHM sender caches are replaced. Receiver caches remain;
the tradeoff is additional frontend tensor storage and tensor IPC on each request.

Test Plan

vLLM Version: 0.28.0; PyTorch 2.13.0+cu130, CUDA runtime 13.0, Transformers 5.14.1.

vLLM-Omni Commit: d3a86d70c70ae2e5b4559d56232e951689f1ec22; base main 5e4baea64d0f4cd1b6ce147e4f7a28f0f41d995f.

• L1 CPU: actual LRU and SHM caches with 1/2/3 receivers, receiver eviction, disabled and
  processor-only caches, text-only renderers, newly joined receivers, and engine-to-builder
  wiring for static and dynamic deployments. Only model/tokenizer setup is mocked.
• Fixed-replica GPU serving: Qwen3-Omni-30B-A3B-Instruct thinker-only on two H200 GPUs,
  TP=1, GPU memory utilization 0.60, prefix caching enabled, eager execution disabled.
• Dynamic serving: start the head with one replica, warm the frontend audio cache, attach
  another headless replica, then resend byte-identical audio with round-robin routing.
  Use --disable-log-stats to isolate cache behavior from the separate metrics failure below.

CPU commands, from the repository root (requires compatible vLLM dependencies, pytest and
pytest-mock; no model weights or GPU execution needed):

python -m pytest -q -o addopts='' tests/engine/test_async_omni_engine_stage_init.py
python -m pytest -q -o addopts='' tests/engine/test_async_omni_engine_stage_init.py -m 'core_model and cpu'

The existing Buildkite L1 engine CPU sweep already collects this module; no pipeline change.

Minimal serving reproduction: save the following as /tmp/omni-replicas.yaml:

pipeline: qwen3_omni_moe_thinker_only
distributed_executor_backend: mp
enable_prefix_caching: true
stages:
  - stage_id: 0
    devices: "0,1"
    num_replicas: 2
    tensor_parallel_size: 1
    gpu_memory_utilization: 0.60
    max_num_seqs: 64
    enforce_eager: false
    async_scheduling: false

CUDA_VISIBLE_DEVICES=0,1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni --port 8099 --deploy-config /tmp/omni-replicas.yaml

After /health returns 200, run this against a fresh server using a short WAV file:

import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

url = "http://127.0.0.1:8099"
with urlopen(url + "/v1/models") as response:
    model = json.load(response)["data"][0]["id"]
audio = base64.b64encode(Path("sample.wav").read_bytes()).decode()
body = {"model": model, "modalities": ["text"], "temperature": 0, "max_tokens": 128,
        "messages": [{"role": "user", "content": [
            {"type": "audio_url", "audio_url": {"url": "data:audio/wav;base64," + audio}},
            {"type": "text", "text": "Transcribe the audio briefly."}]}]}
for _ in range(3):
    request = Request(url + "/v1/chat/completions", data=json.dumps(body).encode(),
                      headers={"Content-Type": "application/json"})
    with urlopen(request, timeout=120) as response:
        choice = json.load(response)["choices"][0]
    print(choice["finish_reason"], len(choice["message"].get("content") or ""))

Test Result

• CPU module: 72 passed.

• Full changed-file pre-commit, including mypy-3.10: passed.

• Negative controls: disabling cache replacement makes the eviction regression fail with
  MultiModalCacheMissError; dropping the dynamic flag makes the engine wiring test fail.

• Before fix: 10 errors in 28 HTTP requests, 10 cache-miss warnings. Three fresh clips worked;
  repeated C returned error, stop, error and repeated D returned error, stop.

• Revised fixed-replica serving: 76/76 requests passed, including 64 repeated audio requests
  at concurrency 8; zero cache misses.

• Dynamic serving with log stats disabled: 12/12 requests passed before joining, then
  76/76 passed after joining, including 64 concurrent repeated audio requests; zero cache misses.

• With default log stats, joining succeeds but processing the first remote output raises
  KeyError: (0, 1) at orchestrator.py's _stage_replica_to_engine_idx lookup. This mapping
  contains only initial replicas (tracked in [Bug]: Adding a headless replica crashes output processing with log stats enabled #7296). That file is unchanged from the base; this separate issue
  requires its own fix. Default-stats dynamic serving is not validated as working.

SHM is covered by CPU cache tests; the GPU serving runs use LRU. These are correctness checks,
not throughput or general transcription-accuracy benchmarks.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/observability.md, docs/design/module/engine_orchestration.md.

Module owners: @lishunyang12 @vraiti @fake0fan

Routing: @lishunyang12 via module of the changed files; @vraiti via module of the changed files; @fake0fan via semantic router, CODEOWNERS

@leegangtoe, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit d3a86d70c70a produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[leegangtoe]

Self-review completed on head d3a86d7:

• Reviewed the complete three-file diff against the base and matched the committed files to the source hashes from the validation runs.
• Traced the OpenAI chat path through rendering and replica routing: frontend cache hits can remove tensors before routing, so raw UUID scoping cannot repair the already-rendered request. The replacement retains preprocessing results while supplying tensors to each receiver.
• Checked startup selection for static replicas and distributed runtimes with 0/1 initial replicas. single_stage_mode selects the dynamic case before the membership controller exists. Local single-replica, disabled-cache, processor-only, and text-only paths remain unchanged.
• Reviewed the real LRU/SHM regression tests for repeated hits, receiver eviction, late joins, and engine-to-builder wiring. The module passed 72 tests; the negative controls fail when cache replacement or dynamic-flag propagation is removed. Full changed-file pre-commit, including mypy, passed.
• Rechecked the two-H200 results: the baseline had 10 errors in 28 requests; the fix passed 76/76 fixed-replica requests with zero cache misses. Dynamic serving passed 12/12 before joining and 76/76 afterward with --disable-log-stats, isolating the separate failure in [Bug]: Adding a headless replica crashes output processing with log stats enabled #7296 addressed by [Bugfix] Register statistics for dynamically added replicas #7298.
• Checked the tradeoff and limits: frontend tensor storage and per-request IPC increase; SHM has CPU coverage only, and these runs do not establish throughput, transcription accuracy, or multi-node correctness. GitHub pre-commit, both wheel builds, DCO, and docs checks are passing.

[leegangtoe]

@Gaohan123 Thanks for merging #7298!
Could you clarify why #7297 was closed?
These address separate bugs: #7298 fixes statistics registration for dynamically added replicas (#7296),
while #7297 fixes missing multimodal tensors across stage-0 replicas (#7295).
Their changed files do not overlap, and the cache fix is not included in #7298.
Was this closure intentional, or could we reopen #7297 for review?

[armaanamatya]

@Gaohan123 I think this may have been closed unintentionally. It closed 3 seconds after #7298 merged, and #7298's description contains "multimodal cache fix #7297", which GitHub treats as a closing keyword.

From reading current main, #7295 still looks present. The chat path renders through the shared frontend sender cache before a replica is chosen, and _scope_stage0_multimodal_cache_to_replica skips rendered prompts because they have no multi_modal_data. I have not reproduced it on GPU.

My #8062 touches the raw-prompt scoping path only and does not fix this, so the two are complementary. This PR would need a rebase (_initialize_stages moved to omni_engine_base.py). Could it be reopened for review?