[Core] Make async output reads event-driven; signal engine death via queue shutdown by zeningc · Pull Request #4177 · vllm-project/vllm-omni

zeningc · 2026-06-05T08:10:34Z

Summary

Replace the busy-poll / drain-thread read of the engine output queue with a
plain event-driven await on the queue's async side, and route orchestrator
death through the queue via shutdown(). The motivation is async correctness,
a cleaner death/hang-safety contract, and lower idle CPU.

Rebased onto current main (vLLM 0.30.0). Since this PR was opened,
#5221 landed a flag-gated event-driven orchestration loop plus a
drain-thread workaround for the serving-side final-output drain
(get_output_blocking_async, gated by VLLM_OMNI_EVENT_DRIVEN_ORCH), and
#7413 moved the engine body into OmniEngineBase / AsyncOmniBase. This PR
now sits on top of both: the orchestration-loop flag is untouched, while the
final-output drain becomes event-driven unconditionally and the drain thread
goes away. See "Relationship to #5221" below.

Background

The Orchestrator drives the multi-stage pipeline in its own background thread
with its own asyncio loop. The server (OmniEngineBase plus the API layer)
runs on the main asyncio loop.

They communicate over janus queues, which expose one underlying queue as a
sync_q (for thread code) and an async_q (for event-loop code). The rule:
whichever side waits for the next message should await async_q, and the side
running in a thread should push with sync_q.put_nowait.

The request queue (server → orchestrator) already follows this: the
orchestrator is the consumer and awaits async_q on its loop, while the server
pushes with sync_q.put_nowait. The output queue (orchestrator → server:
per-request tokens / audio chunks, the one this PR touches) is the mirror image,
with the server as the consumer. It should await async_q too, but the
pre-change code had the orchestrator awaiting the async side, which left the
server unable to await the queue at all.

Why

The output reader is a hot path that every generated token/audio chunk flows
through. Two reasons to change how it works:

The server can await instead of poll (or thread-hop). A janus queue's
async_q attaches to a single event loop (the first one to use it) and can
only be touched from that loop. The old code had the orchestrator do
await async_q.put(), which attached async_q to the orchestrator's loop.
The server runs on a different loop, so it could not await the queue at
all. Legacy path: poll sync_q.get_nowait() with a 1 ms sleep. [Perf][Engine] Event-driven orchestration loop (opt-in) — S1 of #4855 #5221 path:
block on sync_q.get(timeout=1) inside a dedicated executor thread. Both
only exist because the queue was bound to the wrong loop. Now the
orchestrator pushes with sync_q.put_nowait(), which attaches nothing, so
async_q attaches to the server's loop the first time the server awaits it,
and the server reads with a plain await async_q.get().
Idle cost at fleet scale. The poll loop wakes ~900×/s and burns ~5% of a
core per engine while idle. Parking on get() drops it to ~0, with no extra
thread per engine.

Design / what changed

 BEFORE (legacy / #5221)                 AFTER: event-driven
 ───────────────────────                 ───────────────────
 orch thread     server loop             orch thread     server loop
 (orch loop)     (consumer)              (orch loop)     (consumer)
     │               │                       │               │
 await async_q.put   │ poll get_nowait   sync_q.put_nowait    │ await async_q.get()
     │               │ + sleep 1ms, or       │               ▲  (parks until a msg)
     ▼               │ sync_q.get(1s) in     ▼               │
 ┌─────────────┐     │ a drain thread    ┌─────────────┐     │
 │ janus queue │◄────┤                   │ janus queue │─────┘
 └─────────────┘     │                   └─────────────┘
 async_q bound to    │                   async_q bound to
 ORCH loop ──────────┘                   CONSUMER loop (awaits)
 (consumer can't await it)               put_nowait needs no loop

A parked await get() cannot notice the engine dying on its own, so the
orchestrator thread signals through the queue on any exit (Exception,
BaseException, or clean return):

 sync_q.put_nowait(ErrorMessage(fatal=True))   # crash only, delivered first
 output_queue.shutdown()                        # drains queued msgs, then
                                                #   parked get() -> QueueShutDown
                                                #   -> RuntimeError -> clean teardown

The RPC output queue is unchanged: a crash still enqueues a fatal
ErrorMessage there, which CorrelatedRpcClient's router already broadcasts
to waiters.

Files:

engine/omni_engine_base.py: try_get_output_async parks on async_q.get()
and maps a shut-down queue to the existing RuntimeError; try_get_output
(sync) maps it too; _bootstrap_orchestrator enqueues the fatal message and
shutdown()s the output queue in finally; get_output_blocking_async and
its lazily created drain executor are removed.
engine/orchestrator.py: pushes outputs with put_nowait on the sync side;
param/attribute renamed output_async_queue → output_sync_queue, typed
janus.SyncQueue, contrasting with the still-async request_async_queue /
rpc_async_queue. duplex_orchestrator.py / duplex/session/manager.py
follow the rename (the session manager already preferred put_nowait).
engine/membership_controller.py: pushes with put_nowait; type narrowed.
entrypoints/async_omni_base.py: the final-output drain is a single
await engine.try_get_output_async(); the None/sleep branch, the
VLLM_OMNI_EVENT_DRIVEN_ORCH drain branch, and both interval constants are
gone.
docs: VLLM_OMNI_EVENT_DRIVEN_ORCH now documents only the orchestration
loop; the drain is always event-driven.
tests: engine output tests over a real janus queue (event-driven wakeup,
fatal-drain-then-raise, shutdown wakes a parked getter, bootstrap death
signaling on Exception and BaseException); orchestrator fixtures pass the
sync side; the [Perf][Engine] Event-driven orchestration loop (opt-in) — S1 of #4855 #5221 drain-thread unit tests are removed with the method;
fake engines in entrypoint/streaming tests park instead of returning None.
benchmarks/engine/: standalone queue-handoff microbenchmark.

Relationship to #5221

#5221 fixed the orchestrator-side poll (one reader task per stage replica) and
is kept as is, flag and all. For the serving-side drain it could not await the
queue because of the loop-binding problem above, so it added a drain thread
gated by the same flag. This PR removes the need for that thread by fixing the
queue direction, so the drain is event-driven for every pipeline with no flag
and no extra thread. _orchestration_loop_event_driven and its parity suite
are untouched.

Testing

Re-run on vLLM 0.30.0 after merging main at 6daf5b30f.

Unit tests (all of tests/engine/, including #5221's event-driven parity
suite, plus the entrypoint, streaming-interaction, and stat-logger suites that
touch the output queue):

raw

$ .venv/bin/python -m pytest tests/engine/ tests/entrypoints/test_omni_entrypoints.py \
    tests/diffusion/streaming_interactive_generation/ tests/metrics/test_stat_logger.py -q
1450 passed, 2 skipped, 27 warnings in 218.38s (0:03:38)

pre-commit on every changed file: ruff, format, typos, markdownlint, SPDX,
CI-marks and forbidden-imports hooks pass. The mypy-3.10 hook reports the
same 14 pre-existing errors on main for these files (verified on a clean
checkout; none are on changed lines).

Microbenchmark: isolates the queue handoff (producer thread → async consumer),
no model in the loop:

raw

$ .venv/bin/python benchmarks/engine/queue_handoff_latency.py --runs 3
poll interval = 1.0 ms, messages/run = 3000, runs = 3, max inter-message gap = 2.0 ms

== per-message wakeup latency (producer thread -> async consumer) ==
  OLD poll : mean=  581.2us  p50=  583.8us  p99= 1105.5us  max= 1307.2us
  NEW await: mean=  153.9us  p50=  137.8us  p99=  419.1us  max=  649.0us

== idle cost (no messages flowing) ==
  OLD poll : wakeups=  1777 (  888.5/s)  cpu= 129.9 ms over 2s idle
  NEW await: wakeups=     1 (    0.5/s)  cpu=   0.4 ms over 2s idle

Serving smoke on this branch (Qwen3-TTS 0.6B, 1× RTX 4090, vLLM 0.30.0,
/v1/audio/speech, stream:true, pcm):

raw

$ vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --omni --port 8091
$ curl -s -X POST http://localhost:8091/v1/audio/speech -H "Content-Type: application/json" \
    -d '{"model":"Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice",
         "input":"Event driven output queue smoke test on vLLM zero point thirty.","voice":"vivian",
         "language":"English","stream":true,"response_format":"pcm"}' \
    --output smoke.sse -w "HTTP %{http_code} time_total=%{time_total}s bytes=%{size_download}\n"
HTTP 200 time_total=2.563356s bytes=364096
# 4 speech.audio.delta + 1 speech.audio.done; base64 payloads decode to
# 272640 PCM bytes (~5.68s @ 24 kHz 16-bit mono).

# 32 concurrent streaming requests: 32 x HTTP 200, no server-side errors.
# SIGTERM: server exits cleanly in ~6s, 0 tracebacks / ERROR lines in the log
# (exercises the clean-exit queue shutdown -> drain-loop teardown path).

End-to-end A/B (Qwen3-TTS 0.6B, 1× RTX 4090, 3 × 200 = 600 streaming requests/arm
per cell) from the earlier vLLM 0.24.0 run, confirming no regression: every pooled
TTFP/E2E mean sits within ±1.3%. Not re-run on 0.30.0; the mechanism under test
(queue read on the serving side) is unchanged by the rebase.

raw (baseline = 1 ms poll, candidate = event-driven; vLLM 0.24.0)

Client: N concurrent streaming POSTs to /v1/audio/speech (stream:true, pcm);
TTFP = wall time to first PCM byte, E2E = to last byte. Fixed 8-sentence prompt
set, warmup 6, 3 runs × 200 requests (600 pooled) at concurrency 1 / 8 / 16.
Means below are pooled over the full 600 samples/cell.

conc metric       baseline  candidate     delta    change
---------------------------------------------------------
   1 TTFP mean        43.5       43.2      -0.2     -0.6%
   1 E2E mean        611.4      609.5      -1.9     -0.3%
   8 TTFP mean        93.1       92.0      -1.1     -1.1%
   8 E2E mean       1052.5     1046.1      -6.3     -0.6%
  16 TTFP mean       714.9      724.3      +9.4     +1.3%
  16 E2E mean       1794.6     1803.2      +8.6     +0.5%

(0 errors on either arm. Both arms identical except the queue read mechanism:
baseline polls sync_q.get_nowait() + 1 ms sleep, candidate parks on
async_q.get(). p99 tails stay within a few ms of each other and are omitted;
at this sub-second, compute-bound scale they're dominated by model time.)

Note

The serving numbers do not justify this on their own; inference is compute-bound
and the gain hides under model time. Merge it for the async correctness (the poll
loop and the drain thread were both workarounds for binding the queue to the
wrong loop), the cleaner death/hang-safety contract, and the idle-CPU saving
that grows with fleet size.

chatgpt-codex-connector · 2026-06-05T08:10:42Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

zeningc · 2026-06-06T19:58:13Z

@tzhouam would you mind taking a look when you get a chance? This PR replaces the 1ms busy-poll on the output queue with an event-driven await async_q.get(), and routes engine death through queue.shutdown(). All CI is green and unit tests pass. Thanks!

hsliuustc0106 · 2026-06-06T21:49:10Z

The queue-direction fix is correct and the tests look good.

One minor convention note: the PR title uses [Engine] which isn't in the documented prefix list. Should be [Core] per CLAUDE.md ([Core] — for changes in core logic like OmniProcessor, OmniARScheduler, etc.).

hsliuustc0106 · 2026-06-06T21:49:27Z

                if self.rpc_output_queue is not None:
                    self.rpc_output_queue.sync_q.put_nowait(error_msg)
+                    self.rpc_output_queue.shutdown()
            except Exception:


The single except Exception: pass covers both queue blocks. If output_queue.shutdown() raises (e.g. queue already shut down), the rpc_output_queue block is skipped entirely, so its put_nowait+shutdown() never run. Consider splitting into two independent try/except blocks so a failure on one queue does not skip the other.

thanks for pointing it out! I just updated this part

zeningc · 2026-06-06T22:29:40Z

The queue-direction fix is correct and the tests look good.

One minor convention note: the PR title uses [Engine] which isn't in the documented prefix list. Should be [Core] per CLAUDE.md ([Core] — for changes in core logic like OmniProcessor, OmniARScheduler, etc.).

got it! updated the title, thanks!

zeningc · 2026-06-12T03:55:34Z

Hi @hsliuustc0106 , both review comments have been addressed, would you mind taking another look when you get a chance? Thanks!

…queue shutdown Replace the 1ms busy-poll on the engine output queue with an event-driven await. The async consumer (`_final_output_loop`) read the queue via `sync_q.get_nowait()`, returned `None` on empty, and slept 1ms before retrying. That added up to one sleep-interval of latency per message and burned idle CPU. It now parks on `async_q.get()` and wakes when a message arrives. To make this correct and hang-free: - `_bootstrap_orchestrator` hands the Orchestrator the queue's *sync* side, and it pushes outputs with `put_nowait()` (it never awaits output), so `async_q` binds to the consumer loop that awaits it. Renamed the param/attr `output_async_queue` -> `output_sync_queue` and narrowed the type to `janus.SyncQueue`, which contrasts with the still-async `request_async_queue` / `rpc_async_queue`. - On a crash, `_bootstrap_orchestrator` enqueues the fatal `ErrorMessage` and then calls `output_queue.shutdown()`. With `immediate=False`, shutdown drains the fatal message first (so the caller still sees `fatal=True`), then wakes any still-parked getter with `QueueShutDown`. So `await get()` can never hang on a dead engine, whether or not the put landed. `try_get_output*` and `collective_rpc` convert `QueueShutDown` to the existing RuntimeError. Tests: rewrite the async output tests over a real janus queue (event-driven wakeup, fatal-drain-then-raise, shutdown wakes a parked getter), update the orchestrator harness to pass the sync side, and add `put_nowait` to the kv-sender mock queue. Merge integration: the new MembershipController pushes its replica-disappeared error through the same sync side (put_nowait), and the crash handler shuts down each queue independently so a failure on one does not skip the other. Signed-off-by: zeningc <zening.chen@yahoo.com>

Standalone, GPU-free microbenchmark isolating the Orchestrator -> server handoff over the janus output queue. Mirrors production (producer thread -> async consumer) and compares the old 1ms busy-poll against the event-driven await, reporting per-message wakeup latency and idle CPU/wakeups. Representative result: mean latency 574us -> 145us (p50 575us -> 96us), and idle wakeups ~900/s (~5% of a core) -> ~0. Signed-off-by: zeningc <zening.chen@yahoo.com>

zeningc · 2026-06-20T00:11:20Z

Hi @hsliuustc0106! Just wanted to follow up on this when you have a moment. The review comments are addressed, it's rebased onto the latest main, and CI is green. Would appreciate another look when you have a moment. Thanks!

hsliuustc0106 · 2026-07-01T02:16:49Z

we have updated with vllm 0.24.0, can you help update the use case results?

hsliuustc0106

The output queue rename is incomplete and leaves live failure paths/tests broken.

hsliuustc0106 · 2026-07-01T03:03:36Z

    ) -> None:
        self.request_async_queue = request_async_queue
-        self.output_async_queue = output_async_queue
+        self.output_sync_queue = output_sync_queue


Please update the remaining output_async_queue references too. git grep still finds live calls at lines 1251 and 1419; since this constructor no longer sets that attr, those paths raise AttributeError instead of sending the terminal output.

Thanks! I just fixed it, sorry for the oversight, this was pulled from new code when I updated the current branch.

hsliuustc0106 · 2026-07-01T03:03:36Z

            orchestrator = Orchestrator(
                request_async_queue=request_queue.async_q,
-                output_async_queue=output_queue.async_q,
+                output_sync_queue=output_queue.sync_q,


Please also update tests/engine/test_orchestrator_stage_input_bridge.py; it still passes output_async_queue=... and is marked core_model/cpu, so it fails with the new constructor signature.

Thanks, fixed as well!

Merging main pulled in new call sites that used the pre-rename output queue API, which this branch had already renamed output_async_queue -> output_sync_queue (put_nowait on the sync side). These were semantic conflicts: git auto-merged them without a textual conflict, so they landed referencing an attribute the constructor no longer sets. - orchestrator.py:1251 (from vllm-project#4079, diffusion request-level batching) and orchestrator.py:1419 (from vllm-project#4257, Aura non-async-chunk path) still called `await self.output_async_queue.put(...)`, which would raise AttributeError on those terminal-output error/edge paths. Convert to `self.output_sync_queue.put_nowait(...)`. - tests/engine/test_orchestrator_stage_input_bridge.py (new file from vllm-project#4257, marked core_model/cpu) constructed Orchestrator with the old `output_async_queue=` kwarg and failed against the new signature. Update to `output_sync_queue=output_q.sync_q`. Tested on vLLM 0.24.0: engine + orchestrator unit tests (41 passed) and a Qwen3-TTS-12Hz-0.6B streaming /v1/audio/speech smoke test. Signed-off-by: zeningc <zening.chen@yahoo.com>

zeningc · 2026-07-01T07:34:18Z

we have updated with vllm 0.24.0, can you help update the use case results?

sure! I've ran all the tests with 0.24.0 and updated the details in PR description

The event-driven output loop change made `AsyncOmniEngine.try_get_output_async` park on `await async_q.get()` and never return None; the loop's per-empty `await asyncio.sleep()` yield point was removed accordingly. Two test doubles still modeled the old contract, returning None synchronously on an empty queue: `FakeAsyncOmniEngine` (tests/entrypoints/test_omni_entrypoints.py) and `_OrchestratorBridgeEngine` (tests/diffusion/test_diffusion_streaming_output.py). Against those doubles `while True: msg = await try_get_output_async()` spins with no real suspension point once the queue drains, starving the event loop so `generate()` never resumes — the loop hangs and CI kills the step on timeout (exit 124: test_async_omni_yields_only_final_stage_outputs and test_streaming_output_reaches_async_omni_from_mock_pipeline). Update both doubles to honor the new contract: park (poll the backing queue and `await asyncio.sleep(0.001)` on empty) until a real message arrives, instead of returning None. Death is still signaled by enqueuing a fatal ErrorMessage, and shutdown() cancels the loop task, so no None sentinel is needed. Production stays event-driven and unchanged. Signed-off-by: zeningc <zening.chen@yahoo.com>

Two gaps in how a dying orchestrator wakes a parked output-queue reader, both now that the reader parks on the queue instead of polling: 1. Death signaling (put fatal ErrorMessage + queue shutdown) lived only in ``_bootstrap_orchestrator``'s ``except Exception``. A BaseException (SystemExit / KeyboardInterrupt / asyncio.CancelledError) or a clean return skipped it, leaving a parked ``await async_q.get()`` hung forever. Move the shutdown() into ``finally`` so any thread exit wakes the reader; the except only records the error text, which finally enqueues as the fatal message first (drained before QueueShutDown, so the caller still sees fatal=True). This also de-duplicates the per-queue loop. 2. janus raises different, unrelated classes on the two sides of a shut-down queue: ``ShutDown`` from the sync side and ``QueueShutDown`` from the async side. The sync readers (``try_get_output``, ``collective_rpc``) only caught ``QueueShutDown``, so a real shutdown leaked a raw ``janus.ShutDown`` instead of mapping to RuntimeError. Catch both via a shared ``_QUEUE_SHUTDOWN_ERRORS`` tuple at all sites. (Surfaced by the janus 2.0 bump; the old test masked it by mocking side_effect=QueueShutDown.) Tests: drive ``_bootstrap_orchestrator`` over real janus queues and assert a BaseException exit still shuts the queues (reader raises RuntimeError, not a hang), an Exception exit drains the fatal message then shuts down, and rewrite the queue-shutdown test to use a real queue instead of a mocked exception. Signed-off-by: zeningc <zening.chen@yahoo.com>

The output queue is now handed to the MembershipController as the queue's sync side (Orchestrator pushes via put_nowait), but the annotations still read asyncio.Queue[EngineQueueMessage], which is neither the real runtime type nor even a janus type. Narrow the attribute and both handler params to janus.SyncQueue[EngineQueueMessage] to match reality and the Orchestrator's own output_sync_queue typing. Signed-off-by: zeningc <zening.chen@yahoo.com>

vllm-omni-review-bot · 2026-08-31T10:37:28Z

Omni ReviewBot triage note

Automated triage of commit b1cdf68cf5c6 produced:

Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

vllm-omni-review-bot · 2026-09-16T08:14:07Z

Omni ReviewBot: no human activity for 15 days

@zeningc this pull request has had no human commit, comment or review since 2026-08-31. Per repository policy it may be closed if it stays inactive.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

vllm-omni-review-bot · 2026-09-23T20:38:56Z

Omni ReviewBot: no human activity for 23 days

@zeningc this pull request has had no human commit, comment or review since 2026-08-31. Please consider marking this PR as draft until work can resume. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

Re-port the event-driven output drain onto the refactored engine layout on main (OmniEngineBase / AsyncOmniBase from vllm-project#7413) and on top of the flag-gated event-driven orchestration loop from vllm-project#5221. - omni_engine_base.py: try_get_output_async parks on output_queue.async_q.get(); a shut-down queue maps to RuntimeError on both the sync and async readers; _bootstrap_orchestrator enqueues the fatal ErrorMessage and shuts the output queue down in `finally` so a parked reader wakes on any thread exit. get_output_blocking_async and its drain-thread executor are removed. - orchestrator.py / duplex_orchestrator.py / duplex session manager / membership_controller.py: push outputs with sync_q.put_nowait; output_async_queue -> output_sync_queue (janus.SyncQueue). - async_omni_base.py: the final-output drain is a single `await engine.try_get_output_async()`; the poll branch, the VLLM_OMNI_EVENT_DRIVEN_ORCH drain branch, and both interval constants are gone. The flag now governs only the orchestration loop. - docs: VLLM_OMNI_EVENT_DRIVEN_ORCH describes only the orchestration loop; the drain is always event-driven. - tests: engine output tests over a real janus queue; orchestrator fixtures pass the sync side; vllm-project#5221's drain-thread unit tests removed with the method; fake engines park instead of returning None. Signed-off-by: zeningc <zening.chen@yahoo.com>

zeningc · 2026-09-24T02:36:51Z

Rebased onto current main (vLLM 0.30.0) and resolved the conflicts. @hsliuustc0106 the earlier review items (split queue signaling, remaining output_async_queue references, the stage-input-bridge test) were addressed in July; this push re-ports the change onto the engine refactor that landed since.

What changed in the re-port:

main moved the engine body into OmniEngineBase / AsyncOmniBase ([Core][Frontend] Unified Full-duplex Framework #7413), so the queue-read and death-signaling changes now live there.
[Perf][Engine] Event-driven orchestration loop (opt-in) — S1 of #4855 #5221 added a flag-gated drain thread (get_output_blocking_async) for the serving-side final-output drain. This PR removes that thread: the orchestrator now pushes on the queue's sync side, so the server can await async_q.get() directly, for every pipeline and with no flag. [Perf][Engine] Event-driven orchestration loop (opt-in) — S1 of #4855 #5221's orchestration-loop flag and parity suite are untouched. Docs for VLLM_OMNI_EVENT_DRIVEN_ORCH updated to say it only governs the orchestration loop.

Validation on 0.30.0: 1450 unit tests pass across tests/engine/ and the entrypoint/streaming/metrics suites, pre-commit is clean on changed files (mypy reports only the same pre-existing errors as main), and a Qwen3-TTS streaming smoke plus a 32-way concurrent burst and a clean SIGTERM shutdown all ran without errors. Details and the microbenchmark numbers are in the updated description.

vllm-omni-review-bot · 2026-10-01T02:47:46Z

Omni ReviewBot: no human activity for 7 days

@zeningc this pull request has had no human commit, comment or review since 2026-09-24. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

vllm-omni-review-bot · 2026-10-02T13:44:15Z

Omni ReviewBot routing record

Assigned Strict on cursor (cursor-grok-4.6-high) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

zeningc requested review from Gaohan123, david6666666, hsliuustc0106, tzhouam, yenuo26 and ywang96 as code owners June 5, 2026 08:10

zeningc force-pushed the feat/event-driven-output-queue branch 2 times, most recently from e573a92 to de7e57b Compare June 5, 2026 08:53

hsliuustc0106 reviewed Jun 6, 2026

View reviewed changes

zeningc changed the title ~~[Engine] Make async output reads event-driven; signal engine death via queue shutdown~~ [Core] Make async output reads event-driven; signal engine death via queue shutdown Jun 6, 2026

zeningc added 2 commits June 19, 2026 16:43

zeningc force-pushed the feat/event-driven-output-queue branch from 283bf7c to 26d97a7 Compare June 19, 2026 23:44

Merge branch 'main' into feat/event-driven-output-queue

f0e04b9

hsliuustc0106 added the core related to core module: cache, scheduler, engine, worker, modelrunner label Jul 1, 2026

hsliuustc0106 requested changes Jul 1, 2026

View reviewed changes

zeningc and others added 2 commits June 30, 2026 23:22

Merge branch 'main' into feat/event-driven-output-queue

2cd5591

zeningc requested a review from hsliuustc0106 July 1, 2026 07:34

Merge branch 'main' into feat/event-driven-output-queue

f6f7353

zeningc requested review from Sy0307 and alex-jw-brooks as code owners July 4, 2026 18:25

hsliuustc0106 added the ready label to trigger buildkite CI label Jul 4, 2026 — with ChatGPT Codex Connector

hsliuustc0106 requested a review from linyueqian July 4, 2026 21:00

zeningc and others added 7 commits July 4, 2026 15:34

Merge branch 'main' into feat/event-driven-output-queue

9fe0bc6

Merge branch 'main' into feat/event-driven-output-queue

8a4ab19

Merge branch 'main' into feat/event-driven-output-queue

0a46ec3

Merge branch 'main' into feat/event-driven-output-queue

be7539a

LOGO127 mentioned this pull request Sep 6, 2026

[Refactor][TTS]: Problems and improvement directions (async-chunk reliability, serving_speech, runtime hot path) #4855

Open

Sy0307 mentioned this pull request Sep 6, 2026

[Frontend] Scale Omni serving across API processes #6923

Merged

zeningc requested a review from NickCao as a code owner September 24, 2026 02:36

Conversation

zeningc commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Why

Design / what changed

Relationship to #5221

Testing

Note

Uh oh!

chatgpt-codex-connector Bot commented Jun 5, 2026

Uh oh!

zeningc commented Jun 6, 2026

Uh oh!

hsliuustc0106 commented Jun 6, 2026

Uh oh!

hsliuustc0106 Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

zeningc Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

zeningc commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeningc commented Jun 12, 2026

Uh oh!

zeningc commented Jun 20, 2026

Uh oh!

hsliuustc0106 commented Jul 1, 2026

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

zeningc Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

zeningc Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

zeningc commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vllm-omni-review-bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot triage note

Uh oh!

vllm-omni-review-bot commented Sep 16, 2026

Omni ReviewBot: no human activity for 15 days

Uh oh!

vllm-omni-review-bot commented Sep 23, 2026

Omni ReviewBot: no human activity for 23 days

Uh oh!

zeningc commented Sep 24, 2026

Uh oh!

vllm-omni-review-bot commented Oct 1, 2026

Omni ReviewBot: no human activity for 7 days

Uh oh!

vllm-omni-review-bot commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zeningc commented Jun 5, 2026 •

edited

Loading

zeningc commented Jun 6, 2026 •

edited

Loading

zeningc commented Jul 1, 2026 •

edited

Loading

vllm-omni-review-bot commented Aug 31, 2026 •

edited

Loading