[alex-jw-brooks]

Purpose

Currently, running generator with py_generator=True shuts down the engine when we evaluate the generator. I.e.,

import os

os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

from vllm_omni.entrypoints.omni import Omni

if __name__ == "__main__":
    omni = Omni(model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", trust_remote_code=True, log_stats=False)
    prompt = {
        "prompt_token_ids": [0] * 50,
        "additional_information": {
            "task_type": ["CustomVoice"],
            "text": ["Hello."],
            "language": ["English"],
            "speaker": ["Vivian"],
            "instruct": [""],
            "max_new_tokens": [256],
        },
    }

    list(omni.generate([prompt], py_generator=True))
    list(omni.generate([prompt], py_generator=True))

the second generate call will blow up because we unconditionally close + shut down the engine when py_generator=True.

Traceback (most recent call last):
  File "/home/alex-jw-brooks/scratch/repro_omni_close_on_generator.py", line 27, in <module>
    list(omni.generate([prompt], py_generator=True))
  File "/home/alex-jw-brooks/vllm-omni/vllm_omni/entrypoints/omni.py", line 104, in _run_generation_with_generator
    yield from gen
  File "/home/alex-jw-brooks/vllm-omni/vllm_omni/entrypoints/omni.py", line 153, in _run_generation
    self.engine.add_request(
  File "/home/alex-jw-brooks/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 1335, in add_request
    self.request_queue.sync_q.put(msg)
  File "/home/alex-jw-brooks/venvs/omni/lib64/python3.12/site-packages/janus/__init__.py", line 372, in put
    raise SyncQueueShutDown
janus.ShutDown

We also hold a weakref to the engine and shut it down through that, so added a couple of tests to ensure calling generate with a generator doesn't shut down the engine

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/entrypoints.md.

Module owners: @alex-jw-brooks @linyueqian @NickCao

@alex-jw-brooks, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 20 days

@alex-jw-brooks this pull request has had no human commit, comment or review since 2026-08-19. Per repository policy it may be closed if it stays inactive.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[NickCao]

@vllm-omni-review-bot

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Direct under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Resolved as of ceacaaf92afb: the high-priority or low-quality signal noted on an earlier commit no longer applies.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

Consuming Omni.generate(..., py_generator=True) no longer shuts down the shared engine when the generator finishes. The old _run_generation_with_generator path ran self.close() in a finally, so a fully evaluated generator left the request queue dead and a second generate raised janus.ShutDown. The fix yields through _run_generation only, keeps exception-path/__del__/explicit shutdown cleanup, and updates unit tests to assert the engine stays alive after generator use while still shutting down on Omni deletion.

Change flow

flowchart TD
  A["[EXISTING] Caller list(Omni.generate py_generator=True)"]:::existing --> B["[CHANGED] Omni._run_generation_with_generator<br/>remove finally self.close()"]:::changed
  B --> C["[EXISTING] Omni._run_generation / engine queues"]:::existing
  C --> D["[EXISTING] OmniBase weakref.finalize / explicit close()"]:::existing
  B --> E["[NEW] test_omni_pygenerator_does_not_kill_engine<br/>+ flipped shutdown_called asserts"]:::new
  D --> F["[NEW] test_del_shutsdown_engine"]:::new
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

Findings

• [P1] PR body missing Test Plan/Result vs Ready entrypoints guard — .buildkite/cuda/test-ready.yml:23
  Frozen PR body has Purpose/repro text but no Test Plan or Test Result naming what was run. Ready CI guard step "Simple · Engine&Entrypoints Test" runs pytest -sv tests/entrypoints tests/engine -m 'core_model and cpu' and is the lane that covers tests/entrypoints/test_omni_entrypoints.py (pytestmark = core_model, cpu). Please record that selector (or an equivalent tests/entrypoints/test_omni_entrypoints.py -m 'core_model and cpu' run) plus results; informal "added a couple of tests" is not enough for CI alignment.