[Refactor] P0.2: Migrate API server helpers out of api_server

[herotai214]

[Refactor] P0.2: Migrate API server helpers out of api_server

cc @linyueqian
See if/how a P0.1: Add API server guardrails should be added prior merging this PR.
This PR tgt with P0.3 serves as the fundamental structure of our whole refactor project.

Summary

This is Refactor [P0.2] for #5227.

It moves non-route helpers and request models out of vllm_omni/entrypoints/openai/api_server.py into endpoint-owned or server-owned modules. The goal is to stop api_server.py from being the place other code imports helper implementation from.

This PR does not migrate endpoint route bodies yet. That is intentional: Phase 0 is split into:

• P0.1: API server guardrails/tests
• P0.2: helper migration, this PR
• P0.3: endpoint route-owner extraction, immediately after this PR

What Changed

Moved API-server helpers into clearer owners:

• openai/app_state.py for app-state accessors
• openai/chat_template.py for chat-template bootstrap helpers
• openai/diffusion.py for shared diffusion-stage generation/sampling helpers
• openai/lora.py for OpenAI LoRA request parsing
• openai/images/helpers.py for image-only helpers
• openai/models/serving.py for diffusion-only /v1/models serving shim
• serve/utils/errors.py for server-level engine exception handling
• serve/utils/routes.py for app/router route-table mutation helpers
• serve/profile/protocol.py and serve/profile/utils.py for profiler route support
• serve/omni_control/protocol.py for sleep/wakeup request models

This follows upstream vLLM's current ownership direction where practical:

• server/app mechanics live under entrypoints/serve/utils;
• OpenAI-compatible response shaping stays under entrypoints/openai;
• model-list serving helpers live near entrypoints/openai/models;
• endpoint-family helpers move toward their owning OpenAI endpoint packages instead of remaining in the server composition root.

Updated tests and serving imports to use the new helper locations.

Why api_server.py Still Imports These Helpers

This PR intentionally does not move endpoint bodies yet, so api_server.py temporarily imports helpers from their new owners.

For example:

from vllm_omni.entrypoints.openai.images.helpers import (
    _check_max_generated_image_size,
    _extract_images_from_result,
    _load_input_images,
)
from vllm_omni.entrypoints.openai.lora import _parse_lora_request
from vllm_omni.entrypoints.serve.utils.errors import (
    _create_engine_error_json_response,
)

That looks a little inverted today because the route bodies are still in api_server.py. The next PR, Refactor [P0.3], will move those route bodies beside these helpers, so these temporary imports disappear or become local endpoint-package imports.

Out Of Scope

This PR does not move decorated endpoint bodies such as:

• /v1/chat/completions
• /v1/images/generations
• /v1/images/edits
• /v1/videos*
• /v1/realtime
• /v1/video/chat/stream
• /v1/duplex
• /v1/omni/sleep
• /v1/omni/wakeup

It also does not move duplex_capability.py, because that helper was already outside api_server.py on latest main.

Relationship To Phase 0

This PR is the middle step of Phase 0:

P0.1 tests/guardrails
  -> P0.2 helper migration
  -> P0.3 endpoint router migration

The endpoint migration should start immediately after this PR. Keeping P0.2 focused on helpers keeps this review smaller and makes the P0.3 route-body move mostly mechanical.

Test Plan

Focused API-server pytest passed in a local development environment with matching vllm / vllm-omni versions:
vLLM Version: 0.25.0

vLLM-Omni Commit: d688aa8

pytest -q tests/entrypoints/openai_api/test_image_server.py \
  tests/entrypoints/openai_api/test_video_server.py \
  tests/entrypoints/openai_api/test_serving_speech.py \
  tests/entrypoints/openai_api/test_omni_sleep_wakeup.py

(Those 4 are only the focused regression set we ran for this PR, not the only tests in the repo.

They were chosen because they most directly hit the helpers moved out of api_server.py:

• test_image_server.py — image helpers / diffusion / LoRA / models serving
• test_video_server.py — video generation helpers / store / form parsing
• test_serving_speech.py — speech error-response helpers
• test_omni_sleep_wakeup.py — sleep/wakeup protocol + app-state init)

Result:

357 passed, 18 warnings in 13.22s

Additional checks run:

python - <<'PY'
from pathlib import Path

for path in [
    "vllm_omni/entrypoints/openai/api_server.py",
    "vllm_omni/entrypoints/openai/app_state.py",
    "vllm_omni/entrypoints/serve/utils/errors.py",
    "vllm_omni/entrypoints/serve/utils/routes.py",
]:
    compile(Path(path).read_text(), path, "exec")

print("compile ok")
PY

Also checked IDE lints on touched helper files; no linter errors were reported.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[herotai214]

Additional Test Plan

pytest -q tests/entrypoints -m "core_model and cpu"

Result: 1049 passed, 2 skipped, 165 deselected (~31s)

Scope: all CPU core_model tests under tests/entrypoints/ (38 files / 1051 collected).

Covered modules

• Audio: generate, speech, speech stream, audio format, decode URL
• Images: image server
• Video: generation server, video stream, video output stream, API utils, storage
• Chat: sampling params, multistage, metrics, speaker
• Duplex / realtime: handler, capability, protocol, session, async duplex, fence, realtime helpers
• OpenPI: connection, serving
• Serve / omni core: sleep/wakeup, profiler, stage utils, async omni, diffusion config, snapshot download, utils, modality validation

Files

38 files

openai_api/

• test_audio_format.py, test_decode_audio_url.py
• test_duplex_capability.py, test_duplex_handler.py
• test_image_server.py, test_modality_validation.py
• test_omni_sleep_wakeup.py
• test_openpi_connection.py, test_openpi_serving.py
• test_serving_audio_generate.py
• test_serving_chat_metrics.py, test_serving_chat_multistage_generation.py, test_serving_chat_sampling_params.py, test_serving_chat_speaker.py
• test_serving_speech.py, test_serving_speech_stream.py
• test_serving_video_output_stream.py, test_serving_video_stream.py
• test_storage_manager.py, test_video_api_utils.py, test_video_server.py

openai/

• test_duplex_protocol.py, test_duplex_session_attachment.py

entrypoints/

• test_async_omni.py, test_async_omni_diffusion_config.py, test_async_omni_duplex.py, test_async_omni_prompt_update.py
• test_duplex_fence_propagation.py
• test_omni_base_profiler.py, test_omni_entrypoints.py, test_omni_new_request_data.py, test_omni_snapshot_download.py
• test_realtime_connection_helpers.py, test_resolve_dreamzero_config.py
• test_serve.py, test_stage_utils.py, test_stream_finish_reason.py, test_utils.py

```

[linyueqian]

Reviewed the diff and validated it end to end on an H100 node (merge-base vs head, vllm 0.26). The zero-behavior-change claim holds.

Runtime equivalence

• Routes identical: 42 vs 42, empty set diff.
• OpenAPI schema deep-equal after key sorting (373 definitions each).
• Live endpoints match: /health, /ping, /version, /v1/models, /v1/audio/voices (same 9 voices), and POST /v1/audio/speech returning byte-identical audio (53804 B, 1.12 s, 24 kHz mono) on both trees.
• All 10 error probes return identical status codes and byte-identical JSON bodies, which covers the moved exception handlers.
• All 43 names that left api_server's namespace resolve from their new modules; nothing unaccounted for. Import time is flat (-0.5%, measured interleaved in one process).

Tests — tests/entrypoints/, two full runs per tree: head never regresses. The 8 differing tests all go base-fail to head-pass, and base disagrees with itself on exactly those 8 across its own two runs, so they are flaky under shared-GPU pressure rather than affected by this PR. Five failures are pre-existing on both trees.

Structure — the direction is right: none of the 13 new modules import the composition root, so the dependency graph stays acyclic, and keeping app_state / models.serving reachable as module aliases preserves existing monkeypatch targets. I also AST-compared every moved symbol against main: 46 of 47 are byte-identical, the only delta being a docstring (inline comment below).

One blocker, with a verified one-line fix. The ReadTheDocs build fails on this PR. It is green on 10 of 10 other currently-open PRs, so it is attributable here. Details and the fix are in the inline comment on serve/omni_control/protocol.py.

On your question about P0.1: I would land the guardrails before this. The route manifest and the OpenAPI snapshot are exactly what made this review conclusive, and having them in CI means later phases do not depend on someone re-running this by hand. Happy to review that PR as well.

[linyueqian]

Blocking: this PR breaks the ReadTheDocs strict build. Reproduced locally on both trees — merge-base builds clean (exit 0), this head aborts:

WARNING - mkdocs_autorefs: api/README.md: Could not find cross-reference target
  'vllm_omni.entrypoints.serve.omni_control.protocol.OmniSleepRequest'
  'vllm_omni.entrypoints.serve.omni_control.protocol.OmniWakeupRequest'
  'vllm_omni.entrypoints.serve.profile.protocol.ProfileRequest'
Aborted with 3 warnings in strict mode!

Two things collide. docs/mkdocs/hooks/generate_api_readme.py AST-scans the tree and emits [dotted.path][] autorefs for every public class; its exclude list covers vllm_omni.entrypoints.openai but not vllm_omni.entrypoints.serve, so these three pydantic models get links. Meanwhile mkdocs.yml sets api-autonav: on_implicit_namespace_package: skip, and the 8 new package directories have no __init__.py, so autonav skips them, no API page is generated, the anchors never exist, and fail_on_warning: true in .readthedocs.yml turns the dangling references into a failed build.

Fix, verified: add __init__.py to the 8 new directories (serve/, serve/utils/, serve/profile/, serve/omni_control/, openai/images/, openai/models/, openai/video/, openai/video/generation/). With those files present, mkdocs build --strict exits 0 with zero warnings. Adding vllm_omni.entrypoints.serve to the hook's exclude list also silences it, but the __init__.py route is better: it matches the other 206 packages in the tree and gives these modules real API pages.

For the record, packaging is not affected either way — I built a wheel from this head and all 13 new modules are present, because [tool.setuptools.packages.find] defaults to namespaces = true. This is a docs-only failure.

[herotai214]

Agree; I'll add the __init__.py for them.

[linyueqian]

The move dropped the body of this docstring. On main it explained why Omni overrides the upstream handler at all:

The upstream engine_error_handler is designed for AsyncLLM (single EngineCore process). Omni uses a multi-stage orchestrator with different health semantics, so we register our own handlers that log multi-stage diagnostic info (orchestrator liveness, per-stage health) when an EngineDeadError is caught, call terminate_if_errored, and return an OpenAI-compatible error JSON response.

That paragraph is the rationale for this module existing, and it is more valuable here than it was in api_server.py. Worth restoring verbatim. This is the only content delta I found across all 47 moved symbols, the code itself is byte-identical.

[herotai214]

Good catch! I'll add the docstring body back.

[linyueqian]

Worth a note for whoever touches the video tests next: VIDEO_STORE and STORAGE_MANAGER are now bound in two modules, here and in api_server.py, because the route bodies stay put until P0.3. Production behavior is unaffected since both are from ... import bindings of the same singletons in openai/stores.py and openai/storage.py, but a test that patches only one of the two targets will silently fail to isolate the other path, which is why test_video_server.py now has to patch both. P0.3 dissolves this once the routes move next to these helpers, so no change requested here.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 290476c0635d produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[herotai214]

Omni ReviewBot triage note

Automated triage of commit c90a2805317e produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

Refactor update base on v0.28.0 will be ready soon within these 1-2 days.

[herotai214]

Resolved another conflict-minimal merge onto latest main. Only api_server.py conflicted; Incoming was #4728 (image/video benchmark metrics).
@Gaohan123 @hsliuustc0106 — Could you help land this sooner if the extract looks right? main still has these helpers in api_server.py, so every nearby video/image change rebases as a dump-vs-empty conflict and we have to re-forward the real edit frequently.

Resolved conflict details:

Same pattern as MAGI / Cosmos3: we kept this PR’s empty side after edit_images and forwarded the real #4728 edits into video/generation/helpers.py:

• _unpack_video_generation_result (new)
• _run_video_generation_job now unpacks the 4- or 5-tuple and stores metadata
• _parse_video_form accepts return_stage_metrics
• video_response_from_request now fills fps / num_frames / duration_s (this one was missed in the merge commit and forwarded in a follow-up so queued /v1/videos matches main)

Route bodies stay in api_server.py: generate_images / edit_images already auto-merged the metrics wiring; create_video / create_video_sync still call the helpers.

_build_image_generation_response and _build_image_response_metrics are still in api_server.py on purpose. They are internal helpers used only by the image endpoint functions (generate_images / edit_images), not public API and not a missing images/serving.py. P0.2 left the first one next to the route; #4728 added the second beside it. Both move in the next PR, P0.3, together with those endpoint bodies.

P0.2 scope is otherwise unchanged: no route-body moves in this PR. New request/job helpers should land in the owning package (openai/video/generation/helpers.py, openai/images/helpers.py, entrypoints/duplex/, …), not in api_server.py.

[herotai214]

If possible, plz refrain from merging PRs touching the api_server for 1-2 days & merge this PR first....

Resolved another conflict-minimal merge onto latest main. Three files conflicted (api_server.py, test_serving_speech.py, check_forbidden_imports.py). Could you help land this sooner if the extract looks right? main still has these helpers in api_server.py, so every nearby change rebases as a dump-vs-empty conflict and we have to re-forward the real edit.

Same pattern as MAGI / Cosmos3 / #4728: Incoming dumped the already-extracted helper block after profiler_router. We kept this PR’s empty side and forwarded the real edits:

• Add vLLM-Omni library info to Hugging Face Hub requests #5381 (Hub tagging) — _load_model_chat_template_json now calls hf_api().hf_hub_download in openai/chat_template.py (matches main AST). The huggingface allowlist takes main’s short list; api_server.py / chat_template.py are not grandfathered.
• [Frontend] Add server-side VAD for turn-based omni models #6618 (server-side VAD) — already auto-merged in the route/init bodies (omni_init_app_state duplex kwargs + /v1/realtime default-on). Left as-is.
• [Feat][OmniVoice]Support Varlen Attn, Request-Batch and Step-Execution #6408 — test_serving_speech.py keeps both imports: this PR’s serve_errors patch target and Incoming’s OmniDiffusionSamplingParams.

_build_image_generation_response and _build_image_response_metrics are still in api_server.py on purpose. They are internal helpers used only by the image endpoint functions, not public API. Both move in P0.3 together with those endpoint bodies.

P0.2 scope is otherwise unchanged: no route-body moves in this PR. New request/job helpers should land in the owning package, not in api_server.py.

[herotai214]

AGAIN

Resolved another conflict-minimal merge onto latest main. Only api_server.py conflicted; Incoming was #6963 (image pixel limits on video refs). Could you help land this sooner if the extract looks right? main still has these helpers in api_server.py, so every nearby video change rebases as a dump-vs-empty conflict and we have to re-forward the real edit.

Same pattern as MAGI / Cosmos3 / #4728 / #5381: Incoming dumped the already-extracted helper block after edit_images. We kept this PR’s empty side and forwarded the real #6963 edits into video/generation/helpers.py:

• _validate_minimax_h3_image_payload now runs _validate_image_pixel_limit and maps InvalidInputReferenceError / DecompressionBombError
• _persist_uploaded_media_references decodes uploads via _decode_image_bytes

video_api_utils.py and the video tests already auto-merged. api_server.py does not import those decode helpers — helpers → utils only.

[Gaohan123]

Please resolve conflicts

[herotai214]

Resolved another conflict-minimal merge onto latest main. Only api_server.py conflicted; Incoming was #7230 (rebase to vLLM 0.29).

This one was not a helper dump. #7230 only moved vLLM import paths on api_server.py. We kept this PR’s extracted helper imports and took the 0.29 homes for names that file still uses:

• RequestResponseMetadata ← generate.base.protocol
• make_arg_parser ← launchers.cli_args
• serve_http ← launchers.launcher
• get_uvicorn_log_config ← launchers.utils.server_utils
• ErrorResponse ← serve.engine.protocol

We did not pull Incoming’s ModelCard / terminate_if_errored / create_error_response / 0.28 try/except back into api_server.py — those already live in the extract.

Follow-up on this branch (same import rewrite #7230 could not see):

• serve/utils/errors.py and video/generation/helpers.py — launchers.launcher.terminate_if_errored + from vllm.entrypoints.serve import create_error_response
• models/serving.py — serve.engine.protocol for ModelCard / ModelList / ModelPermission
• openai/errors.py — serve.engine.protocol for ErrorResponse

The real 0.29 behavior changes (video_api_utils pyav → opencv, AsyncOmni.finish_weight_update(weight_version=...)) already auto-merged. No new routes or helpers.

---

Full L1 set (guards + image/video/speech/audio/sleep/log-stats): 620 passed. (After below)

One local fallout from the pyav → opencv switch, not from the extract:

vLLM 0.29 dropped the pyav decoder; video_api_utils._decode_video_bytes now uses backend="opencv". opencv-python-headless still dlopens libxcb.so.1 (and libGL / libX11 / libXext / libSM / libICE). On a slim GPU container those packages were missing, so 14 test_video_server.py cases initially failed on my side. The user-visible error is a 400 InvalidInputReferenceError: ... not a valid video, with ImportError: libxcb.so.1: cannot open shared object file as the cause — easy to misread as bad bytes.

After installing libxcb1 libx11-6 libxext6 libgl1 libsm6 libice6, video tests went 14 failed → 111 passed.

CI (vllm/vllm-openai:v0.29.0) may already ship these. Worth a glance at #7230 / the image so other slim or custom boxes don’t hit the same wrap. Not something P0.2 should paper over in api_server.py.

P0.2 scope is otherwise unchanged: no route-body moves. New request/job helpers should land in the owning package, not in api_server.py.

[herotai214]

The failing CIs buildkite with amd, intel and npu should be all due to vllm version mismatch after the rebase v0.29 commit; and not introduced by this PR or api server related files.

For example, the ImportError: cannot import name 'compute_layout_strides' from 'vllm.v1.kv_cache_interface' (/usr/local/lib/python3.12/dist-packages/vllm/v1/kv_cache_interface.py here is far from our api_server/entrypoints refactor

The buildkite/vllm-omni passed.

[Gaohan123]

LGTM. Thanks