[apex-mochen]

Fixes #8139.

Update since the first revision. This started as a guard for /tokenize and /detokenize only, matching the report in #8139. While auditing the same code path I assembled the real app in pure-diffusion mode and found the identical failure on four more route families, so the fix is now stated once for the class instead of four times per route. Title, description and tests below have been updated; the single previous commit is still in this branch's history. The branch name still says tokenize-routes because it predates the generalization — renaming it here would close and reopen the PR, so I left it alone.

Problem

omni_run_server_worker() assembles the app with upstream build_app(args, supported_tasks). For supported_tasks=("generate",) that mounts the whole generate family — /v1/completions, /v1/responses (plus retrieve and cancel), /v1/messages — on top of the always-mounted /tokenize and /detokenize routes.

A serving mode only initializes the handlers it can serve. Pure diffusion and duplex deliberately leave app.state.openai_serving_completion, openai_serving_responses, anthropic_serving_messages and serving_tokenization unset or None (_DIFFUSION_MUST_BE_NONE freezes this for diffusion; duplex nulls its handlers in a loop), and no route-level null check covers them:

• /v1/completions, /v1/responses (+ retrieve/cancel), /v1/messages, /tokenize, /detokenize → AttributeError → HTTP 500.
• /v1/messages is doubly affected: it null-checks anthropic_serving_messages, but then builds its error through serving_tokenization.create_error_response, which diffusion sets to None — so even its existing check cannot produce the intended error.

#8139 reported this for /tokenize from a Cosmos3 diffusion startup log. The same root cause reaches three more route families, so the fix here is stated once for the class rather than four times per route.

Reproduction (CPU, no GPU needed)

Assemble the real app the way omni_run_server_worker() does — upstream build_app(args, ("generate",)) plus the real omni_init_app_state() for a pure-diffusion engine — then probe it with TestClient:

probe	before	after
POST /v1/completions	500 AttributeError	400 BadRequestError
POST /v1/responses	500 AttributeError	400 BadRequestError
GET /v1/responses/{id}	500 AttributeError	400 BadRequestError
POST /v1/responses/{id}/cancel	500 AttributeError	400 BadRequestError
POST /v1/messages	500 AttributeError	400 BadRequestError
POST /tokenize	500 AttributeError	400 BadRequestError
POST /detokenize	500 AttributeError	400 BadRequestError
GET /v1/models, /load, /version (controls)	200	200

Fix

Reuse the mechanism this repo already has for exactly this job — shutdown_unsupported_routes(), added by #4762 for /v1/completions and /v1/images/edits — and derive its input from app.state instead of hard-coding routes:

• config/endpoint_policy.py — register the newly reachable capabilities (RESPONSES, RESPONSES_RETRIEVE, RESPONSES_CANCEL, MESSAGES) next to the existing COMPLETIONS, and replace the two-branch special case with one UNWIRED_HANDLER_ROUTES table mapping each capability to the app.state attribute that serves it, plus unwired_endpoint_restrictions(state, already_restricted=...).
• entrypoints/serve/utils/routes.py — route_is_mounted(app, path, methods), so a restriction is only applied to a route that is actually mounted.
• entrypoints/openai/api_server.py — _shutdown_routes_without_handlers(app, already_restricted=...) replaces the previous tokenize-only helper. One call site, unchanged position, right after omni_init_app_state() and the existing shutdown_unsupported_routes(app, engine_client.endpoint_restrictions) call.

Three invariants the helper keeps, each with a test:

1. Wired handlers are never touched. Negative control: a fully-wired app keeps every route object identical (26 -> 26).
2. Routes that are not mounted are not added. route_is_mounted() is checked first, so this never grows the surface — a path the app did not mount still answers 404 rather than gaining a new 400.
3. A pipeline's own reason wins. Capabilities already present in engine_client.endpoint_restrictions are skipped, so a model that declares a restriction keeps the reason that pipeline declared.

Why not set the state attribute to None, or install stub handlers?

Setting the attribute to None gives 501 for completions/responses (upstream null checks) but still 500 for /v1/messages, which needs a ServingTokenization object to build its error. Installing stub handlers contradicts _DIFFUSION_MUST_BE_NONE in tests/entrypoints/openai_api/test_api_server_guards.py, a contract this repo deliberately froze. Route rejection is the only one of the three that fixes the whole class, and it reuses the path the repo already chose for the other unavailable endpoints.

Why 400 here rather than 404

Omni's own routes answer 404 for an unwired handler ("The model does not support …"). These seven routes are different: they are upstream routes, and the repo already has a mechanism for "this route is mounted but not available to this deployment" — shutdown_unsupported_routes(), which answers 400. The same route can already be made unavailable that way today: a pipeline that declares it in endpoint_restrictions (e.g. qwen3_omni/pipeline.py) makes /v1/completions answer 400 on that deployment. Routing the unwired case through the same function keeps one answer per route regardless of why it is unavailable, and keeps this change to deriving the function's input rather than changing what it emits. If you would rather these answer 404 to match omni's own routes, that is a one-line change in build_rejection_handler() and I will happily make it.

Scope: why the table lists exactly these seven routes

The table is only worth having if it is complete. I enumerated the real route surface an omni server builds — upstream register_api_routers() plus the per-task factories (generate, speech_to_text, pooling) plus omni's own router — and checked every route that is not in the table against the app.state attribute it reads:

route	why it is not in the table
/v1/chat/completions, /v1/chat/completions/batch, /v1/audio/speech, /v1/audio/speech/batch, /v1/audio/generate	omni removes the upstream routes and serves these from its own handlers, which already null-check and answer 404 ("The model does not support …")
/cohere/v2/chat, /cohere/v2/chat/render	upstream resolves the handler with getattr(..., None) and answers 501; additionally gated behind VLLM_ENABLE_COHERE_API
/v1/load_lora_adapter, /v1/unload_lora_adapter	backed by openai_serving_models, which every omni mode wires; additionally gated behind VLLM_ALLOW_RUNTIME_LORA_UPDATING
/v1/embeddings, /v1/score, /v1/rerank, /v1/pooling/*	mounted and wired under the identical POOLING_TASKS test on both sides
/v1/audio/transcriptions, /v1/audio/translations	mounted only for transcription; no omni pipeline declares that task
/generative_scoring	the route already handles an unwired handler itself (NotImplementedError, documented as a 404 in current vLLM main). Omni's defect is different in kind: it never initializes state.serving_generative_scoring in any mode, so the attribute read raises before that path is reached. That needs the attribute wired, not the route rejected, so it is deliberately not in this table.

Test Plan

CPU-only, no GPU needed.

pytest -q tests/entrypoints/openai_api/test_api_server_guards.py tests/config/test_endpoint_policy.py

Test Result

• tests/entrypoints/openai_api/test_api_server_guards.py: 74 passed (64 on upstream main; the first revision of this PR added 2 of them, this one replaces those two with 10)
• guard file + tests/config/test_endpoint_policy.py + tests/entrypoints/openai_api/test_duplex_api_server.py + test_video_pipeline_capability.py: 134 passed
• ruff check and ruff format --check on all four changed files: clean
• no file touched outside vllm_omni/; line endings unchanged (0 CR bytes in all four files, same as upstream)
• real-app probe from the table above: 7 routes 500 → 400, three control routes still 200, fully-wired mode untouched, model-declared reason preserved — 0 failures

Regression check (A/B with the patch physically removed). The 15 test files that import the modules this PR touches were run twice, once with the patch and once after replacing the three source files and the test file with the ad025def originals (the replacement was asserted on md5 before the second run):

run	result
patched	1294 passed, 9 failed, 1 xfailed
upstream base	1284 passed, 9 failed, 1 xfailed

The two FAILED sets are identical — no test passes only before this change and none fails only after it. The delta of 10 is exactly the new tests. The 9 failures are environment-only and reproduce at the base too: ModuleNotFoundError: No module named 'cache_dit' via the Cosmos3 model registry (test_config_factory.py, test_image_server.py, test_video_server.py), i.e. an optional dependency missing from this CPU box, not a behaviour change.

Not run: the GPU server-startup / perf job that produced the original log in #8139 — no GPU available here. The path is exercised end-to-end against the real upstream routers and TestClient, but the Cosmos3 diffusion run should be repeated in CI to confirm the startup log is clean there.

vLLM Version: 0.30.0 (the version in the report)

vLLM-Omni Commit: ad025de

AI assistance: used Claude Code to diagnose the failure and draft the patch and the regression tests. I reviewed every change, ran the checks reported above myself, and can explain the design and how it interacts with the existing endpoint-policy mechanism.

Negative control

A/B on the 15 test files that exercise the touched paths, with the three source
files restored to their pre-patch contents by file-level replacement (not
git stash, which would be a silent no-op here) and then re-patched: both sides
fail exactly the same 9 tests — all of them the pre-existing cache_dit
environment failures — and the FAILED sets are identical, i.e. no regression and
no new failure. The head adds 10 new assertions in
tests/entrypoints/openai_api/test_api_server_guards.py (64 → 74 passed).

Alternatives considered

The narrow first revision — reject only /tokenize and /detokenize, which is
what #8139 reported — was replaced rather than shipped: assembling the real app
with build_app(..., ("generate",)) plus the real omni_init_app_state shows
seven routes answering 500 on a pure-diffusion server, so the fix is driven by the
state each route actually needs. Per-route hard-coding was rejected because it
would need editing every time upstream adds a route.

Compatibility

No route is added, removed, or renamed, and no schema changes: only routes that
already answered 500 start answering the loader's existing "unsupported" envelope
(the same JSON fields and the reason the pipeline itself declared). Wired routes
are untouched — an invariant with its own test — and a route that was never
mounted does not appear.

AI assistance

AI assistance (Claude Code) was used to draft the route table, the regression
tests, and this description. I reviewed every change, ran every command reported
here locally, and can explain each invariant above without the tool.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/entrypoints.md.

Module owners: @alex-jw-brooks @linyueqian @NickCao

Routing: @alex-jw-brooks via module of the changed files, module named in the PR description, semantic router, CODEOWNERS; @linyueqian via module of the changed files, module named in the PR description, semantic router, CODEOWNERS; @NickCao via module of the changed files, module named in the PR description, semantic router, CODEOWNERS

@apex-mochen, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[apex-mochen]

Author self-review, as requested.

What the diff changes (3 files, +75/-1)

• vllm_omni/config/endpoint_policy.py (+2) — adds TOKENIZE / DETOKENIZE to OmniServingCapability. Nothing else changes; the enum's existing path / methods contract is untouched.
• vllm_omni/entrypoints/openai/api_server.py (+29/-1) — imports EndpointRestriction + OmniServingCapability, declares the two restrictions, and adds _shutdown_tokenization_routes_if_unwired(app), called from omni_run_server_worker() immediately after omni_init_app_state(). The single modified line is the fastapi import (adds FastAPI for the helper annotation).
• tests/entrypoints/openai_api/test_api_server_guards.py (+44) — the two regression tests.

Edge cases checked

• handler is None (pure diffusion / duplex) → 400 BadRequestError, both routes asserted, not just one.
• handler is wired (multi-stage) → both routes left untouched and still served by the upstream handler — this is the negative control, so the helper cannot pass by tearing down everything.
• the attribute is missing entirely (getattr(..., None)) → same path as None, so an app state without the key cannot raise AttributeError here.
• methods: only POST is targeted, matching what upstream mounts; a GET /tokenize is a 405 before and after, unchanged.

Cross-module impact

• The helper has exactly one caller (omni_run_server_worker()), so no other entrypoint — batch, render, speech/audio routes — changes behaviour.
• The rejection envelope is the one shutdown_unsupported_routes() already produces for /v1/completions and /v1/images/edits, so it is a shape clients already handle.
• No upstream vLLM behaviour changes; the upstream attempt (#58669) was closed deliberately, since handler is None is unreachable there.

Review-type checks before submitting

• Patch coverage: each added branch (is None / is not None) is reached by a test.
• Edge cases: as listed above.
• Cross-backend/provider impact: not applicable here (no provider abstraction); the analogous surface is the other serving modes, covered above.

Not run

The GPU server-startup / perf job that produced the original log. Local evidence is CPU-only: 66 passed in the guards file (64 before), 5 passed in the policy file, ruff check / ruff format --check / mypy clean on added lines, plus an end-to-end probe against the real upstream api_router.tokenize / detokenize (500 → 400, wired control still 200).

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 026e0e065c29 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

In serving modes that never construct a tokenization handler (pure diffusion and duplex), upstream build_app() still mounts POST /tokenize and POST /detokenize, and those handlers used to crash with AttributeError and HTTP 500. After omni_init_app_state(), the worker now reuses shutdown_unsupported_routes() to replace both routes with the existing 400 BadRequestError envelope when app.state.serving_tokenization is missing or None. Multi-stage servers that do wire ServingTokenization are left unchanged, so clients keep using the upstream handlers.

Change flow

flowchart TD
  A["[EXISTING] Client POST /tokenize or /detokenize"]:::existing
  B["[CHANGED] omni_run_server_worker after omni_init_app_state"]:::changed
  C["[NEW] _shutdown_tokenization_routes_if_unwired"]:::new
  D["[CHANGED] OmniServingCapability TOKENIZE and DETOKENIZE"]:::changed
  E["[EXISTING] shutdown_unsupported_routes 400 BadRequestError"]:::existing
  F["[EXISTING] Wired ServingTokenization handler"]:::existing
  A --> B
  B --> C
  C -->|"handler is None"| D
  D --> E
  C -->|"handler wired"| F
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 48868d4012d6 (2026-09-26T20:27:31.765076+00:00): required check(s) blocking: buildkite/vllm-omni (missing), and -5 more.

No actionable findings.

[apex-mochen]

Author self-review of the updated commit 061967a6 (4 files, +263/-1), as the repo asks for. Three passes below.

What the diff changes. config/endpoint_policy.py (+263 across the change set) adds four OmniServingCapability members and one table mapping each affected capability to the app.state attribute that serves it, plus unwired_endpoint_restrictions(); nothing existing changes. entrypoints/serve/utils/routes.py adds route_is_mounted(), a read-only sibling of remove_route_from_app(). entrypoints/openai/api_server.py adds _shutdown_routes_without_handlers() and its single call site; the previous tokenize-only helper and its constant are removed. tests/.../test_api_server_guards.py adds the regression tests. No file outside vllm_omni/ is touched.

Patch Coverage Review — is every added line exercised?

added code	branch	exercised by
OmniServingCapability.{RESPONSES, RESPONSES_RETRIEVE, RESPONSES_CANCEL, MESSAGES}	enum members; .path / .methods are read by route_is_mounted() and shutdown_unsupported_routes()	4 of the 7 parametrized cases in test_unwired_routes_are_rejected_instead_of_crashing (the other 3 cover COMPLETIONS, TOKENIZE, DETOKENIZE)
UNWIRED_HANDLER_ROUTES	all 7 entries	all 7 asserted by the same parametrized test
unwired_endpoint_restrictions	capability already restricted → skip	test_model_declared_restriction_keeps_its_own_reason
	getattr(state, attribute, None) is None → include	5 cases set the attribute to _UNSET_ATTRIBUTE (absent) and 2 set it to an explicit None, so both shapes reach this branch
	handler is wired → exclude	test_wired_routes_are_left_alone
route_is_mounted	True	the 7 parametrized cases
	False	test_unwired_routes_that_are_not_mounted_are_not_added, which asserts 404
_shutdown_routes_without_handlers	if restrictions: is empty	test_wired_routes_are_left_alone — the route must still answer 200, which it could not do if the helper had replaced it

Two branches deliberately have no new test:

• the route.methods & methods mismatch path inside route_is_mounted(). That predicate is copied verbatim from remove_route_from_app(); no serving mode mounts one of these paths under a method that differs from its policy entry, so the branch is unreachable here. The same predicate is covered for the removal side by tests/config/test_endpoint_policy.py.
• already_restricted accepting any Iterable — the only caller passes the tuple it built from engine_client.endpoint_restrictions.

Edge Case Review — normal / None / absent / boundary / exception

• normal value. Fully-wired multistage server: all 7 capabilities resolve to a handler that is not None, the helper returns no restrictions, shutdown_unsupported_routes() is never called and the route objects stay identical (verified on the assembled app: 26 → 26).
• None. Pure diffusion freezes serving_tokenization = None; duplex sets serving_tokenization = None and nulls 18 handlers in a loop, including openai_serving_completion, openai_serving_responses and anthropic_serving_messages. All 7 routes now answer 400.
• absent attribute. The pure-diffusion state has no openai_serving_completion / openai_serving_responses / anthropic_serving_messages key at all. getattr(..., None) treats absent and None identically, so the helper itself cannot raise AttributeError — both shapes are asserted, not just one.
• boundary. A route the app never mounted: route_is_mounted() returns False, so no rejection route is registered and /tokenize on a render-only or TTS-only server stays 404 instead of becoming a new 400 endpoint. This is exactly the invariant shutdown_unsupported_routes() alone would break, because it re-registers every path it is handed.
• exception / ordering. The helper runs after the existing shutdown_unsupported_routes(app, engine_client.endpoint_restrictions) call and skips exactly those capabilities, so a pipeline-declared reason is never overwritten by the generic one. If engine_client has no endpoint_restrictions attribute, the existing branch logs its warning and this call falls back to () — consistent, because nothing was restricted in that case either.
• /v1/messages depends on two handlers. It null-checks anthropic_serving_messages but builds its error envelope through serving_tokenization.create_error_response. The two cannot diverge: _init_duplex_app_state() nulls both, and omni_init_app_state() assigns serving_tokenization = ServingTokenization(...) unconditionally in the same block that assigns anthropic_serving_messages. Keying the table on anthropic_serving_messages alone is therefore sufficient; a second required attribute would be dead configuration.

Cross-Provider / Backend Impact Review

• Serving modes. Multistage: unchanged, no restriction fires. Diffusion: 7 routes move from 500 to 400. Duplex: the same 7 routes move from 500 to 400; duplex clients use /v1/realtime and /v1/chat/completions, neither of which is touched. Render-only / TTS-only: unchanged, and specifically must not gain endpoints — the reason route_is_mounted() exists.
• Entrypoints. omni_run_server_worker() is the only caller. run_batch.py and the render entrypoint assemble their apps elsewhere and are not touched.
• Shared abstractions. shutdown_unsupported_routes, remove_route_from_app and _remove_route_from_router are unchanged — this adds a caller, not a new contract. endpoint_policy gains one public helper (unwired_endpoint_restrictions) and its table; no existing signature changes, so [Core / Bugfix] Add Mechanism for Endpoint Rejection #4762's call path and the model-declared restrictions keep working as before.
• Upstream handlers. No file outside vllm_omni/ is modified, so OpenAIServingCompletion, OpenAIServingResponses, AnthropicServingMessages and ServingTokenization behave exactly as before whenever they are wired.
• Per-model policy. The skip set comes from the same engine_client.endpoint_restrictions the pipeline declares and is evaluated per process, so nothing leaks between models in a multi-model deployment.

[apex-mochen]

Rebased onto main (235c43032) to clear the merge conflict — mergeable_state is no longer dirty.

The conflict was confined to one file and was purely mechanical: both sides added a single import line to api_server.py. No logic was touched, and the resulting change is the same size as before the rebase:

• git diff origin/main HEAD -> 4 files, +263/-1 (unchanged)
• test-name set: upstream 64 is a strict subset of this branch's 74 -> 0 tests lost, exactly 10 added
• pytest tests/entrypoints/openai_api/test_api_server_guards.py -> 74 passed
• ruff check / ruff format --check clean; all 4 files LF

New head: ca033defd. DCO is green.

[apex-mochen]

Rebased onto main again — the merge conflict is cleared (mergeable_state back to blocked).

The only conflict was the import block in tests/entrypoints/openai_api/test_api_server_guards.py: #8287 reshuffled those imports while fixing the duplex.openai import cycle, and this PR adds the endpoint_policy import to the same block. Resolution takes upstream's line and keeps the added import; the duplex.openai import is dropped (upstream removed it, and duplex_openai now has zero references in that file).

No source changes: git diff --stat origin/main HEAD is still 4 files, +263/-1. Re-verified locally after the rebase: guards file 74 passed, ruff check/ruff format --check clean, LF only.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This PR stops pure-diffusion and duplex servers from returning HTTP 500 on upstream routes that stay mounted while their app.state handlers are never wired (/tokenize, /detokenize, completions, responses retrieve/cancel, messages). After normal app-state init and any model-declared endpoint_restrictions, it derives additional restrictions from missing/None handlers, applies them only when the route is actually mounted, and reuses the existing shutdown_unsupported_routes() 400 envelope. Wired deployments and unmounted surfaces stay unchanged; a pipeline-declared reason still wins over the generic unwired reason.

Change flow

flowchart TD
  A["[EXISTING] omni_run_server_worker<br/>after omni_init_app_state"]:::existing
  B["[EXISTING] shutdown_unsupported_routes<br/>model endpoint_restrictions"]:::existing
  C["[NEW] _shutdown_routes_without_handlers"]:::new
  D["[CHANGED] endpoint_policy<br/>UNWIRED_HANDLER_ROUTES + unwired_endpoint_restrictions"]:::changed
  E["[NEW] route_is_mounted"]:::new
  F["[EXISTING] shutdown_unsupported_routes<br/>replace mounted unwired handlers"]:::existing
  G["[CHANGED] unwired probes: AttributeError/500 → 400 BadRequestError"]:::changed
  A --> B --> C
  C --> D
  C --> E
  D --> F
  E --> F
  F --> G
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 8634dc0bb8de (2026-09-30T22:40:55.249371+00:00): required check(s) blocking: buildkite/vllm-omni (missing).

No actionable findings.

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[apex-mochen]

Hi @hsliuustc0106 — polite follow-up on this one. You applied the bug/frontend labels on 09-26, so I wanted to check whether it's still on the radar: it hasn't had a first human review yet.

What it does: build_app mounts routes whose handlers are never wired into state.*, so requests to them fail at request time (500). The fix rejects such mounts at startup, so a broken route fails loudly instead of never working, plus a regression test. Head 8634dc0bb8 (+263/−1, 4 files), unchanged since the last rebase; mergeable: true (blocked only by the label gate); 74 tests passed at the last verification run.

If this isn't the right area for you, could you point me to whoever owns omni's frontend wiring? The four sibling wiring PRs (#8209, #8212, #8242, #8244) are in the same state — happy to consolidate them if that's easier to review. Thanks!

[hsliuustc0106]

[P2] Cover the sibling Messages and tokenizer-info routes

The upstream vLLM 0.30.0 routers also mount POST /v1/messages/count_tokens and, with --enable-tokenizer-info-endpoint, GET /tokenizer_info. Both depend on the same handlers checked here, but neither is included in this table. On pure-diffusion or duplex servers, a valid count-tokens request still reaches the missing/None anthropic_serving_messages or the fallback serving_tokenization.create_error_response(), and tokenizer-info directly calls serving_tokenization.get_tokenizer_info() on None; these paths still produce HTTP 500. Please add both route targets to the restrictions and cover them using the real upstream routers. The new tests register synthetic routes, so they cannot catch omissions from the mounted upstream surface.

[apex-mochen]

Thanks @hsliuustc0106. On 10-09 I said I'd handle this in a separate PR. Since it uses the same table, I've added it to this PR in 026e0e06 instead.

• POST /v1/messages/count_tokens now goes through the guard keyed on anthropic_serving_messages. When the Messages handler is missing, upstream builds its error response from serving_tokenization, which pure diffusion leaves unset and duplex sets to None. That is why the request crashed.
• GET /tokenizer_info now goes through the guard keyed on serving_tokenization.

Tests use upstream's real routers. The new tests mount vllm.entrypoints.anthropic.api_router and vllm.entrypoints.serve.tokenize.api_router (with --enable-tokenizer-info-endpoint) the same way build_app does. They cover /v1/messages, /v1/messages/count_tokens, /tokenize, /detokenize and /tokenizer_info, each with the handler absent (pure diffusion) and set to None (duplex).

• Each case first checks that the unguarded upstream route raises AttributeError on the missing handler.
• It then checks that the guarded route returns 400.
• A separate test checks that every guarded path and method is actually mounted by those upstream routers, so an upstream rename would fail the test.

Verification (local, CPU):

• test_api_server_guards.py: 86 passed on the branch, and 93 passed on a merge with current main (910b8f07).
• Negative control: with only endpoint_policy.py reverted, exactly the 5 new count-tokens/tokenizer-info cases fail (81 passed).
• ruff 0.14.10 check and format are clean.

One more route I found but did not change here. I sent a request to every upstream route build_app mounts on pure-diffusion and duplex state. POST /invocations has the same problem: it dispatches by request body, so a completions-shaped body reaches the missing openai_serving_completion and returns 500. Pure diffusion does wire a chat handler, so a whole-route 400 like the others would also reject requests that work. It needs a per-request check, which I'd rather propose as a separate change. Happy to do it here if you prefer.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on 9212539811ba refers to an earlier head; the pull request now points at d85dd63b139e.

[apex-mochen]

Hi @hsliuustc0106, thanks for the review!

Re: P2 sibling Messages/tokenizer-info routes — good catch. This PR currently guards the routes that were already wired with handlers; extending the guard to /v1/messages/count_tokens and /tokenizer_info is a natural follow-up. I'll open a separate PR for those once this one lands, unless you'd prefer I bundle the coverage here.

[apex-mochen]

Hi maintainers — heads-up on the current pre-commit failure on head d85dd63b1.

The run (pre-commit/action@v3.0.1) failed at the framework level: it exited in ~40s with Process completed with exit code 1 and no file-level hook findings / annotations (only a generic .github:178 step failure + a Node-20 deprecation warning for actions/cache@v4).

Evidence this is not a code problem:

• This PR changes only 4 Python files (test_api_server_guards.py, endpoint_policy.py, api_server.py, routes.py) — no workflows, YAML or Markdown involved.
• The added lines are clean: no trailing whitespace, no debug statements, no mixed line endings, no CR.
• The same action + same config is green on our other vllm-omni PRs at the same time ([Bugfix][Frontend] Forward the serve options omni's handler wiring drops #8209 pre-commit success 00:01Z, [Bugfix] Wire the generative-scoring handler that build_app mounts for every server #8242 success 05:28Z).

Looks like a CI-infra issue (e.g. hook-env cache restore failure). Could you re-run the check? Happy to dig further if the log shows anything else.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on d85dd63b139e refers to an earlier head; the pull request now points at edcae0bcf6ff.

[apex-mochen]

Correction to my note above: this was not a CI-infra issue, and I was wrong to ask for a re-run.

The real cause is in this PR's own diff. Running the ruff version this repo pins (rev: v0.14.10 in .pre-commit-config.yaml) over the full tree at head d85dd63b1 gives:

tests/entrypoints/openai_api/test_api_server_guards.py:62:54: F401 [*] `vllm_omni.entrypoints.openai.realtime_connection` imported but unused
Found 1 error.

That unused import came in with the rebase commit 9212539811 (the base only had import api_server). ruff-check runs with --fix, so any finding makes the hook rewrite the file and the job fail — which is also why the run dies early and the check-runs API shows no file-level annotation: the finding is in the hook log, not in an annotation. Treating "no annotation" as "no finding" was the mistake.

Fix is pushed in edcae0bc: , realtime_connection removed. On the full tree All checks passed! and ruff format --check . reports all 3512 files already formatted; CR bytes 0. All four checks on the new head are green — pre-commit, build (3.11), build (3.12), DCO.

Sorry for the noise. I should have reproduced the pinned linter locally before suggesting the failure was infrastructure.

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 120s (try 2 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 600s (try 3 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; falling back to direct/cursor/auto).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (Cursor exited with 1: ActionRequiredError: You've hit your usage limit You've saved $3501 on API model usage this month with Ultra. Switch to a different model or set a Spend Limit to continue with Auto. Your usage limits will reset when your monthly cycle ends on 10/20/2026.; retrying direct/cursor).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (Cursor exited with 1: ActionRequiredError: You've hit your usage limit You've saved $3501 on API model usage this month with Ultra. Switch to a different model or set a Spend Limit to continue with Auto. Your usage limits will reset when your monthly cycle ends on 10/20/2026.; retrying direct/cursor).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (Cursor exited with 1: ActionRequiredError: You've hit your usage limit You've saved $3501 on API model usage this month with Ultra. Switch to a different model or set a Spend Limit to continue with Auto. Your usage limits will reset when your monthly cycle ends on 10/20/2026.; no fallback: the fallb).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (Cursor exited with 1: ActionRequiredError: You've hit your usage limit You've saved $3501 on API model usage this month with Ultra. Switch to a different model or set a Spend Limit to continue with Auto. Your usage limits will reset when your monthly cycle ends on 10/20/2026.; no fallback: the fallb).

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 7 days

@apex-mochen this pull request has had no human commit, comment or review since 2026-10-15. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 14 days

@apex-mochen this pull request has had no human commit, comment or review since 2026-10-15. Please consider marking this PR as draft until work can resume. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[apex-mochen]

Closing this stale bugfix PR to keep the queue focused on core feature work. It is not a technical rejection: the branch and review history remain available, and it can be reopened if maintainers want it after the core work lands.