[apex-mochen]

vLLM Version: v0.31.0 · vLLM-Omni commit: 2d9909525a8a805ed9c3a56a5287a48687654270 (this branch is two commits on top of it, head 4e27e522)
Test Plan / Result: the exact commands and their results are under "Tests" below; the base-vs-head negative control is under "Negative control".

---

Problem

vllm serve --omni builds its serve parser with vLLM's own (vllm_omni/entrypoints/cli/serve.py
imports make_arg_parser from vllm.entrypoints.launchers.cli_args), so frontend options are
accepted. Omni then constructs the handlers in its own copy of upstream's init_generate_state, and
that copy passes only a subset of the kwargs upstream passes to the same constructors, so some
options are silently ignored.

On main (2d990952), enable_per_request_metrics and enable_flash_late_interaction do not
appear in vllm_omni/entrypoints/openai/api_server.py at all, and OpenAIServingResponses is built
without default_chat_template_kwargs.

Reproduction

The flag reaches the parsed args and then stops:

import argparse, os
os.environ.setdefault("VLLM_PLATFORM", "cpu")
from vllm_omni.entrypoints.cli.serve import _ensure_vllm_platform
_ensure_vllm_platform()
from vllm.entrypoints.launchers.cli_args import make_arg_parser

parser = argparse.ArgumentParser(prog="vllm serve --omni")
parser = make_arg_parser(parser)
ns = parser.parse_args(["demo-model", "--enable-per-request-metrics",
                        "--default-chat-template-kwargs", '{"enable_thinking": false}'])
print(ns.enable_per_request_metrics, ns.default_chat_template_kwargs)

True {'enable_thinking': False}

The handlers then never see enable_per_request_metrics, so the per-request timing metrics that
upstream gates on this flag are not enabled, and OpenAIServingResponses parses responses with its
own template kwargs instead of the server-wide default that the renderer applies.

Fix

Forward the options at the construction sites, matching what upstream init_generate_state
(vLLM v0.31.0) passes:

#	site	option	why it is not a no-op
1	_init_chat_services -> OmniOpenAIServingChat / OmniOpenAIServingChatBatch (shared by the multi-stage and duplex paths)	enable_per_request_metrics	--enable-per-request-metrics: per-request timing metrics
2	OpenAIServingCompletion	enable_per_request_metrics	same flag, completion path
3	OpenAIServingResponses	enable_per_request_metrics	same flag, responses path
4	OpenAIServingResponses	default_chat_template_kwargs	upstream passes it in init_generate_state
5	ServingScores	enable_flash_late_interaction	--enable-flash-late-interaction (True by default upstream, so it matters when disabled)

Upstream passes enable_per_request_metrics to chat, completion and responses and to no other
constructor; all three now receive it.

enable_per_request_metrics and enable_flash_late_interaction are read as
getattr(args, ..., <upstream default>), matching upstream's own
enable_flash_late_interaction=getattr(args, ..., True) in vllm/entrypoints/pooling/factories.py:
callers in this repository build partial argument namespaces
(tests/entrypoints/openai_api/test_duplex_api_server.py does exactly that), so for them an absent
option has to keep meaning its upstream default instead of raising AttributeError.

Tests

tests/entrypoints/openai_api/test_api_server_guards.py gains two regression tests. They replace
the handler classes with recorders and assert which options reach each constructor:

• the multi-stage test runs omni_init_app_state with the options on, then again with them off, and
  checks chat, chat batch, completion and responses for both True and False (so a hardcoded value
  cannot pass), plus default_chat_template_kwargs and enable_flash_late_interaction;
• the duplex test runs _init_duplex_chat and checks True and False the same way.

Local, CPU only:

$ python -m pytest -q tests/entrypoints/openai_api/test_api_server_guards.py
74 passed
$ python -m pytest -q tests/entrypoints/openai_api/test_api_server_guards.py tests/entrypoints/openai_api/test_duplex_api_server.py
122 passed

tests/entrypoints/openai_api/ as a whole: 1674 passed on this branch and 1672 passed on a clean
main checkout, with the same 21 failures and 75 errors on both. Those come from missing CUDA,
cache_dit and model servers in my environment; none are new.

Negative control

main's api_server.py (2d990952, byte-identical) with this PR's tests: exactly the two new
tests fail.

2 failed, 72 passed
FAILED tests/entrypoints/openai_api/test_api_server_guards.py::test_multistage_wiring_forwards_options_upstream_passes
FAILED tests/entrypoints/openai_api/test_api_server_guards.py::test_duplex_wiring_forwards_options_upstream_passes

With the previous head's api_server.py (no completion or responses forwarding), the multi-stage
test fails at completion; with only the responses enable_per_request_metrics line removed, it fails
at responses.

ruff check and ruff format --check (0.14.10), check_test_marks.py and mypy are clean.

GPU: the main CUDA CI has not run on this head yet, so there are no GPU results.

Relation to #7646

#7646 (merged) moved chat, chat-batch and duplex construction into _init_chat_services and passes
log_error_stack on the duplex path, so this PR no longer touches either. Site 1 is now a single
line in that shared helper.

Out of scope

• Upstream also passes tool_strict_level to chat, responses and anthropic, and
  disabled_thinking_effort to anthropic; Omni's wiring passes neither.
• Consuming enable_per_request_metrics inside Omni's own chat generators (raised in review) needs its own change.

AI assistance

This patch was produced with AI assistance (Claude Code). The audit that located the missing
forwarding, the patch and the regression tests were AI-generated; every claim above was verified by
running the tests locally against this PR's base, including the negative control.

Alternative approaches

• Pass the whole parsed args namespace into the handlers instead of the
  named options: those constructors take explicit keyword arguments, so an
  opaque namespace would weaken their contract and hide typos at the call site.
• Call upstream's init_generate_state and inherit its construction
  wholesale: it also builds the entire generate-side surface, which
  omni_init_app_state deliberately builds itself. Forwarding the missing
  options keeps that structure untouched.

Compatibility

No flag is added, removed, or renamed. Each option is passed to the constructor
upstream already passes it to. With the options unset, every constructor receives
exactly what it received before (the upstream defaults), so existing deployments
are unaffected; with an option set, the handler finally sees it.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/error_contracts.md, docs/design/module/profiling.md, docs/design/module/entrypoints.md.

Module owners: @alex-jw-brooks @linyueqian @NickCao

Routing: @alex-jw-brooks via module of the changed files, module named in the PR description, CODEOWNERS; @linyueqian via module of the changed files, module named in the PR description, CODEOWNERS; @NickCao via module of the changed files, module named in the PR description, CODEOWNERS

@apex-mochen, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit d156e9788057 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

Omni already accepts upstream serve flags through make_arg_parser, but its own handler construction in omni_init_app_state and _init_duplex_chat forwarded only a subset of those kwargs. This change passes default_chat_template_kwargs into OpenAIServingResponses, enable_per_request_metrics into chat/completion (including duplex chat), enable_flash_late_interaction into ServingScores, and log_error_stack into the duplex OnlineRenderer. Missing attributes keep the upstream defaults via getattr, so partial test namespaces do not raise. The user-visible effect is that those CLI flags now reach the same constructors upstream uses, instead of being parsed and then ignored.

Change flow

flowchart TD
    A["[EXISTING] Parsed vllm serve --omni args"]:::existing
    B["[CHANGED] omni_init_app_state and _init_duplex_chat kwargs"]:::changed
    C["[EXISTING] Responses/chat/completion/scores/renderer ctors"]:::existing
    D["[NEW] Constructor-capture wiring regression tests"]:::new
    E["[EXISTING] Template kwargs, per-request metrics, flash-LI, error stacks"]:::existing
    A --> B
    B --> C
    C --> E
    D --> B
    classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
    classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
    classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
    classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at e00bc2999893 (2026-09-27T20:13:21.936724+00:00): required check(s) blocking: buildkite/vllm-omni (missing), and -5 more.

No actionable findings.

[apex-mochen]

Author self-review (@apex-mochen), as the review bot asks.

What I checked

• Scope. 2 files; the source change is six one-line additions at six
  construction sites (+6/-0), no other logic touched, plus +141 of tests.
• Root cause, verified against the base rather than inferred. omni wires the
  handlers in its own copies of upstream's init path and passes only a subset of
  the kwargs upstream passes to the same constructors; on the base commit
  enable_per_request_metrics and enable_flash_late_interaction appear nowhere
  in vllm_omni/entrypoints/openai/api_server.py, while the flags themselves parse
  fine (reproduction in the body). I diffed each of the six call sites against the
  upstream constructor call it mirrors, so every forwarded argument is one
  upstream already passes, with upstream's own default.
• Both directions tested. The multi-stage test asserts which options reach each
  handler including the off values, so a hardcoded constant cannot pass it; the
  duplex test does the same for the duplex path.
• Partial-namespace edge case. Two options are read with getattr(args, …, <upstream default>) because callers inside this repo build partial arg
  namespaces (test_duplex_api_server.py does exactly that) — for them an absent
  option must keep upstream's meaning rather than raise AttributeError.
  enable_flash_late_interaction uses upstream's default of True, matching
  vllm/entrypoints/pooling/factories.py.
• Cross-provider / cross-mode impact. No provider, backend, or engine code is
  touched; every option goes to the constructor upstream passes it to, and the
  duplex path is covered explicitly so the two modes cannot diverge.
• Negative control. With upstream's unmodified api_server.py in the same
  tree (byte-verified against this PR's base) and only the tests added, exactly the
  two new tests fail: 2 failed, 64 passed. Re-applying → 66 passed.
• Test runs, local, CPU/WSL/vLLM 0.30.0. guards file 66 passed; duplex file
  40 passed; ruff check and ruff format --check clean.
• Overlap disclosed. [Frontend] Share regular and duplex chat initialization #7646 passes log_error_stack on the duplex path — the
  same omission this PR fixes at site 5. Whoever lands second rebases one line;
  no other overlap.
• Disclosure. AI assistance was used to prepare this change; I reviewed every
  line and ran every command reported above myself.

No GPU was used or needed.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 7 days

@apex-mochen this pull request has had no human commit, comment or review since 2026-09-28. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[apex-mochen]

Current plan and next step, as the bot asks.

State: the PR is complete and unchanged since 09-27 (e00bc299): six one-line kwargs at six construction sites (+6/−0 production, +141 tests). Every check that has run on the head is green (pre-commit, build (3.11), build (3.12), DCO); the buildkite jobs have not been triggered. The author self-review comment above covers scope, both directions of the wiring (including the off-values), the partial-arg-namespace edge case, and the negative control against upstream's unmodified api_server.py.

Blocker: none on my side — this needs a maintainer review/decision. There is no code work pending, and the branch has no conflicts with main.

Plan: I will keep the branch current and rebase if it goes stale. If there is no review by 10-09 I will ask the maintainer who has been handling the frontend/init area for this repo. This PR is one of a family of five wiring fixes from the same audit (#8169, #8209, #8212, #8242, #8244) — if you would prefer them combined, split differently, or reviewed in a particular order, say so and I will restructure.

Next step from me: none until there is reviewer feedback; happy to answer anything specific in the meantime.

[apex-mochen]

Hi @alex-jw-brooks, could you take a look at this one when you have a moment? The branch is a single commit on top of main (0 behind), all checks are green (pre-commit, build 3.11/3.12, DCO), and the change is six one-line kwargs at six construction sites (+6/-0 production, +141 tests) — the handler wiring now forwards the same serve options the CLI parser already accepts. Happy to rebase or adjust if needed.

[apex-mochen]

Still active. The current head is mergeable and I’m keeping the scope limited to forwarding the serve options that omni’s handler wiring currently drops.

My next step is human review; I’ll rebase and rerun the focused frontend tests if main moves in a way that affects this path, and I can address review feedback promptly.

[hsliuustc0106]

[P2] Consume the per-request metrics flag in Omni's response generators

Passing the flag reaches the inherited constructor, but OmniOpenAIServingChat overrides both chat_completion_full_generator() and chat_completion_stream_generator() and neither reads self.enable_per_request_metrics or builds the upstream timing metrics. Instead, response metrics are still filtered by _filter_stage_metrics_detail() solely according to the request's return_stage_metrics extra-body flag. An ordinary chat request with --enable-per-request-metrics enabled therefore still receives no timing metrics, including on the duplex chat path. Please honor this flag when constructing the Omni responses and add a response-level regression test; recording constructor kwargs does not verify the advertised behavior.

[apex-mochen]

Hi @hsliuustc0106, thanks for the review!

Re: P2 per-request metrics flag in response generators — noted. This PR forwards the serve options that the handler wiring drops; consuming the flag inside the overridden generators is a separate extension. Happy to open a follow-up PR for that, or expand scope here if you prefer — just let me know.

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 120s (try 2 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 600s (try 3 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 120s (try 2 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 600s (try 3 of 3)).

[vllm-omni-review-bot]

Omni ReviewBot attempt record

Review attempt ended as failed (failed; falling back to direct/cursor/auto).

[vllm-omni-review-bot]

Omni ReviewBot review

Changes since the previous review

• 0 new inline finding(s); 1 finding(s) below.

CI at 2e0e9b437d61 (2026-10-10T15:52:44.210186+00:00): required check(s) blocking: buildkite/vllm-omni (missing).

Note: The assigned review arm strict/cursor/cursor-grok-4.6-high could not complete this review, so it was produced by the fallback arm direct/cursor/auto. It is excluded from the routing experiment.

Full review analysis

PR description

vllm serve --omni already parses upstream frontend flags, and this patch forwards three of them at Omni's own handler setup. Shared chat and batch construction now passes enable_per_request_metrics, OpenAIServingResponses receives default_chat_template_kwargs, and ServingScores receives enable_flash_late_interaction. Duplex chat goes through the same chat helper, so it gets the metrics flag; the renderer already received log_error_stack. The PR text lists six new call sites, but the frozen diff adds only those three lines, and OpenAIServingCompletion is left unchanged.

Change flow

flowchart LR
  Args["[EXISTING] Parsed serve args"]:::existing
  ChatInit["[CHANGED] _init_chat_services chat kwargs"]:::changed
  Responses["[CHANGED] OpenAIServingResponses template kwargs"]:::changed
  Scores["[CHANGED] ServingScores flash-late-interaction flag"]:::changed
  Completion["[EXISTING] OpenAIServingCompletion"]:::existing
  Tests["[NEW] Constructor-capture regression tests"]:::new
  Args --> ChatInit
  Args --> Responses
  Args --> Scores
  Args --> Completion
  ChatInit --> Tests
  Responses --> Tests
  Scores --> Tests
  Completion --> Tests
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

Findings

• [P1] Completion wiring still drops per-request metrics — vllm_omni/entrypoints/openai/api_server.py:1060

Evidence for Completion wiring still drops per-request metrics

omni_init_app_state builds OpenAIServingCompletion without enable_per_request_metrics, so --enable-per-request-metrics still never reaches /v1/completions. Chat and batch get the flag from _init_chat_services (getattr(args, "enable_per_request_metrics", False) at the shared kwargs), but the completion call only passes return_tokens_as_token_ids, enable_prompt_tokens_details, and enable_force_include_usage. The new test_multistage_wiring_forwards_options_upstream_passes records that constructor and asserts the value is True when the flag is set and False from _minimal_args(). A missing keyword is None, so both assertions fail and the reported 66 passed result does not match this head. Pass the same attribute into OpenAIServingCompletion, using the same default as the chat kwargs.

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[vllm-omni-review-bot]

Omni ReviewBot: finding feedback

[p1] Completion wiring still drops per-request metrics — vllm_omni/entrypoints/openai/api_server.py:1060

See the review for details. If you are the PR author and disagree, react 👎 here; the maintainer will see your disagreement.

[linyueqian]

Option forwarding is incomplete: the new constructor-capture test expects per-request metrics on completions, but the completion constructor still omits the option. Please complete this wiring before merge.

Reviewed 2e0e9b4 statically. The finding below follows from the recorded constructor kwargs; no PR code or tests were executed. Main CUDA has not reported on this head.

[linyueqian]

[important] Forward enable_per_request_metrics to OpenAIServingCompletion as well. The new _recording_ctor stores keyword arguments passed by omni_init_app_state, but the completion constructor still omits this flag. For completion, the assertion at line 1308 sees None instead of True, and the disabled assertion at line 1315 similarly sees None instead of False. The newly added regression test cannot pass with the current wiring, and completions continue to ignore the requested setting. Please add the forwarding argument at the completion construction site and verify both flag values.

[apex-mochen]

Thanks @linyueqian, you're right: the completion wiring was missing on 2e0e9b43.

Root cause. The original revision (e00bc299) forwarded the flag at six construction sites. Main later moved chat, batch and duplex construction into _init_chat_services, and when I rebased onto it, the completion keyword was lost. The constructor-capture test therefore could not pass on 2e0e9b43. The "66 passed" figure in the description predates that rebase and does not hold for that head. Sorry for the noise.

Fixed in 4e27e522 (rebased onto main 2d990952):

• OpenAIServingCompletion now receives enable_per_request_metrics, using the same getattr(..., False) default as the chat kwargs.
• I compared every handler constructor in omni_init_app_state against upstream init_generate_state in vLLM v0.31.0. Upstream passes this flag to chat, completion and responses. OpenAIServingResponses had the same gap, so the flag is now forwarded there as well. Upstream passes it to no other constructor.
• Tests:
  • The wiring test now checks chat, chat batch, completion and responses with the flag both True and False.
  • The duplex test now also covers False.
  • I removed the OpenAIServingRealtime stub, because main no longer constructs that class in omni_init_app_state. With the stub in place, the test raises AttributeError on current main.

Verification (local, CPU only):

• tests/entrypoints/openai_api/test_api_server_guards.py: 74 passed.
• Negative controls:
  • With the previous api_server.py, the wiring test fails at completion.
  • With only the responses line removed, it fails at responses.
• tests/entrypoints/openai_api/: 1674 passed. The 21 failures and 75 errors are the same set as on a clean main checkout (1672 passed). They come from missing CUDA, cache_dit and model servers in my environment, and none are new.
• ruff 0.14.10 check and format, check_test_marks.py, and mypy are clean.

On this head, pre-commit, DCO and CodeQL are green and the builds are queued. Main CUDA has not run yet, so I have no GPU results to report.

Two related items are out of scope for this PR, and I'm happy to follow up on them separately:

• Upstream also passes tool_strict_level to chat, responses and anthropic, and disabled_thinking_effort to anthropic. Omni's wiring passes neither.
• @hsliuustc0106's point about consuming the flag inside Omni's chat generators still needs its own change.

Could you take another look when you have a chance?

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 7 days

@apex-mochen this pull request has had no human commit, comment or review since 2026-10-14. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[apex-mochen]

@linyueqian — following up on your review: the completion wiring you flagged is fixed in 4e27e522 (details in my reply above). The builds that were still queued then have since finished, and every check on that head is green now (build 3.11 / 3.12, pre-commit, DCO, CodeQL, docs). It still merges cleanly into main.

Plan, since the bot asked: no further changes on this PR unless you'd like some. The two follow-ups I listed above (consuming the flag in Omni's chat generators, per @hsliuustc0106, and the tool_strict_level / disabled_thinking_effort wiring) will be separate PRs. Could you take another look when you have a moment?

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on 4e27e5224073 refers to an earlier head; the pull request now points at ce06c589cce5.

[linyueqian]

@apex-mochen the change looks good to me, and Buildkite ran on 4e27e52 after the ready label. Its only failure is test_inventory_matches_reviewed_snapshot_counts (build 18099), which counts one more PUBLIC_OMNI variable than the snapshot it compares against; this PR adds no environment variable, and the same failure shows up on other PRs with older bases. Could you merge main into the branch (please merge, not rebase)? I will approve once Buildkite is green on the new head.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 7 days

@apex-mochen this pull request has had no human commit, comment or review since 2026-10-26. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[apex-mochen]

Hi @linyueqian — merged main into the branch as you asked (merge commit 5e4ea82b, not a rebase). The PR is mergeable again.

One conflict, in vllm_omni/entrypoints/openai/api_server.py: main now resolves default_chat_template_kwargs through resolve_default_chat_template_kwargs(args) and adds enable_log_outputs / enable_log_deltas to the completion handler. I kept upstream's resolved variable and its two new arguments, and kept this PR's forwarded option (enable_flash_late_interaction for the pooling/embedding handler).

Worth flagging: main already carries the per-request metrics wiring for chat / completion / responses, so the unique part of this diff is now that one forwarded option plus the constructor-capture guard test.

buildkite/vllm-omni is running on 5e4ea82b. The test_inventory_matches_reviewed_snapshot_counts failure you mentioned earlier is unrelated to this diff — this PR adds no environment variable.

[apex-mochen]

@linyueqian — all GitHub checks are green on 5e4ea82b (Analyze, build (3.11), build (3.12), pre-commit, CodeQL, DCO, docs).

buildkite/vllm-omni has not started on this head: on the previous commits it only ran after the ready label was applied (e.g. on 4e27e522), and the label is already on the PR, so this push did not re-trigger it. Could you re-apply the ready label (or kick off the build) on 5e4ea82b when you get a chance?

Nothing in the diff changed beyond the merge resolution described above — one line of forwarded option plus the constructor-capture guard test.

[apex-mochen]

Thanks for the review @linyueqian! Pushed 4e27e52240 to forward enable_per_request_metrics to OpenAIServingCompletion as well. The new _recording_ctor test now asserts all three options (enable_per_request_metrics, default_chat_template_kwargs, enable_flash_late_interaction) reach every handler.

[apex-mochen]

@linyueqian — done, merged main ($(.commit.sha.Substring(0,10))) into ix/serve-options-not-forwarded. New head is 748a082b54. Buildkite should re-run on the new head — would appreciate the re-review when it's green. Thanks!

[hsliuustc0106]

@apex-mochen, one current-head test issue on 748a082:

• P2: Update the default-template-kwargs assertion and rerun the focused wiring test. The new assertion expects None, but the pinned vLLM resolver normalizes the unset value to {}. Omni calls that resolver and forwards its result, so is None rejects the correct wiring. Expect {} and record the command, runtime version, and passing result for test_multistage_wiring_forwards_options_upstream_passes on the fix SHA.

This finding follows from source inspection; I did not execute the test or observe it fail in CI.

[apex-mochen]

@hsliuustc0106 — good catch, pushed 761ab28 to fix the assertion. You're right that
esolve_default_chat_template_kwargs normalizes via dict(args.default_chat_template_kwargs or {}), so the unset value comes through as {} rather than None. Changed the assertion from is None to == {} to match the actual wiring.

[hsliuustc0106]

@apex-mochen, the {} assertion is corrected at 761ab28. One new test regression remains:

• P2: Restore the timestamp middleware test's actual dispatch stub. Line 713 now replaces middleware._inner, but the wrapper captures and calls _asgi. Replacing _inner after initialization does not replace that callable, so the HTTP/websocket calls do not reach the recording stub and cannot satisfy the called assertions. Restore the _asgi replacement (or inject the fake before wrapper construction), then record the fix SHA, runtime and passing focused timestamp/wiring tests.

Source-inspected only; I did not execute these tests or observe this failure in CI.

[apex-mochen]

@linyueqian — just a heads-up that all CI checks are green on the new head 95f38f0 (build 3.11/3.12, pre-commit, DCO, CodeQL, Analyze). Also fixed the P2 assertion issue @hsliuustc0106 flagged (is None → == {} to match the resolver normalization). Ready for your review whenever you have a moment.

[apex-mochen]

@hsliuustc0106 — both items are addressed at head d156e978 (parent 95f38f0b9).

P2 (default-template-kwargs assertion). Already in at 95f38f0b9: test_api_server_guards.py now asserts captured["responses"].get("default_chat_template_kwargs") == {}, matching the dict(args.default_chat_template_kwargs or {}) normalization you pointed at.

P2 (timestamp middleware dispatch stub). Fixed in d156e978. Single hunk in tests/entrypoints/openai_api/test_api_server_guards.py: middleware._inner = inner → middleware._asgi = inner, with a comment naming _asgi as the captured dispatch callable. The wrapper still sets _inner in __init__, so __getattr__ delegation and the hasattr(served_wrapper, "_inner") assertion in the routing test are untouched.

What I could execute here:

• python -m py_compile on the committed file: OK.
• Executed the real _TimestampMiddleware source, extracted verbatim from vllm_omni/entrypoints/openai/api_server.py and exec'd against a stub ASGI app (no vLLM import needed):
  • before, _inner replaced: stub called = {'http': False, 'websocket': False}, real app called = {'http': 1, 'websocket': 1} — the called assertions can never pass, exactly as you described;
  • after, _asgi replaced: stub called = {'http': True, 'websocket': True}, request_timestamp is a float on the HTTP scope and absent from the websocket scope.

What I could not execute: the guards module through pytest on this machine. It is CPU-only (python 3.12, torch 2.13.0+cpu, vLLM 0.1.dev1+g157bcb7c editable), and import vllm_omni fails before collection at vllm_omni/patch.py:39 — utils.compute_mm_prefix_ranges does not exist in that vLLM revision. So I have no green pytest log to quote, and I would rather state that than paste a run I did not produce.

On CI: pushing to the branch does not create a new Buildkite build here — the .buildkite/cuda steps are gated on build.pull_request.labels includes "ready", and a build is created by the label-add event, while ready is already present. If you re-add ready (or re-run the Buildkite build for d156e978), it will exercise test_timestamp_middleware_stamps_http_and_passes_websocket and test_multistage_wiring_forwards_options_upstream_passes on the fix SHA and give you the runtime plus passing result you asked for.

Note on the SHA: the diff first landed as 4c795942, but that server-side commit carried the GitHub display name as its author, so DCO failed against the Signed-off-by: apex-mochen trailer. I recreated it with the matching identity and force-updated the branch, so the head is now d156e978 (same parent 95f38f0b9, identical tree) and DCO is green.