fix: include diffusion scheduler wait in queue metrics

[Asthenia0412]

What changed

Requests to diffusion models can wait in the diffusion scheduler after the Orchestrator has already handed them to a stage. That admission delay was recorded separately, so queue_wait_ms and the request queue-wait histogram showed only the earlier Orchestrator wait. This change combines the two sequential waits for the response metric and the aggregate request histogram. The per-stage stage_in_queue_s observation remains available for diagnosing a specific scheduler.

I added regression coverage for the combined value, scheduler-only waits, valid zero waits, and the response/Prometheus metric path. The metrics design doc now describes the combined meaning.

Fixes #5924

Validation

• Python compile check passed for the changed implementation and test files.
• Ruff, formatting, typos, Markdown lint, SPDX, and the other applicable pre-commit hooks passed. The local mypy hook reports existing errors elsewhere in metrics/utils.py and is skipped in the project's CI configuration.
• I could not run pytest locally because this environment does not have the vllm package installed. The focused test command is pytest -q tests/metrics/test_emit_calls.py.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/observability.md, docs/design/module/diffusion/index.md.

Module owners: @david6666666 @Isotr0py @princepride

Routing: @david6666666 via module named in the PR description; @Isotr0py via module named in the PR description; @princepride via module named in the PR description

@Asthenia0412, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 9ab163aa7fca produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[hsliuustc0106]

Minor: the combined value also overlaps stage_gen_time_s / stage_*_gen_ms. The diffusion admission wait is still measured inside the stage's generation window (that is where #5924 observed the missing ~135s), so for diffusion stages queue_wait + stage_gen_time now double-counts it. One sentence noting the overlap would keep consumers from reconstructing E2E latency as a sum of the two.

[hsliuustc0106]

None of the lanes on this PR execute this suite (the GitHub build lanes don't run pytest and no test-trigger label is set), and the PR body notes pytest couldn't run locally — so these assertions, including the 135131.0 / 135.131 expectations, haven't been executed anywhere. Could you run pytest -q tests/metrics/test_emit_calls.py on a vllm-capable environment (or flag which lane will pick it up)?

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Direct under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

Diffusion requests can sit in the stage scheduler after the Orchestrator has already handed them off. This change adds that later admission wait to the existing Orchestrator queue_wait_ms so the response field and the request_queue_wait_s histogram report the full sequential queue time, while stage_in_queue_s stays a per-stage diagnostic. Present zeros are still emitted and missing sources still produce no sample. The user-visible effect is that serialized diffusion jobs no longer look almost unqueued on the request metric while their wait is buried in stage generation time.

Change flow

flowchart TD
  A["[EXISTING] Orchestrator queue_wait_ms and diffusion scheduler_queue_wait_s"]:::existing
  B["[CHANGED] extract_queue_wait_s sums sequential waits"]:::changed
  C["[CHANGED] OmniBase._process_single_result writes combined queue_wait_ms"]:::changed
  D["[EXISTING] Response metrics and request_queue_wait_s histogram"]:::existing
  E["[NEW] Combined-wait regression tests in test_emit_calls.py"]:::new
  F["[CHANGED] docs/design/metrics.md Path 2 wording"]:::changed
  A --> B --> C --> D
  B --> E
  C --> F
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 9ab163aa7fca (2026-09-26T20:24:45.627414+00:00): required check(s) blocking: buildkite/vllm-omni (missing), and -5 more.

Findings

• [P1] Combined-wait regression tests have no execution evidence — tests/metrics/test_emit_calls.py:347
  The new pins in TestQueueWaitExtraction and test_queue_wait_response_and_histogram_include_diffusion_admission are the only check that _process_single_result writes 135131.0 ms and calls observe_queue_wait(135.131). The PR Validation section states those tests were not run locally because vllm is missing. Frozen CI for this head at 2026-09-26T20:24:45Z shows pre-commit and Read the Docs passed; the required buildkite/vllm-omni check is missing, and the GitHub checks that did run do not execute pytest. .buildkite/** is not materialized here and git show could not be executed, so this does not name a lane or selector. The assertions still need a vllm-capable pytest run, or the required Buildkite check needs to report, before the 135s combine path is trusted.
  Existing thread: #8170 (comment)
• [P3] Combined queue_wait still undocumented as overlapping stage_gen_time — docs/design/metrics.md:78
extract_queue_wait_s now adds scheduler_queue_wait_s into response queue_wait_ms and request_queue_wait_s. The catalog still describes stage_gen_time_s as including in-stage queueing, which is where the missing ~135s admission wait was observed. Summing the two families therefore double-counts that wait. Path 2 at this line describes the combine but does not warn consumers; the table row for request_queue_wait_s still says "Orchestration-layer queue wait", which no longer matches the implementation. Add one sentence that the combined wait already sits inside stage generation time, and update the table row to the combined meaning.
  Existing thread: #8170 (comment)

[vllm-omni-review-bot]

Omni ReviewBot: finding feedback

[p1] Combined-wait regression tests have no execution evidence — tests/metrics/test_emit_calls.py:347

The new pins in TestQueueWaitExtraction and test_queue_wait_response_and_histogram_include_diffusion_admission are the only check that _process_single_result writes 135131.0 ms and calls observe_queue_wait(135.131). The PR Validation section states those tests were not run locally because vllm is missing. Frozen CI for this head at 2026-09-26T20:24:45Z shows pre-commit and Read the Docs passed; the required buildkite/vllm-omni check is missing, and the GitHub checks that did run do not execute pytest. .buildkite/** is not materialized here and git show could not be executed, so this does not name a lane or selector. The assertions still need a vllm-capable pytest run, or the required Buildkite check needs to report, before the 135s combine path is trusted.

This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 7 days

@Asthenia0412 this pull request has had no human commit, comment or review since 2026-09-26. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 14 days

@Asthenia0412 this pull request has had no human commit, comment or review since 2026-09-26. Please consider marking this PR as draft until work can resume. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.