[lm10-arg]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Localize the intermittent MiniCPM-o 4.5 Stage-2 Code2Wav hang seen in AMD nightly CI and prevent one permanent hang from producing repeated 900-second client-timeout waves until the job-level timeout.

2. What is the highest-leverage variable for that outcome?

Code2Wav phase and owner-thread stack observability at the stall, together with benchmark cancellation immediately after the first structurally identified request timeout.

3. How will this work move that variable?

Add an opt-in, per-model no-progress watchdog that records phase transitions and reports the blocked Python stack, plus an opt-in benchmark circuit breaker that preserves completed outputs and cancels every remaining queued or in-flight request after the first timeout.

4. Whose customer or user needs does this address, and what are those needs?

CI maintainers and model contributors need actionable evidence about the blocked Stage-2 operation and bounded accelerator time, while retaining partial benchmark metrics and artifacts for diagnosis.

Purpose

The soft failure in AMD build 14607 is part of an intermittent pattern also seen in builds 14528, 14598, 14604, and 14606 across different nodes. Builds 14541 and 14600 are passing controls. The failure boundary is a permanent Stage-2 Code2Wav stall under the nightly's ROCm concurrency of 10; the exact blocked operation is not identifiable from the current logs. The later 900-second client TimeoutError waves are downstream symptoms.

Before this change:

• a stalled Code2Wav forward has no phase or owner-thread stack evidence;
• after ten active requests time out, the benchmark semaphore admits another wave of queued requests;
• the cycle can repeat until the 165-minute job timeout.

After this change, when explicitly enabled:

• VLLM_OMNI_CODE2WAV_WATCHDOG_TIMEOUT_S reports setup, encoder, CFM mode, cache materialization, HiFT, or output-copy phase details and the blocked owner-thread Python stack once per operation without interrupting it;
• --omni-fail-fast-on-timeout is limited to immediately dispatched openai-chat-omni workloads (--request-rate inf, non-self-timed). Unsupported dispatch modes are rejected before the benchmark starts. Python and aiohttp server timeouts are marked structurally; remaining tasks are cancelled and awaited; completed outputs stay in request order; and timeout/cancellation counts are recorded;
• the AMD MiniCPM-o accuracy job enables a 60-second watchdog and its Seed-TTS benchmark enables the circuit breaker. Seed-TTS validation fails closed when any request times out or is cancelled, so a high-scoring partial subset cannot pass. The existing 900-second request timeout and 165-minute job timeout are unchanged.

This PR does not claim to fix the underlying ROCm/GPU hang. It adds the evidence needed to identify the blocked operation and bounds the repeated-timeout failure mode.

The public environment-variable guide is updated. Existing benchmarking, observability, model-integration, and autoregressive-runtime design pages are draft contracts; this opt-in diagnostic behavior does not change their serving or output semantics.

Implementation and PR preparation were assisted by OpenAI Codex and reviewed with the repository's full precheck-pr workflow.

Test Plan

Reproduction in the affected AMD CI environment uses the existing nightly node at concurrency 10:

pytest -s -v \
  tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py::test_minicpmo_4_5_seed_tts_wer_bench \
  -m "full_model and omni and rocm and MI325 and cards_1" \
  --run-level full_model

Added regression coverage verifies:

• Python and aiohttp.ServerTimeoutError chat deadlines are marked with timed_out=True;
• the first timeout cancels pending work, awaits cancellation cleanup, retains completed outputs, and returns ordered placeholders;
• unsupported backend, finite-rate, and self-timed fail-fast modes are rejected before benchmark dispatch;
• Seed-TTS accuracy rejects otherwise high-scoring partial results when timeout/cancellation counters are nonzero;
• the CLI flag defaults off and parses on;
• the watchdog does not report completed work, reports phase/details/owner stack after no progress, resets its deadline on phase updates, replaces stale details between phases, and terminates its monitor thread on close.

vLLM Version: v0.32.0 API baseline

vLLM-Omni Commit: 761b86ade040ad3da82552212b46c3b5b05a14ba

Test Result

Passed:

• pre-commit run --from-ref origin/main --to-ref HEAD
  • Ruff check/format, typos, Markdown lint, Python 3.10 mypy, test marks, SPDX, forbidden imports, new torch.cuda guard, and Buildkite schema validation all passed.
• python3 -m py_compile for every changed Python file.
• Standalone watchdog validation: phase/details/owner-stack report fired once, phase updates replaced stale details, and close() terminated the daemon monitor thread.
• git diff --check origin/main...HEAD.

Not run locally:

• Focused pytest collection is unavailable on this Mac because the local Python 3.14 environment has no PyTorch/vLLM (ImportError: Error importing plugin "tests.helpers.fixtures.config": No module named 'torch').
• The remaining required runtime gate is the exact-head AMD MiniCPM-o accuracy job. If the intermittent hang recurs, it must emit the new Stage-2 phase/stack report and stop after the first 900-second timeout rather than admitting additional request waves.

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-25T17:17:20.142614Z We couldn’t complete this review. Comment @codex review to try again.	5ebde0d	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/ar_runtime.md, docs/design/module/benchmarking.md, docs/design/module/model_integration.md, docs/design/module/observability.md.

Module owners: @alex-jw-brooks @tzhouam @Bounty-hunter

Routing: @alex-jw-brooks via module of the changed files, module named in the PR description, CODEOWNERS; @tzhouam via module of the changed files, module named in the PR description, CODEOWNERS; @Bounty-hunter via module of the changed files, module named in the PR description

@lm10-arg, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 761b86ade040 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[hsliuustc0106]

• P2 — stop the request producer when the first timeout completes. @lm10-arg, at the new gather call, _gather_request_outputs starts only after get_request(...) has emitted the entire workload. With finite-rate or self-timed arrivals, an earlier request can time out while that producer is still waiting; subsequent requests continue to be dispatched until the last scheduled arrival, despite --omni-fail-fast-on-timeout. Monitor timeout completion concurrently with production and stop/cancel the producer as well as outstanding requests. Add a regression through the benchmark dispatch path with delayed arrivals: after the first timeout, assert no later request is sent, cancellation cleanup is awaited, completion is bounded, and completed outputs retain their order. The direct gather-helper tests do not exercise this producer/monitor ordering.

Source-only inspection at 5ebde0d8f95ed6e0efd81c5c3e92c66e1b083355; no local runtime or GPU test executed.

[amy-why-3459]

Reviewed the opt-in Code2Wav watchdog and --omni-fail-fast-on-timeout path on tip 5ebde0d8.

Direction is sound for the AMD Stage-2 hang diagnosis goal: phases reset the deadline, reports fire once without interrupting the hung op, details are bounded, and the gather helper preserves completed outputs in request order while cancelling the rest. Default-off (timeout_s=0 / flag off) keeps serving semantics unchanged. Unit coverage for report/phase/close and the gather-helper contract looks solid.

Points:

1. Producer vs fail-fast (already flagged). _gather_request_outputs only runs after the request-production loop has emitted the full workload. With delayed / rate-limited arrivals, later requests can still be dispatched after an earlier request has already timed out. Fail-fast needs to observe timeout completion concurrently with production and stop the producer, with a regression through that path — the direct gather tests do not cover it.

2. Timeout classification. timed_out=True is set only on asyncio.TimeoutError / TimeoutError. If the shared session raises aiohttp.ServerTimeoutError (or another ClientError subclass) for the same total/sock deadline, the request lands in the generic client-error path without timed_out, so fail-fast never trips. Please confirm which exception the 900s omni session actually raises and treat that family as structural timeouts too.

3. Watchdog detail merge. phase() does {**operation.details, **new}, so keys from earlier phases linger on later reports (e.g. codec_frames still present under decode/cfm). Prefer replacing details per phase (or namespacing) so a stall report reflects the blocked phase only.

Covered: code2wav_watchdog.py, BatchedToken2Wav / Code2Wav phase hooks, and the benchmark gather/CLI wiring. No local ROCm run.

[lm10-arg]

Author self-review at 761b86ade040ad3da82552212b46c3b5b05a14ba:

• Narrowed --omni-fail-fast-on-timeout to the mode this PR actually implements: immediate openai-chat-omni dispatch with --request-rate inf; finite-rate, self-timed, and other backends now fail before benchmark dispatch.
• Seed-TTS accuracy now fails closed when timeout or post-timeout cancellation counters are nonzero, preventing a high-scoring partial subset from passing.
• Explicitly classifies aiohttp.ServerTimeoutError as a structural timeout and added regression coverage.
• Watchdog phase updates now replace stale phase details, with a regression covering the report payload.

Validation: full pre-commit run --from-ref origin/main --to-ref HEAD, py_compile, standalone watchdog validation, and git diff --check origin/main...HEAD passed. Focused pytest remains unavailable on this Mac because the repository test plugin imports torch, which is not installed locally.

No known source-level blocker remains from this self-review. The exact-head AMD MiniCPM-o accuracy run is still the required runtime gate; this PR continues not to claim that it fixes the underlying ROCm/GPU hang.

[amy-why-3459]

Re-reviewed the delta on tip 761b86a ("Close timeout diagnostic review gaps").

Prior points look addressed:

1. Fail-fast scope. _validate_fail_fast_on_timeout now rejects anything other than immediate openai-chat-omni + --request-rate inf + non-self-timed before set_fail_fast_on_timeout / main_async, with CLI help and unit coverage. That closes the producer/delayed-arrival gap for the MiniCPM-o Stage-2 path without claiming fail-fast works under rate-limited or self-timed dispatch.

2. Timeout classification. aiohttp.ServerTimeoutError is in the structural timeout catch, with a regression asserting timed_out=True.

3. Watchdog details. phase() replaces details via _bounded_details(details) (handles None → {}), and the new unit test checks stale keys do not linger across phases.

Also good: Seed-TTS _validate_seed_tts fails closed when timed_out_requests / cancelled_after_timeout_requests are nonzero, with a high-scoring partial-result regression.

No new issues from this tip. Source-only inspection; no local ROCm run.

[andyluo7]

Approved exact head 761b86a. The exact synthetic merge with main 14874d0 passed GitHub checks plus CUDA build 18080, AMD build 14632, and NPU build 9825. AMD and NPU MiniCPM-o accuracy each completed all 1,088 Seed-TTS requests with zero failures, timeouts, or cancellations, and both configured accuracy suites passed. The prior review thread is resolved.