[Feature] Failure message shows more details

[wuhang2014]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

#2392 's P1 fixing:

Phase 1: stop the bleeding

• Ensure all accepted async jobs reach a terminal state
• Add watchdog / timeout / engine-health mapping for async jobs
• Never leave job-status APIs permanently in in_progress

In the latest commit, OOM failures are now shown to the user, but the error message in the response isn’t very clear. This PR improves the message to make it clearer.

Test Plan

Start a serving instance of qwen3-tts, then hog the spare vram.

# serving of qwen3-tts
python3 -m vllm_omni.entrypoints.cli.main serve Qwen/Qwen3-TTS-12Hz-0.6B-Base --omni --host 127.0.0.1 --port 8091 --stage-init-timeout 600 --trust-remote-code --disable-log-stats --allowed-local-media-path /tmp/

# hog the spare vram with torch.empty

Burst 3 requests and see what respond.

Test Result

Before

[req1] HTTP code: 500
[req1] Content-Type: application/json
[req2] HTTP code: 500
[req2] Content-Type: application/json
[req3] HTTP code: 500
[req3] Content-Type: application/json
[req1] Response body: {"error":{"message":"('EngineCore encountered an issue. See stack trace (above) for the root cause.', \"('EngineCore encountered an issue. See stack trace (above) for the root cause.', 'EngineCore encountered an issue. See stack trace (above) for the root cause.')\")","type":"InternalServerError","param":null,"code":500}}
[req3] Response body: {"error":{"message":"('EngineCore encountered an issue. See stack trace (above) for the root cause.', \"('EngineCore encountered an issue. See stack trace (above) for the root cause.', 'EngineCore encountered an issue. See stack trace (above) for the root cause.')\")","type":"InternalServerError","param":null,"code":500}}
[req2] Response body: {"error":{"message":"('EngineCore encountered an issue. See stack trace (above) for the root cause.', \"('EngineCore encountered an issue. See stack trace (above) for the root cause.', 'EngineCore encountered an issue. See stack trace (above) for the root cause.')\")","type":"InternalServerError","param":null,"code":500}}

After

[req1] HTTP code: 400
[req1] Content-Type: application/json
[req3] HTTP code: 400
[req3] Content-Type: application/json
[req2] HTTP code: 400
[req2] Content-Type: application/json
[req1] Response body: {"error":{"message":"Speech generation failed: Orchestrator thread crashed. Failure point: RuntimeError: Orchestrator failed while polling stage-0 outputs via StageEngineCoreClient. Likely upstream failure in StageEngineCoreProc(stage-0). Original exception: EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.. Root cause: EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause..","type":"BadRequestError","param":null,"code":400}}
[req3] Response body: {"error":{"message":"Speech generation failed: Orchestrator thread crashed. Failure point: RuntimeError: Orchestrator failed while polling stage-0 outputs via StageEngineCoreClient. Likely upstream failure in StageEngineCoreProc(stage-0). Original exception: EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.. Root cause: EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause..","type":"BadRequestError","param":null,"code":400}}
[req2] Response body: {"error":{"message":"Speech generation failed: Orchestrator thread crashed. Failure point: RuntimeError: Orchestrator failed while polling stage-0 outputs via StageEngineCoreClient. Likely upstream failure in StageEngineCoreProc(stage-0). Original exception: EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.. Root cause: EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause..","type":"BadRequestError","param":null,"code":400}}

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[Copilot]

Pull request overview

Improves user-visible failure messages for orchestrator/engine crashes (e.g., OOM-related engine death) so clients receive clearer diagnostic context instead of a generic “orchestrator crashed” error.

Changes:

• Make AsyncOmni surface orchestrator error payloads as readable strings rather than dumping full dicts.
• Add richer context when orchestrator polling fails, and preserve chained exceptions for root-cause reporting.
• Cache and propagate an orchestrator crash reason through AsyncOmniEngine output polling, with new unit tests covering the behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vllm_omni/entrypoints/async_omni.py	Raises a clearer RuntimeError message from orchestrator error payloads.
vllm_omni/engine/stage_engine_core_proc.py	Improves handshake failure error text when the stage core subprocess exits.
vllm_omni/engine/orchestrator.py	Wraps polling failures with more explicit component/stage context and exception chaining.
vllm_omni/engine/async_omni_engine.py	Adds crash-reason formatting, caches it, and uses it when the orchestrator is dead.
tests/entrypoints/test_async_omni_abort.py	Adds coverage for string error payload handling in _process_orchestrator_results.
tests/engine/test_async_omni_engine_outputs.py	Adds coverage for cached crash-reason propagation and formatter behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lishunyang12]

@amy-why-3459 @ZeldaHuang PTAL.

[amy-why-3459]

LGTM

[lishunyang12]

Solve conflicts.