[RFC] Offload blocking TTS/speech ops to thread pool to unblock event loop

[scyyh11]

Purpose

Offload synchronous CPU-bound TTS preprocessing operations to a shared ThreadPoolExecutor(max_workers=1) to prevent blocking the asyncio event loop under concurrent TTS requests. This is the same class of fix as vllm-project/vllm#34789 applied to vllm-omni's TTS/speech serving layer.

Offloaded methods (called from async def _prepare_speech_generation()):

• _build_voxtral_prompt: sync tokenizer.encode_speech_request()
• _build_fish_speech_prompt: lazy AutoTokenizer.from_pretrained() + tokenization + np.save()
• _estimate_prompt_len: lazy AutoTokenizer.from_pretrained() + model computation

Intentionally kept synchronous:

• _build_tts_params: offloading would break atomicity with validation for uploaded-voice requests, since delete_voice() can mutate self.uploaded_speakers between the await point and the dict access.

max_workers=1 serializes tokenizer access to avoid Rust RefCell "Already borrowed" errors from concurrent use.

Closes #2476

Test Plan

Unit tests:

pytest tests/entrypoints/openai_api/test_serving_speech.py -v --timeout=60

Event loop responsiveness benchmark (Qwen3-TTS, 500 reqs @ concurrency 50):

# Terminal 1: TTS benchmark
python benchmarks/qwen3-tts/vllm_omni/bench_tts_serve.py \
    --port 8000 --num-prompts 500 --max-concurrency 50 \
    --config-name "async_chunk" --result-dir results/

# Terminal 2: /health latency during TTS load
for i in $(seq 1 100); do
  curl -w "%{time_total}\n" -s -o /dev/null http://localhost:8000/health
  sleep 0.1
done

Test Result

Environment: A100-SXM4-80GB, Python 3.11.5, vllm 0.19.0, Qwen3-TTS-12Hz-0.6B-CustomVoice

Unit Tests

112 passed, 0 failed (3.37s)

Event Loop Responsiveness — /health latency during 500 reqs @ concurrency 50

/health Metric	Main	Fix	Improvement
P50	0.93ms	1.01ms	~same
P95	3.34ms	2.58ms	23% better
P99	5.66ms	4.37ms	23% better
Max	18.96ms	10.77ms	43% better

Throughput — 500 reqs @ concurrency 50

Metric	Main	Fix	Diff
Request throughput (req/s)	0.79	0.82	+3.4%
Audio throughput (audio-s/s)	4.46	4.62	+3.4%
P99 E2E (ms)	72483	70594	-2.6%
P99 RTF	16.66	15.07	-9.5%

No regression. Fix branch is slightly faster (~3-4% throughput, ~10% better P99 RTF).

Shutdown

SIGTERM under load: server exited within <1s, no hung workers. shutdown(wait=False, cancel_futures=True) cancels queued work cleanly.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

[scyyh11]

@linyueqian PTAL

[scyyh11]

@hsliuustc0106 Hi, I think this PR is ready for review now, PTAL, thank you.

[linyueqian]

LGTM