[Frontend] Opt-in WebSocket TTS split_granularity and session seed

[rk9595]

Summary

• After [Bugfix] Stream Qwen3-TTS WebSocket input as one request #4731, WebSocket /v1/audio/speech/stream synthesizes one request per input.done. Docs still described per-sentence splitting from [Feature][TTS] Streaming Text Input for Qwen3-TTS via WebSocket #1230.
• Adds opt-in split_granularity (none default / sentence / clause) so STT/LLM clients can start audio before input.done, without changing the default path.
• Sentence/clause boundaries include Latin .!?, CJK, Indic danda (।॥), and Arabic marks.
• Forwards seed from WebSocket session.config (HTTP speech already had it).

This is a new opt-in feature plus a docs/API alignment fix, not a crash bug. No dedicated issue.

AI assistance: implementation drafted with Cursor; human submitter is rk9595. Please review splitter edge cases (decimals, Latin abbreviations).

Test plan

• pytest tests/entrypoints/openai_api/test_speech_text_splitter.py tests/entrypoints/openai_api/test_serving_speech_stream.py -m "core_model and cpu"
• Optional: streaming_speech_client.py --split-granularity sentence with Indic text such as नमस्ते। कैसे हो?
• Confirm default (no split_granularity / none) still one request per flush

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/entrypoints.md.

Module owners: @alex-jw-brooks @linyueqian @NickCao

@rk9595, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit e4c89b34de2a produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[hsliuustc0106]

Review findings:

• The splitter mishandles punctuation runs/closing quotes, abbreviations, and long incremental buffers.
• Session configuration can change after a sentence has already been emitted but before input.done, producing mixed settings within one utterance.

GitHub currently reports this PR as non-mergeable; a local three-way merge also conflicts in the docs and stream test file. py_compile and whitespace checks pass, but targeted pytest collection was blocked by the environment's incompatible Torch/NumPy installation.

[hsliuustc0106]

[P1] Reject reconfiguration after a split unit has already been emitted. In sentence/clause mode, feed() can return a unit while leaving _buf empty, so this check allows session.config before input.done. The next unit then uses different voice/settings but keeps the same utterance_index and total_sentences; track an active flush separately and reset it only after input.done.

[hsliuustc0106]

[P1] Preserve punctuation runs and closing delimiters when flushing. Because flush makes every terminator complete independently, Wait... becomes three units and He said "Hello." leaves the quote as a separate unit. That causes extra TTS requests and malformed text for common input.done payloads; consume the punctuation/closing-delimiter run before emitting one unit.

[hsliuustc0106]

[P2] Add actual abbreviation handling. Requiring whitespace after . does not distinguish sentence endings from Dr. Smith, e.g. this, or initials, so these are emitted as separate TTS requests despite this comment identifying abbreviations as a concern.

[hsliuustc0106]

[P2] Avoid rescanning the entire unfinished buffer on every input.text message. A long punctuation-free sentence sent one token/word at a time makes extract_complete_units O(n^2) and can consume substantial CPU; retain a scan cursor/state for incremental parsing.

[rk9595]

Self-review

Rebased onto main (conflict was in docs/serving/speech_api.md and the TTS README; resolved to main's list/table formatting). Pushed as 4f9cad2, which addresses all four review findings. Thanks @hsliuustc0106 — the punctuation-run and reconfiguration cases were both real.

[P1] session.config after a split unit was emitted. Correct: in sentence/clause mode feed() can return a unit and leave _buf empty, so has_buffered_text() let a reconfiguration through mid-utterance and the rest of the same utterance_index would have been synthesized with a different voice. The handler now tracks the open utterance explicitly (utterance_open: set on the first non-empty input.text, cleared only on input.done) instead of inferring it from the buffer. has_buffered_text() is gone since nothing else used it. New test: test_session_config_rejected_after_a_split_unit_was_emitted (rejected mid-utterance, accepted again after input.done). The existing "while input is buffered" message is now "while an utterance is in progress"; its test was updated.

[P1] Punctuation runs and closing delimiters. The splitter now absorbs the run of terminators and closing delimiters that follows a boundary, so Wait... is one unit and He said "Hello." keeps its quote instead of emitting it as a separate request. One deliberate consequence: a unit only closes once the character after the run is known (or on input.done), because a closing delimiter can still arrive. That delays a script-terminator unit by one chunk — Latin already waited for whitespace, so this makes the two paths consistent. test_cjk_period_completes_once_the_next_char_is_known covers it.

[P2] Abbreviations. Added a real check: the token before a . is rejected as a boundary if it is a known abbreviation (Dr., Mr., Prof., e.g., i.e., etc., U.S., a.m., …) or a single letter, so initials like J. R. R. Tolkien stay in one unit. While there I generalized the decimal check to cover thousands separators — clause mode was splitting 1,000 into 1, and 000.

[P2] O(n²) rescanning. extract_complete_units now takes/returns a scan cursor and SpeechTextSplitter carries it across feeds, so a long punctuation-free sentence arriving token by token is scanned once rather than rescanned per input.text. Only backward-looking checks are the decimal/abbreviation lookbacks, and both are evaluated at the terminator, which is only decided once the following character exists — so parking the cursor never freezes a decision that later text could change. Measured locally: 20k single-word feeds into one unterminated sentence, 0.05 s.

What I checked

• tests/entrypoints/openai_api/test_speech_text_splitter.py — 15 passed. torch/vllm are not installable on this machine, so I ran the file standalone (the splitter module has no vLLM imports); the same env problem blocked the reviewer.
• The WebSocket handler path can only run under the real conftest, so I drove handle_session locally against stubbed fastapi/vllm/service objects and confirmed: reconfiguration after an emitted unit is rejected; reconfiguration between utterances still applies; empty input.text does not open an utterance; नमस्ते। कैसे हो? emits two units under one utterance_index. The equivalent assertions are in TestWebSocketSentenceSplitting, which CI will run for real.
• ruff check and ruff format clean on all touched files; check-spdx-header, check-test-marks, markdownlint-cli2 (0 errors), git diff --check all pass.
• Default path unchanged: with no split_granularity (or none), it is still one request per input.done, sentence_index: 0 of total_sentences: 1.

Not addressed: nothing from the review. Remaining known limitation is that abbreviation detection is a fixed list plus a single-letter rule, not locale-aware — over-suppression at worst merges two sentences into one request, and input.done always flushes whatever is left.

[linyueqian]

The default buffering behavior and session-seed forwarding look correct. In sentence and clause mode, abbreviation detection can repeatedly rescan the full prefix of one accepted text frame, making the synchronous splitter quadratic and able to stall the shared WebSocket event loop. That needs a bounded scan before merge; the Arabic semicolon mismatch is only a small follow-up.

[linyueqian]

[important] Bound this backward scan or track the current token start. For input such as a.a.a.a... without whitespace, every period walks the entire preceding prefix, making the splitter quadratic. One accepted WebSocket text frame can approach 128 KiB, so sentence or clause mode can perform billions of synchronous character checks and stall every session sharing this event loop.

[linyueqian]

[suggestion] Add U+061B ARABIC SEMICOLON (؛) here or correct the module contract. The docstring says Arabic semicolons are covered, but clause mode includes the Arabic comma and question mark only, so text ending at ؛ stays buffered until another boundary or input.done.

[rk9595]

Done in e4c89b3: ؛ U+061B is now in _SCRIPT_CLAUSE_TERMINATORS, so it also flows into the run-absorption set. The split_granularity row in docs/serving/speech_api.md omitted it too and is corrected; covered by test_clause_mode_splits_on_arabic_semicolon.

[rk9595]

Round 2

Pushed e4c89b3 on top of 4f9cad2 addressing @linyueqian's review.

[important] Unbounded abbreviation lookback. Correct, and a regression from the round-1 abbreviation fix. _is_abbreviation now walks back at most _MAX_ABBREVIATION_LEN characters (6, from approx); if it hits that limit while still on a token character, the token is longer than any known abbreviation and it returns False immediately. Every abbreviation fits inside the limit, so the cap can only rule a token out, never produce a false positive; Dr., e.g., initials, decimals and thousands separators all behave as before. New test test_abbreviation_lookback_is_bounded monkeypatches _is_token_char and asserts the number of character checks on a 40k-char a.a.a... frame is at most len(text) * (_MAX_ABBREVIATION_LEN + 1). Against 4f9cad2 that same input takes 400,000,000 checks; it is a count rather than a timing assertion so it cannot flake on a loaded runner. test_long_token_before_period_is_not_an_abbreviation guards the early-return path.

[suggestion] Arabic semicolon. Added ؛ U+061B to _SCRIPT_CLAUSE_TERMINATORS (it flows into the run-absorption set automatically), and corrected the split_granularity row in docs/serving/speech_api.md, which also omitted it. New test test_clause_mode_splits_on_arabic_semicolon.

Checked: 18 splitter tests pass standalone (torch still not installable on this machine, same as round 1); ruff check / ruff format clean at v0.14.10; markdownlint-cli2 0 errors; git diff --check clean; branch merges cleanly into main.

@hsliuustc0106 your round-1 review is still marked changes-requested; all four findings were addressed in 4f9cad2, would you mind taking another look?

fixed

[linyueqian]

The abbreviation lookback is now bounded, and the Arabic semicolon handling matches the docs. Both review findings are fixed, with regression tests for the long-token case, the scan bound, and Arabic clause splitting.

CI at this head. The lanes had never run when this was first reviewed, because the PR carried no ready label; with it added, general build 14829, Intel 7953, and NPU 6570 are green. AMD build 11553 hard-failed exactly one step, mi300_1: Diffusion · Model Test, with its other 13 steps passing.

That AMD failure is inherited rather than caused here, and today gives it unusually strong corroboration: five unrelated PRs hit the identical single step in builds 11556, 11558, 11561, 11562, and 11563, across frontend, TTS, and refactor changes with disjoint file sets. It is the known SenseNova paged-decode failure on a ROCm image lacking the CUDA flash-attention extensions. This PR adds opt-in WebSocket TTS split_granularity and touches no diffusion code.

Validation: static review of the delta at e4c89b34, the abbreviation scan bound, the clause-splitting paths, and the new regression tests, plus public Buildkite step metadata for all four lanes. Codex and Claude agree both findings are resolved. I did not execute the fork's code.