[Feature][TTS] Add Speech API streaming metrics

[gxxx-hum]

Motivation

Improve production monitoring for Speech TTS services with first-audio latency, playback continuity, and visibility into interrupted streams.

Changes

• Reuse existing audio metrics for raw HTTP, SSE, and WebSocket Speech streaming.
• Measure server-side TTFP at the first non-empty PCM payload, excluding WAV headers.
• Separate first-packet timing from the emission guard so label resolution can retry without changing the original TTFP.
• Record underrun and continuity only for normally completed streams, using the actual audio channel count.
• Add Speech-only completed and aborted stream counters, with bounded interruption reasons.
• Explicitly close nested audio generators on response interruption so cleanup and interruption counting do not depend on garbage collection.
• Keep non-streaming Speech outside TTFP, continuity, and stream lifecycle counters.
• Update documentation and regression tests.

Metrics

Metric	Purpose
vllm_omni:audio_ttfp_s	Request arrival to first non-empty PCM payload at the server
vllm_omni:audio_rtf	Existing generation-time / audio-duration metric
vllm_omni:audio_underrun_s	Worst playback-buffer deficit for completed streams
vllm_omni:audio_continuity_ok_total	Completed streams within the continuity threshold
vllm_omni:speech_stream_aborted_total	Interrupted Speech audio generators, by reason
vllm_omni:speech_stream_completed_total	Normally completed Speech audio generators

The two new counters use model-level labels; aborted reasons are cancelled, closed, engine_dead, and error. They cover started audio generators, including interruption before first PCM. WebSocket counts each sentence generation. Completion does not confirm client receipt.

Interruption ratio: aborted / (aborted + completed), using increments over the same time window and summing reasons per model.

Type-checking note

The remaining edits involving ref_audio, optional adapters, tokenizers, and correlated audio-output values are casts and local type narrowing required by the changed-file mypy hook. They do not add request validation or change reference-audio cardinality semantics. Ming-TTS multi-reference handling remains unchanged.

Validation

• Added tests for first-packet label retry, single emission, invalid timestamps, interruption before/after PCM, generator closure propagation, task cancellation, and non-streaming exclusion.
• Latest local changes pass Ruff checks, Ruff formatting, and git diff --check.
• Full pytest and deployment-environment validation of the latest changes are pending; the local Python environment lacks pytest.

The review follow-up changes described above are prepared locally and have not yet been committed/pushed.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/observability.md.

Module owners: @lishunyang12 @vraiti

@gxxx-hum, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vraiti]

LGTM

[hsliuustc0106]

The protocol validator (protocol/audio.py) explicitly accepts ref_audio as a list of strings, but this silently keeps only ref_audio[0] and drops the rest — silent data loss for users passing multiple clips. Either reject len > 1 with a clear error or document single-clip semantics.

[hsliuustc0106]

This ref_audio is None guard + identical message now appears 4x across the MOSS builders (_build_moss_tts_params voice_clone and realtime branches, plus the tts/local and ttsd _encode_ref call sites). A single check at the top of voice-clone param building (or inside _resolve_ref_audio) would cover all of them.

[hsliuustc0106]

Beyond the metrics work, this bundles several unrelated None-safety / protocol fixes (_coerce_ref_audio_str list support, the four MOSS ref_audio-is-None guards, _estimate_ref_code_len / voxcpm2 adapter-None guards, ming spk_emb None guard, _find_tts_stage str cast). They look like real fixes to the str | list[str] | None protocol contract that landed before the handlers caught up, but the PR body does not mention them and they change error behavior (new 400s where crashes/confusing errors were). Please add them to the PR description (and consider [Bugfix] in the title) so the change is reviewable as a unit — or split them into a separate PR.

[gxxx-hum]

Beyond the metrics work, this bundles several unrelated None-safety / protocol fixes (_coerce_ref_audio_str list support, the four MOSS ref_audio-is-None guards, _estimate_ref_code_len / voxcpm2 adapter-None guards, ming spk_emb None guard, _find_tts_stage str cast). They look like real fixes to the str | list[str] | None protocol contract that landed before the handlers caught up, but the PR body does not mention them and they change error behavior (new 400s where crashes/confusing errors were). Please add them to the PR description (and consider [Bugfix] in the title) so the change is reviewable as a unit — or split them into a separate PR.

Thanks for pointing this out. I re-reviewed the changes and removed the unrelated runtime/protocol fixes from this PR.

In particular, I removed _coerce_ref_audio_str, the duplicate MOSS ref_audio guards, and the additional defensive None branches. The remaining edits in these areas are type-only narrowing required by the changed-file mypy hook and do not change runtime behavior.

_resolve_ref_audio() retains its single-string contract, while Ming-TTS continues to handle multiple references through its model-specific _resolve_ref_audio_many() path.

I also updated the PR description to keep the scope focused on Speech API streaming metrics.

[gxxx-hum]

Hi, @hsliuustc0106 @lishunyang12 @vraiti all requested changes and merge conflicts have been done, and the streaming/non-streaming metrics have been verified. Would it be possible to review and merge this within the next couple of days. We have a downstream deployment depending on it. Thanks!

[linyueqian]

This does what it says and does not duplicate the existing Prometheus work. The seven audio families already lived in metrics/modality.py and were fed only by the Chat path; serving_speech.py made zero calls to them on the merge base and makes three here. The change to modality.py is five lines, a channels passthrough and a licence header, so no new families are introduced and no existing series changes shape.

The streaming details that are easy to get wrong are handled. TTFP is taken from the first non-empty PCM payload rather than the WAV header, empty payloads do not arm the once-guard, and the guard is backed by both a generator-local flag and req_state.first_audio_ts, so there is no path that observes twice. Labels come from the audio-producing result with the same stage_pools fallback the Chat path uses, and this path declines to invent stage 0 when they are missing, which is stricter than Chat. The per-chunk accumulation is generator-local rather than stored on the request state, so it is bounded by one request's chunk count and released when the generator closes, including on cancellation.

I went in expecting the finalize hook to be the problem and it is not. Skipping it on cancellation and error is deliberate: the tests pin it, the Chat path behaves the same way, and docs/design/metrics.md in this PR states the contract. On reflection I think it is also correct, because audio_underrun_s and audio_continuity_ok_total only mean something over a completed playback, and folding partial timelines into them would change what the histograms measure while making the two entrypoints disagree. My suggestion there is about the missing denominator rather than about moving the call.

One thing I checked and got wrong on the first pass, recorded because it changes how the flush path should be read: I thought last_audio_result could still be None when the resampler released everything in the final flush, which would have dropped the whole request's metrics. It cannot. new_chunks comes from the model output, not from the resampler, and the resampler runs per chunk inside that loop, so the producer is retained for every model chunk that carries audio. The flush_only parametrisation of test_resampled_speech_records_flush_with_audio_producer covers exactly that case.

Two suggestions inline, nothing blocking.

Validation: static review of the nine-file diff at 26c5b64d against merge base 6e79d582, following the TTFP guard and both record sites, the finalize placement against the try and finally structure, the label resolution against the Chat path, the continuity byte math in audio_continuity.py, and the new test file's assertions. I did not execute the branch.

[linyueqian]

[suggestion] The once-guard is consumed before the observation is known to have succeeded, so a first chunk whose labels cannot be resolved costs the request all of its streaming metrics with no second chance.

first_audio_chunk_s is set on the line above unconditionally, then _observe_speech_audio_ttfp can return (None, None) when arrival_ts, stage_id or replica_id is missing. Later chunks no longer enter this branch, so TTFP is never retried, and because audio_stage_id and audio_replica_id stay None the finalize guard further down also fails. One unresolvable first chunk silently drops TTFP, underrun and continuity together.

In practice res.replica_id is populated on the normal path, so this is a defensive edge rather than something you should expect to see. Setting the guard only once labels resolve, or retrying on the next chunk, would make the failure cost one sample instead of the whole request.

[gxxx-hum]

Good point, The guard violates the single-responsibility principle. I'll separate the two responsibilities.

[linyueqian]

[suggestion] Skipping finalize on abort is the right call and I am not asking for it to move into finally. Conditioning audio_underrun_s and audio_continuity_ok_total on a completed stream is what keeps them comparable, and folding partial timelines in would quietly change what the histogram measures. It also matches the Chat path, so the family is not split across two semantics, and docs/design/metrics.md states the contract.

What is missing is the denominator. Aborted streams currently leave no trace in this metric family at all, so a deployment cannot tell a healthy continuity rate from one computed over a shrinking population of survivors. The failure mode that matters most, a stream that stalls badly enough that the client gives up, is exactly the one that never appears.

A counter incremented in the except paths, labelled by reason, would close that without touching the histogram semantics: continuity stays completion-conditioned, and the rate of streams that never got to finalize becomes visible next to it.

[gxxx-hum]

Good suggestion. I'll try to track the failures

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit f415ccd5d566 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[gxxx-hum]

@linyueqian Both changes are now implemented, and I’ve verified streaming, non-streaming, and client cancellation scenarios. The metrics match the expected counts, and the PR description has been updated.