feat(voice): streaming, conversational TTS with barge-in across all surfaces

[OutThisLife]

Voice replies now speak sentence-by-sentence while the model is still generating — on every surface, with every TTS provider — and you can interrupt them by talking or typing. This is the token → sentence → TTS pipelining that realtime voice stacks (Pipecat's SentenceAggregator, LiveKit Agents, HF speech-to-speech) converge on: buffer LLM deltas, cut at sentence boundaries, hand each sentence to TTS immediately, keep one audio clock so sentences butt seamlessly.

What was wrong

TTS was batch-only almost everywhere: the agent finished its entire reply, the full text went to the provider, one big audio file came back, then playback started — 10–30s of dead air on long replies. The one streaming path that existed was hardcoded to ElevenLabs + CLI. Once audio started there was no way to stop it short of /voice off.

What this does

Provider-agnostic streaming core — tools/tts_streaming.py (new)

• StreamingTTSProvider ABC + registry + resolve_streaming_provider(). ElevenLabs (pcm_24000) and OpenAI (response_format=pcm) stream chunked PCM; every other provider keeps its configured voice and gets per-sentence sync synthesis. No silent provider swaps.
• SentenceChunker — the one incremental sentence cutter every surface shares: cuts at sentence boundaries the moment they arrive in the delta stream, strips <think> blocks split across deltas, merges short fragments ("Ha!") into the following sentence so they never become tiny stalling clips.
• stream_tts_to_speaker (tools/tts_tool.py) is the single dispatcher over both paths; markdown and emoji are stripped per sentence (emoji made providers stall or read out "smiling face").

CLI — streaming gate broadened from ElevenLabs-only to any working provider. In continuous voice mode a mic monitor runs during playback; speaking over the agent cuts TTS at the moment of detection while the monitor keeps recording the interruption and queues its transcript as the next turn. Push-to-talk and typed input drain the pipeline mid-sentence.

TUI — turn deltas feed a per-turn pipeline in tui_gateway/server.py; the post-complete speak_text call survives only as a fallback. session.interrupt, /voice toggles, and new turns cut in-flight speech; VAD barge-in emits voice.interrupted, then the captured interruption goes out as voice.transcript — the same event the TUI already submits as a spoken turn.

Desktop — /api/audio/speak-stream WebSocket is a per-reply speech session: the renderer opens it when a voice-mode reply starts and feeds raw LLM deltas as they arrive ({"text": ...} frames, {"done": true} at the end); the server cuts sentences with the shared chunker and streams int16 PCM back while generation continues. One socket + one Web Audio clock per reply — speech overlaps generation ChatGPT-style, with no per-sentence connection/synthesis gaps (the old per-chunk approach paid a fresh WS + AudioContext + provider TTFB for every sentence). Read-aloud uses the same session one-shot. Falls back to the POST data-URL path for old backends / non-chunked providers. VAD barge-in (echo-cancelled mic monitor, noise floor calibrated against playback) and prompt-submit both silence in-flight speech.

Shared — tools/voice_mode.listen_for_speech(): sustained-RMS speech detector with playback-aware floor calibration (CLI + TUI). With capture=True it keeps a rolling pre-roll buffer and records through to a silence endpoint, so the interruption is transcribed from its first syllable. New voice.barge_in config key, default true.

Barge-in captures the whole interruption

Detection alone loses the opening words: sustained-speech VAD needs ~300ms to trip, and re-opening the mic afterwards costs more — "stop, actually—" arrived as "actually—". Every surface now records through the barge instead of re-listening after it:

• Desktop (voice-barge-in.ts) — a MediaRecorder runs on the monitor's stream the whole time (recorder rotated every 5s while quiet to bound pre-roll), and once tripped keeps rolling until you go quiet; the complete blob goes straight to transcription and submit, no re-record round trip.
• CLI / TUI — listen_for_speech(capture=True) does the same server-side; the CLI queues the transcript as pending input, the TUI emits it as voice.transcript.

Silence is not an error

Providers that hear no words (ElevenLabs/xAI "empty transcript") now flag no_speech; transcribe_recording() maps that to a successful empty result and /api/audio/transcribe returns 200 with "" instead of a 400. The desktop voice loop treats it as a quiet turn and re-listens — no more error toast spam on every pause.

Why the fallbacks are cheap

Streaming and barge-in degrade independently: an old backend gets the data-URL path, a non-chunked provider gets per-sentence sync, a missing mic just means no VAD (typing still interrupts). Nothing new is required for things to keep working exactly as before.

Tests

• tests/tools/test_tts_streaming.py — SentenceChunker (boundary cuts, fragment merging, cross-delta think blocks), ABC/registry/resolver, both streamers, dispatcher streaming + sync-fallback paths
• tests/hermes_cli/test_web_server_speak_stream.py — WS session protocol (incremental text frames, done, stop/barge-in, start/pcm/end/fallback), auth, splitter
• tests/test_tui_gateway_server.py — pipeline lifecycle, barge-in, interrupt wiring
• tests/tools/test_voice_mode.py — listen_for_speech calibration/trigger/stop, capture pre-roll (every loud block lands in the WAV, including pre-detection ones), no-speech mapping
• tests/tools/test_voice_cli_integration.py — broadened gate, barge-capture submit + mic hand-back
• tests/hermes_cli/test_web_server.py — no-speech → 200/"" contract

1,180+ tests green across the touched Python files, desktop vitest suites, tsc + eslint + ruff clean.

[github-actions]

૮ >ﻌ< ა ci review

_{running on 83456d0}

looks good to me!