feat(gateway): streaming TTS adapter contract and consumer - #73358
Closed
ctaylor86 wants to merge 1 commit into
Closed
feat(gateway): streaming TTS adapter contract and consumer#73358ctaylor86 wants to merge 1 commit into
ctaylor86 wants to merge 1 commit into
Conversation
…rch#60671) Add an opt-in streaming-audio adapter seam to BasePlatformAdapter so voice-capable gateway platforms (LiveKit, Discord voice, future adapters) can consume LLM output as streaming PCM audio before the full response completes, dropping perceived voice latency from ~2-3.5s to ~500-800ms. Adapter contract (gateway/platforms/base.py): - AudioFormat dataclass: declared sample_rate, channels, sample_width - StreamingTTSHandle: opaque handle with audible/aborted flags - supports_streaming_tts / begin_streaming_tts / write_streaming_tts / finish_streaming_tts / abort_streaming_tts - All default to unsupported/no-op so existing adapters are source-compatible - Per-turn _streaming_tts_completed_chats set suppresses duplicate whole-file auto-TTS when streaming succeeded; cleared after turn completion Gateway consumer (gateway/streaming_tts_consumer.py): - StreamingTTSConsumer: bridges sync agent deltas to async adapter audio sink - Uses existing SentenceChunker (no competing parser) - Thread-safe bounded queue; on_delta never blocks the agent worker thread - Resolves configured streaming provider via resolve_streaming_provider() - Serialises clause playback in order; flushes tail on completion - Pre-audio failure: completed=False (falls back to whole-file TTS) - Post-audio failure: completed=True, partial=True (no replay from start) - Abort is idempotent; late chunks silently dropped - Per-turn state isolated across concurrent chats Gateway integration (gateway/run.py): - message_type parameter threaded through _run_agent -> _run_agent_inner - StreamingTTSConsumer created when voice input + auto-TTS + provider active - Delta callback teed to both text stream consumer and TTS consumer - TTS-only delta callback installed when text streaming is off - finish() called from executor; wait_complete() in async context after - Barge-in aborts the consumer at all three interrupt detection points - Runner-level _send_voice_reply suppressed when streaming TTS completed Tests (tests/gateway/test_streaming_tts_consumer.py): - 15 focused tests: adapter defaults, lifecycle, ordered chunks, unsupported/No-streamer fallback, abort idempotency, late-chunk drop, pre/post-audio failure, concurrent-turn isolation, think-block suppression, queue backpressure Does not touch desktop/TUI code or add config flags. Plugin TTS provider stream() metadata gap (NousResearch#47896) is explicitly out of scope — built-in ElevenLabs/OpenAI PCM streamers are the first consumers. Refs: NousResearch#60671, NousResearch#47896
This was referenced Jul 28, 2026
Closed
teknium1
added a commit
that referenced
this pull request
Jul 29, 2026
Contributor
|
Merged via PR #73862 (4de9e53) — cherry-picked clean with your authorship preserved in git history, and live-tested by Teknium working flawlessly. Your StreamingTTSConsumer (clause-boundary detection, backpressure, duplicate suppression, whole-file fallback) landed intact as the gateway half of the streaming subsystem, composed with the StreamingTTSProvider registry from #47588 for synthesis. Nice work writing it against the registry from the start — the two halves composed with zero seam changes. The plugin-stream()/opus half (#47896) is noted as the follow-up you scoped out. Thanks for the contribution! |
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
prmartinow
pushed a commit
to prmartinow/hermes-agent
that referenced
this pull request
Aug 26, 2026
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds gateway-level streaming TTS so voice platform adapters can consume the LLM token stream and synthesise audio clause-by-clause, instead of waiting for the full response before starting TTS. This closes the consumer gap referenced in #47896 and implements the feature requested in #60671.
Why this approach: The existing TTS path buffers the entire assistant response, then calls the TTS provider once. For voice interactions this adds latency proportional to response length. This PR introduces an opt-in streaming contract on
BasePlatformAdapterthat adapters can implement to receive PCM chunks incrementally as the LLM produces text. Non-streaming adapters are unaffected - the fallback to whole-file TTS is preserved.Related Issue
Fixes #60671
Type of Change
Changes Made
gateway/streaming_tts_consumer.py(new, 423 lines):StreamingTTSConsumerclass that sits between the LLM token stream and the adapter. Iterates provider output, detects clause boundaries, and forwards PCM chunks to the adapter as they arrive. Handles timeouts, abort, fallback to whole-file TTS, and queue backpressure.gateway/platforms/base.py(+175 lines): Adds thesupports_streaming_ttsflag and thebegin_streaming_tts/write_streaming_tts_chunk/finish_streaming_tts/abort_streaming_tts/get_streaming_tts_audio_formatmethods toBasePlatformAdapter. All default to no-ops/false so existing adapters are unaffected.gateway/run.py(+198/-52 lines): Wires the consumer intoGatewayRunner._run_agent. When the adapter supports streaming TTS, the consumer is created on the gateway event loop and fed tokens from the LLM stream. Per-turn duplicate suppression prevents whole-file fallback from replaying after streaming has begun. Think blocks are suppressed (not synthesised).gateway/config.py(-2 lines): Minor cleanup.How to Test
pytest tests/gateway/test_streaming_tts_consumer.py -v- 27 tests covering adapter contract defaults, consumer lifecycle, timeout/abort/fallback semantics, duplicate suppression, concurrent turn isolation, queue backpressure, and the finish-sentinel race.pytest tests/gateway/test_streaming_tts_gateway_regression.py -v- 2 regression tests exercising the real_run_agentfinalisation path (noNameErrorin voice or text turns).pytest tests/gateway/ -qAll 29 tests pass (verified on macOS 15, Python 3.11).
Checklist
Code
feat(gateway): ...)pytest tests/gateway/ -qand all tests passDocumentation & Housekeeping
Design Notes
supports_streaming_ttsdefaults toFalse. No existing adapter changes required.