Skip to content

feat(gateway): streaming TTS adapter contract and consumer - #73358

Closed
ctaylor86 wants to merge 1 commit into
NousResearch:mainfrom
ctaylor86:feat/gateway-streaming-tts-60671
Closed

feat(gateway): streaming TTS adapter contract and consumer#73358
ctaylor86 wants to merge 1 commit into
NousResearch:mainfrom
ctaylor86:feat/gateway-streaming-tts-60671

Conversation

@ctaylor86

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds gateway-level streaming TTS so voice platform adapters can consume the LLM token stream and synthesise audio clause-by-clause, instead of waiting for the full response before starting TTS. This closes the consumer gap referenced in #47896 and implements the feature requested in #60671.

Why this approach: The existing TTS path buffers the entire assistant response, then calls the TTS provider once. For voice interactions this adds latency proportional to response length. This PR introduces an opt-in streaming contract on BasePlatformAdapter that adapters can implement to receive PCM chunks incrementally as the LLM produces text. Non-streaming adapters are unaffected - the fallback to whole-file TTS is preserved.

Related Issue

Fixes #60671

Type of Change

  • ✨ New feature (non-breaking change that adds functionality)

Changes Made

  • gateway/streaming_tts_consumer.py (new, 423 lines): StreamingTTSConsumer class that sits between the LLM token stream and the adapter. Iterates provider output, detects clause boundaries, and forwards PCM chunks to the adapter as they arrive. Handles timeouts, abort, fallback to whole-file TTS, and queue backpressure.
  • gateway/platforms/base.py (+175 lines): Adds the supports_streaming_tts flag and the begin_streaming_tts / write_streaming_tts_chunk / finish_streaming_tts / abort_streaming_tts / get_streaming_tts_audio_format methods to BasePlatformAdapter. All default to no-ops/false so existing adapters are unaffected.
  • gateway/run.py (+198/-52 lines): Wires the consumer into GatewayRunner._run_agent. When the adapter supports streaming TTS, the consumer is created on the gateway event loop and fed tokens from the LLM stream. Per-turn duplicate suppression prevents whole-file fallback from replaying after streaming has begun. Think blocks are suppressed (not synthesised).
  • gateway/config.py (-2 lines): Minor cleanup.

How to Test

  1. pytest tests/gateway/test_streaming_tts_consumer.py -v - 27 tests covering adapter contract defaults, consumer lifecycle, timeout/abort/fallback semantics, duplicate suppression, concurrent turn isolation, queue backpressure, and the finish-sentinel race.
  2. pytest tests/gateway/test_streaming_tts_gateway_regression.py -v - 2 regression tests exercising the real _run_agent finalisation path (no NameError in voice or text turns).
  3. Full suite: pytest tests/gateway/ -q

All 29 tests pass (verified on macOS 15, Python 3.11).

Checklist

Code

  • I have read the Contributing Guide
  • My commit messages follow Conventional Commits (feat(gateway): ...)
  • I searched for existing PRs to make sure this is not a duplicate
  • My PR contains only changes related to this feature
  • I have run pytest tests/gateway/ -q and all tests pass
  • I have added tests for my changes
  • I have tested on my platform: macOS 15.2 (Tahoe), Python 3.11

Documentation & Housekeeping

  • N/A - no new config keys, no architecture/workflow changes, no tool behaviour changes
  • Cross-platform: uses only stdlib asyncio primitives (no platform-specific code)

Design Notes

  • Opt-in by default: supports_streaming_tts defaults to False. No existing adapter changes required.
  • Prompt-cache safe: No changes to system prompt, toolset, or message history mid-conversation.
  • Fallback preserved: If streaming fails before any audio is audible, whole-file TTS runs as before. After audio starts, fallback is suppressed to prevent replay from the beginning.
  • Per-turn isolation: Duplicate suppression markers are scoped per-turn and cleared before recursive turns.
  • Event-loop safe: Provider iteration runs off the gateway event loop. PCM chunks are written immediately (not buffered per clause).

…rch#60671)

Add an opt-in streaming-audio adapter seam to BasePlatformAdapter so
voice-capable gateway platforms (LiveKit, Discord voice, future adapters)
can consume LLM output as streaming PCM audio before the full response
completes, dropping perceived voice latency from ~2-3.5s to ~500-800ms.

Adapter contract (gateway/platforms/base.py):
- AudioFormat dataclass: declared sample_rate, channels, sample_width
- StreamingTTSHandle: opaque handle with audible/aborted flags
- supports_streaming_tts / begin_streaming_tts / write_streaming_tts
  / finish_streaming_tts / abort_streaming_tts
- All default to unsupported/no-op so existing adapters are source-compatible
- Per-turn _streaming_tts_completed_chats set suppresses duplicate whole-file
  auto-TTS when streaming succeeded; cleared after turn completion

Gateway consumer (gateway/streaming_tts_consumer.py):
- StreamingTTSConsumer: bridges sync agent deltas to async adapter audio sink
- Uses existing SentenceChunker (no competing parser)
- Thread-safe bounded queue; on_delta never blocks the agent worker thread
- Resolves configured streaming provider via resolve_streaming_provider()
- Serialises clause playback in order; flushes tail on completion
- Pre-audio failure: completed=False (falls back to whole-file TTS)
- Post-audio failure: completed=True, partial=True (no replay from start)
- Abort is idempotent; late chunks silently dropped
- Per-turn state isolated across concurrent chats

Gateway integration (gateway/run.py):
- message_type parameter threaded through _run_agent -> _run_agent_inner
- StreamingTTSConsumer created when voice input + auto-TTS + provider active
- Delta callback teed to both text stream consumer and TTS consumer
- TTS-only delta callback installed when text streaming is off
- finish() called from executor; wait_complete() in async context after
- Barge-in aborts the consumer at all three interrupt detection points
- Runner-level _send_voice_reply suppressed when streaming TTS completed

Tests (tests/gateway/test_streaming_tts_consumer.py):
- 15 focused tests: adapter defaults, lifecycle, ordered chunks,
  unsupported/No-streamer fallback, abort idempotency, late-chunk drop,
  pre/post-audio failure, concurrent-turn isolation, think-block suppression,
  queue backpressure

Does not touch desktop/TUI code or add config flags. Plugin TTS provider
stream() metadata gap (NousResearch#47896) is explicitly out of scope — built-in
ElevenLabs/OpenAI PCM streamers are the first consumers.

Refs: NousResearch#60671, NousResearch#47896
@alt-glitch alt-glitch added type/feature New feature or request comp/gateway Gateway runner, session dispatch, delivery tool/tts Text-to-speech and transcription area/streaming Streaming responses: gateway delivery, provider wire sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P3 Low — cosmetic, nice to have labels Jul 28, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #73862 (4de9e53) — cherry-picked clean with your authorship preserved in git history, and live-tested by Teknium working flawlessly. Your StreamingTTSConsumer (clause-boundary detection, backpressure, duplicate suppression, whole-file fallback) landed intact as the gateway half of the streaming subsystem, composed with the StreamingTTSProvider registry from #47588 for synthesis. Nice work writing it against the registry from the start — the two halves composed with zero seam changes. The plugin-stream()/opus half (#47896) is noted as the follow-up you scoped out. Thanks for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/streaming Streaming responses: gateway delivery, provider wire comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages tool/tts Text-to-speech and transcription type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway streaming TTS: let voice platform adapters consume the LLM token stream for clause-by-clause synthesis (closes the #47896 consumer gap)

3 participants