Skip to content

feat(plugins): pluggable TTS backends + Volcengine (Doubao) reference provider - #17589

Closed
Hypnus-Yuan wants to merge 11 commits into
NousResearch:mainfrom
Hypnus-Yuan:feat/pluggable-tts-backends
Closed

Hypnus-Yuan wants to merge 11 commits into
NousResearch:mainfrom
Hypnus-Yuan:feat/pluggable-tts-backends

Conversation

@Hypnus-Yuan

Copy link
Copy Markdown
Contributor

Summary

Mirrors the image_gen plugin refactor for TTS:

  • New TtsProvider ABC + tts_registry in agent/
  • PluginContext.register_tts_provider() hook
  • Prepend-style dispatcher in text_to_speech tool (mirror of image_generation_tool._dispatch_to_plugin_provider)
  • Picker integration (hermes_cli/tools_config.py), setup-wizard status probe (hermes_cli/setup.py)
  • First reference provider: Volcengine (Doubao seed-tts-2.0) for Chinese-native TTS — plugins/voice/volcengine/

Why

Adding a new TTS backend today requires editing tools/tts_tool.py (1500+ LOC). This PR makes that a plugin drop, matching how image_gen already works. The dispatcher is prepend-style so all existing users see zero change — only tts.provider: <non-legacy-name> routes through the plugin path.

What's NOT in this PR

  • STT — separate follow-up PR (Volcengine STT already exists locally, will ship after this merges)
  • Porting existing hardcoded TTS providers (edge/openai/elevenlabs/...) to the plugin interface — one at a time in future PRs
  • Streaming TTS via plugins — stays ElevenLabs-only for now (regression test tests/tools/test_tts_streaming_untouched.py locks this in)

Behavior preservation

  • Users with tts.provider: edge / elevenlabs / openai / minimax / mistral / gemini / xai / kittentts / neutts → zero change
  • Dispatcher fires only when tts.provider is set to a non-legacy name
  • Existing test suite: all ~17k tests pass
  • New test coverage: 8 test files, ~55 test cases covering ABC contract, registry, plugin hook, dispatcher, requirements probe, streaming regression guard, picker injection, setup status probe, Volcengine synthesis (mocked)

How to test locally

uv pip install -e ".[all,dev,voice-volcengine]"
echo 'VOLCENGINE_APP_ID=your_app_id' >> ~/.hermes/.env
echo 'VOLCENGINE_ACCESS_TOKEN=your_token' >> ~/.hermes/.env
# Set tts.provider: volcengine in ~/.hermes/config.yaml
hermes -q "use text_to_speech to say 你好"

Platforms tested

  • macOS 15.x (Apple Silicon)
  • CI: Ubuntu 24.04 + Python 3.11 (via this PR's tests.yml run)

Design notes

  1. TtsProvider.synthesize() returns a Dict[str, Any] (mirror of ImageGenProvider.generate()) rather than writing and raising. This lets providers carry native_opus / voice_compatible metadata forward without a hardcoded name list in _finalize_tts_response.

  2. tts_registry.get_active_provider() returns None when tts.provider is unset — deliberate divergence from image_gen_registry (which auto-selects FAL). The divergence preserves the existing unset-config → Edge-default flow; documented in the registry module docstring.

  3. Volcengine's client.py WebSocket streaming code is lazily imported inside synthesize() so machines without the optional voice-volcengine extra installed can still discover the plugin without ImportError.

  4. Output path derivation in Volcengine provider: when audio_format=ogg_opus, the provider writes to a .ogg path (not the caller-suggested .mp3) and returns the actual path in file_path.

Closes

(no open issue — happy to split / revert / rework if you prefer a different architecture)

Pure refactor in preparation for pluggable TTS backends. Moves the
post-synthesis finalization block (file existence check, Opus conversion,
MEDIA tag construction, response JSON) into a dedicated helper.

No behavior change — all existing TTS tests pass unchanged.
Mirrors agent/image_gen_provider.py in shape. Concrete backends subclass
TtsProvider, implement name + synthesize(text, output_path, config), and
optionally override is_available / max_text_length / list_voices /
default_voice / get_setup_schema.

synthesize() returns a result dict documenting file_path, format,
native_opus, and voice_compatible so the text_to_speech tool can route
Telegram voice-bubble delivery without hardcoded provider lists.
Mirrors agent/image_gen_registry.py with one deliberate divergence:
get_active_provider() returns None when tts.provider is unset, rather
than auto-selecting a legacy-default or sole-registered backend. This
preserves the existing Edge TTS default for users who never configured
tts.provider — the text_to_speech dispatcher treats None as 'fall
through to the hardcoded branch'.

Thread-safe under a module-level Lock. _reset_for_tests() provided for
pytest isolation.
Mirrors register_image_gen_provider at line ~423. Validates the provider
is a TtsProvider subclass; delegates to agent.tts_registry. Plugins call
ctx.register_tts_provider(MyTtsProvider()) from their register(ctx)
function to attach a new TTS backend without touching core tool code.
…atcher

Add LEGACY_TTS_PROVIDERS frozenset and _dispatch_to_plugin_tts_provider
function that fires before the legacy if-elif chain in text_to_speech_tool.

The dispatcher returns None for empty or legacy provider names so
existing users (tts.provider: edge/elevenlabs/openai/etc.) see zero
change. Non-legacy names consult agent.tts_registry; unregistered
names surface a helpful 'run hermes plugins list' error.

_finalize_tts_response_from_plugin consumes the result dict returned
by TtsProvider.synthesize() — file_path, native_opus, voice_compatible
— so plugin backends can produce any format without hardcoded
gateway voice-bubble drift.

check_tts_requirements consults the plugin registry after all
hardcoded checks, so users with only a plugin backend configured
still see TTS as 'available'.

_resolve_max_text_length falls through to the plugin registry when
provider is not in the hardcoded PROVIDER_MAX_TEXT_LENGTH table.
stream_tts_to_speaker must not consult the plugin registry or the
prepend dispatcher. Two source-level assertions lock this in: no
tts_registry reference, no _dispatch_to_plugin_tts_provider reference.
Add _plugin_tts_providers() row builder mirroring _plugin_image_gen_providers.
Plugin providers with names in LEGACY_TTS_PROVIDERS are skipped to avoid
duplicate picker entries.

_visible_providers injects TTS plugin rows when cat is 'Text-to-Speech'.
_is_provider_active recognizes 'tts_plugin_name' marker.
_configure_provider writes tts.provider when tts_plugin_name is present.
When tts.provider names a non-legacy backend, the setup wizard's
tool-status section now consults the TTS plugin registry to determine
availability. Shows display_name and 'missing env vars' when
is_available() returns False, falls through to Edge TTS default
when the named provider is not registered.
…backend

Implements the first plugin-registered TTS provider using the new
TtsProvider ABC + tts_registry infrastructure.

- plugins/voice/volcengine/client.py: 868-line WebSocket bidirectional
  streaming client (seed-tts-2.0 protocol), production-tested
- plugins/voice/volcengine/tts.py: VolcengineTtsProvider implementing
  the ABC contract; returns result dict; lazy-imports .client inside
  synthesize() so discovery doesn't require websockets
- plugins/voice/volcengine/__init__.py: register(ctx) hook
- plugins/voice/volcengine/plugin.yaml: manifest

Design notes:
- is_available() checks only VOLCENGINE_APP_ID + VOLCENGINE_ACCESS_TOKEN
  via get_env_value (profile-aware), never imports .client/websockets
- All os.getenv calls replaced with get_env_value; _reload_dotenv removed
- audio_format=ogg_opus → native_opus=True, voice_compatible=True
- Config > env > default resolution via _env_or helper
- pyproject.toml: add voice-volcengine = ["websockets>=12.0,<15"]
  optional dependency and include it in the [all] extra
- scripts/release.py: add hypnus.yuan@gmail.com → Hypnus-Yuan to AUTHOR_MAP
- hermes_cli/tools_config.py: add tts_plugin_name handling to
  _reconfigure_provider so reconfigure flow correctly sets
  tts.provider for plugin backends
- hermes_cli/setup.py: distinguish unregistered non-legacy plugin
  names from the Edge TTS default fallback in status probe;
  show 'plugin not registered' with helpful guidance
- plugins/voice/volcengine/tts.py: derive correct output path
  based on audio_format (ogg_opus → .ogg, wav → .wav) to avoid
  writing Ogg/Opus bytes into a .mp3 file
- pyproject.toml: tighten websockets version to >=14.0,<16
  (additional_headers API requires websockets 14+)
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins tool/tts Text-to-speech and transcription labels Apr 29, 2026
@Hypnus-Yuan

Copy link
Copy Markdown
Contributor Author

Closing this PR — saw that #17843 (command-type provider registry) shipped
the same day from a different angle, and it subsumes most of what this PR
was trying to solve for TTS extensibility.

Migration path for the Volcengine work:

I've reshaped the 868-line Volcengine WebSocket-streaming client into a
thin CLI that plays nicely with the #17843 config-only interface:

tts:
  provider: volcengine
  providers:
    volcengine:
      type: command
      command: 'python ~/.hermes/scripts/volcengine_tts.py
                --text-file {input_path} --out {output_path}
                --voice {voice} --speed {speed} --format {format}'
      output_format: mp3
      max_text_length: 1024
      timeout: 30

This works for non-streaming use cases (cold-start ~1-2s per call to open
the WebSocket). If a native Python-ABC plugin path is ever wanted for
low-latency streaming backends (Volcengine bidi-stream, MiniMax realtime,
Azure Speech WS, etc.), the TtsProvider + tts_registry + PluginContext
scaffolding in this branch is 100% mirrored on the existing image_gen
plugin system and can be revisited then. No hard feelings closing it — just
wanted to note the scaffolding is available if it becomes useful.

Branch stays on my fork at Hypnus-Yuan/hermes-agent:feat/pluggable-tts-backends
for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have tool/tts Text-to-speech and transcription type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants