Skip to content

fix(tts): transcode OpenAI-compatible TTS to OGG/Opus when backend lacks opus - #56873

Closed
Bartok9 wants to merge 1 commit into
NousResearch:mainfrom
Bartok9:fix/54589-openai-tts-opus-transcode
Closed

fix(tts): transcode OpenAI-compatible TTS to OGG/Opus when backend lacks opus#56873
Bartok9 wants to merge 1 commit into
NousResearch:mainfrom
Bartok9:fix/54589-openai-tts-opus-transcode

Conversation

@Bartok9

@Bartok9 Bartok9 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • _generate_openai_tts() hardcoded response_format="opus" for any .ogg target, breaking OpenAI-compatible backends that can't encode opus.
  • Honor an optional tts.openai.response_format; when set to a non-opus format for an .ogg target, synthesize in that format and transcode to OGG/Opus locally via ffmpeg.

Motivation

Closes #54589.

Self-hosted OpenAI-compatible TTS backends (e.g. Speaches/Kokoro) only support mp3/flac/wav/pcm. Because Telegram/Matrix voice bubbles use an .ogg output path, _generate_openai_tts() sent response_format="opus" straight to the endpoint, which rejected it — so no voice bubble was delivered. This mirrors the Edge provider, which already synthesizes MP3 and transcodes to OGG/Opus via _convert_to_opus().

Fix

  • When tts.openai.response_format is set to a non-opus format and the target is .ogg: synthesize to a temp file in that format, transcode to .ogg with _convert_to_opus(), and clean up the intermediate file. If ffmpeg transcode fails, raise a clear error.
  • Default behavior is unchanged: without the option, .ogg targets still request opus directly (real OpenAI API path).

Verification

  • python3 -m pytest tests/tools/test_tts_openai_opus_transcode.py tests/tools/test_tts_opus_routing.py — 4 passed
  • test_ogg_target_defaults_to_opus proves the default path is untouched (still requests opus, no transcode).
  • test_ogg_target_transcodes_when_backend_lacks_opus proves the new path asks the backend for mp3, transcodes to .ogg, and removes the temp mp3.

Did NOT change

  • The opus-preference/want_opus routing in text_to_speech_tool() — this fix is scoped to the synthesis format negotiation inside _generate_openai_tts().

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused compatibility fix. The premise remains present on current main: tools/tts_tool.py:1114-1130 requests the extension-derived opus format, while tools/tts_tool.py:2372-2373 selects .ogg for Telegram/OpenAI. The temporary-file conversion matches the existing tools/tts_tool.py:907-941 same-stem OGG/Opus helper.

Problems

  • The new tts.openai.response_format setting is only described in the default-config comment. website/docs/user-guide/features/tts.md:54-58 documents the OpenAI configuration surface but omits it, and :172-179 currently presents OpenAI as always natively Opus-capable.

Suggested changes

  • Add a documented OpenAI-compatible backend example using response_format: mp3, state that ffmpeg performs the OGG/Opus conversion for Telegram, and retain the default native-Opus behavior when unset.

This is an automated hermes-sweeper review.

…cks opus

Closes NousResearch#54589

_generate_openai_tts() derives response_format from the output extension,
so any .ogg target requests opus. OpenAI-*compatible* backends that only
encode mp3/flac/wav/pcm (e.g. self-hosted Speaches/Kokoro) reject that and
no voice bubble is delivered.

Honor an optional tts.openai.response_format. When set to a non-opus format
for an .ogg target, synthesize in that format and transcode to OGG/Opus
locally via ffmpeg (mirroring the Edge provider's _convert_to_opus path).
Default behavior is unchanged. Rebased onto the refactored _generate_openai_tts
(3-tuple _resolve_openai_audio_client_config) and documented the option in
website/docs/user-guide/features/tts.md per review.
@Bartok9
Bartok9 force-pushed the fix/54589-openai-tts-opus-transcode branch from 403bd07 to 5fd0dc2 Compare July 15, 2026 12:55
@Bartok9

Bartok9 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @teknium1 — addressed both the CI failure and the docs gap:

  • CI (slice 3/8 unpack error): rebased onto current main. The branch predated the _generate_openai_tts refactor, so _resolve_openai_audio_client_config() now returns a 3-tuple (api_key, base_url, is_managed) and the test's monkeypatch was returning 2. Reapplied the transcode logic on top of the new _tts_response_format_from_path() structure and updated the test stub. Both regression tests pass locally and ruff is clean.
  • Docs: added the tts.openai.response_format: mp3 example to website/docs/user-guide/features/tts.md (config block + Telegram voice-bubble section), stating ffmpeg performs the OGG/Opus conversion and that the default stays native-Opus when unset.

Default behavior unchanged: .ogg targets still request opus directly unless response_format is pinned to a non-opus format.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 15, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Resolved at the class level by PR #73072. Rather than a per-provider transcode, text_to_speech_tool now sniffs the audio magic bytes once after every synthesis and repairs any MP3/WAV-bytes-in-.ogg centrally (in-place ffmpeg transcode to real Ogg/Opus, honest-extension rename fallback when ffmpeg is missing) — covering this provider and all others, including command providers and plugins. Your diagnosis of the container mismatch was correct and helped shape the central fix — thanks for the contribution.

@teknium1 teknium1 closed this Jul 28, 2026
@Bartok9

Bartok9 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Perfect — a central magic-byte sniff + repair in text_to_speech_tool is the right layer for this; covering every provider (command providers and plugins included) beats a per-provider transcode. Glad the container-mismatch diagnosis was useful. Closing this in favor of #73072. Thanks @teknium1!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/tts Text-to-speech and transcription type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: OpenAI-compatible TTS backends without opus support fail on voice bubbles (_generate_openai_tts hardcodes response_format="opus" for .ogg)

3 participants