Skip to content

fix(tts): class-level .ogg container repair + multi-platform Opus voice detection - #73072

Merged
teknium1 merged 1 commit into
mainfrom
fix/tts-ogg-container-repair
Jul 28, 2026
Merged

fix(tts): class-level .ogg container repair + multi-platform Opus voice detection#73072
teknium1 merged 1 commit into
mainfrom
fix/tts-ogg-container-repair

Conversation

@teknium1

@teknium1 teknium1 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Broken TTS voice bubbles are fixed at the class level: text_to_speech_tool now sniffs the audio file's magic bytes once after synthesis and repairs any "MP3/WAV bytes in a .ogg path" centrally — covering every current and future provider — and want_opus now recognizes all Ogg/Opus voice-bubble platforms, not just Telegram.

Root cause (two class defects, ~15 open one-off PRs each patching one provider or one platform):

  1. Several backends silently ignore the requested opus format — Edge only emits MP3, Piper writes WAV, xAI writes MP3, some OpenAI-compatible servers (Speaches/Kokoro) reject or ignore response_format="opus". The bytes land in a .ogg path and Telegram/Matrix/Feishu render a broken 0-second voice bubble.
  2. want_opus = (platform == "telegram") — Matrix, Feishu, WhatsApp and Signal auto-TTS replies were synthesized as MP3 and delivered as broken attachments.

Addresses #57048, #54589, #57213, #58845, #14841, #45557 (the container half), #57049 (the platform half).

Changes

  • tools/tts_tool.py:
    • _sniff_audio_container() — magic-byte detection (ogg/wav/mp3/flac).
    • _repair_ogg_container() — central post-synthesis check in the dispatcher: ffmpeg transcode in place to real Ogg/Opus; if ffmpeg is unavailable, rename to the honest extension so platforms deliver a working file instead of a dead bubble.
    • _ffmpeg_transcode_to_opus() — shared transcode helper (safe in-place via temp file + os.replace, forced -f ogg muxer); _convert_to_opus now delegates to it.
    • OPUS_VOICE_PLATFORMS = {telegram, matrix, feishu, whatsapp, signal} replaces the Telegram-only check.

Validation

Check Result
tests/tools/test_tts_container_repair.py (new, 13 tests) pass — includes a LIVE ffmpeg round-trip: real MP3 misnamed .ogg → sniffed mp3 → repaired → sniffed ogg
Full TTS suite (14 files) 282 passed
No-ffmpeg fallback renames v.ogg (MP3 bytes) → v.mp3

Infographic

TTS container repair

@teknium1
fix(tts): class-level .ogg container repair + multi-platform opus voi…
ddd7118 
…ce detection

Root-cause fix for the 'TTS voice bubble broken' issue family (#57048,
#54589, #57213, #58845, #14841, #45557, #57049). Two class-level defects:

1. Several backends silently write MP3/WAV bytes into a .ogg output path
   (Edge only emits MP3, Piper writes WAV, xAI writes MP3, some
   OpenAI-compatible servers ignore response_format=opus). Platforms that
   need real Ogg/Opus render 0-second/broken voice bubbles. Instead of
   per-provider patches, text_to_speech_tool now sniffs magic bytes once
   after synthesis (_sniff_audio_container) and repairs the container
   centrally (_repair_ogg_container): ffmpeg transcode in place, or rename
   to the honest extension when ffmpeg is unavailable. Covers every
   current and future provider, including command providers and plugins.

2. want_opus only recognized Telegram, so Matrix/Feishu/WhatsApp/Signal
   auto-TTS voice replies were synthesized as MP3 and delivered as broken
   attachments. New OPUS_VOICE_PLATFORMS set covers all voice-bubble
   platforms.

_convert_to_opus refactored onto a shared _ffmpeg_transcode_to_opus that
supports safe in-place transcodes (-f ogg forced muxer, temp file +
os.replace).

Tests: tests/tools/test_tts_container_repair.py (13 tests incl. a live
ffmpeg round-trip); full TTS suite green (153 + 187 passed).
@github-actions

github-actions Bot commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on ddd7118

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

1 visual diff.

inline evidence upload failed.

Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso)

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/tools Tool registry, model_tools, toolsets tool/tts Text-to-speech and transcription labels Jul 28, 2026
@teknium1
teknium1 merged commit fae29c8 into main Jul 28, 2026
40 checks passed
@teknium1
teknium1 deleted the fix/tts-ogg-container-repair branch July 28, 2026 03:28
This was referenced Jul 28, 2026
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Merged
Merged
Merged
teknium1 pushed a commit that referenced this pull request Jul 28, 2026
@malaiwah @teknium1
fix(matrix): enforce Ogg/Opus at send_voice boundary, probe metadata …
c289d80 
…off-loop

Salvaged from PR #68063 (@malaiwah). MatrixAdapter.send_voice now transcodes
any non-Ogg audio to Ogg/Opus at the adapter boundary (best-effort — the
original file is sent unchanged when ffmpeg is unavailable), so MSC3245
voice bubbles render even when a caller hands the adapter MP3/WAV audio.
_matrix_voice_metadata_for_file probing now runs via asyncio.to_thread so
ffprobe/ffmpeg subprocess timeouts can't stall the adapter event loop.

The PR's tools/tts_tool.py want_opus hunk was dropped: main's
OPUS_VOICE_PLATFORMS set (PR #73072) already includes matrix; the Matrix
opus-routing test is kept.

Refs #14841
teknium1 pushed a commit that referenced this pull request Jul 28, 2026
Gilad Bauman @teknium1
fix(gateway): platform-aware auto-TTS output path for native voice bu…
ce71b68 
…bbles

Salvaged from PR #62040 (@giladbau), simplified per post-#73072 main: the
central _repair_ogg_container transcode makes an explicit .ogg output path
sufficient — no target_platform plumbing through the TTS tool needed.

Root cause (class-level): both gateway auto-TTS delivery call sites relied
on the TTS tool reading HERMES_SESSION_PLATFORM to pick Ogg/Opus vs MP3,
but that contextvar is cleared by _clear_session_env before the base
adapter's post-handler auto-TTS block runs, so want_opus was always False
on that path → MP3 → Telegram sent an audio attachment instead of a native
voice bubble (#57049, #36685). The runner's _send_voice_reply had the
sibling bug: it hardcoded .ogg for Telegram only, leaving Matrix and
Feishu runner voice replies as MP3 (#14841, #45557).

Fix: new build_auto_tts_output_path(platform) in gateway/platforms/base.py
hands an explicit .ogg temp path when the platform is in the TTS tool's
OPUS_VOICE_PLATFORMS set (single source of truth — telegram/matrix/feishu/
whatsapp/signal), .mp3 otherwise. Used by BOTH delivery call sites:
- BasePlatformAdapter auto-TTS block (also honors the tool's success flag
  and cleans up requested + returned paths)
- GatewayRunner._send_voice_reply (replaces the telegram-only ternary)

Fixes #57049
Fixes #36685
Refs #14841 #45557
teknium1 added a commit that referenced this pull request Jul 28, 2026
@teknium1
refactor: extract shared audio container sniffer to tools/audio_conta…
156edc5 
…iner.py

One sniffer owns magic-byte container detection (Teknium's one-concept-
one-owner rule): the new tools/audio_container.py is used by

- gateway/platforms/base.py _sniff_audio_ext (inbound cache — PR #36166's
  central sniffer, now covering AAC/ADTS, MP4-brand disambiguation, webm)
- gateway/platforms/signal.py _guess_extension (audio/AV branches
  delegated; RIFF/WAVE fix from PR #50690 and M4A-brand fix from
  PR #72490 now live centrally)
- tools/tts_tool.py _sniff_audio_container (outbound repair, PR #73072)

cache_audio_from_url inherits the sniff via cache_audio_from_bytes.
Adds tests/tools/test_audio_container.py covering every magic-byte type,
wrong-extension repair on the inbound cache, unknown passthrough, the
URL path, and Signal's delegation.
teknium1 pushed a commit that referenced this pull request Jul 28, 2026
@LeonSGP43 @teknium1
fix(tts): honor tts.openai.api_key and base_url from config.yaml
efc81a1 
Salvaged from PR #26233 (@LeonSGP43), rebased onto the current 3-tuple
_resolve_openai_audio_client_config (is_managed flag, post-#73072 layout).
Same fix independently submitted earlier in PR #26209 (@zccyman) — credit
to both.

Resolution order now mirrors the STT resolver: tts.openai.api_key/base_url
from config.yaml -> VOICE_TOOLS_OPENAI_KEY/OPENAI_API_KEY env (still
honoring config base_url) -> managed gateway. _has_openai_audio_backend
also counts a config api_key as an available backend.

Fixes #26175
teknium1 pushed a commit that referenced this pull request Jul 28, 2026
@malaiwah @teknium1
fix(matrix): enforce Ogg/Opus at send_voice boundary, probe metadata …
33313e8 
…off-loop

Salvaged from PR #68063 (@malaiwah). MatrixAdapter.send_voice now transcodes
any non-Ogg audio to Ogg/Opus at the adapter boundary (best-effort — the
original file is sent unchanged when ffmpeg is unavailable), so MSC3245
voice bubbles render even when a caller hands the adapter MP3/WAV audio.
_matrix_voice_metadata_for_file probing now runs via asyncio.to_thread so
ffprobe/ffmpeg subprocess timeouts can't stall the adapter event loop.

The PR's tools/tts_tool.py want_opus hunk was dropped: main's
OPUS_VOICE_PLATFORMS set (PR #73072) already includes matrix; the Matrix
opus-routing test is kept.

Refs #14841
teknium1 pushed a commit that referenced this pull request Jul 28, 2026
Gilad Bauman @teknium1
fix(gateway): platform-aware auto-TTS output path for native voice bu…
1753369 
…bbles

Salvaged from PR #62040 (@giladbau), simplified per post-#73072 main: the
central _repair_ogg_container transcode makes an explicit .ogg output path
sufficient — no target_platform plumbing through the TTS tool needed.

Root cause (class-level): both gateway auto-TTS delivery call sites relied
on the TTS tool reading HERMES_SESSION_PLATFORM to pick Ogg/Opus vs MP3,
but that contextvar is cleared by _clear_session_env before the base
adapter's post-handler auto-TTS block runs, so want_opus was always False
on that path → MP3 → Telegram sent an audio attachment instead of a native
voice bubble (#57049, #36685). The runner's _send_voice_reply had the
sibling bug: it hardcoded .ogg for Telegram only, leaving Matrix and
Feishu runner voice replies as MP3 (#14841, #45557).

Fix: new build_auto_tts_output_path(platform) in gateway/platforms/base.py
hands an explicit .ogg temp path when the platform is in the TTS tool's
OPUS_VOICE_PLATFORMS set (single source of truth — telegram/matrix/feishu/
whatsapp/signal), .mp3 otherwise. Used by BOTH delivery call sites:
- BasePlatformAdapter auto-TTS block (also honors the tool's success flag
  and cleans up requested + returned paths)
- GatewayRunner._send_voice_reply (replaces the telegram-only ternary)

Fixes #57049
Fixes #36685
Refs #14841 #45557
@teknium1 teknium1 mentioned this pull request Jul 29, 2026
Closed
2 tasks
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@teknium1
refactor: extract shared audio container sniffer to tools/audio_conta…
21b8297 
…iner.py

One sniffer owns magic-byte container detection (Teknium's one-concept-
one-owner rule): the new tools/audio_container.py is used by

- gateway/platforms/base.py _sniff_audio_ext (inbound cache — PR NousResearch#36166's
  central sniffer, now covering AAC/ADTS, MP4-brand disambiguation, webm)
- gateway/platforms/signal.py _guess_extension (audio/AV branches
  delegated; RIFF/WAVE fix from PR NousResearch#50690 and M4A-brand fix from
  PR NousResearch#72490 now live centrally)
- tools/tts_tool.py _sniff_audio_container (outbound repair, PR NousResearch#73072)

cache_audio_from_url inherits the sniff via cache_audio_from_bytes.
Adds tests/tools/test_audio_container.py covering every magic-byte type,
wrong-extension repair on the inbound cache, unknown passthrough, the
URL path, and Signal's delegation.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@LeonSGP43 @teknium1
fix(tts): honor tts.openai.api_key and base_url from config.yaml
ffc0e7f 
Salvaged from PR NousResearch#26233 (@LeonSGP43), rebased onto the current 3-tuple
_resolve_openai_audio_client_config (is_managed flag, post-NousResearch#73072 layout).
Same fix independently submitted earlier in PR NousResearch#26209 (@zccyman) — credit
to both.

Resolution order now mirrors the STT resolver: tts.openai.api_key/base_url
from config.yaml -> VOICE_TOOLS_OPENAI_KEY/OPENAI_API_KEY env (still
honoring config base_url) -> managed gateway. _has_openai_audio_backend
also counts a config api_key as an available backend.

Fixes NousResearch#26175
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
@malaiwah @teknium1
fix(matrix): enforce Ogg/Opus at send_voice boundary, probe metadata …
bb51f62 
…off-loop

Salvaged from PR NousResearch#68063 (@malaiwah). MatrixAdapter.send_voice now transcodes
any non-Ogg audio to Ogg/Opus at the adapter boundary (best-effort — the
original file is sent unchanged when ffmpeg is unavailable), so MSC3245
voice bubbles render even when a caller hands the adapter MP3/WAV audio.
_matrix_voice_metadata_for_file probing now runs via asyncio.to_thread so
ffprobe/ffmpeg subprocess timeouts can't stall the adapter event loop.

The PR's tools/tts_tool.py want_opus hunk was dropped: main's
OPUS_VOICE_PLATFORMS set (PR NousResearch#73072) already includes matrix; the Matrix
opus-routing test is kept.

Refs NousResearch#14841
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
Gilad Bauman @teknium1
fix(gateway): platform-aware auto-TTS output path for native voice bu…
a46adb5 
…bbles

Salvaged from PR NousResearch#62040 (@giladbau), simplified per post-NousResearch#73072 main: the
central _repair_ogg_container transcode makes an explicit .ogg output path
sufficient — no target_platform plumbing through the TTS tool needed.

Root cause (class-level): both gateway auto-TTS delivery call sites relied
on the TTS tool reading HERMES_SESSION_PLATFORM to pick Ogg/Opus vs MP3,
but that contextvar is cleared by _clear_session_env before the base
adapter's post-handler auto-TTS block runs, so want_opus was always False
on that path → MP3 → Telegram sent an audio attachment instead of a native
voice bubble (NousResearch#57049, NousResearch#36685). The runner's _send_voice_reply had the
sibling bug: it hardcoded .ogg for Telegram only, leaving Matrix and
Feishu runner voice replies as MP3 (NousResearch#14841, NousResearch#45557).

Fix: new build_auto_tts_output_path(platform) in gateway/platforms/base.py
hands an explicit .ogg temp path when the platform is in the TTS tool's
OPUS_VOICE_PLATFORMS set (single source of truth — telegram/matrix/feishu/
whatsapp/signal), .mp3 otherwise. Used by BOTH delivery call sites:
- BasePlatformAdapter auto-TTS block (also honors the tool's success flag
  and cleans up requested + returned paths)
- GatewayRunner._send_voice_reply (replaces the telegram-only ternary)

Fixes NousResearch#57049
Fixes NousResearch#36685
Refs NousResearch#14841 NousResearch#45557
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists tool/tts Text-to-speech and transcription type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@teknium1 @alt-glitch