fix(gateway): transcribe voice replies to a pending clarify prompt

[Adridot]

What does this PR do?

The gateway's clarify-reply interception only reads (event.text or "").strip(). A voice reply to a pending clarify prompt has empty text, so it falls through: the clarify never resolves and the audio is processed as an unrelated new turn — the user's answer is silently dropped.

This transcribes a voice reply via the existing _enrich_message_with_transcription pipeline and resolves the clarify with the raw transcript, echoing it back as 🎙️ "<transcript>" so the user can confirm what was understood. If STT yields nothing usable, the clarify stays pending and the user is asked to reply in text, instead of the answer being lost.

Design note: this deliberately calls _enrich_message_with_transcription (low-level) rather than the canonical _prepare_inbound_message_text. The latter returns the agent-facing wrapper text (...voice message..., sender prefixes, channel context) and has a native-image-buffer side effect — both wrong for a value read only as a clarify answer. Prior art #31525 took the _prepare_inbound_message_text route; this avoids the wrapper pollution and the side effect, and stays voice-targeted so an image/document reply isn't mistaken for the answer.

Related Issue

No single tracked issue to close. Related (clarify replies interrupted / not accepted): #27564, #39694. Prior art for the same gap (self-closed by its author, unmerged): #31525.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• gateway/run.py — in the clarify-reply interception, transcribe a voice reply and resolve the pending clarify with the raw transcript (echoed as 🎙️ "..."); keep the clarify pending with a text-reply nudge when STT yields nothing usable.
• tests/gateway/test_telegram_audio_vs_voice.py — two tests: a voice reply resolves a pending clarify with the transcript, and a failed transcription keeps the clarify pending.

How to Test

1. Trigger a clarify prompt (agent asks a blocking question).
2. Reply with a voice message instead of text.
   • Before: clarify stays pending; the voice note is handled as a brand-new message; the answer is lost.
   • After: the voice note is transcribed, echoed as 🎙️ "<transcript>", and resolves the clarify.
3. Reply with an unintelligible/empty voice note → clarify stays pending and the user is asked to reply in text (no silent drop).

Automated: pytest tests/gateway/test_telegram_audio_vs_voice.py -q → 7 passed.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):)
• I searched for existing PRs to make sure this isn't a duplicate (found Fix Telegram voice replies during clarify #31525, self-closed — referenced above)
• My PR contains only changes related to this fix (single commit, two files)
• I've run pytest tests/ -q and all tests pass — ran the affected file tests/gateway/test_telegram_audio_vs_voice.py (7 passed) + py_compile gateway/run.py; did not run the full suite in this environment
• I've added tests for my changes
• I've tested on my platform: Ubuntu (Linux), running on a production Telegram deployment

Documentation & Housekeeping

• I've updated relevant documentation — N/A (no doc/docstring impact)
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (no config keys)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) — N/A (no platform-specific code; pure async over existing primitives)
• I've updated tool descriptions/schemas if I changed tool behavior — N/A (no tool/schema change)

Screenshots / Logs

$ pytest tests/gateway/test_telegram_audio_vs_voice.py -q
.......                                                                  [100%]
7 passed in 0.87s

[Adridot]

Hello @liuhao1024,
Any chance this PR could be merged ? It seems like the issue is being called out by many people

[LauraGPT]

I tested the Matrix/SenseVoice case from #52998 against current main (915f1bf1bc2f939356226250e0eade3c9b3c5960) with this PR merged.

There is one remaining gap: Matrix voice events can carry a cached filename such as voice_message_123.ogg in event.text. This PR only enters the STT branch when the raw clarify reply is empty, so that filename is currently resolved as the user's answer instead of the transcript. The failed-transcription case also needs to avoid resolving any enriched failure marker.

I added two end-to-end GatewayRunner._handle_message regressions and ran them against this PR merged onto current main: 2 failed, 1 passed. The failures were:

• successful STT resolved voice_message_123.ogg instead of use the local model
• failed STT did not return the text-reply retry while keeping the clarify pending

A tested implementation is available in 1f0694aa. It applies the normal inbound voice/audio classification before clarify resolution even when event.text contains a filename, accepts only raw non-empty transcripts, and leaves the clarify pending when transcription fails.

Verification on that commit:

• canonical test wrapper: 89 passed, 0 failed across clarify, STT, Matrix voice, Telegram audio/voice, queue, and button tests
• ruff check, format check, py_compile, git diff --check, and Windows-footgun check all pass

For one-click integration, I opened Adridot/hermes-agent#1 against the exact author branch. It contains only the tested delta at bc8ad6a: 84/84 related tests pass on the author head, and 91/91 pass after merging it with current upstream main at 861d69c. The upstream review remains consolidated here.

[Copilot]

Pull request overview

This PR fixes a gateway edge case where a user’s voice reply to a pending clarify prompt could not be captured (because event.text is empty), causing the clarify to remain unresolved and the reply to be mishandled as a new turn. The change extends the clarify interception path in gateway/run.py to transcribe voice media and resolve the pending clarify using the raw transcript, with a fallback message when transcription yields no usable text.

Changes:

• Add a voice-aware clarify interception path that detects voice-like audio attachments, runs _enrich_message_with_transcription, and resolves the clarify with the raw transcript.
• Echo back the transcript to the chat (🎙️ "...") so the user can confirm what was understood.
• Add tests validating that (a) a voice reply resolves a pending clarify via transcript and (b) failed transcription keeps clarify pending and prompts for text.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
gateway/run.py	Transcribes voice replies during pending-clarify interception and uses transcript to resolve (or prompts for text if transcription fails).
tests/gateway/test_telegram_audio_vs_voice.py	Adds async tests covering clarify-resolution via voice transcript and non-resolution on transcription failure.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[teknium1]

Thanks for covering the pending-clarify voice path and incorporating the Matrix cached-filename follow-up from the discussion. The premise is still present on current main: gateway/run.py:9090-9107 only resolves a clarify from non-empty event.text.

Problems

• gateway/run.py:7512 on the PR head unconditionally posts the raw transcript. This bypasses stt.echo_transcripts: false, whose documented purpose is to avoid posting raw transcripts (website/docs/user-guide/configuration.md:1552). Existing echo paths are gated by _should_echo_stt_transcripts() (gateway/run.py:10463, 15329), and tests/gateway/test_stt_transcript_echo_config.py:57-70 enforces that invariant.

Suggested changes

• Gate the new clarify echo with _should_echo_stt_transcripts() and add a regression that voice clarify still resolves while no echo is sent when the setting is false.

Automated hermes-sweeper review.

[teknium1]

Please gate this echo with _should_echo_stt_transcripts(). stt.echo_transcripts: false is documented to suppress raw transcript posts, and existing gateway echo paths all preserve that setting.

[LauraGPT]

I verified both currently unresolved review findings against the generated merge of current main (569b912d7) and this PR head (77a9be56c).

Red:

• The existing test_all_gateway_transcript_echo_sends_are_gated regression fails specifically on the new clarify 🎙️ send.
• A behavioral regression with GatewayConfig(stt_echo_transcripts=False) confirms that the voice transcript still resolves the pending clarify correctly, but adapter.send is awaited once.
• A Telegram DM-topic regression confirms the clarify transcript resolves correctly but the echo receives only {"thread_id": "42"}, dropping direct_messages_topic_id, the reply anchor, and the topic fallback flag.

The minimal production changes are:

-                            _echo_meta = {"thread_id": source.thread_id} if source.thread_id else None
+                            _echo_meta = self._thread_metadata_for_source(
+                                source, self._reply_anchor_for_event(event),
+                            )
-                            if _echo_adapter:
+                            if _echo_adapter and self._should_echo_stt_transcripts():

The two behavioral regressions assert the complete contract: _handle_message() acknowledges the reply, clarify_gateway receives the raw transcript, a disabled echo setting sends nothing, and an enabled Telegram DM-topic echo preserves direct_messages_topic_id plus its reply anchor.

Green on the current-main merge tree:

• 87/87 tests pass across clarify, STT configuration/echo, Matrix voice, Telegram audio/voice, queue, and clarify-button paths.
• Ruff lint, py_compile, and both staged/working-tree diff checks pass.

The source branch predates stt_echo_transcripts, _should_echo_stt_transcripts(), and the current topic-aware metadata helper, so it should first merge/rebase current main, then apply these focused changes and regressions. That keeps the final implementation on the shared helpers instead of adding old-base compatibility paths that would be dead after merge.

[Adridot]

Addressed the two outstanding echo findings in b5ba9fd (after merging current main into the branch):

• The clarify 🎙️ echo is now gated behind _should_echo_stt_transcripts(), so stt.echo_transcripts: false is honored like every other echo path.
• The echo metadata is built with _thread_metadata_for_source(source, _reply_anchor_for_event(event)), so Telegram DM-topic echoes keep direct_messages_topic_id, the reply anchor, and the topic fallback flag.

Added the two behavioral regressions (echo disabled → clarify still resolves, no send; DM-topic echo → full routing metadata preserved). tests/gateway/test_stt_transcript_echo_config.py (incl. test_all_gateway_transcript_echo_sends_are_gated) now passes, and the related suites (clarify, STT, Matrix voice, Telegram audio/voice, queue, buttons) are green: 223 passed, 22 skipped. ruff check, py_compile, and git diff --check all pass.