fix(gateway): transcribe voice replies before clarify resolution and busy steering - #73518
Merged
Merged
Conversation
… sent as documents [[audio_as_voice]] is message-global but was applied to every media file in a message. A non-audio file flagged is_voice is excluded from the embedded-photo batch and falls through to send_document, so an image in a message that also carries a voice note arrives as a file attachment instead of an inline photo. Gate the voice flag on the file extension so one message can carry an inline image AND a voice bubble. Also wrap the path append in try/except so a crafted ~\x00 path is skipped rather than aborting extraction of all attachments.
_invalidate_pending_stt_cache() clears the gateway-side transcription
cache when merge_pending_message_event() folds a follow-up message into a
still-pending event, so the next transcription picks up the merged text
and attachments. It also cleared _gateway_pending_stt_echo_sent, but that
flag is not derived state — it records that the transcript was already
delivered to the user.
Dropping it makes the re-run transcription echo the earlier notes a second
time. Both merge branches are affected, including the text-only follow-up
case where no new audio arrived at all: there the cache is invalidated,
the same voice note is transcribed again (a second paid STT call) and the
same line is echoed again.
Sequence:
1. voice note arrives, interrupt monitor transcribes it and echoes
'🎙️ "hello"'
2. user sends a follow-up while the turn is still pending, so it merges
3. drain path re-transcribes and echoes '🎙️ "hello"' a second time
Keep the ledger out of the invalidation set and track it as a count of
already-echoed transcripts instead of a single boolean. A count is what
the merge case actually needs: re-running transcription over the extended
media list returns the earlier transcripts as a prefix of the new one, so
echoing only the unsent tail suppresses the repeat while still surfacing a
newly merged voice note. A count rather than a set of seen values, so two
separate notes that transcribe identically stay two distinct deliveries —
covered by test_pending_stt_merge_echoes_two_identical_transcripts.
The guard stays within the 12-line window that
test_all_gateway_transcript_echo_sends_are_gated enforces over run.py.
…point Follow-up for salvaged #65023/#53020: _prepare_busy_steer_text now calls _transcribe_and_echo_pending_voice (the same helper the interrupt monitor and pending-drain paths use) instead of a private transcription+echo copy, so out-of-band voice pays one STT call per platform message and the echo respects the count-based ledger from #67281. can_steer now accepts events whose attachments are all STT-eligible voice media, completing the steer half of #58780. Adds extract_media gating tests for #44826 and the contributor mapping for chefboyrdave21.
Contributor
૮ >ﻌ< ა ci reviewran on a3f970c ℹ️ InfoDesktop E2E visual evidence · View test artifacts · View job1 visual diff. inline evidence upload failed. Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso) |
alt-glitch
added
type/bug
This was referenced Jul 29, 2026
Closed
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(gateway): voice messages during agent activity are first-class — transcribe before clarify + steer, gate audio_as_voice, keep the echo ledger
Consolidated salvage of four contributor PRs. Class-level framing: out-of-band voice (a voice note that arrives while the agent is busy or waiting on a clarify) now routes through one transcription choke point —
_transcribe_and_echo_pending_voice/_transcribe_pending_audio_event_once— so each platform message pays at most one STT call, and the transcript echo respects a count-based ledger that survives pending-media merges.What was broken on main (all premises re-verified on f228e14)
gateway/run.pyreadevent.textraw. A voice reply arrives with the filename as text (e.g.voice_message_1.ogg), so the clarify was "answered" with a filename, or dropped. Fixes Voice message replies to pending clarify are sent as filename, not transcript #52998, Voice messages ignored when Telegram clarify tool is waiting for user response #56739.can_steerrequiredmessage_type == TEXTwith zero media, so withdisplay.busy_input_mode: steera voice follow-up quietly fell back to queue. Fixes the steer half of Telegram: out-of-band messages (text and voice) interrupt current task instead of being queued #58780.[[audio_as_voice]]tainted non-audio files — the directive is message-global; an image sharing a message with a voice note gotis_voice=True, was excluded from the embedded-photo batch, and arrived as a document attachment._invalidate_pending_stt_cache()deleted_gateway_pending_stt_echo_sentwhen a follow-up merged into a pending event, so the re-run transcription re-echoed (and re-paid for) the earlier voice note.Commits (contributor authorship preserved via cherry-pick)
ebe57ca7fe_prepare_clarify_reply_text: transcribe pending voice audio before resolving a clarify; retain the pending clarify when STT yields no usable text (a failure marker is not an answer).97f2a91f53_prepare_busy_steer_text: transcribe busy voice follow-ups before the steer decision.6f8ccd5136[[audio_as_voice]]on audio file extensions inextract_media; wrapexpanduserin try/except so a crafted~\x00path is skipped, not fatal.92818ae028_invalidate_pending_stt_cache; track echoed transcripts as a count so only the unsent tail is echoed after a merge (two identical transcripts stay two deliveries).a3f970c16c_prepare_busy_steer_textnow routes through the shared_transcribe_and_echo_pending_voicechoke point (event-cached STT + count-ledger echo, same as interrupt/drain);can_steeraccepts all-voice-media events so the transcript actually steers; extension gating also applied to extensionless MEDIA-tag resolution; addsextract_mediagating tests + contributor mapping.Duplicate PRs to close with credit (earliest submitter first)
Clarify-voice trio — earliest was #50925 by @Adridot (June 22); #53020 (@izumi0uu, June 26) won on diff quality (reuses
_pending_event_audio_paths, retains the clarify on empty STT); #67014 (@dpowrepo, July 18) same idea later.Steer-voice pair — earliest was #49949 by @the3asic (June 21); #65023 (@canorionen, July 15) won on diff quality (mirrors the inbound STT contract instead of inlining media classification).
Issues
display.busy_input_modedocs answer the rest)Tests
tests/gateway/test_unknown_command.py: clarify-with-voice — transcript resolves the clarify (incl. numeric choice coercion), failed STT leaves the clarify pending (2 new tests, parametrized).tests/gateway/test_busy_session_ack.py: steer-with-voice — voice follow-up is transcribed and steered, never queued.tests/gateway/test_platform_base.py:[[audio_as_voice]]gating — image+voice message keeps the image inline; video/pdf never flagged (2 new tests; verified they FAIL when the gate is reverted).tests/gateway/test_telegram_voice_v0_regressions.py: 4 new ledger tests incl.test_pending_stt_merge_echoes_two_identical_transcriptsand no-re-echo across text-only merges.tests/gatewaysuite: 11297 passed; the only failures (test_voice_command.py::TestVoiceReceptionDAVE decrypt pair + env-dependent errors) reproduce identically on cleanorigin/main— pre-existing, unrelated.Merge method: rebase (preserves contributor authorship per-commit).
Infographic