Skip to content

fix(gateway): handle silent audio and media producers - #65745

Open
dhansxd wants to merge 1 commit into
NousResearch:mainfrom
dhansxd:fix/gateway-audio-edge-cases
Open

fix(gateway): handle silent audio and media producers#65745
dhansxd wants to merge 1 commit into
NousResearch:mainfrom
dhansxd:fix/gateway-audio-edge-cases

Conversation

@dhansxd

@dhansxd dhansxd commented Jul 16, 2026

Copy link
Copy Markdown

Summary

  • Drop successful empty STT transcripts as valid no-speech before invoking the agent.
  • Transcribe voice follow-ups before busy-session handling, so pending silence is discarded without interrupting active work.
  • Recover allowed media producers from explicit tool-result tool_name / name when the paired assistant tool_calls row is absent.
  • Log unsuccessful auto voice-reply SendResult values instead of silently ignoring them.

Generic symptoms

Command-backed STT can exit successfully with an empty transcript for silence. Fresh silent voice previously reached the LLM as a quoted empty string, while silent voice received during an active turn could queue and interrupt that turn.

Post-stream media extraction also required a paired assistant tool-call row. Persisted/reconstructed tool results that retained explicit producer metadata but lost that row could therefore fail to auto-deliver valid generated media.

Security invariant

Automatic file delivery remains fail-closed: explicit tool-result metadata only identifies the producer; producer must still match _AUTO_APPEND_MEDIA_TOOL_NAMES, and path matching plus delivery-path filtering remain unchanged. Arbitrary tools such as terminal cannot auto-deliver files.

Test evidence

Strict RED observed before implementation:

  • tests/gateway/test_stt_config.py: 1 failed, 7 passed ('" \\n"' is None)
  • tests/gateway/test_busy_session_ack.py: 1 failed, 26 passed (silent voice queued/interrupted)
  • tests/gateway/test_media_extraction.py: 2 failed, 17 passed (explicit tool_name / name ignored); arbitrary producer cases passed
  • tests/gateway/test_send_voice_reply_notify.py: 1 failed, 2 passed (unsuccessful send result not logged)

GREEN:

scripts/run_tests.sh tests/gateway/test_stt_config.py tests/gateway/test_busy_session_ack.py tests/gateway/test_media_extraction.py tests/gateway/test_send_voice_reply_notify.py -q
57 passed, 0 failed

scripts/run_tests.sh tests/gateway/test_telegram_audio_vs_voice.py tests/gateway/test_stt_transcript_echo_config.py tests/gateway/test_voice_command.py tests/gateway/test_tts_media_routing.py tests/gateway/test_telegram_documents.py tests/gateway/test_telegram_voice_duration.py -q
250 passed, 0 failed

scripts/run_tests.sh tests/gateway/test_run_tool_media_re.py tests/gateway/test_media_metadata_contract.py tests/gateway/test_platform_base.py -q
237 passed, 0 failed

uv run ruff check gateway/run.py tests/gateway/test_stt_config.py tests/gateway/test_busy_session_ack.py tests/gateway/test_media_extraction.py tests/gateway/test_send_voice_reply_notify.py
All checks passed

.venv/bin/python -m py_compile gateway/run.py tests/gateway/test_stt_config.py tests/gateway/test_busy_session_ack.py tests/gateway/test_media_extraction.py tests/gateway/test_send_voice_reply_notify.py
passed

git diff --check
passed

ruff format --check was also inspected but not used as a gate: current gateway/run.py and touched legacy test files are already not ruff-formatted on main; formatting whole files would create a large unrelated diff.

Co-Authored-By: Claude <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery tool/tts Text-to-speech and transcription P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 16, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

Fix handles silent audio and media producers in gateway (+169 -6). Good edge case handling improvement.


Reviewed by Hermes Agent

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for covering real gateway edge cases. The silent-STT, producer-metadata, and failed-send premises still exist on current main (gateway/run.py:15806, 1215, and 13730).

Problems

  • The new busy-session voice branch calls adapter.send(...) before the local adapter binding. In current main, the binding remains later in the same function at gateway/run.py:5631; the PR inserts its echo call before that point. A normal non-empty transcript with STT echo enabled therefore raises instead of reaching busy handling. The added test only exercises the empty-transcript early return.

Suggested changes

  • Move/resolve the adapter lookup before the new transcription block and retain the existing absent-adapter fallback.
  • Add a busy non-empty-voice test with transcript echo enabled, asserting both the echo and subsequent busy-mode behavior.

Automated hermes-sweeper review.

Comment thread gateway/run.py
event.media_urls = []
event.media_types = []
if self._should_echo_stt_transcripts():
echo_meta = self._thread_metadata_for_source(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adapter is assigned only later in this function's normal busy-case setup. Resolve it before this STT block (and preserve the no-adapter fallback), otherwise a non-empty transcript with echo enabled raises here.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 18, 2026
@andrexibiza

Copy link
Copy Markdown
Contributor

Vox Lockin lane 07 verification — partially superseded on main; residual gaps shipped in #78196.

Probed against origin/main @ 70db671fac:

Already on main (not re-applied):

  • Empty-transcript handling: fix(gateway): guard empty/inaudible voice transcripts in enrichment #41603's sentinel note (gateway/run.py:21335+) plus the pending-STT cache (f76b2b47aa, _transcribe_pending_audio_event_once) already cover "silent audio" — the busy-path reorder and text is None drop from this PR would regress that deliberate design.
  • Transcript echo before busy handling is covered by _transcribe_and_echo_pending_voice at the interrupt/queue/drain sites.

Genuine residuals still missing on main (shipped in #78196, commit 2):

  1. Producer recovery_collect_auto_append_media_tags (gateway/run.py:1502) still rejects tool results whose paired assistant tool_calls row is absent; persisted/reconstructed tool rows that name their producer via tool_name/name fall out of the auto-delivery allowlist and their TTS/audio MEDIA: tags are silently dropped. This PR's fallback is correct and safe (allowlist still enforced — test_gateway_auto_append_rejects_unallowed_explicit_tool_name).
  2. SendResult logging_send_voice_reply (gateway/run.py:19080) ignores adapter.send_voice() failures.

This PR's head (f2554a9e57) is stale (merge base 5d9a72b7c2) and CONFLICTING/DIRTY against main. Cherry-picked the residual hunks with authorship preserved (Co-Authored-By: Dani Ubaidillah Husain), added regressions in tests/gateway/test_media_extraction.py (producer recovery ×2) and tests/gateway/test_tts_media_routing.py (delivery-failure logging). All targeted suites green locally: 61 passed.

andrexibiza added a commit to andrexibiza/hermes-agent that referenced this pull request Aug 4, 2026
…ribution

Maps the author emails of the two salvaged gateway voice commits (from
NousResearch#74051 @mglavinic86 and NousResearch#65745 @dhansxd) so the check-attribution CI
gate can resolve them to their GitHub identities.

Signed-off-by: andrexibiza <84248988+andrexibiza@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data tool/tts Text-to-speech and transcription type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants