fix(gateway): surface and log voice auto-transcription failures instead of stalling

[Finn763]

What Problem This Solves

Reported in #105315: a Telegram voice note is downloaded and cached, the assistant shows
"typing…" forever, no reply ever arrives, and the gateway logs stop right after
Cached user voice at … — nothing at INFO level explains what happened. Manual
transcription of the same file from the CLI works in seconds.

The voice actually reaches the automatic STT pipeline (the reporter's inbound message … msg='' line is the voice turn); what it reaches is a path that cannot report failure.
Two independent defects in gateway/run_inbound.py:

1. The automatic STT call had no bound. _transcribe_one_clip awaited
   asyncio.to_thread(transcribe_audio, path, None, "gateway") (gateway/run_inbound.py:1934
   on main) with no timeout, and the same for the local fallback (:1936). A backend that
   never returns — local whisper/Faster-Whisper wedged in model load or CUDA init, a stalled
   provider socket — leaves the inbound turn awaiting that future forever. The platform's
   typing task is driven by the event loop, which is still healthy, so the loop watchdog
   never fires and the indicator is refreshed indefinitely. Nothing is raised, so the
   [voice message could not be transcribed automatically; …] note that every failure branch
   emits was never produced, and no reply was ever generated.

2. The one log line that would have distinguished "STT is wedged" from "STT never ran"
   was DEBUG. logger.debug("Transcribing user voice: %s", path) (:1987) is invisible in
   the reporter's default INFO logs, which is exactly the ambiguity raised in the issue
   thread. The "transcription failed" line (:1941) was INFO too, with no traceback, so even
   a fast failure was easy to miss.

A third, smaller defect: _transcribe_one_clip only tried the local-STT recovery when the
configured provider returned success=False. If the provider raised (e.g. CUDA libs
failing to dlopen, a network exception), the exception escaped to the caller's handler at
:1994 and the local fallback the code was written to provide never ran.

The fix

• _bounded_stt_call (new) awaits each blocking STT call off-loop under
  asyncio.wait_for(..., timeout=stt.timeout_seconds), default 180s, and logs an explicit
  ERROR naming the stage, the timeout and the file ("…did not return within 180.0s … a wedged
  local model load / CUDA init is the usual cause"). On timeout the clip gets the existing
  neutral _untranscribed_audio_note marker, so the agent still gets a turn and the user gets a
  reply instead of an endless typing indicator.
• A timed-out backend is not retried through the second backend: wait_for can only abandon
  the future, not the worker thread, so stacking the local fallback on top would just pile a
  second wedged call behind the first (and, with the local provider configured, behind the same
  model lock).
• The start of automatic transcription is now logged at INFO
  (Auto-transcribing inbound voice: <path> (timeout 180s)), and the final failure line at
  WARNING — so an operator can tell "wedged" from "never called" from a default log.
• A raising configured provider is now treated as a failed result (WARNING "Configured STT raised for …") and the local fallback still runs, i.e. the recovery path actually recovers.
• The caller's unexpected-error handler logs the path and a traceback
  (Transcription error for <path>: …, exc_info=True).
• The bound is configurable via stt.timeout_seconds (documented in cli-config.yaml.example)
  for slow first-time model loads on weak GPUs; missing/unparsable/zero values keep the 180s
  default.

Partial fix for #105315 — the closing keyword is intentionally not used here (this PR does not close issue #105315): the gateway-side
silent stall is fixed and reproduced locally, but the reporter's wedged Faster-Whisper/CUDA
call itself needs their Windows/GTX 1650 Ti host to confirm (see the evidence boundary). What
this PR guarantees on any host: the turn can no longer hang behind STT unnoticed, and the logs
now say which stage wedged.

Evidence

No Telegram, no real audio, no STT model: the fake download/STT fixtures drive
GatewayRunner._enrich_message_with_transcription (the single choke point every automatic
inbound-voice path routes through — the pending/interrupt _transcribe_and_echo_pending_voice
and the dispatch-time _enrich_inbound_voice both end here).

New test file tests/gateway/test_voice_stt_silent_failure_105315.py, run with
HERMES_PYTHON=… bash scripts/run_tests.sh tests/gateway/test_voice_stt_silent_failure_105315.py
(and directly per-file with the repo venv):

• RED before the change — 3 failed in 9.22s:
  • test_hanging_stt_call_is_bounded_and_marked → asyncio.exceptions.TimeoutError from the
    test's own 8s guard: a fake transcribe_audio that blocks forever (the stand-in for the
    wedged CUDA call) means _enrich_message_with_transcription never returns — the
    typing-forever stall, reproduced without a GPU.
  • test_auto_transcription_start_is_logged_at_info → captured records []: nothing at INFO.
  • test_raising_provider_still_recovers_with_local_stt → recorded
    ERROR gateway.run:run_inbound.py:1995 Transcription error: cuda lib load failed and an
    empty transcript list: the exception skipped the local fallback.
• GREEN after the change — 4 passed in 2.23s (3 stall/log/recovery tests + the
  stt.timeout_seconds parsing test); the 3 behaviour tests alone went from
  3 failed in 9.22s to 3 passed in 1.44s — the same hang now costs one bounded wait and
  yields the neutral marker instead of an 8s test timeout.
• Sabotage verification (each fix piece reverted in isolation, restored file confirmed
  byte-identical afterwards):
  • drop timeout=timeout_s → test_hanging_stt_call_is_bounded_and_marked FAILED
    (1 failed, 2 passed)
  • re-raise instead of falling back → test_raising_provider_still_recovers_with_local_stt
    FAILED (1 failed, 2 passed)
  • log the start marker at DEBUG again → test_auto_transcription_start_is_logged_at_info
    FAILED (1 failed, 2 passed)
• Related suites green (per-file subprocesses, repo venv, TZ=UTC PYTHONHASHSEED=0):
  tests/gateway/test_telegram_voice_v0_regressions.py 3 passed,
  tests/gateway/test_telegram_audio_vs_voice.py 2 passed, tests/gateway/test_stt_config.py
  4 passed, tests/gateway/test_stt_transcript_echo_config.py 2 passed,
  tests/gateway/test_mixed_attachment_routing.py 4 passed,
  tests/gateway/test_feishu_voice_message_type.py 1 passed,
  tests/gateway/test_busy_session_ack.py 11 passed,
  tests/gateway/test_run_progress_interrupt.py 3 passed,
  tests/gateway/relay/test_wire_voice_media.py 19 passed,
  tests/gateway/relay/test_relay_media.py 8 passed, tests/tools/test_transcription.py
  16 passed, tests/tools/test_stt_idle_unload.py 17 passed,
  tests/gateway/test_voice_command.py 70 passed, 7 skipped (one PCM→WAV test hit its own 10s
  ffmpeg subprocess timeout on a cold ffmpeg start in the first run; the whole file passes in
  21.54s on the warm re-run and the test is unrelated to this change).

Evidence boundary. The machine used here has no NVIDIA GPU, so no claim is made about
running Faster-Whisper/large-v3-turbo, CUDA init or any real STT model — the wedged call is
simulated with a blocking fake. What is proved locally is the gateway behaviour: with an STT
backend that never returns, main produces zero diagnostics, zero user-visible output and an
inbound turn that never completes; with this change the turn completes, the failure is logged at
ERROR with the stage that wedged, and the user still gets a reply. Confirming why the
reporter's local model hangs (and that 180s is the right cap for a 4GB GTX 1650 Ti cold load)
still needs their host; stt.timeout_seconds exists to tune that without a code change.

[alt-glitch]

This was generated by AI during triage.

Related: #74051 (open, bounds the same STT await with a 45s configurable deadline) and its salvage #78196 (STT/TTS reliability slice). Competing fixes for #105315; this PR additionally raises the transcription log lines to INFO/ERROR and routes raised exceptions into the local-STT fallback. Maintainer should pick one deadline mechanism.

[kvnloo]

Exact-head KEEP on #107738 head 769267c.

KEEP surfacing/logging automatic voice STT failures on the inbound path so Telegram voice notes no longer sit on "typing…" forever after Cached user voice at … with no INFO explanation (#105315). KEEP regression tests/gateway/test_voice_stt_silent_failure_105315.py.

CHECK: failure path still returns a user-visible error (not only a log line). CHECK: successful STT path unchanged. CHECK: config example docs in cli-config.yaml.example stay accurate. Author Finn763 not kvnloo. No competing PR.