feat(voice): chat UX polish — busy-aware silence, stop hint, thinking sounds, barge-in fix

[teknium1]

Voice chat UX polish — four live-tested fixes (one branch)

Branch: feat/voice-chat-ux-polish @ c407f90 (5 commits off current main)

Summary

Four voice-chat UX improvements from Teknium's live testing of the new voice
stack, plus a salvage of PR #71083 (@beardedeagle, authorship preserved):

1. Silence never ends the chat while the agent is busy or TTS is playing.
2. "Say "stop" to end the voice chat." notice on voice-mode start — CLI,
   TUI, and desktop, i18n'd, sourced from voice.stop_phrases.
3. Ambient "thinking" sound — calm synthesized bubble blips while the
   agent works with no audio flowing (config off-switch, volume-respecting,
   macOS-TCC-safe).
4. Spoken barge-in fixed on every TTS playback path — CLI whole-file
   fallback, gateway fallback speak, voice.tts RPC; plus fix(voice): rolling-window VAD to prevent false barge-in during TTS #71083's
   rolling-window VAD against false barges during streaming playback.

---

Item 1 — silent cycles never end the chat while the agent works

Root cause. Both no-speech counters (the shared continuous loop in
hermes_cli/voice.py _CONTINUOUS_NO_SPEECH_LIMIT=3, and cli.py's
_no_speech_count >= 3) counted every silent capture cycle unconditionally.
During a multi-minute agent turn or while TTS plays, the user is CORRECTLY
silent — three cycles later the voice chat shut off under them.

Fix.

• hermes_cli/voice.py: new set_voice_busy_probe(probe) seam +
  _voice_activity_held() — held while TTS plays (existing _tts_playing
  Event) or the registered probe says the agent is mid-turn. Both strike
  paths (continuous-loop and force-transcribe single-shot) skip counting
  while held; they keep listening. Fail-open: a broken probe counts cycles
  as before, so the chat can never become immortal.
• tui_gateway/server.py: registers _any_session_running() (any session's
  running flag — voice is process-global) as the probe on voice.record
  start.
• cli.py: classic-CLI strike path skips counting while _agent_running
  or _voice_tts_done is unset.
• Stop phrase and barge-in still work during the hold (dedicated tests).

Live-test. Start a voice chat, ask for something that runs tools for
2+ minutes, stay silent → chat stays alive; after the reply, stay silent for
3 cycles → chat ends as before. Say "stop" mid-turn → chat ends immediately.

Item 2 — stop-phrase notice on voice-mode start

One owner for the wording: voice_stop_hint() in tools/voice_mode.py —
Say "<first stop phrase>" to end the voice chat.; returns "" when
stop_phrases: [] (feature disabled → no hint anywhere).

• CLI: printed in /voice on output (dim, style-matched).
• TUI: voice.toggle action=on response now carries stop_hint; the Ink
  client renders it in the "Voice mode enabled" block. Older gateways omit
  the field — no hint, no crash.
• Desktop: the renderer voice loop never touches tools/voice_mode.py,
  so the phrase is read from config (voice.stop_phrases →
  $voiceStopPhrase store, seeded in use-hermes-config) and shown as an
  info toast when a voice conversation starts.

i18n coverage: desktop en, ja, zh, zh-hant, ar
(notifications.voice.sayStopToEnd(phrase)). ui-tui has no i18n catalog
(all transcript strings are English) — hint follows suit.

Live-test. /voice on (CLI/TUI) and desktop voice button → hint shows;
set voice.stop_phrases: ["goodbye hermes"] → hint shows the custom phrase;
set [] → no hint.

Item 3 — ambient "thinking" sound

New: while the agent processes a voice-mode turn and no speech audio is
flowing, quiet repeating "blub blub" bubbles play so the user knows the agent
is alive.

• Synthesis (tools/voice_mode.py): pure numpy — two alternating low
  pitches (G4/E4) with a downward pitch glide, smooth attack + exponential
  decay (no clicks), ~0.8–1.2 s randomized spacing. No binary assets.
• Volume: voice.beep_volume × 0.5 (deliberately quieter than beeps).
• Config: voice.thinking_sound: true|false (default true) — added
  to DEFAULT_CONFIG; desktop reads the same key.
• macOS TCC: _sounddevice_output_allowed() gate → the loop exits
  silently on macOS (per-second afplay churn judged worse than silence;
  documented in the code).
• Lifecycle: daemon thread with a stop Event.
  • CLI: started when a voice-mode turn begins in chat(); per-blip gate
    skips while TTS speaks / mic records / barge capture owns the mic;
    stopped in the finally on every exit path.
  • Gateway: same seams around _run_prompt_submit (voice mode on), gated
    on the new is_audio_output_active() ref-count + continuous capture.
  • New mark_audio_output_active() ref-count brackets play_audio_file
    and the streaming OutputStream sentence writes, so "audio is flowing"
    is accurate rather than inferred from per-turn events.
• Desktop decision: the desktop voice conversation renders audio in the
  RENDERER (WebRTC mic + WebAudio/audio-element playback), not through the
  backend — so it gets a matching renderer implementation
  (src/lib/thinking-sound.ts, WebAudio oscillators, same envelope/pitches)
  driven by conversation status === 'thinking'; honors
  voice.thinking_sound and the shared sound-mute toggle; stops instantly
  on speaking/listening/end. No voice.thinking gateway event needed.

Live-test. Voice chat, ask a long question → soft bubbles while it
thinks; they stop the instant speech starts; voice.thinking_sound: false
silences them; on macOS CLI they are (deliberately) absent.

Item 4 — spoken barge-in

Premise check on live main (what actually existed vs. what was broken):

Path	Barge monitor on main?	Verdict
CLI streaming pipeline (chat() use_streaming_tts)	YES (_voice_barge_in_monitor)	worked
CLI whole-file fallback (_voice_speak_response_async — fires whenever streaming can't start: sounddevice missing, probe fails)	NO	broken — nothing armed; speech uninterruptible
Gateway streaming pipeline (_tts_stream_begin)	YES (#73933)	worked
Gateway fallback speak (tts_queue is None → speak_text) + voice.tts RPC	NO	broken — speak_text's internal streaming dispatch even created a PRIVATE stop event nothing could reach
Desktop renderer conversation (voice-barge-in.ts)	YES (live-stream + fallback both wired via openBargeMonitor)	verified, no change
False barges during TTS (stale one-shot VAD calibration)	broken class	fixed by salvaged #71083

This composition explains "interruption does not work in practice": whenever
the streaming pipeline didn't engage (very common — sounddevice missing or
provider probe failing routes to fallback), speech was un-barge-able; and
when streaming DID engage, the old one-shot VAD floor caused false trips or
mis-calibration.

Fixes.

• Salvaged fix(voice): rolling-window VAD to prevent false barge-in during TTS #71083 (@beardedeagle, cherry-picked with authorship):
  rolling-window VAD (90th-percentile deque floor, 8× multiplier, 4000-RMS
  trigger ceiling, min-floor clamp), voice.barge_in_grace_seconds
  (default 2.0 s, now documented in DEFAULT_CONFIG) before the mic opens,
  duplicate-render suppression, TTS-CUT diagnostics. Fixture follow-up: the
  fake tools.tts_tool module needed _load_tts_config (main's
  tts_streaming now imports it).
• CLI fallback: _voice_speak_response_async arms
  _voice_barge_in_monitor in continuous voice mode (stop_playback()
  kills the file player, so the same machinery covers this path).
• Gateway: new _speak_text_with_barge() — runs
  _tts_stream_barge_in_monitor beside the speak thread; used by both the
  fallback whole-reply path and the voice.tts RPC.
  hermes_cli.voice.speak_text / _speak_text_streaming accept an external
  stop_event so a barge cuts an internal streaming dispatch too
  (mark_speech_interrupted latch preserved; stop-phrase check inherited
  from the shared monitor — merged fix(voice): saying or typing 'stop' ends the voice chat on every surface #73933 behavior).

Live-test. CLI /voice on + TTS with and WITHOUT a streaming provider
configured → talk over the reply → playback cuts, your interruption is
transcribed and submitted; say "stop" over the reply → chat ends. Desktop:
same over both streamed and fallback speech.

Validation

Check	Result
tests/hermes_cli/test_voice_wrapper.py (59)	✅
tests/tools/test_voice_mode.py (129, incl. #71083's 4 new VAD tests)	✅
tests/tools/test_voice_stop_phrase.py (+3 hint tests)	✅
tests/tools/test_voice_thinking_sound.py (13, new)	✅
tests/tools/test_voice_cli_integration.py (+3 fallback-barge tests)	✅
tests/tools/test_tts_streaming.py + _e2e	✅
tests/test_tui_gateway_server.py (491, incl. +1 stop_hint, +2 barge)	✅ (full file)
Combined targeted run	864 passed
Sabotage-verified	item 1 (hold removed → 3 tests fail), item 2 (hardcoded phrase → custom-phrase test fails), item 3 (TCC gate removed → macOS test fails), item 4 (monitor arming removed → CLI + gateway tests fail)
ruff (all touched py)	✅ clean
desktop npm run typecheck	✅
desktop vitest (voice suites + composer, 31 files / 223 tests)	✅
desktop eslint	0 errors (67 pre-existing warnings)
ui-tui typecheck + vitest (131 files / 1417 tests)	✅
Pre-push gate	rebased onto main, behind=0, only intended files in diff

Note: two full-tests/hermes_cli runs showed thousands of OSError errors —
traced to /tmp tmpfs filling up mid-run (No space left on device), not
code; scoped re-runs of every affected file pass.

Config keys added / documented

• voice.thinking_sound: true (new, default true) — ambient thinking sound
  off-switch.
• voice.barge_in_grace_seconds: 2.0 (introduced by fix(voice): rolling-window VAD to prevent false barge-in during TTS #71083's code, now
  documented in DEFAULT_CONFIG).

i18n coverage

• Desktop: sayStopToEnd added to en, ja, zh, zh-hant, ar
  (+ types.ts contract).
• ui-tui: no i18n catalog exists; hint is backend-provided English text like
  every other transcript system note.

Decisions / deferred

• Desktop thinking sound is renderer-side (WebAudio), not a
  voice.thinking gateway event — desktop voice audio renders entirely in
  the renderer, and the backend blips would play on the wrong machine for
  remote gateways. Matches the wake-sound/completion-sound pattern.
• Two stop matchers remain (backend is_voice_stop_phrase exact-match
  vs desktop voice-stop-word.ts wider list) — known consolidation target
  per the voice skill; NOT unified here (out of scope, the notice sources
  each surface's own config phrase).
• macOS thinking sound: silent skip (TCC gate). Routing per-blip audio
  through afplay would spawn a subprocess every second; judged worse than
  silence. Revisit if macOS users ask for it.
• fix(voice): rolling-window VAD to prevent false barge-in during TTS #71083: salvaged in full (all 5 files, clean cherry-pick, author
  beardedeagle). Original PR can be closed with credit once this merges.

Salvage credit

• PR fix(voice): rolling-window VAD to prevent false barge-in during TTS #71083 — @beardedeagle (fix(voice): rolling-window VAD to prevent false barge-in during TTS) — cherry-picked as commit e4b9d07d1c with
  authorship preserved.

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 7e22f78}

all good!