fix(voice): full-duplex turn listener — interrupt by voice during generation AND playback

[teknium1]

fix(voice): full-duplex agent-turn listener — interrupt by voice during generation AND playback

Summary

Teknium's live test after #74000: voice interruption doesn't work at all — (a) not while the LLM is generating (before any audio), (b) not while TTS audio is playing.

Root cause is a half-duplex design, confirmed on main:

1. No listener during generation. After a voice utterance is submitted, the continuous recorder is not re-armed until the whole response + TTS finish (feedback-loop guard), and the barge monitor only spawns when TTS playback starts (cli.py streaming + whole-file paths; gateway _tts_stream_begin). During LLM generation — often the longest part of the turn — there is literally no microphone open. That's (a).
2. The playback monitor calibrates against its own speaker bleed. listen_for_speech builds its noise floor from the first ~800ms of input while TTS is already blasting, then multiplies the (bleed-inflated) floor by 8× and demands 1 full second of strictly consecutive 30ms blocks above trigger — any intra-word energy dip resets the counter. On real hardware the trigger is effectively unreachable for normal speech. The 2.0s grace additionally swallows early interjections. That's (b).

The fix — ONE full-duplex agent-turn listener

tools/voice_mode.full_duplex_listen() runs for the entire agent turn: armed at utterance-submit, disarmed when the turn is fully done (response + TTS finished). Wired identically in the CLI (cli.py _voice_full_duplex_listener) and the gateway (tui_gateway/server.py _full_duplex_listener / _arm_full_duplex_listener), so TUI (and desktop backend-mic mode) inherit it.

Phase	Mic	On user speech
Generation (LLM working, no TTS audio)	live	Interrupt the in-flight turn via the SAME seam as typed/Ctrl+C interrupt (agent.interrupt(); gateway: every running session's agent), cut any pending TTS pipeline so the stale reply never plays, capture the interjection (pre-roll — first syllable kept), submit it as the next message
Playback (TTS audio flowing)	live (same listener instance — no re-arm race at the transition)	Cut TTS (streaming pipeline stop + fallback speak stop events + file player), capture + submit
Stop phrase ("stop")	both phases	Mid-generation: interrupt the turn and end the voice chat ("stop everything"). Mid-playback: cut speech and end the chat. Explicit stop signal, never sent to the agent

Deafness fixes (detection math)

• Pre-playback calibration: the quiet-room noise floor is established at turn start — before any TTS exists — and held through playback. The floor is never recalculated against active speaker bleed.
• Phase-aware trigger: generation phase = quiet_floor × voice.barge_in_threshold_multiplier; playback phase additionally clamps the trigger up to a 1500-RMS minimum (bleed through air is typically well under this; direct speech is 3000–8000 RMS) and down to a 4000-RMS ceiling so speech is always reachable.
• Windowed majority (≥80% of a 300ms window above trigger) replaces the strictly-consecutive counter, so intra-word dips no longer reset detection.
• Grace applies to playback ONSET only (suppresses the start transient), not to opening the mic — and only when playback starts after a ≥1s gap so inter-sentence flapping can't chain grace windows.

Config keys (new defaults)

Key	Old	New	Meaning
voice.barge_in_threshold_multiplier	— (hardcoded 8×)	3.0	Speech trigger = quiet-room floor × this. 8× only made sense against a bleed-inflated floor; against a genuine quiet floor (50–300 RMS) the synthetic-frame tests show 3× separates speech from ambient while staying reachable (300×3=900 vs 3000+ RMS speech)
voice.barge_in_grace_seconds	2.0	0.5	Trip suppression right after playback starts (onset transient). The mic itself is live the whole turn
voice.barge_in	true	true	unchanged master switch

Diagnostics for live tuning

Every decision point logs at debug level and — with HERMES_VOICE_DEBUG=1 — mirrors to stderr: calibrated floor value (+pct90 + block count), per-block RMS whenever above 50% of trigger (with floor/trigger/window state/phase), TRIPPED with full reason, grace suppressions with remaining blocks, playback-start grace arming.

Live-test script (Teknium)

Setup: /voice on + continuous mode, speakers at normal volume. Optionally HERMES_VOICE_DEBUG=1 hermes to watch the VAD stream.

1. Interject during generation: ask something that thinks/tools for a while ("search the web for X and summarize five sources"). ~3–5s in, before any audio, say "actually stop, just tell me a joke". Expect: turn interrupts (same as typing over it), your interjection runs as the next turn, the stale reply never speaks.
2. Interrupt during playback: ask for something long ("explain TCP in ten sentences"). Once it's speaking, talk over it at normal volume mid-reply. Expect: audio cuts within ~0.5s of you starting to speak (0.5s grace applies only at the very start of playback), your words are transcribed from the first syllable and submitted.
3. Stop phrase, generation phase: start another long generation, say exactly "stop" before audio starts. Expect: turn interrupted AND voice chat ends ("stop everything").
4. Stop phrase, playback phase: say "stop" over playing audio. Expect: audio cuts, voice chat ends.
5. False-positive check: let a long reply play at loud speaker volume without talking. Expect: no self-trigger (playback trigger floor is 1500 RMS + held quiet baseline). If it self-triggers, raise voice.barge_in_threshold_multiplier; if it misses your speech, check the HERMES_VOICE_DEBUG stream — the per-block lines show exactly where your RMS lands vs the trigger.

Diagnosis confirmations (which suspects were real)

• ✅ Half-duplex gap: confirmed — no mic between submit and playback start on all CLI/gateway paths.
• ✅ Calibration-against-bleed: confirmed as the primary playback deafness — floor built from active playback, ×8, 4000-ceiling only partially rescued it.
• ✅ Strict consecutive-blocks requirement (1000ms) — real speech has intra-word dips; counter reset constantly.
• ✅ 2.0s grace swallowing early interjections — real, default now 0.5s.
• ➖ sounddevice full-duplex contention: not implicated on Linux/Pulse (output stream + input stream coexist; the old monitor did open a stream during playback and got frames, they just never tripped). Kept in mind for Windows follow-up if live testing shows dead input during playback.
• ➖ Monitor thread races: the private-stop-event / tts_done early-exit races are mooted by the single-listener design (one owner, turn-scoped lifecycle, double-arm refused).

Notes / decisions

• _tts_stream_barge_in_monitor kept as a shim that arms the new listener (stray callers get the new behavior).
• Gateway _speak_text_with_barge registers its (stop, done) pair in _fd_speak_pipelines; the listener cuts fallback speaks and keeps listening while they're pending.
• Desktop: the Electron renderer owns its own mic (voice-barge-in.ts, getUserMedia) — it does NOT use the backend voice.* mic path, so it does not inherit this fix; renderer-side generation-phase interjection is a follow-up. Desktop connected to a gateway with HERMES_VOICE=1 (backend-mic mode) inherits it.
• Docs updated (website/docs/user-guide/features/voice-mode.md barge-in section).

Validation

• New TestFullDuplexListen synthetic-RMS suite (10 tests): speech-over-bleed trips / bleed alone doesn't / quiet floor held through 100+ blocks of loud bleed / grace suppresses onset but not later speech / multiplier math (3× trips at 1200 RMS where 8× doesn't) / windowed-majority tolerates dips / brief spike rejected.
• CLI TestVoiceFullDuplexListener (7 tests): generation-phase interrupt seam invocation, playback cut without agent interrupt, lifecycle spans generation→playback→done, double-arm refused, config forwarding, barge_in=false never opens mic, stop-phrase-mid-generation interrupts + ends chat. Generation-interrupt test sabotage-verified (removing the interrupt call fails the test).
• Gateway: generation-phase interrupt + stop-phrase-mid-generation tests; existing playback barge tests updated to the new listener seam.
• 1020 targeted tests green (voice_mode, voice_cli_integration, voice_wrapper, tui_gateway_server ×493, tts_streaming, hermes_cli config); ruff clean; Windows-footgun checker clean. Wider tests/tools failures verified pre-existing on pristine origin/main.

Live verification (virtual-audio harness, this box)

Real audio through a PipeWire virtual mic (pw-loopback + injected WAVs), real full_duplex_listen with real sounddevice/PortAudio:

Scenario	Result
GENERATION phase: quiet room (floor=200), speech injected	TRIPPED 8/8 windows @ RMS 3.2k-8.3k vs trigger 600 → interrupted → 4.56s utterance captured (RMS 2254)
PLAYBACK phase: 8%-volume speaker bleed only	NO trip (bleed under the 1500-RMS playback clamp) — false-positive rejection verified
PLAYBACK phase: real speech over bleed	TRIPPED 8/8 windows @ RMS 3.2k-8.1k vs trigger 1500 → 4.53s capture

Debug traces (HERMES_VOICE_DEBUG=1) confirm: quiet-floor calibrated pre-playback and held, 480ms grace applied at playback onset, windowed-majority detection.

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 4a025db}

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

1 visual diff.

_{inline evidence is publishing...}