fix(desktop): support voice barge-in during TTS - #67425
Conversation
Related to #59637: both seek interruptible Desktop TTS, while this implementation specifically keeps the recorder alive during playback and routes first speech into the normal turn-close path. |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for targeting a real Desktop voice-conversation gap: current main waits at apps/desktop/src/app/chat/composer/hooks/use-voice-conversation.ts:228 for TTS completion before it can re-enter the recorder loop.
Problems
apps/desktop/src/app/chat/composer/hooks/use-voice-conversation.ts:235-237stops playback and startshandleTurn(true), butspeak()then resumes and unconditionally setspendingStartRefandstatusto idle at lines 245-247. The idle loop at lines 378-384 can start listening while that same recorder is still stopping/transcribing.stopVoicePlayback()resolves the active playback promise (apps/desktop/src/lib/voice-playback.ts:40-50), andplaySpeechText()reports the cancellation asfalseat lines 134-141, but this result is currently discarded.- The diff adds no behavior test for interruption versus normal completion.
Suggested changes
- Preserve the
playSpeechText()completion result (or a barge-in ref) and bypass the normal re-arm path when playback was interrupted; lethandleTurn(true)own that transition. - Add tests for one-shot first-speech detection, no concurrent recorder/timeout during barge-in transcription, and normal TTS re-arming.
Automated hermes-sweeper review.
| idleSilenceMs: 12_000, | ||
| onSpeech: () => { | ||
| stopVoicePlayback() | ||
| void handleTurn(true) |
There was a problem hiding this comment.
stopVoicePlayback() resolves playSpeechText(), so this handleTurn(true) races the enclosing speak() finally block, which unconditionally marks the conversation idle and re-arms listening. Track the interruption and skip that normal-completion re-arm path so this turn exclusively owns recorder stop/transcription.
|
Closing as superseded by the merged voice playback/barge-in rework #69511 ( If anything in this area still misbehaves on current main, please open a fresh PR against the new stack. |
Summary
Root cause
The previous voice-conversation implementation waited for
playSpeechText()to finish before starting the next recorder. As a result, users could not barge in during TTS, and the next turn depended on a later UI interaction. The recorder now observes the first speech threshold during playback and interrupts the current TTS path.Test plan
git diff --checkpassedapps/desktop/node_modulesdoes not contain executabletsc/eslint; no dependency installation was performed