Skip to content

fix(desktop): support voice barge-in during TTS - #67425

Closed
PrCu wants to merge 1 commit into
NousResearch:mainfrom
PrCu:fix/desktop-voice-conversation-lifecycle
Closed

fix(desktop): support voice barge-in during TTS#67425
PrCu wants to merge 1 commit into
NousResearch:mainfrom
PrCu:fix/desktop-voice-conversation-lifecycle

Conversation

@PrCu

@PrCu PrCu commented Jul 19, 2026

Copy link
Copy Markdown

Summary

  • expose first-speech detection from the microphone recorder
  • keep a microphone recorder active while voice-conversation TTS is playing
  • stop TTS and close the current turn when the user starts speaking

Root cause

The previous voice-conversation implementation waited for playSpeechText() to finish before starting the next recorder. As a result, users could not barge in during TTS, and the next turn depended on a later UI interaction. The recorder now observes the first speech threshold during playback and interrupts the current TTS path.

Test plan

  • git diff --check passed
  • source-level lifecycle review completed
  • full Desktop TypeScript/lint tests are blocked in this checkout because apps/desktop/node_modules does not contain executable tsc/eslint; no dependency installation was performed
  • requires Desktop manual verification for TTS interruption, consecutive turns, and microphone cleanup

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/desktop Electron desktop app (apps/desktop/*) tool/tts Text-to-speech and transcription labels Jul 19, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #59637: both seek interruptible Desktop TTS, while this implementation specifically keeps the recorder alive during playback and routes first speech into the normal turn-close path.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for targeting a real Desktop voice-conversation gap: current main waits at apps/desktop/src/app/chat/composer/hooks/use-voice-conversation.ts:228 for TTS completion before it can re-enter the recorder loop.

Problems

  • apps/desktop/src/app/chat/composer/hooks/use-voice-conversation.ts:235-237 stops playback and starts handleTurn(true), but speak() then resumes and unconditionally sets pendingStartRef and status to idle at lines 245-247. The idle loop at lines 378-384 can start listening while that same recorder is still stopping/transcribing. stopVoicePlayback() resolves the active playback promise (apps/desktop/src/lib/voice-playback.ts:40-50), and playSpeechText() reports the cancellation as false at lines 134-141, but this result is currently discarded.
  • The diff adds no behavior test for interruption versus normal completion.

Suggested changes

  • Preserve the playSpeechText() completion result (or a barge-in ref) and bypass the normal re-arm path when playback was interrupted; let handleTurn(true) own that transition.
  • Add tests for one-shot first-speech detection, no concurrent recorder/timeout during barge-in transcription, and normal TTS re-arming.

Automated hermes-sweeper review.

idleSilenceMs: 12_000,
onSpeech: () => {
stopVoicePlayback()
void handleTurn(true)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stopVoicePlayback() resolves playSpeechText(), so this handleTurn(true) races the enclosing speak() finally block, which unconditionally marks the conversation idle and re-arms listening. Track the interruption and skip that normal-completion re-arm path so this turn exclusively owns recorder stop/transcription.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 19, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Closing as superseded by the merged voice playback/barge-in rework #69511 (93e9061f15, merged 2026-07-22) plus #69602: the desktop voice stack was rewritten around speech-stream sessions with barge-in capture and native resume-listening, and the code this PR patches no longer exists. Barge-in during TTS shipped as part of #69511 — thanks for pushing this capability.

If anything in this area still misbehaves on current main, please open a fresh PR against the new stack.

@teknium1 teknium1 closed this Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/desktop Electron desktop app (apps/desktop/*) P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages tool/tts Text-to-speech and transcription type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants