fix(desktop): isolate voice turn lifetimes

[kvnloo]

Summary

• isolate Desktop continuous-voice work by capture/turn lifetime so stale async completion cannot submit, stop, or re-arm a superseded turn
• keep capture active while the assistant is busy, then use sustained-speech barge-in with adaptive ambient thresholds instead of a fixed instantaneous level
• make mute, stop, unmount, supersession, playback completion, and React StrictMode replay deterministically invalidate or restore the right lifetime

This is the current-main Desktop safety slice of #94462. It ports the relevant behavior from the zer0-voice donor architecture into Hermes' existing Desktop hooks; it does not import donor code, add a runtime dependency, or introduce a second voice control plane. #77111 is architecture context only.

Problem

The existing Desktop path could let late getUserMedia, transcription, playback, or busy-state completion act on a turn that had already been muted, stopped, unmounted, or superseded. Listening also stopped during generation/playback, preventing natural barge-in, while a fixed input threshold was brittle across microphones and rooms.

Approach

The change stays within seven Desktop files:

• monotonic capture and turn lifetimes reject stale asynchronous work
• cancellation checks cover every post-await boundary and invalidate pending submission/re-arm work
• recording continues through the busy phase so sustained speech can interrupt playback and the active request
• a bounded ambient noise floor produces adaptive start/continue thresholds
• monitor setup/teardown is idempotent and React StrictMode effect replay restores mounted ownership without allowing real-unmount resurrection

Hermes remains authoritative for prompt submission, cancellation, durable history, tools, approvals, and actions. The acoustic monitor consumes local amplitude samples only; it adds no network path, transcript sink, credential path, or server-side tool authority.

Current-main overlap and scope

This branch is based on current upstream main at f751a8c5467c41500e505d90cb0eb8b70929080f. A fresh issue/PR sweep found adjacent voice work (including provider transcription, silence-duration, playback, and re-arm changes), but no open PR owning this exact lifetime + adaptive-barge behavior. The diff is Desktop-only: 7 files, 461 insertions, 30 deletions.

Verification

The regression suite was exercised RED/GREEN, including sabotage runs that restored the old lifetime behavior and confirmed the new tests fail before passing with the fix.

• focused Desktop Vitest: 4 files, 22/22 tests passed
  • StrictMode effect replay and real-unmount cleanup
  • late getUserMedia cleanup
  • end/mute/unmount/supersede after busy settlement
  • cancellation and stale-turn invalidation
  • barge-in, stop, monitor idempotency, playback re-arm, and adaptive acoustics
• Desktop typecheck passed
• targeted ESLint with --max-warnings=0 passed
• targeted Prettier check passed
• git diff --check passed

The repository's Python wrapper tests could not run in the review worktree because no pytest-capable virtualenv was available; those tests do not execute the changed Desktop TypeScript paths.

Risk and rollback

The main risk is browser media timing across effect replay and teardown. Tests cover stale completion, StrictMode replay, and each cancellation boundary. Rollback is the two commits in this PR; there is no schema, migration, config, or backend dependency.

Attribution and disclosure

Behavior and architecture were selectively reimplemented from kvnloo/zer0-voice, authored by Kevin Loo (kvnloo), behind Hermes-native Desktop contracts. No zer0-voice source or runtime dependency was copied into this repository.

This contribution was developed and reviewed with AI assistance from Hermes Agent; Kevin Loo directed the work, supplied the donor architecture, and is the human contributor/submitter.

Part of #94462.

[kvnloo]

Voice stack (from #94462): merge order — #95193 browser base → #95191 Android STT → #95180 desktop lifetimes → #95147 realtime provider seam. Review/merge in that order; later PRs depend on earlier ones landing first.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

This PR implements rigorous lifetime isolation for voice turns in the Desktop app, addressing a class of bugs where stale async resolutions (from getUserMedia, STT, or settle waits) could mutate shared state after the user ended, muted, or superseded a turn. The lifetimeRef counter pattern — incremented on every turn-invalidating action and checked at every await boundary — is the correct React pattern for this. The test coverage is exceptional: the parametrized ['end', 'mute', 'supersede', 'unmount'] cancellation matrix and the late-STT-rejection test pin the exact invariants the implementation protects.

A few observations:

1. AdaptiveAcousticThreshold EMA adaptation rate (voice-acoustics.ts:25): The noise floor uses floor * 0.92 + level * 0.08 — an 8% weight EMA that adapts slowly. With 40 quiet samples (as in the test), the floor rises gradually. In practice at 60fps rAF cadence, 40 samples ≈ 0.67s of quiet observation. If a user speaks briefly then pauses, the floor could still be adapting from the pre-speech baseline. Consider whether the adaptation should pause during speech (the observeQuiet guard if (level >= this.startThreshold) return handles this, but the caller must also avoid calling it during speech — the use-mic-recorder.ts code does this correctly with if (!heardSpeechRef.current && normalized < speechThreshold)).

2. Separate start/end thresholds (voice-acoustics.ts:17-22): startThreshold = floor + 0.045 and endThreshold = floor + 0.02. The lower end threshold prevents premature silence cutoffs during intra-word pauses. The use-mic-recorder.ts change to check normalized < endThreshold (instead of normalized < speechThreshold) for silence detection is the right application of this. However, if ambient noise rises between the start and end of an utterance, the end threshold could be lower than the current ambient level, making silence detection impossible. The EMA's slow adaptation mitigates this, but it's worth noting.

3. Unmount useEffect with handle dependency suppression (use-voice-conversation.ts:802-817): The eslint-disable react-hooks/exhaustive-deps for handle is justified — handle is recreated every render, so depending on it would tear down the active turn. The comment explains this well. However, the cleanup calls handle.cancel() which captures the handle from the first render (when the effect ran). If handle.cancel() behavior changed between renders, the unmount cleanup would use the stale handle. In practice, cancel is a stable identity from useMicRecorder's useCallback, so this is safe — but worth a comment noting that handle.cancel must remain stable across renders.

The voice-barge-in.ts refactor to use AdaptiveAcousticThreshold instead of manual floor sampling (with the median-of-sorted-samples approach) is cleaner and shares the same adaptation logic between capture and barge-in paths.

[kvnloo]

Comment assumes head bee79171eacf6eb810a208ec8d3154a8891bb2e0; void if moved.

KEEP — Desktop voice turn lifetimes alone (#94462 surface)

• Capture/turn lifetime isolation + sustained-speech barge-in + adaptive ambient thresholds is the right Desktop safety slice: stale async completion must not submit/stop/re-arm a superseded turn.
• Scope boundary in the PR body is correct: Desktop hooks only; no donor import; no second voice control plane; [RFC] RealtimeVoiceProvider ABC — four competing duplex-voice PRs need an interface, not a merge queue #77111 is context, not this vehicle.
• Prefer this as the single Desktop lifetimes fix; do not fold it into web (feat(web): add mobile browser voice conversation #95193/feat(web): add verified Android STT bridge #95191) or duplex ABC (feat(voice): add realtime provider session contract #95147).

CHECK

• Still draft; GitHub reports MERGEABLE but mergeStateStatus: BLOCKED and empty live status rollup — rebase onto current main, undraft, re-run focused Desktop Vitest (mic-recorder / voice-conversation / voice-acoustics / barge-in) + typecheck before merge desk.
• Body’s older “merge order” chaining web → desktop → duplex is stale vs current consolidation: Rank1 = feat(voice): add realtime provider session contract #95147 duplex; Rank2 = feat(web): add mobile browser voice conversation #95193 web; this PR stays Rank3 alone under Discord voice mode has extreme latency or poor STT accuracy #94462.
• Do not fold Rank1 duplex (feat(voice): add realtime provider session contract #95147/feat(tui): add native realtime voice sessions #101034), Rank2 web (feat(web): add mobile browser voice conversation #95193/feat(web): add verified Android STT bridge #95191), or cascaded TTS (feat(tts): add Groq Orpheus as a first-class TTS provider #98105) into this PR.
• Wrong-face: open Desktop/live-voice neighbors that touch the same hooks (e.g. feat(desktop): add ElevenLabs realtime voice transcription #94113 ElevenLabs scribe, feat(voice): add Codex OAuth STT, TTS, and live voice #106640 Codex OAuth STT/TTS/live) should adapt to these lifetime contracts or rebase after this merge tip — not compete as a second lifetimes vehicle or absorb this fix.

One-click close / ordering

• Close competing Desktop turn-lifetime / adaptive-barge duplicates only after this lands and they are true duplicates; otherwise rebase them onto this tip.
• Do not close Discord voice mode has extreme latency or poor STT accuracy #94462 when this merges — umbrella stays open for Discord latency/STT and other surfaces.
• Do not close or absorb feat(voice): add realtime provider session contract #95147 / feat(web): add mobile browser voice conversation #95193 / feat(tts): add Groq Orpheus as a first-class TTS provider #98105 into this PR.

Landing-evidence

• When merged, please reply with the merge commit SHA on main (not “completed”).