Skip to content

fix(desktop): dedupe auto-spoken replies by content fingerprint - #75649

Closed
lilShawtty-byte wants to merge 1 commit into
NousResearch:mainfrom
lilShawtty-byte:fix/desktop-auto-speak-fingerprint-dedupe
Closed

fix(desktop): dedupe auto-spoken replies by content fingerprint#75649
lilShawtty-byte wants to merge 1 commit into
NousResearch:mainfrom
lilShawtty-byte:fix/desktop-auto-speak-fingerprint-dedupe

Conversation

@lilShawtty-byte

Copy link
Copy Markdown
Contributor

Problem

Auto-speak (voice.auto_tts) sometimes reads the same reply aloud twice.

Root cause: spoken-reply dedupe tracked the last spoken message by its id, but message ids are NOT stable across the post-turn hydrate:

  1. During streaming, the bubble carries a runtime id (assistant-stream-<ts>-<seq>)
  2. On message.complete, completeAssistantMessage settles that bubble and auto-speak reads it (first playback)
  3. The same handler then calls hydrateFromStoredSessiontoChatMessages rebuilds the list with persisted ids (<timestamp>-<index>-<role>)
  4. `` emits again → pendingReply sees a 'new' id ≠ `lastSpokenIdRef` → plays the same text a second time

Both existing dedupe layers fail: lastSpokenIdRef (id rewritten by hydrate) and ownsAmbientCue (1s window, the two triggers are a network round-trip apart).

Fix

  • lib/chat-messages.ts: add speechFingerprint() (trimmed text) and latestSpokenReply() — an auto-speak selector that dedupes on content, which survives the hydrate id rewrite, and returns the fingerprint as the ambient-cue key.
  • use-composer-voice.ts: auto-speak path (pendingResponse / consumePendingResponse) uses the fingerprint ref. The voice-conversation path keeps id semantics — its live speech session binds to a message id for the turn and depends on it staying stable while streaming.

Verification

  • vitest run src/lib/chat-messages.test.ts → 57/57 (5 new tests incl. the streaming→hydrated id-rewrite case)
  • vitest run src/app/chat/composer/hooks/use-voice-conversation*.test.tsx → 11/11 (voice-conversation path unaffected)
  • tsc --noEmit: no new errors (11 pre-existing errors in user-edit-composer.tsx, unrelated @assistant-ui version drift)

Spoken-reply dedupe tracked the last spoken message by its id, but message
ids are not stable across the post-turn hydrate: during streaming the bubble
carries a runtime id (assistant-stream-<ts>-<seq>), and hydrateFromStoredSession
rebuilds it with a persisted id (<timestamp>-<index>-<role>). The id-based
check then misses the same reply and reads it aloud a second time.

Add speechFingerprint()/latestSpokenReply() to lib/chat-messages.ts and switch
the auto-speak path (pendingResponse/consumePendingResponse in
use-composer-voice.ts) to fingerprint-based dedupe. The voice-conversation path
keeps id semantics: its live speech session binds to a message id for the turn
and depends on it staying stable while streaming.

Adds unit tests covering the streaming→hydrated id rewrite case.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/desktop Electron desktop app (apps/desktop/*) tool/tts Text-to-speech and transcription labels Jul 31, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the runtime-to-hydrated id transition; current main does have the id-only selector at apps/desktop/src/app/chat/composer/hooks/use-composer-voice.ts:75-109, and hydration replaces the message list at apps/desktop/src/app/contrib/wiring.tsx:321-327.

Problems

  • apps/desktop/src/lib/chat-messages.ts:211 makes trimmed text the entire persistent spoken identity. A later distinct reply with the same content as the preceding reply is therefore returned as null and never read aloud. useAutoSpeakReplies is intended to read each completed assistant turn (apps/desktop/src/app/chat/composer/hooks/use-auto-speak-replies.ts:29-34).

Suggested changes

  • Rework the hydration dedupe to retain or reconcile a per-turn identity rather than using content alone, and add a regression case for two distinct same-text replies. The current new-reply test uses different text, so it does not exercise this collision.

Automated hermes-sweeper review.


const fingerprint = text

if (fingerprint === lastSpokenFingerprint) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This permanently treats equal text as the same reply. A later, distinct turn returning the same text (for example, two separate Done. replies) will be suppressed rather than spoken. Please retain or reconcile a per-turn identity across hydration and add a same-text, distinct-turn regression case.

@OutThisLife

Copy link
Copy Markdown
Collaborator

Superseded by #89815.

That salvage keeps the stream-id rewrite fix, rebased onto current main, and keys spoken state on the assistant-turn ordinal instead of reply text so two Done. turns in one session still both speak. Manual Read Aloud marks the same book after a successful play.

You're credited via Co-authored-by. Thanks for tracing the live-to-durable id swap.

euntaek-hong pushed a commit to wrongbutworks/hermes-agent that referenced this pull request Aug 19, 2026
Supersedes NousResearch#75649, NousResearch#86637, NousResearch#87672, NousResearch#88642.

Fixes NousResearch#86601
Fixes NousResearch#87652
Fixes NousResearch#87823

Co-authored-by: Charmmy <lilShawtty@qq.com>
Co-authored-by: chelsealong <chelsealong@126.com>
Co-authored-by: Olympusbuildz <Olympus.roots@outlook.com>
Co-authored-by: Ricardo Mendes <ricardo.mendes@maiolabs.ai>
lisajlau pushed a commit to lisajlau/hermes-agent that referenced this pull request Aug 20, 2026
Supersedes NousResearch#75649, NousResearch#86637, NousResearch#87672, NousResearch#88642.

Fixes NousResearch#86601
Fixes NousResearch#87652
Fixes NousResearch#87823

Co-authored-by: Charmmy <lilShawtty@qq.com>
Co-authored-by: chelsealong <chelsealong@126.com>
Co-authored-by: Olympusbuildz <Olympus.roots@outlook.com>
Co-authored-by: Ricardo Mendes <ricardo.mendes@maiolabs.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/desktop Electron desktop app (apps/desktop/*) P3 Low — cosmetic, nice to have sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/tts Text-to-speech and transcription type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants