Skip to content

feat(voice): "Hey Hermes" wake word to start a hands-free session - #58539

Closed
omid3098 wants to merge 13 commits into
NousResearch:mainfrom
omid3098:bb/wake-word
Closed

feat(voice): "Hey Hermes" wake word to start a hands-free session#58539
omid3098 wants to merge 13 commits into
NousResearch:mainfrom
omid3098:bb/wake-word

Conversation

@omid3098

@omid3098 omid3098 commented Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Replacement for #53378, rebased onto current main while preserving Brooklyn Nicholson's contributor commits and authorship.

What this adds

  • Local wake-word detection for CLI, TUI, and desktop.
  • Hands-free voice capture after detection, with configurable fresh-session behavior.
  • OpenWakeWord and Porcupine providers through the existing lazy-dependency/config paths.

Review feedback addressed

  • surface: auto now makes surfaces eligible but grants a sticky lease to the first claimant.
  • Later clients cannot replace the owner's callback, pause its listener, stop it, or steal event delivery.
  • A stdlib advisory file lock prevents separate Hermes processes from competing for the microphone.
  • Ownership is released on explicit stop, websocket disconnect, detector startup/runtime failure, or process shutdown.
  • Voice capture temporarily pauses the detector and reliably rearms it after transcript, silence, stop, or client failure.
  • Desktop wake handling is ported to the current ContribWiring/composer architecture; the obsolete controller is not restored.

Verification

  • uv lock --check
  • Ruff and Python syntax checks on affected files
  • 349 passed: wake-word, TUI gateway, and websocket lifecycle tests
  • Desktop composer wake tests: 9 passed; affected ESLint checks pass
  • TUI gateway-event tests: 83 passed; TUI typecheck and affected ESLint checks pass
  • git diff --check origin/main...HEAD

The full desktop typecheck is currently blocked locally by unrelated installed @assistant-ui/react / react-streamdown export mismatches in existing files; the affected desktop files pass their targeted tests and lint checks.

@omid3098
omid3098 requested a review from a team July 4, 2026 22:39
@alt-glitch alt-glitch added type/feature New feature or request comp/cli CLI entry point, hermes_cli/, setup wizard comp/desktop Electron desktop app (apps/desktop/*) comp/tui Terminal UI (ui-tui/ + tui_gateway/) tool/tts Text-to-speech and transcription P3 Low — cosmetic, nice to have labels Jul 4, 2026
teknium1
teknium1 reviewed Jul 15, 2026
View reviewed changes

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the substantial cross-surface implementation. The feature is not present on current main, but the default ownership path needs correction before it is safe to salvage.

Problems

  • tools/wake_word.py:106 lets every surface pass surface: auto; both clients then call wake.start (ui-tui/src/app/createGatewayEventHandler.ts:322, apps/desktop/src/app/desktop-controller.tsx:1035). tui_gateway/server.py:12518-12524 retains only one event transport/callback target, so the later caller takes wake-event ownership. The process-local singleton also cannot prevent separate CLI/TUI processes from competing for the same microphone.
  • tests/tools/test_wake_word.py:30-35 encodes that permissive behavior and does not exercise gateway ownership or event routing.

Suggested changes

  • Make auto acquire one deterministic owner and reject or explicitly hand off later callers; release it on disconnect/stop. Use a machine-scoped lock if multiple processes are supported.
  • Add TUI/desktop competing-client and reconnect ownership tests.

Automated hermes-sweeper review.

Comment thread tools/wake_word.py
@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
OutThisLife and others added 13 commits July 15, 2026 22:41
@OutThisLife @omid3098
feat(voice): add "Hey Hermes" wake word to start a hands-free session
0f1d135 
Adds an opt-in, on-device hotword listener for the CLI. With
wake_word.enabled (or /wake on), Hermes listens in the background for a
wake phrase; on detection it starts a fresh session, captures one
utterance through the existing voice pipeline, and answers — the
"Hey Siri" pattern.

- tools/wake_word.py: provider-pluggable detector (openWakeWord, free
  local default; Porcupine, premium) over the shared 16 kHz sounddevice
  capture path. Background daemon thread with pause/resume so it yields
  the mic during a voice turn.
- CLI wiring: startup listener (off-thread), on-wake flow, an idle
  watchdog that resumes the detector after each turn, cleanup hook, and
  a /wake [on|off|status] command.
- config.yaml wake_word section; PORCUPINE_ACCESS_KEY as an optional
  secret. Engines lazy-install via the [wake] extra.
- Hands a transcript to the input queue exactly like voice mode, so no
  system-prompt/cache mutation. No new core model tool.
- Tests (mocked, no live audio/network) + feature docs.
@OutThisLife @omid3098
feat(voice): extend "Hey Hermes" wake word to TUI + desktop GUI
64b7535 
Makes the wake word a tri-surface feature with one configurable owner.

- wake_word.surface ("auto" | "cli" | "tui" | "gui") + shared
  wake_surface_enabled() gate consulted by every surface, so exactly one
  place owns the listener and the new session it opens.
- tui_gateway: wake.start/stop/pause/resume/status RPCs + a wake.detected
  event, sharing one server-side detector for both TUI and desktop. The
  detector yields the mic to voice.record (pause on capture start, resume
  on terminal) and to the desktop's browser mic (wake.pause/resume).
- TUI (Ink): arm wake.start on gateway.ready; on wake.detected open a
  fresh session and start voice capture.
- Desktop (Electron): arm wake.start on connect; on wake.detected open a
  fresh session.
- CLI now gates on wake_surface_enabled("cli"); /wake status shows surface.
- Tests for the surface gate; docs cover the surface knob + cross-surface.
@OutThisLife @omid3098
feat(desktop): full back-and-forth voice on "Hey Hermes" wake
0e595b9 
On wake, the desktop GUI now opens a fresh session AND starts the
browser voice conversation (continuous, with TTS), matching the CLI/TUI
hands-free flow instead of just opening a session.

- Add an explicit requestVoiceStart() intent to the composer bus
  (idempotent start; toggle could stop an active loop).
- Composer owns mic hand-off: pause the server-side wake detector while
  the browser voice loop is live, resume after (server no-ops when the
  wake word isn't armed) — via the $gateway store accessor.
- Controller fires startFreshSessionDraft() + requestVoiceStart() on
  wake.detected.
@OutThisLife @omid3098
chore(voice): log wake-word lifecycle at INFO for diagnosability
041d221 
The detector logged listen/detect/close at debug, invisible at the
default level. Promote listen-start, phrase-detected, stream-closed, and
the wake.start outcome (disabled / unavailable / listening) to INFO, and
log wake.detected emission, so a non-triggering setup is diagnosable from
gateway/gui.log without flipping global log levels.
@OutThisLife @omid3098
fix(desktop): handle wake.detected on the canonical event pipeline
fe076ed 
The GUI armed the detector (wake.start) and the gateway fired
wake.detected, but the desktop never reacted: detection was wired through
a side-registered gatewayRef.current.on('wake.detected', …) listener that
was instance/timing-fragile (and silently dead across reconnects/HMR),
even though the raw events were arriving on the socket.

Route wake.detected through handleGatewayEventWithWake — the same onEvent
pipeline every gateway socket already feeds via useGatewayBoot — and open
a fresh session + start back-and-forth voice there. Drop the separate
.on() listener; the open-effect now only arms wake.start.
@OutThisLife @omid3098
fix(desktop): deliver wake.detected over the websocket, not stdio
9ba8569 
write_json routes via the request-scoped transport ContextVar, but the
wake detector's callback runs on a background thread where that var is
unset — so wake.detected fell back to _stdio_transport and was dumped to
the backend's stdout (visible as raw [hermes] {...} frames in desktop
logs) instead of crossing the desktop/dashboard websocket. The TUI was
unaffected because it IS stdio.

Capture the arming request's transport at wake.start and bind it around
the emit in _wake_on_detect so the background thread routes to the right
peer. Re-armed on each wake.start, so reconnects pick up the new socket.
@OutThisLife @omid3098
fix(desktop): re-arm wake detector after a manual voice end
56b7361 
Ending a voice conversation manually left the wake detector paused for
good, so the wake word couldn't be used again. The composer paused the
detector on voice start but only resumed on the voiceConversationActive
-> false render; if ending voice tore the composer down first, that
render never landed and the resume was skipped.

Resume on unmount as well (latched on wakePausedRef so it fires exactly
once), and stop early-returning when the $gateway atom is momentarily
null. Add wake.pause/resume INFO logs for visibility.
@OutThisLife @omid3098
fix(desktop): start voice on wake via a latched store, not a window e…
5c34afe 
…vent

Wake opened a fresh session but voice didn't start: the start intent was
a fire-once window CustomEvent, and the fresh-session remount tore down /
recreated the composer's subscription, so the deferred dispatch landed in
the gap and was lost.

Replace it with a latched nanostore ($voiceConversationStartRequest +
takeVoiceConversationStart): the controller sets it on wake.detected, and
the composer claims it once on (re)mount when the gateway is open, waiting
out any transient `disabled`. Drop the now-unused composer voice-start
window event.
@OutThisLife @omid3098
fix(voice): stop wake re-fire loop and empty-transcript error spam
f05f424 
Two bugs surfaced by the desktop wake conversation:

1. Runaway loop: wake -> voice -> resume -> wake fired again within
   ~200ms. openWakeWord keeps its rolling feature buffer across
   pause/resume, so on resume it immediately re-scored the "hey jarvis"
   captured before the pause and re-fired, reopening a session and
   restarting voice in a tight cycle. Reset the engine buffer on every
   detector (re)start so resume begins from clean audio.

2. Empty-transcript toast: a silent re-listen returns
   success:false / "… STT returned empty transcript", which the desktop
   transcribe endpoint turned into a 400 -> thrown error -> "Voice
   transcription failed" notification on every silent gap. Treat an empty
   transcript as no-speech: return {ok, transcript: ""} so the voice loop
   quietly re-listens. Real failures still 4xx/5xx.
@OutThisLife @omid3098
chore(voice): tidy wake_word — drop dead SURFACES, unused np, stale d…
7e73d00 
…ocstring
@OutThisLife @omid3098
fix(voice): honor wake_word.start_new_session on every surface
d14948e 
start_new_session was respected only by the CLI; the TUI and desktop GUI
always opened a fresh session on wake, ignoring the config. The gateway
now carries the flag in the wake.detected payload and both clients honor
it (open a fresh session vs. continue the current one), matching the CLI.
@OutThisLife @omid3098
fix(ci): sync uv.lock and repair wake-word docs MDX
ef64fff 
Regenerate uv.lock for the [wake] extra (openwakeword, pvporcupine,
onnxruntime) so uv lock --check passes. Replace angle-bracket URLs in
wake-word.md with markdown links — MDX treats <https://...> as JSX.
@omid3098
fix(wake-word): enforce single-owner lifecycle
695e844
@teknium1 teknium1 added the area/sessions Session lifecycle, resume, persistence, history label Jul 19, 2026
This was referenced Jul 24, 2026
Closed
Merged
@benbarclay benbarclay mentioned this pull request Jul 28, 2026
Closed
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@teknium1 teknium1 teknium1 left review comments

Assignees

No one assigned

Labels

area/sessions Session lifecycle, resume, persistence, history comp/cli CLI entry point, hermes_cli/, setup wizard comp/desktop Electron desktop app (apps/desktop/*) comp/tui Terminal UI (ui-tui/ + tui_gateway/) P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/tts Text-to-speech and transcription type/feature New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@omid3098 @teknium1 @alt-glitch @OutThisLife