feat(voice): "Hey Hermes" wake word to start a hands-free session by OutThisLife · Pull Request #53378 · NousResearch/hermes-agent

OutThisLife · 2026-06-27T02:55:08Z

Summary

Adds an opt-in, fully on-device "Hey Hermes" wake word — the "Hey Siri" pattern — across all three local surfaces (CLI, TUI, desktop GUI), with one configurable owner. Say the phrase → Hermes opens a fresh session and starts hands-free voice; talk back-and-forth; end it and the wake word re-arms.

"Hey Hermes" works out of the box — a trained openWakeWord model ships with Hermes and is the default. Off by default though; nothing listens until you turn it on.

Footprint

No new core model tool. Config + a CLI command + gateway RPCs (the cheap rungs of the ladder). Zero added model-tool schema footprint.
Cache-safe. On wake we hand a transcript to the normal input path — a user message, never a system-prompt/toolset mutation.
.env only for the secret. All behavior is in config.yaml; only PORCUPINE_ACCESS_KEY (a credential) is an env var.

How it works

tools/wake_word.py is the shared, engine-pluggable detector over the existing 16 kHz sounddevice capture path (background daemon thread, pause()/resume() so it yields the mic during a voice turn, reset() on every (re)start so a resume can't re-fire on stale audio).

CLI runs the detector in-process; on wake → new session + single-utterance voice capture via the existing pipeline.
TUI + desktop GUI share the Python tui_gateway, which runs the detector server-side and exposes wake.start/stop/pause/resume/status + a wake.detected event. Clients arm it on connect and react to wake.detected (open a fresh session, start voice). Events route back over the same transport that armed it (WS for the desktop), and the desktop pauses/resumes the detector around its browser-mic loop.

Configuration

wake_word:
  enabled: false
  surface: auto               # which surface owns it: "auto" | "cli" | "tui" | "gui"
  provider: openwakeword      # "openwakeword" (free, local) | "porcupine" (premium)
  phrase: "hey hermes"        # cosmetic label; detection is keyed by the model/keyword below
  sensitivity: 0.5            # 0.0-1.0 (higher = stricter)
  start_new_session: true
  openwakeword:
    model: hey_hermes         # bundled default; OR a built-in name OR a path to a custom .onnx/.tflite
    inference_framework: onnx
  porcupine:
    keyword: jarvis           # built-in keyword OR path to a custom .ppn

surface is the single gate (wake_surface_enabled) every surface consults, so exactly one owns the listener and the new session it opens.

Using a different phrase

"Hey Hermes" is the default. To wake on something else:

openWakeWord (free, local) — name a built-in (hey_jarvis, alexa, …) or train your own (~75–90 min on a free Colab GPU) and point openwakeword.model at the .onnx. Refs: openWakeWord, 2026 training Colab. Tip: a distinctive 2-word phrase generalizes far better than common words.
Porcupine (custom keyword in seconds) — make a keyword in the Picovoice Console, download the .ppn, set PORCUPINE_ACCESS_KEY in ~/.hermes/.env, and point porcupine.keyword at it.

Both engines lazy-install on first use, or: uv pip install 'hermes-agent[wake]'. On first use openWakeWord fetches its shared feature-extraction models (a small one-time download).

Requirements

Mic + the sounddevice/numpy stack (shared with voice mode).
An STT provider for the spoken command (local faster-whisper works out of the box).
The wake engine deps (auto-installed, or the [wake] extra).

/wake status (CLI) reports exactly what's missing.

Notes / limits

Local surfaces only (CLI/TUI/GUI) — not the messaging gateway, which has no mic.
One mic at a time — the detector yields to voice capture and reclaims it after.

Also in this PR

Bundled "hey hermes" model (tools/wakewords/, trained with the openWakeWord pipeline, Apache-2.0) + packaging so it ships in the wheel and sdist.
Base-model fetch fix: a custom .onnx path used to skip download_models(), so a fresh install crashed on a missing melspectrogram.onnx; the base feature models are now ensured for any model.
Folds in the standalone ElevenLabs voice-list 401 log-spam fix (fix(dashboard): stop ElevenLabs voice-list 401 log spam #53386) — the wake logs were drowned by it during testing.
Empty STT transcript (silence) is treated as no-speech (quiet re-listen), not a "transcription failed" toast.

Test plan

scripts/run_tests.sh tests/tools/test_wake_word.py — config/surface gate, engine dispatch, requirements probe, detector fire/cooldown/reset, singleton lifecycle, bundled-model resolution + base-model fetch (all mocked, no live audio/network).
tests/test_packaging_metadata.py guards the bundled model shipping in both wheel and sdist.
tests/tools/test_lazy_deps.py, tests/hermes_cli/test_commands.py, tests/hermes_cli/test_config.py green.
tsc --noEmit clean for both ui-tui and apps/desktop.
Manual: desktop GUI — "hey hermes" → fresh session + back-and-forth voice → end → re-arms (verified via wake.start/detected/pause/resume in gui.log).

github-actions · 2026-06-27T03:19:17Z

🔎 Lint report: `bb/wake-word` vs `origin/main`

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11717 on HEAD, 11697 on base (🆕 +20)

🆕 New issues (35):

Rule	Count
`unresolved-attribute`	13
`unsupported-operator`	11
`unresolved-import`	6
`invalid-argument-type`	5

First entries

tools/wake_word.py:167: [unresolved-import] unresolved-import: Cannot resolve imported module `openwakeword`
tools/wake_word.py:168: [unresolved-import] unresolved-import: Cannot resolve imported module `openwakeword.model`
cli.py:11162: [unresolved-attribute] unresolved-attribute: Attribute `invalidate` is not defined on `None` in union `None | Unknown`
tools/wake_word.py:223: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `None` in union `Any | None | dict[Unknown, Unknown]`
tools/wake_word.py:114: [unresolved-import] unresolved-import: Cannot resolve imported module `numpy`
hermes_cli/mcp_startup.py:66: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tools/browser_tool.py:1321: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/cron/test_suggestions.py:213: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["monitor"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/tools/test_browser_lightpanda.py:242: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["engine"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
hermes_cli/cli_commands_mixin.py:2678: [unresolved-attribute] unresolved-attribute: Object of type `Self@_handle_wake_command` has no attribute `_start_wake_word_listener`
tests/tools/test_wake_word.py:12: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/hermes_cli/test_destructive_slash_confirm_gate.py:32: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/hermes_cli/test_aux_config.py:47: [unsupported-operator] unsupported-operator: Operator `not in` is not supported between objects of type `Literal["session_search"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/tools/test_refresh_agent_mcp_tools.py:257: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/hermes_cli/test_aux_config.py:54: [unresolved-attribute] unresolved-attribute: Attribute `keys` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/cli/test_reasoning_command.py:552: [invalid-argument-type] invalid-argument-type: Argument to bound method `TestCase.assertIn` is incorrect: Expected `Iterable[Any] | Container[Any]`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
hermes_cli/cli_commands_mixin.py:2676: [unresolved-attribute] unresolved-attribute: Object of type `Self@_handle_wake_command` has no attribute `_stop_wake_word_listener`
tests/cli/test_resume_display.py:716: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["resume_display"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
hermes_cli/cli_commands_mixin.py:2680: [unresolved-attribute] unresolved-attribute: Object of type `Self@_handle_wake_command` has no attribute `_show_wake_word_status`
tools/wake_word.py:213: [unresolved-import] unresolved-import: Cannot resolve imported module `pvporcupine`
tests/tools/test_web_providers.py:219: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["extract_backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/hermes_cli/test_kanban_core_functionality.py:3431: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tools/wake_word.py:115: [unresolved-import] unresolved-import: Cannot resolve imported module `sounddevice`
hermes_cli/config.py:5264: [unresolved-attribute] unresolved-attribute: Attribute `items` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
tests/tools/test_browser_console.py:341: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["record_sessions"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 37 union elements`
... and 10 more

✅ Fixed issues (23):

Rule	Count
`unsupported-operator`	11
`unresolved-attribute`	8
`invalid-argument-type`	4

First entries

tools/browser_tool.py:1321: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/cli/test_fast_command.py:484: [invalid-argument-type] invalid-argument-type: Argument to bound method `TestCase.assertIn` is incorrect: Expected `Iterable[Any] | Container[Any]`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_destructive_slash_confirm_gate.py:32: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/agent/test_curator.py:1208: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["curator"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_refresh_agent_mcp_tools.py:257: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_aux_config.py:54: [unresolved-attribute] unresolved-attribute: Attribute `keys` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_kanban_core_functionality.py:3431: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
hermes_cli/config.py:5233: [unresolved-attribute] unresolved-attribute: Attribute `items` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/run_agent/test_in_place_compaction.py:257: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_web_providers.py:217: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
hermes_cli/config.py:5243: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_mcp_reload_confirm_gate.py:33: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str`, `list[Unknown]`, `list[str]`, `None`, `int`, `float` in union `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/gateway/test_whatsapp_reply_prefix.py:119: [unsupported-operator] unsupported-operator: Operator `>=` is not supported between objects of type `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements` and `int`
tests/cron/test_suggestions.py:213: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["monitor"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/cli/test_reasoning_command.py:552: [invalid-argument-type] invalid-argument-type: Argument to bound method `TestCase.assertIn` is incorrect: Expected `Iterable[Any] | Container[Any]`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_browser_console.py:341: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["record_sessions"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_browser_lightpanda.py:242: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["engine"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/cli/test_resume_display.py:716: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["resume_display"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_aux_config.py:37: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["title_generation"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_web_providers.py:219: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["extract_backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
hermes_cli/mcp_startup.py:66: [invalid-argument-type] invalid-argument-type: Argument to constructor `float.__new__` is incorrect: Expected `str | Buffer | SupportsFloat | SupportsIndex`, found `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/tools/test_web_providers.py:218: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["search_backend"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`
tests/hermes_cli/test_aux_config.py:47: [unsupported-operator] unsupported-operator: Operator `not in` is not supported between objects of type `Literal["session_search"]` and `str | dict[Unknown, Unknown] | list[Unknown] | ... omitted 36 union elements`

Unchanged: 6118 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

tonydwb

Code Review Summary

Verdict: Comment (high surface area)

This PR adds 1,433 lines (68KB diff) introducing voice features across desktop, gateway, and agent layers. Changes span:

Desktop: New voice start event, composer integration, wake word handling
Gateway: Voice session management, gateway state updates
Agent: Voice provider integration

While the feature is well-scoped to voice interaction, the cross-cutting nature (desktop + gateway + agent) and the 1,433-line addition size warrant human review. Key concerns:

New event system (VOICE_START_EVENT) needs integration verification
Gateway state management changes need testing
Voice session lifecycle needs validation

Recommendation: Defer to human review for this cross-cutting feature.

Reviewed by Hermes Agent

github-actions · 2026-07-21T20:46:43Z

૮ >ﻌ< ა ci review

_{ran on 17dc350}

all good!

… model tools/wake_word.py is a shared, engine-pluggable detector (openWakeWord default, free/local; Porcupine premium) over the existing 16 kHz sounddevice capture. A background daemon thread with pause()/resume() yields the mic during a voice turn, and reset() on every (re)start keeps a resume from re-firing on stale audio. Ships a bundled "hey hermes" openWakeWord model (tools/wakewords/, Apache-2.0) as the default; a built-in name or a custom .onnx/.tflite path still works. download_models() runs for any model so a fresh install fetches the shared feature models instead of crashing on a missing melspectrogram.onnx. The wake deps lazy-install on first use, or via the [wake] extra. Packaging ships the bundled model in both wheel and sdist, guarded by a metadata test.

Adds the wake_word config block (surface, provider, phrase, sensitivity, and per-engine options) with wake_surface_enabled() so exactly one surface owns the listener and the session it opens. In the CLI the detector runs in-process; on wake it opens a fresh session and captures a single utterance through the existing voice pipeline, with an idle watchdog that re-arms the mic. The /wake [on|off|status] command reports what is configured and what is missing.

…-free voice The TUI and desktop GUI share the Python tui_gateway, which runs the detector server-side and exposes wake.start/stop/pause/resume/status plus a wake.detected event routed back over the same transport that armed it. Clients arm it on connect; on wake the desktop opens a fresh session, starts voice, and hands the mic between the detector and its browser voice loop. An empty STT transcript (silence) is treated as a quiet re-listen rather than a "transcription failed" toast.

Adds the wake-word user guide (setup, config, per-engine phrase changes, requirements, limits) and links it from the features overview.

teknium1 · 2026-07-24T02:48:19Z

Closing in favor of #58539, which is an explicit replacement for this PR: it rebases these commits onto current main with @OutThisLife's authorship preserved, and additionally fixes the surface: auto ownership race flagged in review (sticky first-claimant lease + cross-process mic lock). This branch is ~300 commits behind and can't be merged directly.

All the substantive work here — the shared detector, bundled "hey hermes" model, /wake command, and the CLI/TUI/desktop wiring — carries forward in #58539. Thanks Brooklyn!

…box default From #53378: ships hey_hermes.onnx/.tflite (openWakeWord pipeline, Apache-2.0) under tools/wakewords/, resolves the default (and hey_hermes aliases) to the bundled file, ensures openWakeWord base feature models are fetched for custom paths too, and updates config defaults + docs from hey_jarvis to hey hermes.

…box default From NousResearch#53378: ships hey_hermes.onnx/.tflite (openWakeWord pipeline, Apache-2.0) under tools/wakewords/, resolves the default (and hey_hermes aliases) to the bundled file, ensures openWakeWord base feature models are fetched for custom paths too, and updates config defaults + docs from hey_jarvis to hey hermes.

OutThisLife requested a review from a team June 27, 2026 02:55

alt-glitch added type/feature New feature or request comp/cli CLI entry point, hermes_cli/, setup wizard tool/tts Text-to-speech and transcription P3 Low — cosmetic, nice to have labels Jun 27, 2026

github-actions Bot mentioned this pull request Jun 27, 2026

🦞 OpenClaw 生态日报 2026-06-27 litang9/big_model_radar#130

Open

tonydwb reviewed Jun 27, 2026

View reviewed changes

This was referenced Jul 4, 2026

[Feature]: Voice wake word for Hermes Desktop (Hey Hermes) #49383

Open

feat(voice): "Hey Hermes" wake word to start a hands-free session #58539

Closed

teknium1 added the area/sessions Session lifecycle, resume, persistence, history label Jul 19, 2026

OutThisLife force-pushed the bb/wake-word branch from 71c2e51 to 6f19888 Compare July 21, 2026 20:45

OutThisLife force-pushed the bb/wake-word branch 4 times, most recently from 6462977 to c7cd8f0 Compare July 23, 2026 06:48

OutThisLife mentioned this pull request Jul 23, 2026

fix(voice): speak the whole desktop turn and idle-flush held narration #69936

Merged

OutThisLife force-pushed the bb/wake-word branch from c7cd8f0 to 5d64d30 Compare July 23, 2026 06:54

OutThisLife added 4 commits July 23, 2026 02:06

docs(voice): document the wake word feature

17dc350

Adds the wake-word user guide (setup, config, per-engine phrase changes, requirements, limits) and links it from the features overview.

OutThisLife force-pushed the bb/wake-word branch from 5d64d30 to 17dc350 Compare July 23, 2026 07:06

teknium1 closed this Jul 24, 2026

teknium1 mentioned this pull request Jul 24, 2026

feat(voice): on-device wake words with open-vocabulary phrases and multi-profile voice routing #70509

Merged

Navigation Menu

Uh oh!

feat(voice): "Hey Hermes" wake word to start a hands-free session - #53378

feat(voice): "Hey Hermes" wake word to start a hands-free session#53378
OutThisLife wants to merge 4 commits into
mainfrom
bb/wake-word

OutThisLife commented Jun 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 27, 2026 •

edited

Loading

Uh oh!

tonydwb left a comment

Uh oh!

github-actions Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

teknium1 commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

OutThisLife commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Footprint

How it works

Configuration

Using a different phrase

Requirements

Notes / limits

Also in this PR

Test plan

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔎 Lint report: bb/wake-word vs origin/main

ruff

ty (type checker)

Uh oh!

tonydwb left a comment

Choose a reason for hiding this comment

Code Review Summary

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

૮ >ﻌ< ა ci review

Uh oh!

teknium1 commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

OutThisLife commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

🔎 Lint report: `bb/wake-word` vs `origin/main`

github-actions Bot commented Jul 21, 2026 •

edited

Loading