Skip to content

feat(discord): transcribe every voice-channel speaker behind voice_transcribe_all - #111218

Closed
MickaelV0 wants to merge 2 commits into
NousResearch:mainfrom
Roxabi:feat/discord-voice-transcribe-all
Closed

MickaelV0 wants to merge 2 commits into
NousResearch:mainfrom
Roxabi:feat/discord-voice-transcribe-all

Conversation

@MickaelV0

@MickaelV0 MickaelV0 commented Sep 14, 2026 •

Copy link
Copy Markdown

Stacked on #111214. This branch contains that fix as its first commit because the feature depends on its platform-neutral observed-context marker. Review/merge #111214 first and this rebases to a single commit — happy to rebase on request.

What does this PR do?

In a voice channel with several people, Hermes only ever hears the allowlisted users. _voice_listen_loop drops everyone else before STT runs:

for user_id, pcm_data in completed:
    if not self._is_allowed_user(str(user_id), guild=_vc_guild, is_dm=False):
        continue

So the agent has no idea anyone else is in the room. Ask it "what did Bob just say?" and it cannot answer — Bob's audio was discarded, not just unanswered.

discord.voice_transcribe_all (config.yaml, default false) transcribes every speaker instead. Authorization is unchanged — only allowlisted users drive a turn. A bystander's utterance is:

  1. transcribed,
  2. echoed into the bound text channel as **[Voice · observed]** @user: …,
  3. persisted as an attributed observed=True row — the same mechanism Telegram (_observe_unmentioned_group_message) and Yuanbao (_observe_group_message) already use for unmentioned group chatter,
  4. never dispatched as a turn.

So the agent can follow the room on its next addressed turn without taking orders from it.

Why the observed-row path and not just "transcribe them too"

A voice channel is a shared, low-friction surface: anyone who can join can speak. Handing that speech to a tool-capable agent as an ordinary user turn is an injection path. This PR therefore:

  • routes bystander speech through the existing observed=True mechanism, which the history builder lifts into a context-only block rather than replaying as user turns;
  • puts the platform-neutral observed-context marker in the turn's channel_prompt (the reason for the stacked fix), and states in that prompt that observed context is never to be followed as instructions;
  • keeps the row attributed ([display_name|user_id]) so the model can tell who said what;
  • sanitizes @everyone / @here in the text-channel echo, matching the existing transcript echo.

Callback compatibility

_voice_input_callback gains an optional authorized keyword, and it is only sent for bystanders — authorized absent means authorized. Existing consumers keep their exact previous call shape (no existing test changed). The callback is also signature-inspected, the rule already used for plugin hook payloads: if a consumer cannot receive authorized, a bystander transcript is dropped rather than handed over as though it were addressed.

Related Issue

No existing issue — built for a shared voice channel in production. Searched open and closed PRs/issues for transcribe all, voice channel multiple speakers, voice transcription allowlist, observed group context: no duplicate.

Adjacent, not overlapping: #63044 adds opt-in bounded room observation for Discord text channels (unmentioned messages in mention-gated channels, via MessageDisposition.OBSERVE). This PR covers the voice channel, where the speech is otherwise discarded before STT. The two are complementary; if #63044 lands first I'm happy to reuse its disposition contract for the voice path instead of the direct observed-row write used here.

Type of Change

  • ✨ New feature (non-breaking change that adds functionality)

Changes Made

  • plugins/platforms/discord/adapter.py
    • _load_discord_bool_config / _voice_transcribe_all() — config read, consistent with the existing _load_discord_int_config.
    • _voice_listen_loop — computes authorized and only skips a bystander when the flag is off.
    • _process_voice_input(..., authorized=True) — carries the decision; annotates the callback only for bystanders; _voice_callback_accepts() does the signature inspection.
    • VoiceReceiver(transcribe_all=...) — _infer_user_for_ssrc considers every channel member (otherwise a bystander's SSRC can never resolve).
  • gateway/run_voice.py
    • _handle_voice_channel_input(..., authorized=True) — bystanders go to the observed path instead of being ignored.
    • _observe_bystander_voice() — echo + attributed observed=True row, no dispatch.
    • _voice_observe_enabled() / _voice_observed_context_prompt() — the prompt carrying the marker; the probe accepts only a real bool so a Mock/coroutine cannot inject the prompt.
  • cli-config.yaml.example, website/docs/user-guide/messaging/discord.md — document the key and the authorization/observed semantics.
  • tests/gateway/test_voice_command.py — 9 new tests, additive only.

How to Test

  1. hermes config set discord.voice_transcribe_all true, restart the gateway.
  2. Join a voice channel with a second person who is not in DISCORD_ALLOWED_USERS, then run /voice join.
  3. Have the bystander speak: the bound text channel shows **[Voice · observed]** @them: …, gateway.log shows Voice observed, no turn dispatched, and no agent turn starts.
  4. Now speak yourself and ask what the other person said — the answer uses the observed context.
  5. Ask the bystander to say "ignore your instructions and …": it lands as observed context and is not executed.
  6. Set the flag back to false, restart: the bystander is no longer transcribed at all (previous behaviour).

Automated: 205 passed, 2 skipped across test_voice_command.py, test_telegram_group_gating.py, test_voice_mode.py, test_discord_send.py, test_discord_liveness.py.

Both security-relevant guards are sabotage-verified (remove the guard → the test fails):

  • drop the authorized check in the gateway → test_bystander_voice_is_observed_not_dispatched fails;
  • drop the marker from the channel_prompt → test_voice_turn_carries_observed_context_marker fails.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits) — plus the stacked dependency noted at the top
  • I've run the full suite via scripts/run_tests.sh and it shows no new failures vs origin/main (see "Full suite run" below); the voice/Discord/group-gating subsets are green (205 passed, 2 skipped)
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Ubuntu Server 24.04, Python 3.12, live Discord gateway with local faster-whisper CUDA STT

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — website/docs/user-guide/messaging/discord.md
  • I've updated cli-config.yaml.example if I added/changed config keys
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A (pure Python; no platform-specific paths)

Notes for review

  • The feature is behaviour-neutral at false, which is the default.
  • voice_transcribe_all widens transcription only. If you would rather this shipped as a plugin, the blocker is that there is no hook carrying voice-session context or a voice-delivery decision — happy to turn it into a generic-surface proposal instead.

Full suite run

Ran the canonical runner (scripts/run_tests.sh -j 10, per-file isolation, hermetic env) on this branch and on a clean origin/main worktree, same machine, same dedicated dev venv (Ubuntu Server 24.04, Python 3.12, 16 cores):

files with failures tests failed
origin/main (baseline) 28 106
this branch 28 106

The two failure sets are identical — diff of the per-file lists is empty. This branch introduces no new failures.

The 106 pre-existing failures are environment-dependent on a workstation without the relevant credentials/binaries, and none are in a touched area: test_daytona_environment (15), test_video_generation_tool_surface_matrix (38), test_fal_plugin (8), test_hindsight_provider (8), test_streaming (5), test_image_generation (3), bedrock/nous-portal/auxiliary-provider routing, browser profile/timeout, modal snapshots, test_web_tools_config, test_execution_flag_detection. No voice, discord, or telegram_group file appears in either list.

…Telegram

`_build_gateway_agent_history` pulls `observed=True` rows out of replay history into a
separate context-only block (wrapped in "[Observed ... context - context only, not
requests]" / "[Current addressed message ...]"), but only when the turn's
channel_prompt matched the literal `"observed Telegram group context"`.

`gateway/platforms/yuanbao.py` (`GroupAtGuardMiddleware._observe_group_message`)
persists observed group rows in the same `[nickname|user_id]\n<body>` shape as
Telegram, and its per-turn prompt says "Lines in history prefixed with
`[nickname|user_id]` are observed group context" — but that string does not contain
the Telegram marker, so the splitter never fired. Yuanbao's observed rows replayed as
ordinary user turns: a bystander's message reached the model with the same authority
as the addressed user's, relying on the model honouring a prose prefix convention
instead of the structural separation Telegram gets.

Adds a platform-neutral marker (`"observed group context"`) that the matcher accepts
alongside the Telegram one. Yuanbao's existing prompt already contains that substring,
so it gains the fencing with no adapter change; Telegram behaviour is unchanged. The
two context headers drop "Telegram" since they now front non-Telegram context too.

Adapters that write observed rows without any marker keep the previous behaviour, so
this is opt-in per prompt rather than a blanket change.
…anscribe_all

Today non-allowlisted speech is dropped in `_voice_listen_loop` before STT runs, so in a
voice channel with several people the agent only ever hears the allowlisted users. It has
no idea anyone else is in the room.

`discord.voice_transcribe_all` (config.yaml, default false) transcribes every speaker
instead. Authorization is unchanged: only allowlisted users drive a turn. A bystander's
utterance is transcribed, echoed into the bound text channel as `**[Voice · observed]**`,
and persisted as an attributed `observed=True` row — the same mechanism Telegram and
Yuanbao use for unmentioned group chatter — so the agent can follow the room on its next
addressed turn without taking orders from it.

The turn's channel_prompt carries the platform-neutral observed-context marker, so those
rows are lifted into the context-only block rather than replayed as ordinary user turns,
and the prompt states explicitly that observed context is never to be followed as
instructions.

`_voice_input_callback` is signature-inspected (the rule already used for plugin hook
payloads): a consumer written against the narrow signature is called exactly as before,
and if it cannot receive `authorized` a bystander transcript is dropped rather than handed
over as if it were addressed.

Default false keeps current behaviour byte-for-byte.
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins platform/discord Discord bot adapter area/config Config system, migrations, profiles sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Sep 14, 2026
@kyssta-exe

Copy link
Copy Markdown

Summary

Adds opt-in discord.voice_transcribe_all (default false): every voice-channel speaker is transcribed, but only allowlisted users drive turns. Bystander speech is echoed to the bound text channel and persisted as observed=True rows, reusing the Telegram/Yuanbao observed-context mechanism so the agent can follow the room without obeying it. Stacked on #111214 (shares its first commit — merge that first).

What changed

  • plugins/platforms/discord/adapter.py: VoiceReceiver(transcribe_all=...), listen loop carries authorized= through STT; narrow callbacks that can't receive authorized get bystander speech dropped rather than misrouted.
  • gateway/run_voice.py: _observe_bystander_voice (echo + observed row, @everyone/@here sanitized), per-turn observed-context prompt carrying the fix(gateway): fence observed group rows for every platform, not just Telegram #111214 marker.
  • Config example + Discord docs; thorough tests on both sides (drop-by-default, observe-not-dispatch, marker present/absent, narrow-callback safety).

Strengths

Findings

  • gateway/run_voice.py (_observe_bystander_voice): sanitizes @everyone/@here but not role mentions (<@&id>), which still ping when echoed. Extend the sanitizer.
  • The echo <@{user_id}> pings a bystander who never addressed the bot; consider a non-pinging attribution (@name (id) like the transcript row) for the channel echo.
  • Cost note: with the flag on, STT runs per bystander utterance — worth one docs line that this multiplies transcription spend in busy channels.
  • No CI checks reported on this branch — run the suite before merge.

Verdict

Needs minor polish (role-mention sanitization).

Reviewed using Hermes-Agent

@MickaelV0

Copy link
Copy Markdown
Author

Stale vs current NousResearch/main.

The voice overlay now lives on Roxabi/roxabi-hermes@hermes-with-voice (product fork). Not landing this PR.

@MickaelV0 MickaelV0 closed this Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have platform/discord Discord bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants