Skip to content

feat(weixin): tag transcribed voice messages and enforce echo on outbound - #27235

Open
aha-lin wants to merge 2 commits into
NousResearch:mainfrom
aha-lin:feat/weixin-voice-transcript-handling
Open

feat(weixin): tag transcribed voice messages and enforce echo on outbound#27235
aha-lin wants to merge 2 commits into
NousResearch:mainfrom
aha-lin:feat/weixin-voice-transcript-handling

Conversation

@aha-lin

@aha-lin aha-lin commented May 17, 2026

Copy link
Copy Markdown

Problem

Weixin's iLink callback delivers auto-transcribed voice messages as plain text, indistinguishable from a typed message. Two pain points fall out of that:

  1. Agents can't apply voice-specific reply conventions (e.g. echoing the transcript so the user can verify ASR accuracy, or switching tone) because they have no signal that the inbound text was a voice transcript.
  2. Even when the agent is told to add an echo prefix, LLMs drift on this surprisingly often — they forget on long turns, after tool calls, or when the user message is short. There's no mechanical guarantee.

Fix

Two small, configurable changes in gateway/platforms/weixin.py:

1. Inbound: tag transcribed voice messages

_extract_text() now returns transcripts with a stable prefix:

 if item.get("type") == ITEM_VOICE:
     voice_text = str((item.get("voice_item") or {}).get("text") or "")
     if voice_text:
-        return voice_text
+        return f"[voice:transcribed] {voice_text}"

This gives every downstream consumer (agent prompt, tools, hooks) a reliable detection signal. The prefix is plain ASCII so it survives any encoding path, and is distinctive enough to not collide with normal text.

2. Outbound: mechanical echo enforcement (opt-out)

When the inbound message was a voice transcript, the adapter stashes it on a per-chat_id map. The outbound send() checks that map and prepends a > 🎤 你说:"…" blockquote echo if the LLM didn't already include one. One-shot: popped on send so it never bleeds into a later turn.

Toggleable via:

  • voice_echo_enforce in platform config
  • WEIXIN_VOICE_ECHO_ENFORCE env var
  • defaults to on (the safer behaviour for human-facing voice gateways)

Operators who already enforce this in their system prompt and don't want the safety net can set voice_echo_enforce=false.

Why both, and why together

These two commits are paired:

  • The prefix (commit 1) makes voice messages detectable.
  • The echo enforcement (commit 2) uses that detection to provide a guarantee, and is the actual user-visible benefit.

I split them as separate commits for review clarity, but they're a unit — without the prefix, the enforcer has nothing to key on; without the enforcer, the prefix is just metadata. Happy to squash on merge if maintainers prefer.

Behaviour examples

Voice message, agent forgets to echo:

User → [voice:transcribed] 帮我看一下今天的天气
Agent draft: "今天东京晴,22-28°C..."
Adapter sends: "> 🎤 你说:"帮我看一下今天的天气"\n\n今天东京晴,22-28°C..."

Voice message, agent already echoed:

User → [voice:transcribed] 帮我看一下今天的天气
Agent draft: "> 🎤 你说:"帮我看一下今天的天气"\n\n今天东京晴..."
Adapter sends: agent draft unchanged (idempotent — startswith check)

Typed message:

User → 帮我看一下今天的天气
Agent draft: "今天东京晴..."
Adapter sends: agent draft unchanged (no transcript stashed)

Edge cases handled

  • Quotes inside transcripts are escaped (safe = transcript.replace('"', '\\"')) so the blockquote parses cleanly.
  • Empty/media-only outbound is skipped (if content and content.strip()) — wouldn't render usefully.
  • Idempotent prefix detection uses content.lstrip().startswith("> 🎤 你说") so manual echoes by the agent (any whitespace prefix) don't get double-stamped.
  • One-shot consumption via dict.pop() — never accidentally re-stamps a later unrelated reply.

Tests

No new tests — like the surrounding adapter code, this path isn't currently covered by the suite. Existing tests still pass. Happy to add coverage if reviewers want; the natural scaffolding would be a small async test that:

  1. Feeds a synthetic voice item through _extract_text → asserts the prefix.
  2. Drives _handle_inbound_message with a transcribed text → asserts _pending_voice_echo populated.
  3. Calls send() with non-prefixed content → asserts the outbound payload starts with the echo and the map was popped.

Manual verification

Running on production gateway for ~2 weeks. The enforcer log line

[weixin] voice-echo enforcer prepended prefix for u**** (LLM forgot)

fires perhaps 10–15% of voice turns when the agent gets distracted by tool use or long context. With it in place, the user-side experience is finally consistent: every voice message comes back with its own transcript echoed, every typed message comes back without one.

Compatibility

  • No config required — defaults preserve the new behaviour, but it can be disabled with one env var.
  • No breaking changes to existing senderssend() signature and contract unchanged; prefix is only added when both (a) the feature is enabled and (b) a transcript was stashed for this chat_id.
  • Prefix [voice:transcribed] is a new convention — happy to bikeshed the exact string (e.g. some structured tag like <voice transcribed=1>), but it does need to be something distinctive that downstream agents/skills can pattern-match on.

aha-lin added 2 commits May 17, 2026 10:57
Weixin delivers auto-transcribed voice messages as plain text via the
callback, indistinguishable from typed input. Prefixing the transcript
lets the agent reliably detect voice messages and apply voice-specific
reply conventions (e.g. echo the transcript so the user can verify
ASR accuracy).
When the inbound message was an auto-transcribed voice (prefixed with
[voice:transcribed]), guarantee the outbound reply starts with the
`> 🎤 你说:"…"` echo so the user can verify ASR accuracy. The LLM
should add this prefix per the wechat-voice-gateway skill, but
enforcing it mechanically removes a recurring drift mode.

One-shot: stash on inbound, pop on outbound. Toggleable via
WEIXIN_VOICE_ECHO_ENFORCE env or voice_echo_enforce config (default on).
@aha-lin
aha-lin force-pushed the feat/weixin-voice-transcript-handling branch from 521f6bc to 6eb6c47 Compare May 17, 2026 02:58
@cardtest15-coder

This comment was marked as spam.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery labels May 17, 2026
@cardtest15-coder

This comment was marked as spam.

@teknium1

teknium1 commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution — the premise still holds on current main: gateway/platforms/weixin.py:963-966 returns voice_item.text directly, so auto-transcribed voice is currently indistinguishable from typed text.

Problems

  • In the PR, the transcript is stashed under sender_id (gateway/platforms/weixin.py:1415 at 6eb6c47e63cd) but later popped by outbound chat_id (gateway/platforms/weixin.py:1702). On current main, group chat_id comes from room_id/to_user_id (gateway/platforms/weixin.py:361-367) and is used as source.chat_id (gateway/platforms/weixin.py:1450-1453), so group echoes will miss the pending transcript.
  • The pending state is not truly one-shot for all sends: the PR only pops inside the nonblank text branch (gateway/platforms/weixin.py:1701-1702 at 6eb6c47e63cd). Media-only sends, empty sends, or no-reply turns can leave stale echo state for a later outbound text.
  • There is no regression coverage, although tests/gateway/test_weixin.py already has adapter/config/send scaffolding suitable for this behavior.

Suggested changes

  • Stash by effective_chat_id after _guess_chat_type(...), not by sender_id.
  • Pop or TTL-expire pending echo state even when no text is delivered.
  • Add tests for voice prefixing, DM/group echo keying, idempotence, and stale-state cleanup.

This is an automated hermes-sweeper review; human maintainers make the final merge decision.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution — current main still returns transcribed Weixin voice text without a source marker at gateway/platforms/weixin.py:963-966, so the inbound tagging premise is valid.

Problems

  • The pending transcript is stored by sender_id in this patch (gateway/platforms/weixin.py:1415), while outbound delivery retrieves it by chat_id. Group messages use effective_chat_id from room_id/to_user_id (gateway/platforms/weixin.py:361-367, 1453-1458), so group echoes will miss.
  • The pending map is neither correlated to a specific reply nor expired. Because the hook sits in generic send(), an unrelated outbound message to the same chat can consume and prepend a stale transcript.
  • WEIXIN_VOICE_ECHO_ENFORCE is a new non-secret behavior setting, contrary to the config.yaml policy in AGENTS.md:102-107; the option is also absent from website/docs/user-guide/messaging/weixin.md:111-127.
  • tests/gateway/test_weixin.py has no coverage for this inbound-to-outbound behavior.

Suggested changes

  • Key state by effective_chat_id after _guess_chat_type() and bind it to the specific reply lifecycle, with bounded cleanup.
  • Keep the setting in gateway.platforms.weixin.extra, document it, and remove the environment override.
  • Add DM/group, idempotence, unrelated-send, and cleanup regression tests.

This is an automated hermes-sweeper review.

if self._voice_echo_enforce and text.startswith("[voice:transcribed] "):
transcript = text[len("[voice:transcribed] "):].strip()
if transcript:
self._pending_voice_echo[sender_id] = transcript

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is keyed by sender_id, but send() looks up by chat_id. For groups, current main derives the outbound/source chat ID from room_id or to_user_id in _guess_chat_type() (gateway/platforms/weixin.py:361-367), so this entry cannot be found. Compute effective_chat_id first and key the pending state with it.

self._dedup = MessageDeduplicator(ttl_seconds=MESSAGE_DEDUP_TTL_SECONDS)
# Voice echo enforcement: cache last voice transcript per chat_id so the
# outbound send() can guarantee the `> 🎤 你说:"…"` echo prefix even when
# the LLM forgets to add it. One-shot: consumed on next outbound send.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not add a new user-facing environment-variable setting for this behavior. AGENTS.md:102-107 requires non-secret behavioral configuration to live in config.yaml; retain platforms.weixin.extra.voice_echo_enforce, document it, and remove this fallback.

# is one-shot regardless).
if self._voice_echo_enforce and content and content.strip():
transcript = self._pending_voice_echo.pop(chat_id, None)
if transcript and not content.lstrip().startswith("> 🎤 你说"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This consumes process-local state on the next generic nonblank send() for the chat, not necessarily the response to this inbound voice event. An unrelated proactive send can receive the echo, while no reply leaves state unbounded. Correlate this to the inbound request/reply lifecycle or add bounded expiry and explicit cleanup.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants