fix(gateway): inbound voice classification/routing for Feishu, DingTalk, LINE, QQ, Photon, WhatsApp, Weixin

[teknium1]

fix: inbound voice classification & routing across 7 platforms (consolidated salvage)

Consolidated salvage of inbound voice-note handling fixes for Feishu, DingTalk, LINE, QQ, Photon (iMessage), WhatsApp, and Weixin. Each platform is its own commit(s) with the original contributor's authorship preserved via cherry-pick; hunks are independent per platform.

Per-platform changes

Feishu — @wuli666 (#29235) + @seamusmore (#40592, trimmed)

• Classify Lark native audio msg_type as MessageType.VOICE so the gateway auto-transcribes voice notes instead of treating them as untranscribable AUDIO attachments (fixes Feishu voice messages incorrectly classified as AUDIO (file attachment), skipping STT transcription #30822). Feishu was the only platform still classifying native voice as AUDIO (follow-up to fix(gateway): transcribe native voice notes (Discord + DingTalk) #28993). Dedicated regression test file.
• From fix(feishu): attach ffprobe-derived duration to voice uploads #40592 (duration/dedup hunks only — the voice-classification hunk duplicated fix(feishu): classify native voice messages as VOICE for auto-transcription #29235 and the outbound Opus send_voice rewrite is out of scope here): ffprobe-derived duration attached to voice uploads (fixes feat: Include audio duration in Feishu voice message payload #16524, Feishu outbound audio upload omits duration, causing incorrect/0s duration display #8300), TTS dedup narrowed to the current turn, and audio-path dedup in _enrich_message_with_transcription.

DingTalk — @rayjerrywoo (#50014) + richText-reset port from #38276's analysis

• Extract ASR recognition text from extensions["content"], route native msgtype=voice/audio correctly (voice → VOICE/auto-STT; generic audio uploads stay AUDIO), EXT_MAP for file/image classification, card/interactiveCard handling (fixes DingTalk voice callbacks are skipped before STT #38211, DingTalk native voice messages are not routed to STT #38219).
• Follow-up commit: the msg_type_str == "richText" branch reset msg_type to PHOTO/TEXT after the rich-text item scan had already promoted it to VOICE, dropping voice notes from the STT path. Now only re-derives when the scan left the type at TEXT. This ports the root-cause analysis from fix(dingtalk): route native voice media to STT #38276 (stale PR against the deleted gateway/platforms/dingtalk.py) onto the live plugin adapter, with regression tests.

LINE — @kronexoi (#27142)

• Normalize inbound media types and cache routing: audio → cache_audio_from_bytes (was cache_image_from_bytes for all media), video → cache_video_from_bytes, files → cache_document_from_bytes with the original filename, and real per-type MIME types on the event (fixes the remainder of LINE adapter: inbound voice/video/file messages are dropped — all media routed through the image cache #57882 — the audio→VOICE map itself already landed on main). Conflict resolution kept main's _LINE_MESSAGE_TYPES dict (the PR's equivalent _line_message_type() helper was dropped as redundant). fix(line): route audio/video/file payloads to cache_audio_from_bytes #57884 is a 6-line subset → close as dupe; fix(line): inbound media caching + outbound media serving (allowed-roots copy, 24h token TTL) #44019 (488-line media-serving rework) deferred for its own design review.

QQ — @Zioywishing (#35705) + @Zhekinmaksim (#23313)

• Stop routing explicit file uploads (content_type='file') through the STT pipeline; keep the extension fallback for genuine voice detection (fixes QQ Bot: Audio files sent via file transfer are incorrectly routed to STT pipeline #35704). Full 5-commit sequence picked (later commits fix earlier ones).
• Always unlink the temp STT wav — previously leaked on any exception after creation (only the happy path cleaned up).

Photon (iMessage) — @FlorianVal (#66326)

• Map audio/x-caf → .caf (was .mp3 — corrupt mapping breaking every iMessage voice note), promote .caf attachments to MessageType.VOICE for STT routing, handle the U+FFFC object-replacement placeholder with a deferred wait so voice notes don't interrupt active tasks (fixes Photon iMessage voice notes bypass STT pipeline and interrupt active tasks #66325), and CAF→WAV conversion in transcription_tools for cloud STT providers. Chosen over Fix Photon iMessage voice replies and threading #65322 (outbound voice replies — different half, review independently) and fix(photon): make packaged voice transcoding deterministic #65836 (CI/packaging-heavy — defer).

WhatsApp — @VIVAAN-DHAWAN (#66626)

• Preserve voice notes when STT is unavailable/fails: local-STT fallback attempt, then a placeholder carrying the cached audio file path instead of a bare [ptt received] (fixes WhatsApp (Baileys bridge): inbound voice notes arrive as [ptt received] placeholder when STT provider is unavailable #64831). Adapter also strips the bridge's synthetic [ptt received] body when the cached audio is present so the agent doesn't answer the placeholder as user text.

Weixin (inbound only) — @Kewe63 (#47125) + @zgzczzw (#65022)

• Route voice through Hermes STT instead of trusting Tencent Cloud's voice_item.text (garbage for non-Chinese audio — the original report was a Russian voice note returned as English phonemes; fixes WeChat voice messages use Tencent Cloud STT which garbles non-Chinese languages — should route through Hermes' own STT pipeline #27300): always download the raw audio, skip voice items in _extract_text so the central pipeline produces the body.
• Merged with fix(weixin): preserve voice transcript origin #65022's transcript-origin tagging: when there is no raw audio to download and Weixin supplied only its own STT text, use it but prefix [Voice transcription provided by Weixin] so the agent can distinguish it from typed text.

Dependencies / coordination

• SILK decode is NOT in this PR. Weixin caches inbound voice as .silk; STT-side SILK preprocessing lands via the separate STT-normalization salvage PR (fix(stt): preprocess .silk voice notes before transcription #11593's central transcription_tools path). Until that merges, Weixin non-Chinese voice falls into the "could not be transcribed" path rather than getting wrong Tencent text — strictly better, fully fixed once the sibling PR lands. Deliberately not duplicated here.
• Outbound SILK voice bubbles intentionally excluded ([Weixin] TTS audio messages are not delivered as voice bubbles — need OGG → SILK transcoding #9971, [Feature]: 消息平替里面支持直接发送微信语音，而不是一个.mp3文件吗？ #47764 stay open): main's send_voice comment marks the upstream SILK encoder path as not proven-working; needs live verification (feat(weixin): add native voice bubble support with OGG→SILK transcoding #32212 tracked separately).

Verification

• Targeted suites: tests/gateway/test_{feishu,feishu_voice_message_type,dingtalk,line_plugin,qqbot,weixin,whatsapp_formatting,stt_config}.py, tests/plugins/platforms/photon/, tests/tools/test_transcription*.py → 888 passed, 47 skipped. The 3 TestTranscribeMistral failures are pre-existing on origin/main (verified against a clean main worktree).
• Sabotage check: reverting the Feishu VOICE branch makes test_feishu_voice_message_type.py fail (2/3) — tests genuinely cover the fix.
• ruff check clean on all touched files; branch base is current origin/main (stale-base gate: 0 behind, diff shows only intended files).
• Contributor email mappings added under contributors/emails/ for all 8 authors.

Post-merge actions

• Close as fixed: Feishu voice messages incorrectly classified as AUDIO (file attachment), skipping STT transcription #30822 feat: Include audio duration in Feishu voice message payload #16524 Feishu outbound audio upload omits duration, causing incorrect/0s duration display #8300 DingTalk voice callbacks are skipped before STT #38211 DingTalk native voice messages are not routed to STT #38219 QQ Bot: Audio files sent via file transfer are incorrectly routed to STT pipeline #35704 Photon iMessage voice notes bypass STT pipeline and interrupt active tasks #66325 WhatsApp (Baileys bridge): inbound voice notes arrive as [ptt received] placeholder when STT provider is unavailable #64831 WeChat voice messages use Tencent Cloud STT which garbles non-Chinese languages — should route through Hermes' own STT pipeline #27300 (partially — full fix after SILK STT PR) LINE adapter: inbound voice/video/file messages are dropped — all media routed through the image cache #57882 (remainder).
• Close originals with credit: fix(feishu): classify native voice messages as VOICE for auto-transcription #29235 fix(feishu): attach ffprobe-derived duration to voice uploads #40592 fix(dingtalk-platform): extract ASR recognition and file message text from incoming messages #50014 fix(line): normalize inbound media types and cache routing #27142 fix(qqbot): stop routing file uploads through STT pipeline #35705 fix(qqbot): unlink temp wav on STT failure path #23313 fix(photon): route iMessage voice notes through STT pipeline #66326 fix(whatsapp): preserve voice notes when STT fails #66626 fix(gateway/weixin): route voice messages through Hermes STT instead of trusting Tencent Cloud's text #47125 fix(weixin): preserve voice transcript origin #65022.
• Close as dupes with credit: fix(feishu): map audio messages to VOICE for STT transcription (#30822) #30829 fix(feishu): route audio messages through voice STT #70034 fix(feishu): transcribe native voice notes #29295 (dead-target) fix: Feishu voice messages routed to VOICE for STT; .ogg added to nat… #38299 fix: Feishu voice message auto-STT transcription (two patches) #39713 fix(feishu): include audio duration in voice message payload #35168 fix(feishu): include audio duration in uploads #8631 (Feishu dupes), fix(dingtalk): route native voice media to STT #38276 (analysis ported here), fix(line): route audio/video/file payloads to cache_audio_from_bytes #57884 (LINE subset).
• Review independently: Fix Photon iMessage voice replies and threading #65322 (Photon outbound voice), fix(line): inbound media caching + outbound media serving (allowed-roots copy, 24h token TTL) #44019 (LINE media serving), feat(weixin): add native voice bubble support with OGG→SILK transcoding #32212 (Weixin outbound SILK).

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on f85b565}

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

1 visual diff.

_{inline evidence is publishing...}