feat(whatsapp): human-like reply context for any quoted media type - #52875
feat(whatsapp): human-like reply context for any quoted media type#52875dhruvkej9 wants to merge 1 commit into
Conversation
|
@teknium1 @kshitijk4poor @brooklyn-bb-nicholson — could one of you take a look? This fixes WhatsApp reply context so the agent sees the original quoted message text (text, captions, media placeholders, location, etc.) instead of just the message id. Tested locally with inbound replies; 27 WhatsApp formatting tests pass. |
alt-glitch
added
type/feature
897a61f to
58bc7d6
Compare
|
Latest revision includes:
Tested locally with real WhatsApp replies (text + image). All 27 tests in pass. |
dhruvkej9
changed the title
58bc7d6 to
edc80f1
Compare
…aths WhatsApp replies previously referenced only the quoted message id. This change makes the agent see the original quoted message content the way a human would: - Bridge: extract user-visible text from the quotedMessage object (text, captions, document names, audio/voice labels, location, sticker). - Bridge: keep an in-memory store of the last 200 inbound messages so replies to uncaptioned images/videos/documents/audio include the cached local media path. - Adapter: map quotedMessageId to MessageEvent.reply_to_message_id. - Adapter: inject quoted text and quoted media paths into the prompt body. Tested locally: replies now carry original text; image/video replies include the cached file path so vision tools can inspect the original. All 27 tests in tests/gateway/test_whatsapp_formatting.py pass.
edc80f1 to
b1bd3ff
Compare
teknium1
left a comment
teknium1
left a comment
There was a problem hiding this comment.
Thanks for pursuing quoted-media context. Current main has since landed a newer structured WhatsApp quote pipeline in 11627fdcb.
Problems
scripts/whatsapp-bridge/bridge.js:493records only inbound messages. The lookup at lines 501-516 therefore cannot resolve a cached path when a user replies to Hermes's own uncaptioned outbound media, so the claimed any-quoted-media coverage is incomplete.plugins/platforms/whatsapp/adapter.py:1245injects the media reference intoMessageEvent.text. Current main deliberately keeps quote data structured and delegates rendering toGatewayRunnerto prevent duplicate reply prefixes (plugins/platforms/whatsapp/adapter.py:1439-1444).
Suggested changes
- Salvage the media-reference part onto
extractBridgeEvent()and the current structuredMessageEventreply fields instead of restoring body pre-rendering. - Cover replies to both inbound and Hermes outbound uncaptioned media, across image/video/document/audio/voice paths.
Automated hermes-sweeper review.
| } | ||
|
|
||
| // Remember this message so replies can include original text/media context. | ||
| rememberInboundMessage(chatId, msg.key.id, { body, hasMedia, mediaType, mediaUrls }); |
There was a problem hiding this comment.
This store only records inbound messages, so a reply to Hermes's own uncaptioned outbound media cannot resolve a cached local path. Please retain outbound media metadata too, or narrow the advertised scope and add a regression test.
| quoted_media_urls = data.get("quotedMediaUrls") or [] | ||
| quoted_media_type = data.get("quotedMediaType") or "" | ||
| if quoted_media_urls: | ||
| body = f"[Replying to {quoted_media_type or 'media'}: {', '.join(quoted_media_urls)}]\n{body}" |
There was a problem hiding this comment.
Current main now keeps WhatsApp quote data structured and lets GatewayRunner render the reply pointer to avoid duplicate prefixes (adapter.py:1439-1444). Port this media context through that current pipeline rather than pre-rendering it into body.
teknium1
added
sweeper:risk-message-delivery
|
This predates the Reimplemented the same idea (bounded cache of already-downloaded inbound media, resolved on a later quoted reply) against current main in #77660, crediting this PR for the original approach. Feel free to close this one in favour of that if a maintainer agrees, or let me know if you'd rather finish rebasing this instead. |
|
Closing this in favour of #77660 (per @ishangodawatta's suggestion and after checking the latest state). Status check:
Keeping this open would just accumulate conflict noise. Reopen if you'd prefer I rebase this branch instead of relying on #77660. |
|
Closed in favour of #77660 — see comment above for rationale. |
Baileys' contextInfo.quotedMessage only ever carries a thumbnail-sized
stub for media, or nothing at all for an uncaptioned attachment — never
a way to fetch the original file. When a user replies to an earlier
photo/video/document/voice note with no caption on it (e.g. "did you
save this?" quoting an uncaptioned wedding invite image), the agent
saw no text and no media reference at all: it looked like the message
never had an attachment.
Add createQuotedMediaCache, a bounded in-memory cache (keyed by
chatId:messageId) of each inbound message's already-downloaded media
and text, populated as extractBridgeEvent processes every message.
When a later message quotes one of these, extractBridgeEvent resolves
quotedMediaUrls/quotedMediaType from the cache and falls back to a
human-readable quotedText ("sent an image", etc.) when the quote had
no caption to extract. The adapter folds resolved quoted media into
the event's own media_urls/media_types — reusing the existing
vision/audio pipeline and the existing _is_allowed_bridge_path path
validation — rather than adding a parallel reply-media code path.
Reimplements the same feature as NousResearch#52875 (credit: dhruvkej9) against
current main, whose 11627fd refactor (native polls, locations, rich
inbound metadata) moved this code into bridge_helpers.js and made that
PR's diff no longer apply cleanly.
4 participants
WhatsApp replies previously referenced only the quoted message id. This PR makes the agent see the original quoted message content the way a human would, for any media type.\n\nChanges:\n- Bridge: extract user-visible text from the
quotedMessageobject (plain text, extended text, image/video captions, document names, audio/voice labels, location, sticker).\n- Bridge: keep an in-memory store of the last 200 inbound messages so replies to uncaptioned images/videos/documents/audio include the cached local media path.\n- Adapter: mapquotedMessageIdtoMessageEvent.reply_to_message_id.\n- Adapter: inject quoted text and quoted media paths into the agent prompt body.\n\nTested locally:\n- Inbound WhatsApp replies now include the original message text.\n- Image/video/media replies include the cached file path so vision tools can inspect the original.\n- All 27 tests intests/gateway/test_whatsapp_formatting.pypass.