feat(gateway): add opt-in MEDIA_CAPTION directive for native media captions - #58911
feat(gateway): add opt-in MEDIA_CAPTION directive for native media captions#58911ferreiraesilva wants to merge 1 commit into
Conversation
…ptions
Tools can emit MEDIA_CAPTION:{"path","caption","type":"image"|"video"}
to have a local image/video delivered as a native media bubble with the
caption attached to the bubble itself, instead of the caption arriving as
a separate text message before the media.
The directive is opt-in: plain MEDIA: tags and markdown images
() are untouched, so existing flows are unaffected. Each item
requires an explicit type that must agree with the file extension, and the
path goes through validate_media_delivery_path (same safety checks as
MEDIA:), so prompt-injection/credential-exfil sites stay rejected.
Reading order is preserved as [opening] [media+caption ...] [closing]:
the streamed bubble is trimmed to the text before the first directive and
the text after the last directive is sent as a trailing message after the
media (split_captioned_media_text). Falls back to the merged bubble if the
edit fails.
Drafted with AI assistance; reviewed and tested by a human maintainer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for this, and for the clear repro on the After tracing it end-to-end, we're going to fix the underlying problem rather than land a new directive, so I'm closing this PR. The reasoning: The reported behavior is already supposed to work. The The implementation itself was clean — the safety reuse ( Really appreciate the contribution and the diagnosis. |
…e sends hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned bubble instead of a separate text message followed by an uncaptioned bubble. Root cause: the standalone senders (hermes send / cron / send_message tool) stripped the MEDIA: tag, sent the remaining text as its own message, and called the media send with no caption -- even though hermes send's help advertises the captioned form and the bridges/adapters already support a caption. Signal already captioned correctly. - tools/send_message_tool.py: new _media_caption_split() chokepoint decides caption-vs-separate-body (single captionable non-voice file within the platform's message-length cap). Wired into the Telegram, WhatsApp and Discord dispatch paths. - Telegram/WhatsApp/Discord: when the single captioned file is missing, the caption text is delivered as a plain message so it is never silently lost. - Telegram caption send gets a MarkdownV2->plain parse fallback. - Tests: _media_caption_split unit tests + per-platform caption tests (ride, multi-file fallback, voice exclusion, over-limit fallback, missing-file text fallback); updated the 3 tests that asserted the old text-then-media split. Closes the gap reported against #58911 (the MEDIA_CAPTION directive PR); credit to @ferreiraesilva for surfacing the caption behavior.
…e sends hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned bubble instead of a separate text message followed by an uncaptioned bubble. Root cause: the standalone senders (hermes send / cron / send_message tool) stripped the MEDIA: tag, sent the remaining text as its own message, and called the media send with no caption -- even though hermes send's help advertises the captioned form and the bridges/adapters already support a caption. Signal already captioned correctly. - tools/send_message_tool.py: new _media_caption_split() chokepoint decides caption-vs-separate-body (single captionable non-voice file within the platform's message-length cap). Wired into the Telegram, WhatsApp and Discord dispatch paths. - Telegram/WhatsApp/Discord: when the single captioned file is missing, the caption text is delivered as a plain message so it is never silently lost. - Telegram caption send gets a MarkdownV2->plain parse fallback. - Tests: _media_caption_split unit tests + per-platform caption tests (ride, multi-file fallback, voice exclusion, over-limit fallback, missing-file text fallback); updated the 3 tests that asserted the old text-then-media split. Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR); credit to @ferreiraesilva for surfacing the caption behavior.
…e sends hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned bubble instead of a separate text message followed by an uncaptioned bubble. Root cause: the standalone senders (hermes send / cron / send_message tool) stripped the MEDIA: tag, sent the remaining text as its own message, and called the media send with no caption -- even though hermes send's help advertises the captioned form and the bridges/adapters already support a caption. Signal already captioned correctly. - tools/send_message_tool.py: new _media_caption_split() chokepoint decides caption-vs-separate-body (single captionable non-voice file within the platform's message-length cap). Wired into the Telegram, WhatsApp and Discord dispatch paths. - Telegram/WhatsApp/Discord: when the single captioned file is missing, the caption text is delivered as a plain message so it is never silently lost. - Telegram caption send gets a MarkdownV2->plain parse fallback. - Tests: _media_caption_split unit tests + per-platform caption tests (ride, multi-file fallback, voice exclusion, over-limit fallback, missing-file text fallback); updated the 3 tests that asserted the old text-then-media split. Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR); credit to @ferreiraesilva for surfacing the caption behavior.
…e sends hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned bubble instead of a separate text message followed by an uncaptioned bubble. Root cause: the standalone senders (hermes send / cron / send_message tool) stripped the MEDIA: tag, sent the remaining text as its own message, and called the media send with no caption -- even though hermes send's help advertises the captioned form and the bridges/adapters already support a caption. Signal already captioned correctly. - tools/send_message_tool.py: new _media_caption_split() chokepoint decides caption-vs-separate-body (single captionable non-voice file within the platform's message-length cap). Wired into the Telegram, WhatsApp and Discord dispatch paths. - Telegram/WhatsApp/Discord: when the single captioned file is missing, the caption text is delivered as a plain message so it is never silently lost. - Telegram caption send gets a MarkdownV2->plain parse fallback. - Tests: _media_caption_split unit tests + per-platform caption tests (ride, multi-file fallback, voice exclusion, over-limit fallback, missing-file text fallback); updated the 3 tests that asserted the old text-then-media split. Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR); credit to @ferreiraesilva for surfacing the caption behavior.
…e sends hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned bubble instead of a separate text message followed by an uncaptioned bubble. Root cause: the standalone senders (hermes send / cron / send_message tool) stripped the MEDIA: tag, sent the remaining text as its own message, and called the media send with no caption -- even though hermes send's help advertises the captioned form and the bridges/adapters already support a caption. Signal already captioned correctly. - tools/send_message_tool.py: new _media_caption_split() chokepoint decides caption-vs-separate-body (single captionable non-voice file within the platform's message-length cap). Wired into the Telegram, WhatsApp and Discord dispatch paths. - Telegram/WhatsApp/Discord: when the single captioned file is missing, the caption text is delivered as a plain message so it is never silently lost. - Telegram caption send gets a MarkdownV2->plain parse fallback. - Tests: _media_caption_split unit tests + per-platform caption tests (ride, multi-file fallback, voice exclusion, over-limit fallback, missing-file text fallback); updated the 3 tests that asserted the old text-then-media split. Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR); credit to @ferreiraesilva for surfacing the caption behavior.
…e sends hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned bubble instead of a separate text message followed by an uncaptioned bubble. Root cause: the standalone senders (hermes send / cron / send_message tool) stripped the MEDIA: tag, sent the remaining text as its own message, and called the media send with no caption -- even though hermes send's help advertises the captioned form and the bridges/adapters already support a caption. Signal already captioned correctly. - tools/send_message_tool.py: new _media_caption_split() chokepoint decides caption-vs-separate-body (single captionable non-voice file within the platform's message-length cap). Wired into the Telegram, WhatsApp and Discord dispatch paths. - Telegram/WhatsApp/Discord: when the single captioned file is missing, the caption text is delivered as a plain message so it is never silently lost. - Telegram caption send gets a MarkdownV2->plain parse fallback. - Tests: _media_caption_split unit tests + per-platform caption tests (ride, multi-file fallback, voice exclusion, over-limit fallback, missing-file text fallback); updated the 3 tests that asserted the old text-then-media split. Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR); credit to @ferreiraesilva for surfacing the caption behavior.
What does this PR do?
Adds an opt-in
MEDIA_CAPTIONdirective so a tool can have a local image orvideo delivered as a native media bubble with the caption attached to the
bubble itself, instead of the caption arriving as a separate text message
before the media.
Today a tool that wants to send media with explanatory text emits
MEDIA:<path>plus the text, and platforms (Telegram, WhatsApp, …) deliver the text as its own
message before the media bubble. For an assistant sending several photos, each
with its own description, the result is a wall of captions followed by a wall of
images, with no association between them.
With this change a tool can instead emit:
and each file is delivered through the adapter's native
send_image_file/send_videowith the caption attached to that bubble.Type of Change
Changes Made
gateway/platforms/base.pyMEDIA_CAPTION_RE— regex for the directive.extract_captioned_media(content) -> (items, cleaned)— parses directivesinto
{path, type, caption}, requiring an explicittype(image/video)that must agree with the file extension, and routing every path through the
existing
validate_media_delivery_path(same denylist / traversal / symlinkchecks as
MEDIA:). Malformed JSON, unknown types, type/extensionmismatches, and unsafe paths are skipped, leaving that text untouched.
split_captioned_media_text(content) -> (opening, closing)— returns thedisplay text before the first directive and after the last, so callers can
preserve
[opening] [media+caption ...] [closing]order.strip_media_directives_for_displaynow also stripsMEDIA_CAPTION:so theraw directive never shows in the streamed text.
gateway/run.py_deliver_media_from_responseextracts captioned media and delivers eachitem via the adapter's native captioned send, preserving order.
closing text is sent as a trailing message after the media
(
media_caption_trailing), so reading order is preserved. If the bubble editfails it falls back to the existing merged-bubble behavior — media still
arrives.
tests/gateway/test_media_caption_directive.py— unit tests for extraction,ordering, type/extension validation, unsafe-path rejection, malformed input,
the opt-out of markdown images, and the opening/closing split.
Design notes
MEDIA:tags ormarkdown images (
); a response with noMEDIA_CAPTION:takes anearly-return fast path. Adapters that don't implement native captioned sends
are unaffected — tools simply keep using
MEDIA:.validate_media_delivery_path, sothe same prompt-injection / credential-exfil rejections apply
(
/etc/passwd,~/.ssh/id_rsa, the Hermes credential store, …).(e.g. declaring a
.pngasvideo), which would otherwise reach the wrongadapter method.
How to Test
MEDIA_CAPTION:{"path": "<local image/video>", "caption": "...", "type": "image"|"video"}directives, with optional text before the first and after the last.
the opening text is its own bubble before the media, and the closing text is a
single message after the media — i.e.
[opening] [media+caption …] [closing].MEDIA_CAPTIONpointing at a denied path (e.g.~/.ssh/id_rsa) isnot delivered and stays as-is in the text.
pytest tests/gateway/test_media_caption_directive.py -q.Platforms tested
native caption, correct opening/closing order.
runtime; the full suite runs in CI.
Checklist
feat(gateway): …)go through the existing
validate_media_delivery_path)