Skip to content

feat(gateway): add opt-in MEDIA_CAPTION directive for native media captions - #58911

Closed
ferreiraesilva wants to merge 1 commit into
NousResearch:mainfrom
ferreiraesilva:feat/media-caption-directive
Closed

feat(gateway): add opt-in MEDIA_CAPTION directive for native media captions#58911
ferreiraesilva wants to merge 1 commit into
NousResearch:mainfrom
ferreiraesilva:feat/media-caption-directive

Conversation

@ferreiraesilva

Copy link
Copy Markdown

What does this PR do?

Adds an opt-in MEDIA_CAPTION directive so a tool can have a local image or
video delivered as a native media bubble with the caption attached to the
bubble itself
, instead of the caption arriving as a separate text message
before the media.

Today a tool that wants to send media with explanatory text emits MEDIA:<path>
plus the text, and platforms (Telegram, WhatsApp, …) deliver the text as its own
message before the media bubble. For an assistant sending several photos, each
with its own description, the result is a wall of captions followed by a wall of
images, with no association between them.

With this change a tool can instead emit:

MEDIA_CAPTION:{"path": "/path/to/planta.png", "caption": "2-bedroom floor plan", "type": "image"}
MEDIA_CAPTION:{"path": "/path/to/tour.mp4", "caption": "Model unit tour", "type": "video"}

and each file is delivered through the adapter's native send_image_file /
send_video with the caption attached to that bubble.

Attribution: this change was drafted with AI assistance and then reviewed,
tested, and validated by a human maintainer before submission. It was verified
end-to-end against a live gateway (Telegram); a human owns and stands behind the
change.

Type of Change

  • ✨ New feature (non-breaking change that adds functionality)

Changes Made

  • gateway/platforms/base.py
    • MEDIA_CAPTION_RE — regex for the directive.
    • extract_captioned_media(content) -> (items, cleaned) — parses directives
      into {path, type, caption}, requiring an explicit type (image/video)
      that must agree with the file extension, and routing every path through the
      existing validate_media_delivery_path (same denylist / traversal / symlink
      checks as MEDIA:). Malformed JSON, unknown types, type/extension
      mismatches, and unsafe paths are skipped, leaving that text untouched.
    • split_captioned_media_text(content) -> (opening, closing) — returns the
      display text before the first directive and after the last, so callers can
      preserve [opening] [media+caption ...] [closing] order.
    • strip_media_directives_for_display now also strips MEDIA_CAPTION: so the
      raw directive never shows in the streamed text.
  • gateway/run.py
    • _deliver_media_from_response extracts captioned media and delivers each
      item via the adapter's native captioned send, preserving order.
    • After streaming, the streamed bubble is trimmed to the opening text and the
      closing text is sent as a trailing message after the media
      (media_caption_trailing), so reading order is preserved. If the bubble edit
      fails it falls back to the existing merged-bubble behavior — media still
      arrives.
  • tests/gateway/test_media_caption_directive.py — unit tests for extraction,
    ordering, type/extension validation, unsafe-path rejection, malformed input,
    the opt-out of markdown images, and the opening/closing split.

Design notes

  • Opt-in and backward-compatible. Nothing changes for MEDIA: tags or
    markdown images (![alt](url)); a response with no MEDIA_CAPTION: takes an
    early-return fast path. Adapters that don't implement native captioned sends
    are unaffected — tools simply keep using MEDIA:.
  • Reuses existing safety. Paths go through validate_media_delivery_path, so
    the same prompt-injection / credential-exfil rejections apply
    (/etc/passwd, ~/.ssh/id_rsa, the Hermes credential store, …).
  • Type/extension agreement prevents a directive from mislabeling a file
    (e.g. declaring a .png as video), which would otherwise reach the wrong
    adapter method.

How to Test

  1. From a tool, return a response containing one or more
    MEDIA_CAPTION:{"path": "<local image/video>", "caption": "...", "type": "image"|"video"}
    directives, with optional text before the first and after the last.
  2. On Telegram/WhatsApp, confirm each media bubble carries its caption natively,
    the opening text is its own bubble before the media, and the closing text is a
    single message after the media — i.e. [opening] [media+caption …] [closing].
  3. Confirm a MEDIA_CAPTION pointing at a denied path (e.g. ~/.ssh/id_rsa) is
    not delivered and stays as-is in the text.
  4. Unit tests: pytest tests/gateway/test_media_caption_directive.py -q.

Platforms tested

  • Ubuntu 24.04 (WSL2), Telegram — validated end-to-end: four images, each with a
    native caption, correct opening/closing order.
  • The added unit assertions were additionally exercised against the live gateway
    runtime; the full suite runs in CI.

Checklist

  • Commit messages follow Conventional Commits (feat(gateway): …)
  • PR contains only changes related to this feature
  • Added tests for the change
  • Considered cross-platform impact (pure-Python string/path handling; paths
    go through the existing validate_media_delivery_path)

…ptions

Tools can emit MEDIA_CAPTION:{"path","caption","type":"image"|"video"}
to have a local image/video delivered as a native media bubble with the
caption attached to the bubble itself, instead of the caption arriving as
a separate text message before the media.

The directive is opt-in: plain MEDIA: tags and markdown images
(![alt](url)) are untouched, so existing flows are unaffected. Each item
requires an explicit type that must agree with the file extension, and the
path goes through validate_media_delivery_path (same safety checks as
MEDIA:), so prompt-injection/credential-exfil sites stay rejected.

Reading order is preserved as [opening] [media+caption ...] [closing]:
the streamed bubble is trimmed to the text before the first directive and
the text after the last directive is sent as a trailing message after the
media (split_captioned_media_text). Falls back to the merged bubble if the
edit fails.

Drafted with AI assistance; reviewed and tested by a human maintainer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/feature New feature or request comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P3 Low — cosmetic, nice to have labels Jul 5, 2026
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Thanks for this, and for the clear repro on the hermes send caption case — that pinned down the real gap precisely.

After tracing it end-to-end, we're going to fix the underlying problem rather than land a new directive, so I'm closing this PR. The reasoning:

The reported behavior is already supposed to work. hermes send --to whatsapp "MEDIA:/path/F22.png This Caption" should arrive as one native image bubble with "This Caption" attached. hermes send's own help even advertises the form hermes send --to telegram "optional caption MEDIA:{file_path}". The bug is that the standalone media senders (WhatsApp, Telegram, and the sibling platforms) currently extract_media() the MEDIA: tag, then send the remaining text as a separate message before the media, and call the media send with no caption — even though the bridge/adapter already support a caption. So the fix is to make the existing, already-documented MEDIA:<path> caption form actually caption the bubble, which needs no new agent-facing surface.

The MEDIA_CAPTION: directive doesn't reach the reported path. This PR adds captioning to the gateway (agent) delivery path in gateway/platforms/base.py + gateway/run.py, but doesn't touch send_cmd.py / send_message_tool.py / the WhatsApp _standalone_send — the exact path hermes send runs through — so the reported command would still split into two parts after merging. It also introduces a second, JSON-based captioning convention alongside MEDIA:, and nothing wires it into the per-platform system-prompt guidance, so the agent has no way to emit it.

The implementation itself was clean — the safety reuse (validate_media_delivery_path, _mask_protected_spans), the type/extension agreement check, and the 12 unit tests were all solid, and your repro is what made the correct fix obvious. We'll credit you in the follow-up PR that fixes the standalone MEDIA: caption path.

Really appreciate the contribution and the diagnosis.

kshitijk4poor added a commit that referenced this pull request Jul 9, 2026
…e sends

hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned
bubble instead of a separate text message followed by an uncaptioned bubble.

Root cause: the standalone senders (hermes send / cron / send_message tool)
stripped the MEDIA: tag, sent the remaining text as its own message, and
called the media send with no caption -- even though hermes send's help
advertises the captioned form and the bridges/adapters already support a
caption. Signal already captioned correctly.

- tools/send_message_tool.py: new _media_caption_split() chokepoint decides
  caption-vs-separate-body (single captionable non-voice file within the
  platform's message-length cap). Wired into the Telegram, WhatsApp and
  Discord dispatch paths.
- Telegram/WhatsApp/Discord: when the single captioned file is missing, the
  caption text is delivered as a plain message so it is never silently lost.
- Telegram caption send gets a MarkdownV2->plain parse fallback.
- Tests: _media_caption_split unit tests + per-platform caption tests
  (ride, multi-file fallback, voice exclusion, over-limit fallback,
  missing-file text fallback); updated the 3 tests that asserted the old
  text-then-media split.

Closes the gap reported against #58911 (the MEDIA_CAPTION directive PR);
credit to @ferreiraesilva for surfacing the caption behavior.
@ferreiraesilva
ferreiraesilva deleted the feat/media-caption-directive branch July 9, 2026 12:41
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…e sends

hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned
bubble instead of a separate text message followed by an uncaptioned bubble.

Root cause: the standalone senders (hermes send / cron / send_message tool)
stripped the MEDIA: tag, sent the remaining text as its own message, and
called the media send with no caption -- even though hermes send's help
advertises the captioned form and the bridges/adapters already support a
caption. Signal already captioned correctly.

- tools/send_message_tool.py: new _media_caption_split() chokepoint decides
  caption-vs-separate-body (single captionable non-voice file within the
  platform's message-length cap). Wired into the Telegram, WhatsApp and
  Discord dispatch paths.
- Telegram/WhatsApp/Discord: when the single captioned file is missing, the
  caption text is delivered as a plain message so it is never silently lost.
- Telegram caption send gets a MarkdownV2->plain parse fallback.
- Tests: _media_caption_split unit tests + per-platform caption tests
  (ride, multi-file fallback, voice exclusion, over-limit fallback,
  missing-file text fallback); updated the 3 tests that asserted the old
  text-then-media split.

Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR);
credit to @ferreiraesilva for surfacing the caption behavior.
justemu pushed a commit to justemu/hermes-agent that referenced this pull request Jul 18, 2026
…e sends

hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned
bubble instead of a separate text message followed by an uncaptioned bubble.

Root cause: the standalone senders (hermes send / cron / send_message tool)
stripped the MEDIA: tag, sent the remaining text as its own message, and
called the media send with no caption -- even though hermes send's help
advertises the captioned form and the bridges/adapters already support a
caption. Signal already captioned correctly.

- tools/send_message_tool.py: new _media_caption_split() chokepoint decides
  caption-vs-separate-body (single captionable non-voice file within the
  platform's message-length cap). Wired into the Telegram, WhatsApp and
  Discord dispatch paths.
- Telegram/WhatsApp/Discord: when the single captioned file is missing, the
  caption text is delivered as a plain message so it is never silently lost.
- Telegram caption send gets a MarkdownV2->plain parse fallback.
- Tests: _media_caption_split unit tests + per-platform caption tests
  (ride, multi-file fallback, voice exclusion, over-limit fallback,
  missing-file text fallback); updated the 3 tests that asserted the old
  text-then-media split.

Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR);
credit to @ferreiraesilva for surfacing the caption behavior.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…e sends

hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned
bubble instead of a separate text message followed by an uncaptioned bubble.

Root cause: the standalone senders (hermes send / cron / send_message tool)
stripped the MEDIA: tag, sent the remaining text as its own message, and
called the media send with no caption -- even though hermes send's help
advertises the captioned form and the bridges/adapters already support a
caption. Signal already captioned correctly.

- tools/send_message_tool.py: new _media_caption_split() chokepoint decides
  caption-vs-separate-body (single captionable non-voice file within the
  platform's message-length cap). Wired into the Telegram, WhatsApp and
  Discord dispatch paths.
- Telegram/WhatsApp/Discord: when the single captioned file is missing, the
  caption text is delivered as a plain message so it is never silently lost.
- Telegram caption send gets a MarkdownV2->plain parse fallback.
- Tests: _media_caption_split unit tests + per-platform caption tests
  (ride, multi-file fallback, voice exclusion, over-limit fallback,
  missing-file text fallback); updated the 3 tests that asserted the old
  text-then-media split.

Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR);
credit to @ferreiraesilva for surfacing the caption behavior.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…e sends

hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned
bubble instead of a separate text message followed by an uncaptioned bubble.

Root cause: the standalone senders (hermes send / cron / send_message tool)
stripped the MEDIA: tag, sent the remaining text as its own message, and
called the media send with no caption -- even though hermes send's help
advertises the captioned form and the bridges/adapters already support a
caption. Signal already captioned correctly.

- tools/send_message_tool.py: new _media_caption_split() chokepoint decides
  caption-vs-separate-body (single captionable non-voice file within the
  platform's message-length cap). Wired into the Telegram, WhatsApp and
  Discord dispatch paths.
- Telegram/WhatsApp/Discord: when the single captioned file is missing, the
  caption text is delivered as a plain message so it is never silently lost.
- Telegram caption send gets a MarkdownV2->plain parse fallback.
- Tests: _media_caption_split unit tests + per-platform caption tests
  (ride, multi-file fallback, voice exclusion, over-limit fallback,
  missing-file text fallback); updated the 3 tests that asserted the old
  text-then-media split.

Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR);
credit to @ferreiraesilva for surfacing the caption behavior.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…e sends

hermes send "MEDIA:/x.png This Caption" now arrives as one native captioned
bubble instead of a separate text message followed by an uncaptioned bubble.

Root cause: the standalone senders (hermes send / cron / send_message tool)
stripped the MEDIA: tag, sent the remaining text as its own message, and
called the media send with no caption -- even though hermes send's help
advertises the captioned form and the bridges/adapters already support a
caption. Signal already captioned correctly.

- tools/send_message_tool.py: new _media_caption_split() chokepoint decides
  caption-vs-separate-body (single captionable non-voice file within the
  platform's message-length cap). Wired into the Telegram, WhatsApp and
  Discord dispatch paths.
- Telegram/WhatsApp/Discord: when the single captioned file is missing, the
  caption text is delivered as a plain message so it is never silently lost.
- Telegram caption send gets a MarkdownV2->plain parse fallback.
- Tests: _media_caption_split unit tests + per-platform caption tests
  (ride, multi-file fallback, voice exclusion, over-limit fallback,
  missing-file text fallback); updated the 3 tests that asserted the old
  text-then-media split.

Closes the gap reported against NousResearch#58911 (the MEDIA_CAPTION directive PR);
credit to @ferreiraesilva for surfacing the caption behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants