fix(gateway,image-routing): stop cross-routing docs as images + transcode uncommon formats

[shashwatgokhe]

What does this PR do?

Hey, second time contributing here so be gentle.

I was hitting HTTP 400: Could not process image from Anthropic on Discord with no obvious cause. The first time it happened my own previous PR (#25745) seemed to cover it -- but then it kept happening, so I dug deeper and it turned out there were actually two different things happening that both surface as the same generic provider error.

I tried fixing both since they share the same user-visible symptom (and the same error message). Happy to split into two PRs if you'd rather review them separately.

The actual bug that hit me

In gateway/run.py, the loop that routes per-attachment media looked like:

if mtype.startswith("image/") or event.message_type == MessageType.PHOTO:
    image_paths.append(path)

The Discord adapter sets the message-level type to PHOTO whenever any one attachment is an image. So if I upload 3 PNGs + 1 .md transcript, the loop sweeps the .md into image_paths too, the gateway base64s it as a vision content part, and Anthropic 400s the whole turn.

I confirmed this in the gateway log: Image routing: native (model supports vision). 4 image(s) will be attached inline. -- but only 3 PNGs were ever cached. The 4th was the document.

Fix: trust the per-attachment MIME when it's known, only fall back to the message-level type when the per-attachment slot is empty (legacy adapters that don't populate media_types still work).

The smaller one (format compatibility)

Anthropic / OpenAI / Gemini only natively accept PNG / JPEG / GIF / WEBP. But Discord lets people upload AVIF (Chromium screenshots), HEIC (iPhone photos), BMP, TIFF, ICO, SVG. Those also produce the same generic 400. Pillow is already used elsewhere in the codebase as a soft dep, so this transcodes anything outside the universal set to PNG before declaring media_type. Falls through cleanly if Pillow / pillow-heif / pillow-avif-plugin isn't installed (image is skipped with a logged warning rather than crashing the turn).

Related Issue

Fixes #25935

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✅ Tests (adding or improving test coverage)

Changes Made

• gateway/run.py -- per-attachment MIME is now authoritative in _prepare_inbound_message_text. Message-level PHOTO / VOICE / AUDIO is only consulted when the per-attachment MIME slot is empty.
• agent/image_routing.py:
  • _sniff_mime_from_bytes extended to recognise AVIF, TIFF (both endians), ICO, SVG (was previously PNG / JPEG / GIF / WEBP / BMP / HEIC)
  • New _UNIVERSALLY_SUPPORTED_MIMES set listing the formats accepted by every major provider
  • New _transcode_to_png helper using Pillow + optional pillow-heif and pillow-avif-plugin
  • _file_to_data_url now transcodes anything outside the safe set to PNG before encoding
  • If transcoding fails (Pillow missing, plugin missing, corrupt bytes) the image is added to skipped rather than sent broken
• tests/gateway/test_mixed_attachment_routing.py -- new file, 5 tests covering: image + document, multi-image + document (the exact failure I hit), unknown-MIME backward compatibility, document-only no-route, image + audio no-cross-route
• tests/agent/test_image_routing.py -- 10 new tests for sniffing AVIF/TIFF/ICO/SVG, BMP→PNG and TIFF→PNG transcode end-to-end, PNG/JPEG pass-through (bytes preserved), corrupted-input handled gracefully

Testing

Locally:

• 48/48 pass across the image-related test files (tests/agent/test_image_routing.py, tests/gateway/test_mixed_attachment_routing.py, tests/gateway/test_native_image_buffer_isolation.py)
• 118/118 pass across all image-related test files including pre-existing ones
• Verified the new routing test fails on the old code (real regression guard)
• End-to-end with my running Hermes gateway: original failure reproduces on old binary, no longer reproduces after restart with the patch

What I sent through manually after restarting the gateway:

• 3 PNGs + 1 .md transcript in one Discord message -> Anthropic accepts, turn completes (previously 400)
• A BMP saved as .jpg (Discord MIME-lying scenario) -> transcoded to PNG, accepted
• A .webp that's actually PNG bytes (the original feat(kanban): add --sort option to 'hermes kanban list' #25745 scenario) -> still works
• Plain PNG / JPEG -> pass through with byte-identical payload (no needless re-encode)

Notes for reviewers

• Pillow is kept a soft dependency. pillow-heif and pillow-avif-plugin are even softer (lazy-registered, swallowed on ImportError). If none are installed, behaviour matches today's: the unsupported image gets reported in skipped and the rest of the turn proceeds, rather than the turn dying with a 400.
• I tried to keep the routing change minimal -- it's 22 lines with comments explaining why, no signature changes, no behaviour change for messages where per-attachment MIME wasn't set.
• The format-compat change touches _file_to_data_url which is the single funnel for native image attachment, so it should also cover non-Discord platforms if they hit similar formats.

Happy to take feedback, trim things, or split this into two PRs if it'd be easier to review one at a time.

[shashwatgokhe]

Pushed 8e0139d -- the two failures attributable to this PR (test_bmp_transcoded_to_png and test_tiff_transcoded_to_png) were from from PIL import Image at the top of the new tests. Since Pillow is a soft dep in this repo, switched to pytest.importorskip so they skip on bare CI runners and still run as full coverage where Pillow is present.

The other 19 failures in that CI run (test_auxiliary_client, test_compression_feasibility, test_plugin_discovery, test_update_autostash, test_provider_parity, test_background_review, test_context_compressor_summary_continuity) appear to be pre-existing on main -- the last few main runs of the same workflow are red too. Happy to look into any of them if you'd like, but didn't want to scope-creep this PR.

[shashwatgokhe]

Latest run after the Pillow fix: 19 failed, 22873 passed, 161 skipped (the 2 extra skips are the Pillow-soft-dep transcode tests skipping correctly when Pillow isn't on the runner).

For sanity I also pulled the most recent Tests workflow run on main itself (run 25883272638, commit cd64bed): 19 failed, 22860 passed, 159 skipped. Same 19 tests, same error messages -- test_auxiliary_client, test_compression_feasibility, test_plugin_discovery, test_update_autostash, test_provider_parity, test_background_review, test_context_compressor_summary_continuity. So all 19 remaining failures pre-exist on main and aren't from this PR.

Diff vs main's run: this PR adds 13 passing tests + 2 conditional-skip tests, removes nothing. Net 0 new failures, net +13 coverage. Hopefully reviewable as-is, but happy to rebase / split / wait for the main-branch flakes to clear if that's preferred.

[teknium1]

Merged via PR #53923 (commit 505bc27d) — #53923

Salvaged onto current main with both bugs fixed (mixed-attachment per-attachment routing + uncommon-format transcode), plus the document-context-note path widened so a doc mixed into a photo message reaches the agent as a readable file. Your work was combined with two other independent fixes for this issue and all three contributors are credited in the PR; @shashwatgokhe's authorship is preserved on the merge commit. Thanks for the fix.