fix(gateway): handle HEIC iMessage attachments

[fabian-lu-ha]

Summary

iMessage photo attachments can arrive as HEIC/HEIF. Hermes was treating those attachments as normal image media, but the downstream vision path only accepts JPEG, PNG, GIF, and WebP.

The user-visible failure is that real iPhone photos can be rejected as invalid image data, while screenshots work. In practice this can also push the agent into trying an ad-hoc terminal conversion, which may ask the user to approve a command just to process an incoming photo.

Changes

• gateway/platforms/base.py: add shared HEIC/HEIF detection and opt-in best-effort JPEG conversion for cached inbound media
• gateway/platforms/bluebubbles.py: route downloaded iMessage attachments through the shared media cache and opt into HEIC conversion
• plugins/platforms/photon/adapter.py: route Photon iMessage attachments through the same async media cache path
• tests/gateway/test_document_cache.py: cover HEIC conversion, fallback, malformed input, and size-limit behavior
• tests/gateway/test_bluebubbles.py: cover BlueBubbles attachment metadata after caching
• tests/plugins/platforms/photon/test_inbound.py: cover Photon direct and grouped HEIC attachments

Conversion uses local tools already available on the host when present: macOS sips, or ffmpeg as a fallback. If conversion is unavailable or fails, Hermes preserves the original attachment as a document instead of crashing or forwarding unsupported image bytes.

Validation

• uv run --no-sync python -m ruff check gateway/platforms/base.py gateway/platforms/bluebubbles.py plugins/platforms/photon/adapter.py tests/gateway/test_document_cache.py tests/gateway/test_bluebubbles.py tests/plugins/platforms/photon/test_inbound.py
• HERMES_HOME=/tmp/hermes-heic-fix/hermes-home uv run --no-sync python -m pytest tests/plugins/platforms/photon/ tests/gateway/test_document_cache.py tests/gateway/test_platform_base.py tests/gateway/test_bluebubbles.py -q
  • 368 passed, 2 skipped

[DanBennettUK]

Tested this branch live on Mimikyu with Photon/iMessage inbound photos.

Good news: the PR path correctly receives the full HEIC attachment bytes from Photon and routes them through the shared cache. In our live test content.read() returned the full declared HEIC size (~1.5 MB), so Photon/Spectrum was not only handing Hermes a thumbnail.

One extra issue showed up with the current ffmpeg conversion fallback: iPhone HEICs can be tiled. ffmpeg decoded one 512x512 tile, so Hermes cached a tiny square crop even though the full HEIC bytes were present. Repro detail from logs:

{"mimeType":"image/heic","declaredSize":1501296,"readBytes":1501296,"firstBytesHex":"00000024667479706865696300000000"}

I tested the same HEIC with pillow-heif; it decoded correctly to the full image, e.g. 1206x2622 for one saved HEIC and 3024x4032 for the later live photo after installing pillow-heif into the gateway venv. Focused tests still passed locally after preferring pillow-heif before ffmpeg:

tests/plugins/platforms/photon/test_inbound.py + tests/gateway/test_document_cache.py
53 passed, 0 failed

Suggested tweak: prefer pillow-heif/libheif for HEIC/HEIF conversion when available, with sips/ffmpeg as fallbacks. ffmpeg alone can produce a valid JPEG that is only a tile/crop for tiled iPhone HEICs, which is hard to detect unless we compare dimensions/byte size.

So overall: PR direction works, but the HEIC converter should probably avoid ffmpeg as the first Linux path for iPhone photos.

[fabian-lu-ha]

Thanks for testing this and for the detailed logs. I pushed an update based on this: HEIC/HEIF conversion now tries pillow-heif first, keeps sips as the macOS fallback, and no longer uses ffmpeg for HEIC because it can return a valid JPEG that is only a tile/crop for tiled iPhone photos.

I also checked it with public HEIC samples from HEIC Digital. Those files decode as 3024x4032 with pillow-heif through the Hermes cache path, while ffmpeg decoded the same samples as 2016x1512.

Focused tests pass locally:

pytest tests/gateway/test_document_cache.py tests/gateway/test_bluebubbles.py tests/plugins/platforms/photon/test_inbound.py -q

114 passed, 7 warnings

So this should address the Photon/iMessage full-byte case you found without relying on ffmpeg's HEIC decode.

[teknium1]

Thanks for tracing both iMessage paths and incorporating the tiled-HEIC feedback from the live Photon test.

Problems

• pyproject.toml:128 adds pillow-heif to unconditional dependencies, but pyproject.toml:39-44 says that set is limited to packages used by every Hermes session; opt-in functionality belongs in an extra/lazy dependency path. The new decoder is only reached by the HEIC paths in gateway/platforms/bluebubbles.py:800-805 and plugins/platforms/photon/adapter.py:1530-1536, so this makes a native decoder part of every installation.

Suggested changes

• Put pillow-heif behind the repository’s established opt-in dependency mechanism, while retaining the existing safe document fallback when it is unavailable.

The underlying bug is still present on main: gateway/platforms/bluebubbles.py:810-821 labels HEIC bytes as .jpg before cache_image_from_bytes rejects their magic bytes. This is an automated hermes-sweeper review.

[teknium1]

This makes a libheif-backed decoder part of every Hermes installation, but the dependency policy immediately above limits base dependencies to packages used by every session. This conversion is only invoked by opt-in iMessage attachment paths; please place it in the established opt-in extra/lazy-dependency mechanism (while keeping the document fallback) or document why it must be core.

[fabian-lu-ha]

Moved pillow-heif out of the base dependencies and into a pinned heic extra backed by tools.lazy_deps (media.heic). The HEIC path now installs it lazily with prompt=False; if installation or decoding is unavailable, the existing document fallback remains unchanged. I also added coverage for the lazy dependency contract and the FeatureUnavailable fallback. The affected Photon, BlueBubbles, and media tests pass locally: 454 passed, 2 skipped.