Skip to content

fix(images): transcode HEIC/HEIF to JPEG at cache and vision boundaries - #44576

Closed
kaantuncel wants to merge 1 commit into
NousResearch:mainfrom
kaantuncel:fix/heic-transcode-vision
Closed

fix(images): transcode HEIC/HEIF to JPEG at cache and vision boundaries#44576
kaantuncel wants to merge 1 commit into
NousResearch:mainfrom
kaantuncel:fix/heic-transcode-vision

Conversation

@kaantuncel

Copy link
Copy Markdown

Symptom

iPhone photos sent over iMessage (Photon plugin) — or any HEIC arriving by email/AirDrop — never reach the model as images:

  1. _looks_like_image() in gateway/platforms/base.py doesn't recognize HEIC magic bytes, so cache_image_from_bytes raises ValueError and the photo is demoted to a document attachment (cache/documents/doc_*.heic). The agent gets a file path instead of a vision attachment.
  2. Even when a .heic file path is fed to vision_analyze or _file_to_data_url directly, it's sent as data:image/heic, which Anthropic and OpenAI reject (HTTP 400).

agent/image_routing.py already sniffs HEIC correctly (line ~391), and bluebubbles.py works around the problem per-adapter with a image/heic → .jpg extension map — evidence this keeps getting solved locally instead of once.

Reproduced on current main by sending iPhone photos through the Photon/iMessage gateway: all four arrived as doc_*.heic documents and required manual sips conversion before vision could read them.

Fix

Transcode at the choke points so every platform adapter and call path benefits:

  • utils.py: looks_like_heic() (ISO-BMFF ftyp brand check, same brand set image_routing already uses) + transcode_heic_to_jpeg() — tries pillow-heif if importable, falls back to macOS sips (ships with the OS), returns None when neither exists so callers keep their pre-existing behaviour. No new required dependency.
  • gateway/platforms/base.py: accept HEIC in _looks_like_image(); transcode in cache_image_from_bytes() — fixes the document-demotion for all gateway platforms at once.
  • agent/image_routing.py: transcode in _file_to_data_url() before base64 — covers HEIC file paths handed to the agent.
  • tools/vision_tools.py: detect HEIC mime + transcode to a temp JPEG in both vision_analyze paths.

Worst case (Linux, no pillow-heif): behaviour is unchanged from today — original bytes pass through and the existing provider-error path surfaces, with an info log suggesting pip install pillow-heif.

Tests

tests/test_heic_transcoding.py — 26 tests:

  • magic-byte detection across all HEIF ftyp brands, negative cases (JPEG/PNG/MP4/short data)
  • wiring tests for all three choke points with a mocked transcoder (no decoder dependency in CI)
  • real-decoder round-trip tests that auto-skip when neither pillow-heif nor sips is available

pytest tests/test_heic_transcoding.py tests/tools/test_vision_tools.py → 104 passed. The 18 failures in a broader tests/agent/ run are pre-existing test-ordering flakiness — identical failure set on untouched main (verified by stash + re-run).

Also verified end-to-end with real iPhone HEIC bytes: detection → transcode (112 KB HEIC → 240 KB JPEG) → cache_image_from_bytes now writes img_*.jpg with valid JPEG magic.

HEIC is the default iPhone camera format, but vision APIs (Anthropic,
OpenAI) reject image/heic payloads. Today an inbound iPhone photo
either fails _looks_like_image() and gets demoted to a document
attachment (Photon/iMessage path), or reaches the provider as
data:image/heic and errors.

Fix at the choke points so every platform adapter benefits:
- utils: looks_like_heic() magic-byte check + transcode_heic_to_jpeg()
  trying pillow-heif (optional dep) then macOS sips (zero install);
  returns None when no decoder exists so callers keep prior behaviour
- gateway/platforms/base: accept HEIC in _looks_like_image and
  transcode in cache_image_from_bytes
- agent/image_routing: transcode in _file_to_data_url before base64
- tools/vision_tools: detect HEIC mime + transcode to a temp JPEG in
  both vision_analyze paths

No new required dependency: pillow-heif is used only if importable;
sips ships with macOS. Tests use synthetic headers + mocked transcoder
(no decoder needed in CI), plus real-decoder tests that skip when
neither backend is present.
@kaantuncel kaantuncel closed this Jun 12, 2026
@kaantuncel
kaantuncel deleted the fix/heic-transcode-vision branch June 12, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant