fix(images): transcode HEIC/HEIF to JPEG at cache and vision boundaries - #44576
Closed
kaantuncel wants to merge 1 commit into
Closed
fix(images): transcode HEIC/HEIF to JPEG at cache and vision boundaries#44576kaantuncel wants to merge 1 commit into
kaantuncel wants to merge 1 commit into
Conversation
HEIC is the default iPhone camera format, but vision APIs (Anthropic, OpenAI) reject image/heic payloads. Today an inbound iPhone photo either fails _looks_like_image() and gets demoted to a document attachment (Photon/iMessage path), or reaches the provider as data:image/heic and errors. Fix at the choke points so every platform adapter benefits: - utils: looks_like_heic() magic-byte check + transcode_heic_to_jpeg() trying pillow-heif (optional dep) then macOS sips (zero install); returns None when no decoder exists so callers keep prior behaviour - gateway/platforms/base: accept HEIC in _looks_like_image and transcode in cache_image_from_bytes - agent/image_routing: transcode in _file_to_data_url before base64 - tools/vision_tools: detect HEIC mime + transcode to a temp JPEG in both vision_analyze paths No new required dependency: pillow-heif is used only if importable; sips ships with macOS. Tests use synthetic headers + mocked transcoder (no decoder needed in CI), plus real-decoder tests that skip when neither backend is present.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
iPhone photos sent over iMessage (Photon plugin) — or any HEIC arriving by email/AirDrop — never reach the model as images:
_looks_like_image()ingateway/platforms/base.pydoesn't recognize HEIC magic bytes, socache_image_from_bytesraisesValueErrorand the photo is demoted to a document attachment (cache/documents/doc_*.heic). The agent gets a file path instead of a vision attachment..heicfile path is fed tovision_analyzeor_file_to_data_urldirectly, it's sent asdata:image/heic, which Anthropic and OpenAI reject (HTTP 400).agent/image_routing.pyalready sniffs HEIC correctly (line ~391), andbluebubbles.pyworks around the problem per-adapter with aimage/heic → .jpgextension map — evidence this keeps getting solved locally instead of once.Reproduced on current
mainby sending iPhone photos through the Photon/iMessage gateway: all four arrived asdoc_*.heicdocuments and required manualsipsconversion before vision could read them.Fix
Transcode at the choke points so every platform adapter and call path benefits:
utils.py:looks_like_heic()(ISO-BMFF ftyp brand check, same brand set image_routing already uses) +transcode_heic_to_jpeg()— triespillow-heifif importable, falls back to macOSsips(ships with the OS), returnsNonewhen neither exists so callers keep their pre-existing behaviour. No new required dependency.gateway/platforms/base.py: accept HEIC in_looks_like_image(); transcode incache_image_from_bytes()— fixes the document-demotion for all gateway platforms at once.agent/image_routing.py: transcode in_file_to_data_url()before base64 — covers HEIC file paths handed to the agent.tools/vision_tools.py: detect HEIC mime + transcode to a temp JPEG in bothvision_analyzepaths.Worst case (Linux, no pillow-heif): behaviour is unchanged from today — original bytes pass through and the existing provider-error path surfaces, with an info log suggesting
pip install pillow-heif.Tests
tests/test_heic_transcoding.py— 26 tests:pytest tests/test_heic_transcoding.py tests/tools/test_vision_tools.py→ 104 passed. The 18 failures in a broadertests/agent/run are pre-existing test-ordering flakiness — identical failure set on untouchedmain(verified by stash + re-run).Also verified end-to-end with real iPhone HEIC bytes: detection → transcode (112 KB HEIC → 240 KB JPEG) →
cache_image_from_bytesnow writesimg_*.jpgwith valid JPEG magic.