Skip to content

fix(photon): transcode inbound HEIC images - #58934

Open
samsungapore wants to merge 1 commit into
NousResearch:mainfrom
samsungapore:fix/photon-heic-inbound-vision
Open

fix(photon): transcode inbound HEIC images#58934
samsungapore wants to merge 1 commit into
NousResearch:mainfrom
samsungapore:fix/photon-heic-inbound-vision

Conversation

@samsungapore

Copy link
Copy Markdown

Summary

  • transcode inbound Photon/iMessage HEIC/HEIF attachment bytes to JPEG via macOS sips
  • propagate the cached MIME (image/jpeg) into MessageEvent.media_types after conversion
  • add a regression test covering native-vision routing for iPhone HEIC photos

Root cause

Inbound iPhone photos can arrive as HEIC/HEIF bytes. Photon cached them with stale image/heic metadata, so native image routing attempted to attach an unreadable HEIC path when pillow-heif was not installed. The model then saw only the iMessage placeholder/attachment marker instead of the image.

Test plan

  • scripts/run_tests.sh tests/plugins/platforms/photon/test_inbound.py tests/agent/test_image_routing.py -q → 111 passed
  • ./venv/bin/python -m py_compile plugins/platforms/photon/adapter.py
  • node --check plugins/platforms/photon/sidecar/index.mjs
  • git diff --check

Live verification

Verified locally with a real inbound iMessage photo: Photon cached a fresh img_*.jpg with JPEG magic ffd8ff, gateway logged native image routing with 1 image attached inline, and Photon sent the model response back successfully.

@samsungapore
samsungapore force-pushed the fix/photon-heic-inbound-vision branch from e9bdc7a to b0efbe0 Compare July 5, 2026 15:31
@alt-glitch alt-glitch added type/bug Something isn't working comp/plugins Plugin system and bundled plugins sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P3 Low — cosmetic, nice to have labels Jul 5, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: #54946 (broader canonical HEIC/HEIF fix — centralizes detection in the shared gateway/platforms/base.py media cache with an octet-stream fallback, covering both bluebubbles and photon). This PR is the narrower photon-only sips transcode; the two overlap on the photon path. Flagging the cluster so a maintainer can pick whether to land the shared-cache approach or this focused one. Prior HEIC transcode attempts #44576 and #25936 are closed-unmerged.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the Photon path and adding a native-vision regression test. The current main path does have the HEIC failure: plugins/platforms/photon/adapter.py:1637-1644 falls back to a document after the shared image cache rejects HEIC magic, while retaining the stale image MIME.

Problems

  • plugins/platforms/photon/adapter.py:1560 calls blocking subprocess.run() from the async inbound dispatch path. A slow sips invocation can block Photon event-loop progress for up to the configured 20 seconds.
  • plugins/platforms/photon/adapter.py:1613-1616 returns None when conversion is unavailable or fails. That causes _normalize_binary_payload to surface only a marker, dropping otherwise available attachment bytes instead of preserving them as a document.
  • The member comment correctly identifies open #54946 as the broader shared-cache approach. It covers Photon and BlueBubbles and includes a document fallback, which avoids duplicating HEIC policy in the Photon adapter.

Suggested changes

  • Rework this onto the shared cache path, with conversion offloaded from the event loop and a document fallback on conversion failure.
  • Add a failure/unavailable-converter regression test alongside the successful JPEG test.

Automated hermes-sweeper review.

src = Path(tmp) / f"source{safe_suffix}"
dst = Path(tmp) / "converted.jpg"
src.write_bytes(raw)
proc = subprocess.run( # noqa: S603 - fixed executable + temp paths

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_transcode_heif_to_jpeg() is reached synchronously from async _dispatch_inbound, so this subprocess.run() blocks Photon event-loop progress for up to 20 seconds. Please offload conversion through an async shared-cache path (for example, asyncio.to_thread) rather than blocking inbound delivery.

if _is_heif_image(mime, suffix):
converted = _transcode_heif_to_jpeg(raw, suffix)
if converted is None:
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning None here drops attachment bytes and makes _normalize_binary_payload emit only a metadata marker. Preserve the original HEIC as a document with a non-image MIME when conversion is unavailable or fails; this keeps the attachment inspectable without routing unsupported bytes to native vision.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants