Skip to content

fix(photon): recognize opaque CAF voice notes and declare transcoder - #91129

Open
mooserini wants to merge 3 commits into
NousResearch:mainfrom
mooserini:fix/photon-voice-ffmpeg
Open

fix(photon): recognize opaque CAF voice notes and declare transcoder#91129
mooserini wants to merge 3 commits into
NousResearch:mainfrom
mooserini:fix/photon-voice-ffmpeg

Conversation

@mooserini

@mooserini mooserini commented Aug 20, 2026

Copy link
Copy Markdown

Summary

  • recognize native iMessage CAF voice notes from their caff signature when Spectrum supplies opaque attachment metadata
  • normalize those events to MessageType.VOICE + audio/x-caf, allowing /voice on to enter the auto-TTS path
  • declare ffmpeg-static as a direct Photon sidecar dependency so Spectrum can transcode non-M4A TTS output when the media transport is functional
  • add focused regression coverage for both independent prerequisites

Reproduction

Opaque inbound CAF

  1. Configure Photon and enable /voice on.
  2. Send an iMessage voice note.
  3. Spectrum can supply valid CAF bytes as an unnamed application/octet-stream attachment. Without byte-signature detection, Hermes classifies it as DOCUMENT, so STT and voice-to-voice handling are bypassed.

Missing transcoder

When Spectrum voice() receives MP3 TTS output without either ffmpeg-static or a system FFmpeg, it fails with:

voice content: input is not m4a/aac and ffmpeg is unavailable.

@spectrum-ts/core treats ffmpeg-static as optional; Photon must declare it if native voice conversion is expected on clean installs.

Scope clarification

This PR supplies two necessary prerequisites, but it does not by itself restore outbound media delivery on the currently pinned spectrum-ts@8.0.0 stack.

Controlled live testing against the same Photon project, credential, registered user, chat GUID, and AAC/M4A fixtures found:

  • Hermes-pinned Spectrum 8: text succeeds; M4A attachment() and voice() fail with target_not_allowed
  • isolated current Spectrum 12.8: the same M4A attachment and native voice sends succeed and are playable on-device
  • current Spectrum 12.8 without FFmpeg: raw MP3 attachment succeeds; raw MP3 voice() fails with the expected transcoder error

A separate companion change is required to migrate the Photon sidecar to the current Spectrum SDK, remove the obsolete 8.x mixed-attachment patch, and validate the complete sidecar path. This PR intentionally keeps the CAF classifier and direct transcoder dependency reviewable independently.

Verification

  • pytest tests/plugins/platforms/photon/test_inbound.py -q → 8 passed
  • pytest tests/test_photon_voice_dependencies.py -q → 1 passed
  • sidecar-local npm ci
  • imported ffmpeg-static and executed the bundled macOS binary
  • sidecar npm audit: 0 vulnerabilities

The live Spectrum 12 media matrix is manual acceptance evidence, not a claim that this branch already runs Spectrum 12.

@mooserini
mooserini requested a review from a team August 20, 2026 22:13
@mooserini mooserini changed the title fix(photon): install voice transcoder fix(photon): restore native voice replies Aug 20, 2026
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins tool/tts Text-to-speech and transcription sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Aug 20, 2026
@mooserini mooserini changed the title fix(photon): restore native voice replies fix(photon): recognize opaque CAF voice notes and declare transcoder Aug 21, 2026
@mooserini

Copy link
Copy Markdown
Author

Review request + fork-CI note.

This is the safe, self-contained first step of a 3-PR voice/Photon sequence:
#90121 (gateway: don't auto-TTS A2A replies) → this PR (#91129) → #91215 (migrate sidecar to Spectrum 12.8.0, remove obsolete v8 patch).

What lands here, running cleanly on the pinned spectrum-ts@8.0.0 stack:

  • Promotes inbound iMessage CAF voice notes to VOICE even when Spectrum supplies opaque application/octet-stream metadata (new caff byte-signature check alongside name/MIME).
  • Declares ffmpeg-static as a direct sidecar dependency so outbound MP3→native-voice transcoding works on clean installs.

Verification (in PR body): pytest tests/plugins/platforms/photon/test_inbound.py (new opaque-CAF case) + pytest tests/test_photon_voice_dependencies.py, npm ci, npm audit: 0 vulnerabilities, and the bundled ffmpeg-static binary was imported and executed.

Note: this is a fork PR, so workflows don't run until a maintainer clicks "Approve and run workflows" on the first run. @teknium1 @shannonsands — could you review and trigger the CI run? Landing this first unblocks inbound voice notes immediately and clears the path for #91215.

@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference, author can ignore or act on any point.

The magic-byte probe is the robust part of this fix: live Spectrum events genuinely do deliver iMessage voice notes as (unnamed) + application/octet-stream, so checking the decoded payload for the caff signature — decoding only the first base64 characters, so cost stays constant regardless of attachment size — promotes correctly where name/MIME checks can't, and stamping the canonical audio/x-caf afterwards keeps downstream transcode decisions consistent. The regression test drives the full _dispatch_inbound path with real CAF bytes and verifies the cached suffix/content, which is the level this deserves.

Two coordination items:

  1. Overlap with build(photon): upgrade spectrum-ts to 12.8.0 #91215: the package.json/package-lock.json changes here add ffmpeg-static (plus its transitive tree) while keeping spectrum-ts at 8.0.0; build(photon): upgrade spectrum-ts to 12.8.0 #91215 adds the same dependency as part of the spectrum-ts 12 upgrade. Whichever merges second will conflict and must regenerate the lockfile — worth coordinating merge order explicitly so one of them doesn't silently drop the other's resolution.
  2. tests/test_photon_voice_dependencies.py reads package.json from disk in a Python test. Repo policy places assertions about package.json/lockfile in the vitest suite — the CI change classifier won't run Python tests for JS-only changes, so a dependency regression lands green on the PR and red on main. Moving the assertion to a vitest test (or folding it into the existing photon JS-side tests) fixes the coverage hole.

@mooserini

mooserini commented Aug 22, 2026

Copy link
Copy Markdown
Author

Addressed the JS-suite point in fb772e1. The direct ffmpeg-static declaration contract now lives in tests-js/photon-sidecar-dependencies.test.ts, so a JS-only sidecar manifest change selects the Vitest lane. Removed tests/test_photon_voice_dependencies.py; the Python CAF _dispatch_inbound regression coverage remains unchanged.

Verified locally: targeted Vitest, full tests-js lint/typecheck/Vitest, and pytest tests/plugins/platforms/photon/test_inbound.py -q (8 passed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages tool/tts Text-to-speech and transcription type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants