BlueBubbles: native iMessage voice memo support + inbound webhook dedup - #11654
Closed
cypres0099 wants to merge 3 commits into
Closed
BlueBubbles: native iMessage voice memo support + inbound webhook dedup#11654cypres0099 wants to merge 3 commits into
cypres0099 wants to merge 3 commits into
Conversation
iMessage only renders an attachment as a voice bubble (waveform + play button) when the underlying send includes the voice-memo marker on the IMCore attachment, and the BlueBubbles AppleScript send path does not set that marker -- only the Private API send path does. Without method=private-api, an audio attachment with isAudioMessage=true still arrives as a generic file attachment even when the bytes are valid Opus-in-CAF. This sets method="private-api" whenever is_audio_message is True so the adapter targets the send path that can actually produce a voice bubble. Non-audio attachments continue to use the server default. Note: this requires the BlueBubbles Private API Helper to be connected on the server (helper_connected: true in /api/v1/server/info). Users without it will see the attachment fall back to a file -- same behavior as before this change -- and should follow the BlueBubbles Private API install docs to enable it.
iMessage's voice-memo renderer is strict about the attachment format: it specifically requires Opus audio inside a CAF container. Any other codec (MP3, AAC, PCM -- including PCM wrapped inside CAF) arrives as a generic file attachment even when the voice-memo flag is set correctly on the send. BlueBubbles-server has its own MP3 -> CAF converter that fires when an .mp3 is sent with isAudioMessage=true, but it currently produces 16-bit PCM in CAF (see BlueBubblesApp/bluebubbles-server#793), which hits the same iMessage fallback. So even with a patched upstream BB, callers that ship any other audio format (AAC, WAV, Opus-in-OGG) still end up with file attachments. This pre-converts anything non-.caf to Opus-in-CAF via afconvert, the native macOS tool. afconvert ships on every macOS version the BlueBubbles server runs on, so there is no new dependency. If afconvert is missing or the conversion fails, the original path is sent unchanged -- matches today's behavior, no regression. This mirrors the Opus-in-OGG transcode already done for Telegram voice messages in tools/tts_tool.py, just targeting the CAF container instead of OGG.
BlueBubbles fires more than one webhook per iMessage in several conditions: 1. A new-message event followed ~1s later by an updated-message event for the same guid (delivery receipt / edit / reaction notification). 2. Group-chat fanout: when the local user is a participant in both a DM and a group chat, a single inbound message can trigger webhooks carrying different chat GUIDs (e.g. "any;-;+phone" and "+phone") but the same message guid. 3. Reconnect / retry paths where BlueBubbles redelivers recent events. Without dedup, each of these duplicate webhooks was being processed by the gateway as an independent inbound turn, producing multiple agent responses for a single user message (most visibly: a voice memo landing in the DM *and* a separate text reply landing in the group chat, both triggered by one user send). This adds a small in-memory LRU (guid -> monotonic timestamp, 60s TTL) that tracks recently-seen iMessage guids and short-circuits the handler if the same guid has already been processed. At-least-once delivery is the norm for HTTP webhooks; idempotent handling on the receiver side is the right place to solve this rather than trying to suppress the duplicate upstream (which would also lose legitimate edit/receipt signal on updated-message events). 60s is generous; iMessage guid collisions are impossible in that window. Eviction is opportunistic on each inbound to keep the dict from growing unbounded.
|
Would love to get the de-dup fix here. |
3 tasks
13 tasks
Contributor
Author
|
Closing this older bundled branch in favor of the dedup-only replacement in #47262. That keeps the maintainer-requested BlueBubbles webhook de-dup fix isolated from the unrelated voice-memo send/transcode work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three related fixes to the BlueBubbles adapter that together make iMessage voice memos work end-to-end and eliminate duplicate inbound turn processing. Each is a standalone commit so they can be cherry-picked or split during review.
The three changes
1. Route voice memos through the `private-api` send path
iMessage only renders an attachment as a voice bubble (waveform + play button) when the underlying IMCore send carries the voice-memo marker. The AppleScript send path in BlueBubbles-server doesn't set this marker -- only the Private API send path does. Setting `method="private-api"` in `_send_attachment` when `is_audio_message=True` fixes this.
Without this change, audio attachments always arrive as generic file attachments even with a correct codec and the `isAudioMessage` flag. See the BlueBubbles adapter source (`packages/server/src/server/api/apple/actions.ts`): the AppleScript `sendMessage` path never forwards `isAudioMessage` into the AppleScript itself.
2. Transcode audio to Opus-in-CAF before voice send
iMessage's voice-memo renderer specifically requires Opus codec inside a CAF container -- MP3, AAC, and PCM-in-CAF all fall back to generic file rendering. BlueBubbles-server's own `convertMp3ToCaf` helper currently produces PCM-in-CAF (tracked in BlueBubblesApp/bluebubbles-server#793), and callers shipping any other audio format (AAC, WAV, Opus-in-OGG) never trigger the converter at all.
This pre-converts anything non-.caf to Opus-in-CAF via `afconvert`, the native macOS tool. Ships on every macOS version BlueBubbles-server runs on, so no new dependency. On conversion failure, the original path is sent unchanged -- matches today's fallback behavior.
Mirrors the Opus-in-OGG transcode `tools/tts_tool.py` already does for Telegram voice messages, just targeting CAF instead of OGG.
3. Dedup inbound webhook events by message guid
BlueBubbles fires more than one webhook per iMessage in several conditions:
Without dedup, each duplicate webhook produced an independent agent turn -- most visibly: a voice memo landing in the DM and a separate text reply landing in the group chat, both triggered by one user send.
This adds a small in-memory LRU (guid -> monotonic timestamp, 60s TTL) that short-circuits the handler if the guid has already been processed. At-least-once delivery is the norm for HTTP webhooks; idempotent handling on the receiver is the right place to solve this rather than trying to suppress duplicates upstream (which would also lose legitimate edit/receipt signal on future `updated-message` handling).
Testing
Verified end-to-end on Apple Silicon macOS 26.3, Hermes v0.10.0 native install, BlueBubbles-server v1.9.9 with Private API helper connected:
Prerequisites for end users
Voice memos require the BlueBubbles Private API Helper to be connected (`helper_connected: true` in `/api/v1/server/info`). On Apple Silicon macOS 14+, this means:
Users without Private API see the same fallback behavior as today (voice send lands as a file attachment), so there's no regression for the non-Private-API path.
Related