Skip to content

fix: deduplicate BlueBubbles webhook messages - #33337

Open
yu-xin-c wants to merge 7 commits into
NousResearch:mainfrom
yu-xin-c:codex/33327-bluebubbles-webhook-dedup
Open

fix: deduplicate BlueBubbles webhook messages#33337
yu-xin-c wants to merge 7 commits into
NousResearch:mainfrom
yu-xin-c:codex/33327-bluebubbles-webhook-dedup

Conversation

@yu-xin-c

@yu-xin-c yu-xin-c commented May 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • reuse the shared MessageDeduplicator in the BlueBubbles adapter
  • fingerprint webhook event shape so exact replays are suppressed while same-GUID text and attachment lifecycle updates remain deliverable
  • reject duplicate events before attachment download/cache work, preventing replayed side effects as well as duplicate dispatch
  • cover exact replay, attachment replay, distinct GUIDs, text updates, and attachment completion

Fixes #33327

Validation

  • uv run --extra dev --extra messaging python -m pytest tests/gateway/test_bluebubbles.py -q -o addopts= (61 passed)
  • uv run --extra dev --extra messaging ruff check gateway/platforms/bluebubbles.py tests/gateway/test_bluebubbles.py scripts/ci/classify_changes.py
  • git diff --check

@yu-xin-c
yu-xin-c force-pushed the codex/33327-bluebubbles-webhook-dedup branch from ed30c7f to d934f5c Compare May 27, 2026 16:12
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery labels May 27, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Competing with #30996 which also wires MessageDeduplicator into BlueBubbles adapter for GUID-based dedup. Both fix #30708. Also overlaps with #18395, #19976, #22116, #27985 (all open BlueBubbles dedup PRs). This PR additionally touches scripts/release.py for contributor email mapping.

@yu-xin-c

yu-xin-c commented Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

Updated this PR to address the scope concern:

  • Removed the unrelated scripts/release.py change and restored its executable mode.
  • Rebased/merged onto latest main so the branch is no longer conflicting.
  • Current diff is limited to gateway/platforms/bluebubbles.py and tests/gateway/test_bluebubbles.py.
  • Local syntax check on the rebased files: python3 -m py_compile /tmp/hermes33337/gateway/platforms/bluebubbles.py /tmp/hermes33337/tests/gateway/test_bluebubbles.py.

The new head is 917511c2ba19702d26691ab6070c3ab746e4724b; CI is running again now.

@Mushy-Snugglebites-badonkadonk

Copy link
Copy Markdown

Thanks for working on this — deduping repeated BlueBubbles webhook deliveries is definitely useful.

One caution from testing a similar path: deduping solely by message GUID can be a little too broad unless the key also accounts for the webhook lifecycle state. BlueBubbles may deliver multiple events for the same GUID as a message transitions or gets enriched, and some of those same-GUID updates can still be meaningful.

Cases worth guarding with regression tests before this lands:

  • exact replay of the same new-message payload should be suppressed;
  • media/attachment completion for the same GUID should not be dropped if the first event did not yet contain usable attachment metadata;
  • edits/retractions should not be suppressed only because they share the original GUID;
  • reactions/Tapbacks should not be suppressed if they arrive as a later same-GUID event shape.

A possible safer shape is to dedupe on a small event fingerprint rather than GUID alone, e.g. message GUID plus event/update type plus the relevant attachment/reaction/edit marker. That still suppresses true webhook replays while preserving legitimate same-message lifecycle updates.

Happy to help adapt tests around those lifecycle cases if useful.

Disclosure: This comment was prepared with AI assistance under human direction and reviewed before posting.

@yu-xin-c

yu-xin-c commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up on the same-GUID lifecycle concern: updated the branch so webhook dedup no longer keys solely on the BlueBubbles message GUID. It now fingerprints the event shape (event type, text, associated/reply fields, item type, and attachment metadata), so exact replay payloads are still suppressed while same-GUID text updates and attachment completion events are allowed through. Added regression coverage for both same-GUID text updates and same-GUID attachment completion.\n\nVerification:\n- uv run --with ruff ruff check gateway/platforms/bluebubbles.py tests/gateway/test_bluebubbles.py\n- uv run --with pytest --with pytest-timeout --with pytest-asyncio --with aiohttp pytest tests/gateway/test_bluebubbles.py -q (60 passed)

@alt-glitch alt-glitch added the duplicate This issue or pull request already exists label Jun 23, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #18395 — the earliest still-open PR wiring MessageDeduplicator into the BlueBubbles adapter for GUID-based inbound dedup (both fix #30708). The previously-cross-referenced #30996 has since been closed, so this PR is now linked to the canonical open one. Saturated cluster also includes #19976, #22116, #27985, #45717 — a maintainer should pick one. (Re-pointed from closed #30996 during re-triage.)

@alt-glitch alt-glitch added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state and removed duplicate This issue or pull request already exists labels Jun 23, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the verified replay-dispatch gap and for updating the key after the same-GUID lifecycle feedback.

Problems

  • gateway/platforms/bluebubbles.py:1056 checks dedup only after the attachment loop at lines 962-996. An exact replay with an attachment still performs the second download/cache side effect before being suppressed. Move the check before that loop and add a replay-with-attachment assertion that _download_attachment runs once.
  • scripts/ci/classify_changes.py:89 is unrelated to BlueBubbles and reverses current main's explicit MCP-catalog skip at line 90. Please remove this hunk from the focused fix.

Suggested changes

  • Preserve the raw event fingerprint before attachment processing, then gate all downstream attachment and dispatch work on it.

Automated hermes-sweeper review.

Comment thread gateway/platforms/bluebubbles.py Outdated
record.get("messageGuid"),
record.get("id"),
)
dedup_key = self._webhook_dedup_key(payload, record, message_id, text)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guard is reached only after the attachment loop at lines 962-996, so an exact replay still downloads and caches every attachment a second time. Build/check the fingerprint after raw text extraction and before attachment processing; add a replay-with-attachment test asserting one download and one dispatch.

Comment thread scripts/ci/classify_changes.py Outdated
ret["deps"] = True

# explicitly skip mcp catalog here. it's not needed unless those files are modified.
ret["mcp_catalog"] = True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This unrelated CI behavior change contradicts the current-main comment that empty/.github diffs should skip the MCP catalog lane unless catalog files changed. Please drop it from this focused BlueBubbles fix.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BlueBubbles webhook conflicts can duplicate or interrupt replies

4 participants