fix(gateway): retry stale dead delivery targets - #63372
Open
necoweb3 wants to merge 1 commit into
Open
Conversation
8 tasks
teknium1
reviewed
Jul 15, 2026
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for identifying the unreachable self-healing branch. The retry-window predicate is coherent inside DeadTargetRegistry, but it currently does not reach the live cron delivery path.
Problems
gateway/delivery.py:275-290is the only production location that callsDeadTargetRegistry.is_dead()before delivery. Current cron delivery instead constructsDeliveryRouteratcron/scheduler.py:1694and calls the private_deliver_to_platform()directly atcron/scheduler.py:1707; it bypasses the changed predicate entirely. Consequently, this PR alone does not retry or clear stale dead targets for scheduled delivery.
Suggested changes
- Please coordinate this retry-window change with the live-path registry wiring described in linked PR #64915, and cover the expired-entry retry through
cron/scheduler.py::_deliver_result().
Automated hermes-sweeper review.
| return False | ||
| try: | ||
| marked_at = float(entry.get("marked_at", 0)) | ||
| except (TypeError, ValueError): |
Contributor
There was a problem hiding this comment.
This expiry predicate only affects callers of DeliveryRouter.deliver(). Current live cron delivery bypasses that method by calling _deliver_to_platform() directly at cron/scheduler.py:1707, so this change alone has no effect on scheduled deliveries; please wire the registry into that path as well (the linked #64915 addresses that gap).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeadTargetRegistryrecords permanently unreachable delivery targets so cron/fanout sends do not hammer deleted groups, blocked bots, or missing chats. However, the registry also claimed to be self-healing: a successful send should clear the dead flag when the user re-adds the bot or restores the chat.That self-healing path was unreachable.
DeliveryRouter.deliver()checksdead_targets.is_dead()before calling the adapter, so a marked-dead target is always short-circuited and the successful-send clear path can never run.Why
A transiently valid dead classification can become stale: for example, a bot is removed from a chat, Hermes marks the target dead, then the user re-adds the bot. Without a retry window, all future scheduled or fanout deliveries to that chat continue returning
dead_targetuntil the user manually deletes the profile-localgateway/dead_targets.json.That leaves delivery permanently wedged even after the platform target is usable again.
Changes
mark_dead()refreshes the timestamp and the target is short-circuited again.Tests