Skip to content

fix(gateway): retry stale dead delivery targets - #63372

Open
necoweb3 wants to merge 1 commit into
NousResearch:mainfrom
necoweb3:fix/dead-target-retry-self-heal
Open

fix(gateway): retry stale dead delivery targets#63372
necoweb3 wants to merge 1 commit into
NousResearch:mainfrom
necoweb3:fix/dead-target-retry-self-heal

Conversation

@necoweb3

Copy link
Copy Markdown
Contributor

Summary

DeadTargetRegistry records permanently unreachable delivery targets so cron/fanout sends do not hammer deleted groups, blocked bots, or missing chats. However, the registry also claimed to be self-healing: a successful send should clear the dead flag when the user re-adds the bot or restores the chat.

That self-healing path was unreachable. DeliveryRouter.deliver() checks dead_targets.is_dead() before calling the adapter, so a marked-dead target is always short-circuited and the successful-send clear path can never run.

Why

A transiently valid dead classification can become stale: for example, a bot is removed from a chat, Hermes marks the target dead, then the user re-adds the bot. Without a retry window, all future scheduled or fanout deliveries to that chat continue returning dead_target until the user manually deletes the profile-local gateway/dead_targets.json.

That leaves delivery permanently wedged even after the platform target is usable again.

Changes

  • Add a retry window for dead-target entries.
  • Keep recent dead targets short-circuited to avoid platform flood/log spam.
  • Allow old dead-target entries to attempt one real send.
  • Preserve the existing clear-on-success behavior: if the retry send succeeds, the dead flag is removed.
  • If the retry send still fails with a dead-target error, mark_dead() refreshes the timestamp and the target is short-circuited again.

Tests

python -m pytest -p no:cacheprovider --basetemp=.pytest_tmp_dead_targets tests/gateway/test_dead_targets.py -q --timeout-method=thread
28 passed in 1.40s

python -m ruff check gateway/dead_targets.py tests/gateway/test_dead_targets.py

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P2 Medium — degraded but workaround exists labels Jul 12, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for identifying the unreachable self-healing branch. The retry-window predicate is coherent inside DeadTargetRegistry, but it currently does not reach the live cron delivery path.

Problems

  • gateway/delivery.py:275-290 is the only production location that calls DeadTargetRegistry.is_dead() before delivery. Current cron delivery instead constructs DeliveryRouter at cron/scheduler.py:1694 and calls the private _deliver_to_platform() directly at cron/scheduler.py:1707; it bypasses the changed predicate entirely. Consequently, this PR alone does not retry or clear stale dead targets for scheduled delivery.

Suggested changes

  • Please coordinate this retry-window change with the live-path registry wiring described in linked PR #64915, and cover the expired-entry retry through cron/scheduler.py::_deliver_result().

Automated hermes-sweeper review.

Comment thread gateway/dead_targets.py
return False
try:
marked_at = float(entry.get("marked_at", 0))
except (TypeError, ValueError):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This expiry predicate only affects callers of DeliveryRouter.deliver(). Current live cron delivery bypasses that method by calling _deliver_to_platform() directly at cron/scheduler.py:1707, so this change alone has no effect on scheduled deliveries; please wire the registry into that path as well (the linked #64915 addresses that gap).

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants