Skip to content

fix(cron): wire confirmed-dead delivery targets into the live-adapter send path - #64915

Open
pierrenode wants to merge 1 commit into
NousResearch:mainfrom
pierrenode:fix/cron-delivery-dead-target-wiring
Open

fix(cron): wire confirmed-dead delivery targets into the live-adapter send path#64915
pierrenode wants to merge 1 commit into
NousResearch:mainfrom
pierrenode:fix/cron-delivery-dead-target-wiring

Conversation

@pierrenode

Copy link
Copy Markdown
Contributor

Summary

gateway/delivery.py::DeliveryRouter.deliver() is the only place that consults gateway/dead_targets.py's DeadTargetRegistry: it skips a target already confirmed dead (deleted group, blocked/kicked bot, deactivated user), and marks/clears the flag around a send's failure/success.

cron/scheduler.py's live-adapter delivery path does not call deliver(). It deliberately calls the private DeliveryRouter._deliver_to_platform() directly instead, because deliver()'s private-chat topic detection demands a reply anchor (thread_id/message_thread_id in metadata) that cron sends — which have no inbound message to reply to — can't supply; routing through _deliver_to_platform() with thread_id passed via the target/metadata bypasses that check (see the existing comment at the call site, #22773/#52060).

Checked every call site in the codebase: deliver()'s only caller anywhere is the test suite. gateway/run.py constructs self.delivery_router and keeps .adapters synced but never calls .deliver() either. The practical effect: a cron job's target that has been definitively dead for weeks is retried — full send attempt, full failure, full log line — on every single tick, on every platform, forever. This is exactly the waste dead_targets.py's own docstring says it exists to prevent ("re-sending to it on every cron tick... wastes a send attempt against the platform's flood-control envelope and spams the logs").

Fix

cron/scheduler.py::_deliver_result() now shares one DeadTargetRegistry instance across a delivery pass:

  1. Before attempting any send for a target (top of the per-target loop), skip it if already marked dead.
  2. Pass the same registry into DeliveryRouter(config, adapters, dead_targets=...) so the live-adapter path's failure handling and my checks stay consistent within one run.
  3. When _deliver_to_platform() raises a real send error, classify it via the existing gateway/delivery.py::_classify_dead_from_error_text() (the same classifier deliver() itself uses) and mark the target dead if it's a whole-chat death.
  4. Right before falling through to the standalone HTTP send path, check again: if the live-adapter failure above just got classified dead this same tick, skip the guaranteed-to-fail standalone attempt too, instead of waiting for the next tick's pre-loop check.
  5. On a successful (or assumed-delivered, in-flight-timeout) live send, clear any stale dead flag — self-healing, mirroring deliver()'s own post-success behavior.

The standalone (non-live-adapter) send path itself (tools/send_message_tool.py::_send_to_platform) is intentionally left untouched — extending dead-target awareness there is a separate, independent surface I haven't audited to the same depth, and is out of scope for this PR. The pre-loop check (item 1) already protects standalone-only ticks (gateway not live) from repeat waste once a target is marked dead via any live-adapter tick.

Test plan

  • test_already_dead_target_is_skipped_without_calling_adapter: pre-marks a target dead, asserts adapter.send is never called and the standalone fallback is never called either.
  • test_forbidden_send_error_marks_target_dead: adapter returns a "Forbidden: bot was blocked by the user" failure; asserts a fresh DeadTargetRegistry() instance sees it marked dead afterward (proves persistence, not just an in-memory object the test happens to hold), and that the standalone fallback is skipped on the same tick.
  • test_transient_send_error_does_not_mark_target_dead: a generic "Connection reset by peer" failure must NOT be classified as dead, and the standalone fallback must still run — guards against over-broad classification.
  • test_successful_send_clears_dead_flag: a successful send calls DeadTargetRegistry.clear() with the right platform/chat_id (spied via patch.object(..., autospec=True)).
  • Mutation-verify: stashed cron/scheduler.py and confirmed 3 of the 4 new tests fail against pre-fix code (the 4th — the transient-error precision guard — correctly passes either way, since "not marking dead" is also true with zero dead-target logic at all; it only has teeth combined with the other three).
  • Full tests/cron/test_scheduler.py (220 tests), tests/gateway/test_dead_targets.py, tests/gateway/test_delivery.py, tests/gateway/test_delivery_silence_filter.py, and tests/cron/test_jobs.py — 443 tests total — all pass.
  • ruff check clean.
  • Fresh competing-PR search: fix(gateway): retry stale dead delivery targets #63372 ("retry stale dead delivery targets") only touches gateway/dead_targets.py's retry-window logic itself, never cron/scheduler.py or gateway/delivery.py — doesn't address this wiring gap. feat(cron): redirect stale-target deliveries to parent/home channel #54598 ("redirect stale-target deliveries to parent/home channel") does touch cron/scheduler.py's standalone-fallback region, but implements a completely independent mechanism (its own is_definitive_delivery_failure() classifier redirecting to a fallback channel) with no reference to DeadTargetRegistry — diffed line-by-line, no functional overlap with this change.

@alt-glitch alt-glitch added type/bug Something isn't working comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 15, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for tracing the live-adapter bypass. The premise is confirmed on current main: cron calls DeliveryRouter._deliver_to_platform() directly at cron/scheduler.py:1707, while the dead-target lifecycle is implemented in DeliveryRouter.deliver() at gateway/delivery.py:269-313.

Problems

  • Live media has the same bypass class but is not covered. _send_media_via_adapter() catches adapter errors at cron/scheduler.py:1324; a media-only forbidden send therefore never reaches the proposed text-send classifier and will still retry on future ticks.
  • The linked fix(gateway): retry stale dead delivery targets #63372 stale-entry issue remains relevant: pre-marked targets are skipped before a send can clear them. Its retry behavior should be exercised through _deliver_result() once these changes are combined.

Suggested changes

  • Feed live media failures through the same whole-chat classification, with a media-only forbidden regression test.
  • Add an _deliver_result() test for an expired dead entry that retries and clears on success when coordinated with fix(gateway): retry stale dead delivery targets #63372.

The changed production file is byte-identical between this PR's base and current origin/main, so salvage should otherwise be mechanical. Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 16, 2026
@pierrenode
pierrenode force-pushed the fix/cron-delivery-dead-target-wiring branch from f7d4b55 to 0119333 Compare July 27, 2026 00:36
… send path

gateway/delivery.py's DeliveryRouter.deliver() checks DeadTargetRegistry
before every send and marks/clears it around success or failure — but
cron's live-adapter path calls the private _deliver_to_platform()
directly, bypassing deliver() entirely (it needs to skip deliver()'s
private-chat reply-anchor requirement, which cron sends have no inbound
anchor to satisfy). The only caller of the public deliver() in the whole
codebase is the test suite, so a target confirmed dead (deleted group,
blocked/kicked bot, deactivated user) was retried on every single cron
tick, on every platform, forever — the exact waste dead_targets.py exists
to prevent.

Share one DeadTargetRegistry instance across a delivery pass:
- Skip a target already marked dead before attempting any send (live or
  standalone), both at the top of the per-target loop and again right
  before the standalone fallback so a target that just got marked dead
  this same tick isn't sent to twice.
- On a real send failure from _deliver_to_platform(), classify it via
  gateway/delivery.py's existing _classify_dead_from_error_text() and
  mark the target dead when it's a whole-chat death.
- On a successful (or assumed-delivered) live send, clear any stale dead
  flag — self-healing, mirroring deliver()'s own behavior.

Extends the same classification to the live-media send path:
_send_media_via_adapter() previously swallowed every per-file failure
internally (just a logger.warning), giving the caller no way to know a
send failed. It now returns the list of failure messages, and the caller
feeds each through the same _classify_dead_from_error_text() used for
text sends. This matters specifically for media-only jobs (no
text_to_send at all): adapter_ok stays at its vacuous default of True
in that case — nothing ever flips it False — so a media-only send
against a confirmed-dead target both never got marked dead AND had any
stale dead flag immediately self-healed away by the very next line. The
self-healing clear() is now guarded on is_dead() so it doesn't erase a
mark the media-failure branch just set earlier in the same pass.

Adds direct test coverage for both the media-only forbidden-send-marks-
dead case and the transient-media-error-does-not-mark-dead negative
case, mirroring the existing text-send test pair.
@pierrenode
pierrenode force-pushed the fix/cron-delivery-dead-target-wiring branch from 0119333 to 3301344 Compare July 29, 2026 14:43
pierrenode added a commit to pierrenode/hermes-agent that referenced this pull request Aug 13, 2026
classify_send_error() didn't recognize Photon's target_not_allowed error
text (shared/free-tier lines permanently rejecting a new outbound thread),
so DeliveryRouter.deliver() never called mark_dead() for it and every
delivery attempt kept re-sending to a target the sidecar had already
rejected — wasting a send against flood control on each try.

Add the 'target_not_allowed' substring to the existing 'forbidden' bucket
(already documented as covering 'lacks permission to post to the target').
No new error kind, no change to DeadTargetRegistry._DEAD_ERROR_KINDS —
mirrors the existing pattern for other permanently-rejected targets.

The scheduler-level wiring this PR originally also carried (cron's
_deliver_result() calling DeliveryRouter._deliver_to_platform() directly,
bypassing deliver()'s dead-target check) is intentionally left to NousResearch#64915,
which already lands a broader fix for that same call site (including the
media-only-job case this PR's version didn't cover) — landing both would
conflict on the same lines with divergent implementations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants