Skip to content

fix(gateway): kanban notifier — honor SendResult(success=False), notify on block_loop_detected, tolerate transient outages - #62712

Closed
rhylryan21 wants to merge 1 commit into
NousResearch:mainfrom
rhylryan21:fix/kanban-notifier-delivery-reliability
Closed

fix(gateway): kanban notifier — honor SendResult(success=False), notify on block_loop_detected, tolerate transient outages#62712
rhylryan21 wants to merge 1 commit into
NousResearch:mainfrom
rhylryan21:fix/kanban-notifier-delivery-reliability

Conversation

@rhylryan21

@rhylryan21 rhylryan21 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What

Three delivery-reliability fixes to the gateway kanban notifier
(gateway/kanban_watchers.py), plus an active-profile adapter routing fix
(gateway/authz_mixin.py):

  1. Soft send-failure was a silent drop (addresses the notifier half of [Bug]: Kanban notifications can be lost for failed sends and decomposed child tasks #31901).
    The delivery loop discarded adapter.send()'s return value. The cursor is
    claimed before send, so an adapter that REPORTS failure via
    SendResult(success=False) without raising (e.g. the Telegram adapter's
    "Not connected" mid-reconnect, or a degraded-send path) advanced the
    subscription past the event — the notification was lost forever, no retry, no
    log above DEBUG. Now a reported failure is raised into the existing
    rewind/retry path. Regression test:
    test_kanban_notifier_rewinds_claim_on_reported_send_failure.

  2. block_loop_detected never notified. When a task re-blocks for the same
    cause past BLOCK_RECURRENCE_LIMIT, block_task routes it to triage for a
    human decision and emits only a block_loop_detected event — a kind absent
    from the notifier's TERMINAL_KINDS. So the one transition that exists to
    force human attention produced zero notification and the task stalled in
    triage silently. Added to TERMINAL_KINDS with a dedicated message.

  3. A brief outage permanently unsubscribed a live channel. MAX_SEND_FAILURES
    was 3 at a 5s tick — a ~15s Telegram/API blip dropped the subscription for
    good (and, with fix Terminal tool #1, soft failures now reach this counter too). Raised to
    12 (~60s); a genuinely dead chat still drops, just later.

  4. Active-profile routing. A single-profile gateway stamps
    notifier_profile=<active> but registers its adapter as primary;
    _authorization_adapter now returns self.adapters[platform] when the
    stamped profile equals the active profile. Related to kanban notify/wake in multi-profile gateways: notifier coupled to dispatch_in_gateway, notifier_profile mis-resolves to 'default', wake targets dispatcher's agent instead of creator #56802 (that issue
    covers the multi-profile topology; this is the single-profile facet).

Why

Each is a silent-failure path: the notifier is the only signal a human gets that
unattended work is held or failed. A dropped/blocked notification means work
piles up behind the review gate invisibly.

Testing

pytest tests/gateway/test_kanban_notifier.py tests/gateway/test_multiplex_profile_authz.py
— green (16 passed), including the new regression test that asserts a reported
send failure leaves the event unseen (rewound) rather than silently consumed.

Not included

Notification-copy/UX wording is intentionally out of scope (reliability only).
The decompose-inheritance half of #31901 (child tasks not inheriting notify
subs) is not addressed here.

- honor SendResult(success=False) instead of discarding it, so an adapter
  that REPORTS (not raises) a soft send failure — e.g. the Telegram adapter's
  "Not connected" mid-reconnect — no longer advances the cursor past an
  undelivered event and silently loses the notification. Addresses the
  notifier half of NousResearch#31901.
- add block_loop_detected to the notifier's TERMINAL_KINDS so a task routed to
  triage for a human decision (re-blocked past the recurrence limit) actually
  pings its subscribers instead of stalling silently.
- raise MAX_SEND_FAILURES 3 -> 12 (~60s at the 5s tick) so a transient
  Telegram/API outage does not permanently unsubscribe a live channel now that
  reported soft-failures also reach this counter.
- route active-profile-stamped subscriptions via the primary adapter on a
  single-profile gateway (self.adapters[platform] when the stamped
  notifier_profile equals the active profile). Related to NousResearch#56802.

Adds test_kanban_notifier_rewinds_claim_on_reported_send_failure asserting a
reported send failure leaves the event unseen (rewound) rather than consumed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Looks good! No obvious issues found.


Reviewed by Hermes Agent

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Looks good! No obvious issues found.


Reviewed by Hermes Agent

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management platform/telegram Telegram bot adapter P3 Low — cosmetic, nice to have sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 11, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the focused reliability fix. The current checkout confirms the reported premises: gateway/kanban_watchers.py:416-418 discards the SendResult, its rewind path is exception-only at :448-472, and block_loop_detected is emitted by hermes_cli/kanban_db.py:4681-4690 but omitted from TERMINAL_KINDS at gateway/kanban_watchers.py:167. The active-profile routing change also matches the primary/secondary adapter topology in gateway/run.py:8478-8491.

Problems

  • The block_loop_detected delivery behavior has no end-to-end notifier regression test. tests/hermes_cli/test_kanban_block_kinds.py:118-130 verifies DB event emission only; it does not run the watcher or verify a subscriber receives the triage notification.

Suggested changes

  • Add a notifier test that drives the recurrence limit, runs one tick, and asserts both the recording-adapter message and cursor behavior.

Automated hermes-sweeper review.

# "status" covers dashboard drag-drop and `_set_status_direct()`
# writes — surface those transitions to subscribers too.
TERMINAL_KINDS = ("completed", "blocked", "gave_up", "crashed", "timed_out", "status", "archived", "unblocked")
TERMINAL_KINDS = ("completed", "blocked", "gave_up", "crashed", "timed_out", "status", "archived", "unblocked", "block_loop_detected")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an end-to-end notifier regression test for this newly claimed kind: drive the block → unblock → same-cause re-block recurrence, run one watcher tick, and assert delivery plus cursor advancement. Existing coverage only confirms event emission in tests/hermes_cli/test_kanban_block_kinds.py.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 11, 2026
teknium1 added a commit that referenced this pull request Jul 26, 2026
…cted e2e coverage

Follow-ups from review of salvaged PRs #59278 and #62712:

* test_kanban_notifier_isolates_per_subscription_failure previously
  created the good subscription first; list_notify_subs() has no
  ORDER BY, so the good delivery happened before the bad claim raised
  and the test passed even without the isolation fix. The bad task is
  now created first AND a deterministic-order shim forces the failing
  subscription to be iterated first, so the test fails on the old
  whole-tick-abort behavior.

* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
  block_loop_detected event through one notifier tick end-to-end,
  asserting the triage ping reaches the adapter and the cursor advances
  (the sweeper review of #62712 flagged that only DB-level emission was
  tested).
teknium1 added a commit that referenced this pull request Jul 26, 2026
…cted e2e coverage

Follow-ups from review of salvaged PRs #59278 and #62712:

* test_kanban_notifier_isolates_per_subscription_failure previously
  created the good subscription first; list_notify_subs() has no
  ORDER BY, so the good delivery happened before the bad claim raised
  and the test passed even without the isolation fix. The bad task is
  now created first AND a deterministic-order shim forces the failing
  subscription to be iterated first, so the test fails on the old
  whole-tick-abort behavior.

* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
  block_loop_detected event through one notifier tick end-to-end,
  asserting the triage ping reaches the adapter and the cursor advances
  (the sweeper review of #62712 flagged that only DB-level emission was
  tested).
@teknium1

Copy link
Copy Markdown
Contributor

Partially merged via PR #72236 — the block_loop_detected notification and transient-outage tolerance halves were cherry-picked with your authorship preserved. The SendResult(success=False) half had already been fixed on main (rewind-on-soft-fail landed earlier), so that part was dropped as implemented-on-main. Thanks!

@teknium1 teknium1 closed this Jul 26, 2026
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…cted e2e coverage

Follow-ups from review of salvaged PRs NousResearch#59278 and NousResearch#62712:

* test_kanban_notifier_isolates_per_subscription_failure previously
  created the good subscription first; list_notify_subs() has no
  ORDER BY, so the good delivery happened before the bad claim raised
  and the test passed even without the isolation fix. The bad task is
  now created first AND a deterministic-order shim forces the failing
  subscription to be iterated first, so the test fails on the old
  whole-tick-abort behavior.

* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
  block_loop_detected event through one notifier tick end-to-end,
  asserting the triage ping reaches the adapter and the cursor advances
  (the sweeper review of NousResearch#62712 flagged that only DB-level emission was
  tested).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have platform/telegram Telegram bot adapter sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants