Skip to content

fix(gateway): isolate per-subscription failures in kanban notifier - #59278

Closed
AlexFucuson9 wants to merge 1 commit into
NousResearch:mainfrom
AlexFucuson9:fix/kanban-notifier-per-sub-isolation
Closed

AlexFucuson9 wants to merge 1 commit into
NousResearch:mainfrom
AlexFucuson9:fix/kanban-notifier-per-sub-isolation

Conversation

@AlexFucuson9

Copy link
Copy Markdown
Contributor

Summary

Fixes the kanban notifier getting permanently stuck when one subscription fails — blocking delivery for ALL other subscriptions.

Root Cause

The _collect() loop in _kanban_notifier_watcher iterates subscriptions without per-subscription error handling. When claim_unseen_events_for_sub raises for one subscription (e.g. DB corruption, lock contention, transaction rollback failure), the entire _collect() call aborts, which means:

  1. ALL other subscriptions in that tick are silently skipped
  2. The outer except logs "kanban notifier tick failed: cannot rollback - no transaction is active" — masking the real error
  3. The notifier retries forever every 5 seconds, hitting the same error
  4. No kanban→platform notifications are delivered for ANY subscription until the offending one is manually removed

Fix

Wrap the per-subscription logic inside the for sub in subs: loop in a try/except block. One bad subscription now logs a warning and continues to the next, instead of aborting the entire tick.

Testing

  • Added regression test test_kanban_notifier_isolates_per_subscription_failure — verifies that when one subscription raises during claim_unseen_events_for_sub, the other subscription still gets delivered
  • All 8 existing kanban notifier tests pass

Closes #59269

The kanban notifier _collect() loop iterates subscriptions without
per-subscription error handling. When claim_unseen_events_for_sub raises
for one subscription (e.g. DB corruption, lock contention), the entire
tick aborts — silently blocking delivery for ALL other subscriptions.

Wrap the per-subscription logic in try/except so one bad subscription
logs a warning and continues to the next, instead of jamming the
entire notifier.

Closes NousResearch#59269
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P3 Low — cosmetic, nice to have labels Jul 6, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the focused notifier-isolation fix. The current collection loop has no per-subscription handler at gateway/kanban_watchers.py:254-293, so the proposed try/except directly addresses a real failure boundary.

Problems

  • tests/gateway/test_kanban_notifier.py:326-332 creates the successful subscription before the failing one. hermes_cli/kanban_db.py:8692 selects subscriptions without ORDER BY; when that scan returns insertion order, current main sends the good notification before the bad claim raises, so this test passes without the fix.

Suggested changes

  • Make the failing subscription deterministically appear first (for example by controlling list_notify_subs in the test), then assert the good subscription is delivered. That makes the regression fail on current main and verifies continuation after a failure.

Automated hermes-sweeper review.

# Create two tasks with subscriptions and complete both.
conn = kb.connect()
try:
tid_good = kb.create_task(conn, title="good task", assignee="worker")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list_notify_subs() uses an unordered SELECT *, and this inserts the good row before the failing row. On an insertion-order scan, current main sends the good event before the bad claim aborts the tick, so this assertion passes without the fix. Please force the bad subscription to be returned first so the test proves continuation after failure.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 15, 2026
teknium1 added a commit that referenced this pull request Jul 26, 2026
…cted e2e coverage

Follow-ups from review of salvaged PRs #59278 and #62712:

* test_kanban_notifier_isolates_per_subscription_failure previously
  created the good subscription first; list_notify_subs() has no
  ORDER BY, so the good delivery happened before the bad claim raised
  and the test passed even without the isolation fix. The bad task is
  now created first AND a deterministic-order shim forces the failing
  subscription to be iterated first, so the test fails on the old
  whole-tick-abort behavior.

* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
  block_loop_detected event through one notifier tick end-to-end,
  asserting the triage ping reaches the adapter and the cursor advances
  (the sweeper review of #62712 flagged that only DB-level emission was
  tested).
teknium1 added a commit that referenced this pull request Jul 26, 2026
…cted e2e coverage

Follow-ups from review of salvaged PRs #59278 and #62712:

* test_kanban_notifier_isolates_per_subscription_failure previously
  created the good subscription first; list_notify_subs() has no
  ORDER BY, so the good delivery happened before the bad claim raised
  and the test passed even without the isolation fix. The bad task is
  now created first AND a deterministic-order shim forces the failing
  subscription to be iterated first, so the test fails on the old
  whole-tick-abort behavior.

* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
  block_loop_detected event through one notifier tick end-to-end,
  asserting the triage ping reaches the adapter and the cursor advances
  (the sweeper review of #62712 flagged that only DB-level emission was
  tested).
@teknium1

Copy link
Copy Markdown
Collaborator

Merged via PR #72236 — your commit was cherry-picked onto current main with your authorship preserved. Your per-subscription isolation fix was the core of the jam repair (#59269); we hardened the regression test on top (the original passed without the fix due to subscription ordering) and added a cursor-snap fix for the related boot-storm. Thanks!

@teknium1 teknium1 closed this Jul 26, 2026
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…cted e2e coverage

Follow-ups from review of salvaged PRs NousResearch#59278 and NousResearch#62712:

* test_kanban_notifier_isolates_per_subscription_failure previously
  created the good subscription first; list_notify_subs() has no
  ORDER BY, so the good delivery happened before the bad claim raised
  and the test passed even without the isolation fix. The bad task is
  now created first AND a deterministic-order shim forces the failing
  subscription to be iterated first, so the test fails on the old
  whole-tick-abort behavior.

* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
  block_loop_detected event through one notifier tick end-to-end,
  asserting the triage ping reaches the adapter and the cursor advances
  (the sweeper review of NousResearch#62712 flagged that only DB-level emission was
  tested).
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
…cted e2e coverage

Follow-ups from review of salvaged PRs NousResearch#59278 and NousResearch#62712:

* test_kanban_notifier_isolates_per_subscription_failure previously
  created the good subscription first; list_notify_subs() has no
  ORDER BY, so the good delivery happened before the bad claim raised
  and the test passed even without the isolation fix. The bad task is
  now created first AND a deterministic-order shim forces the failing
  subscription to be iterated first, so the test fails on the old
  whole-tick-abort behavior.

* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
  block_loop_detected event through one notifier tick end-to-end,
  asserting the triage ping reaches the adapter and the cursor advances
  (the sweeper review of NousResearch#62712 flagged that only DB-level emission was
  tested).
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…cted e2e coverage

Follow-ups from review of salvaged PRs NousResearch#59278 and NousResearch#62712:

* test_kanban_notifier_isolates_per_subscription_failure previously
  created the good subscription first; list_notify_subs() has no
  ORDER BY, so the good delivery happened before the bad claim raised
  and the test passed even without the isolation fix. The bad task is
  now created first AND a deterministic-order shim forces the failing
  subscription to be iterated first, so the test fails on the old
  whole-tick-abort behavior.

* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
  block_loop_detected event through one notifier tick end-to-end,
  asserting the triage ping reaches the adapter and the cursor advances
  (the sweeper review of NousResearch#62712 flagged that only DB-level emission was
  tested).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kanban notifier jams permanently on one bad delivery — masks the real error and blocks all notifications

3 participants