fix(kanban): run notifier independently of gateway dispatch - #67673
Conversation
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the notifier-only use case and adding coverage for disabled embedded dispatch.
Problems
gateway/kanban_watchers.py:134-157removes a deliberate multi-gateway guard. The current contract indocs/kanban/multi-gateway.md:13-17is that the shared gate ensures exactly one process opens board DBs, avoiding WAL-shmreader contention. This patch needs an independent notifier ownership/lease mechanism before all non-dispatch gateways can poll.- The changed configuration semantics are not reflected in
docs/kanban/multi-gateway.md:9-38;website/docs/user-guide/features/kanban.md:863also omits the newschedulednotification event. - GitHub reports the branch conflicting with main, and current main includes notifier-path commits after its base (
246eacea7b,2cc0ff44b6), so the patch needs reconciliation with those paths.
Suggested changes
- Coordinate notifier ownership per board independently of dispatcher ownership, with a multi-gateway regression proving non-owners do not open/poll the shared DB.
- Update the affected Kanban deployment and notification documentation.
Automated hermes-sweeper review.
| purely a fan-out of the single-DB poll. | ||
|
|
||
| Notification delivery is intentionally independent from embedded | ||
| dispatch. A gateway may own the user's chat subscription while task |
There was a problem hiding this comment.
This decouples notification delivery by removing the only multi-gateway DB-access guard. docs/kanban/multi-gateway.md:13-17 defines that guard as the protection against multiple processes opening the same board DB and amplifying WAL -shm contention. Please add notifier-specific per-board ownership/coordination rather than allowing every non-dispatch gateway to poll.
DeliciousHouse
left a comment
There was a problem hiding this comment.
Dev-team merge-gate review at exact head 00d29b89a725c4a3499d0532636d86a0bc867465.
Verdict: changes required before merge (first and only correction cycle).
-
HIGH (confidence 10/10) — independent notifier ownership is missing.
gateway/run.py:7925-7928starts the notifier in every gateway process, and this diff removes the only early ownership gate atgateway/kanban_watchers.py:134-157. Every non-dispatch gateway would therefore enumerate and open every board DB every five seconds. That contradicts the existing multi-gateway contract indocs/kanban/multi-gateway.md:13-17and reintroduces the WAL/open-handle contention that gate prevents. Add notifier ownership/lease coordination independent of dispatcher ownership. Prove the selected notifier gateway delivers withdispatch_in_gateway=false, a non-owner does not open/poll board DBs, both dev boards are seen, anddispatch_onceis never called. -
HIGH (confidence 10/10) — reconcile the live conflict without regressing current notifier delivery. Current
upstream/mainis07e97d2f5dc3d2092cfe693ef07b2527a36cd2d8; the PR is merge-dirty becausegateway/kanban_watchers.pyoverlaps246eacea7and2cc0ff44b. Preserve upstreamadapter_supports_push,SendResult(success=False)handling, API-server self-post-before-cursor-advance, failed-wake rewind/drop behavior, and their tests. Integrate this PR's full blocked/scheduled reason into that path: oncescheduledis globally notifiable, the non-push path must not silently claim it with no self-post, and blocked/scheduled wake delivery must retainASK,WHY GATED,SCOPE,ROLLBACK,REPLY, plus scheduledWINDOW. Add API-server regressions for successful advance, failed-wake rewind, complete decision fields, and final completion unsubscribe. -
MEDIUM (confidence 10/10) — update the affected docs.
docs/kanban/multi-gateway.md:9-39still says notifier and dispatcher share one ownership gate, andwebsite/docs/user-guide/features/kanban.md:861-875omitsscheduled. Document the corrected notifier-owner model and complete scheduled notification behavior.
Evidence on this stale exact head: 43 focused Windows tests passed; 237 related canonical WSL tests passed; Ruff, py_compile, and git diff --check passed. The production diff adds no dispatcher/claim/promote/spawn path, credential, live Telegram/config mutation, or live-board mutation. These results do not clear the merge gate: the corrected exact head still needs green upstream CI, and the live PR currently reports no checks.
DeliciousHouse
left a comment
There was a problem hiding this comment.
Follow-up review of corrected head cab86e38a17afc53e6730a47d1306f6a170f2c55 against current upstream/main 0e2808729e20e09dfe3581b15ab1018f839ff636: not ready. The single developer correction cycle has already been used, so I am not issuing a second bounce; disposition is escalated to dev-lead.
Verified blockers:
-
[P1] The machine-global lease starves disjoint single-profile gateways. Only the lease winner polls (
gateway/kanban_watchers.py:144-180). In the supported one-process-per-profile topology, that process has no adapters for the other profiles;gateway/kanban_watchers.py:319-327skips those subscriptions while every other gateway is locked out. A default winner starves named-profile subscriptions, and a named-profile winner starves default/other-profile subscriptions. Startup order cannot make simultaneous subscriptions for both processes deliver. -
[P1] An active named-profile gateway cannot route even its own stamped subscription. The named profile's primary adapter is in
self.adapters, but notifier delivery passes the name to_authorization_adapter(gateway/kanban_watchers.py:377-400), whose non-default branch consults only_profile_adaptersand returnsNone(gateway/authz_mixin.py:64-89). An isolated two-tick reproduction with active profilebeta, adapter inself.adapters, and a completedbetasubscription producedsends=0and leftcompletedunseen. -
[P1] Multiplex auto-subscription stamps the wrong profile.
/kanban createreads the inbound source but ignoressource.profile, stamping the gateway's startup notifier profile instead (gateway/slash_commands.py:475-493). An isolated reproduction producedsource_profile=betaandstamped_notifier_profile=default, so later delivery can use the wrong same-platform bot. -
[P1] Partial batch failure can duplicate an already delivered decision/scheduled event indefinitely. Claimed events are sent individually, each success clears
failure_count, and a later failure rewinds the whole batch (gateway/kanban_watchers.py:562-592). The next tick re-sends the earlier event and clears the counter before the same later failure, so retry exhaustion and terminal unsubscribe may never occur. This violates the no-double-send/cursor/final-unsubscribe acceptance contract. -
[P2] Adapter loss does not release notifier ownership. Adapter presence is checked before election, but an elected owner with no active platforms only sleeps and retains the lease until its owner loop exits (
gateway/kanban_watchers.py:131-180,269-278). A healthy waiting gateway therefore cannot take over during a prolonged adapter outage.
Verification evidence:
- Exact head unchanged; draft; no CI checks reported, so the green-CI merge precondition is also unmet.
- Current-main merge tree is clean; branch and merge-effect
git diff --checkpass. - Focused Windows notifier/mixin/CLI/Telegram files: 47 passed. The broader Windows platform-base file has five pre-existing path-semantics failures (188 passed, 2 skipped), matching prior evidence.
- Canonical WSL run across seven relevant files: 242 passed.
- Ruff and Python compilation: pass.
- Independent read-only Codex review:
BLOCK, independently confirming the lease/profile-routing and retry-ordering failures.
No code, config, live messaging, or board-runtime mutation was performed by the review.
Summary
kanban.dispatch_in_gatewayis false, without starting or invoking the gateway dispatcherschedulednotificationsASK,WHY GATED,SCOPE,ROLLBACK,REPLY, and scheduledWINDOWfields survive beyond 160 charactersRoot cause
The notifier watcher shared the dispatcher ownership gate and returned before polling subscriptions when in-gateway dispatch was disabled. Blocked reasons were also sliced to 160 characters, and
scheduledwas absent from the notifiable event set.Verification
31 passedacross notifier, dispatcher-gate, watcher-mixin, CLI notification, and Telegram chunking targets188 passedacross the six related test filesruff checkon all changed Python files: passedpy_compileon all changed Python files: passedgit diff --check: passedtest_discord_connect.pygreen;test_discord_liveness.pyretains one pre-existingack_staleversuslatency_non_finiterace. Neither that test nor Discord production code is changed by this branch. CI remains the authoritative Linux full-suite gate.Deployment / rollback
00d29b89a725c4a3499d0532636d86a0bc867465gateway/kanban_watchers.pytests/gateway/test_kanban_notifier.py,tests/gateway/test_kanban_notifier_watcher_dispatch_gate.py,tests/hermes_cli/test_kanban_notify.pyhermes gateway restartkanban.dispatch_in_gateway: false. This PR does not change gateway config; regression coverage proves notifier delivery with both the canonical config gate and legacy env gate false whiledispatch_onceremains uncalled.00d29b89a725c4a3499d0532636d86a0bc867465(restoring prior commite361c5e20), reinstall/redeploy that revision, then runhermes gateway restart.Safety
No Telegram calls, live board status mutations, gateway config changes, extra daemons, cron dispatchers, or alternate claim/promote/spawn paths are introduced. The parent task owns the synthetic subscribed end-to-end test after review/deployment.