Skip to content

fix: isolate kanban dispatcher sqlite board failures - #33929

Open
the24thLetter wants to merge 30 commits into
NousResearch:mainfrom
the24thLetter:fix/kanban-gateway-sqlite-quarantine
Open

the24thLetter wants to merge 30 commits into
NousResearch:mainfrom
the24thLetter:fix/kanban-gateway-sqlite-quarantine

Conversation

@the24thLetter

Copy link
Copy Markdown

Summary

  • isolate gateway-embedded Kanban dispatcher SQLite failures per board so one corrupt/unreadable board does not stop other board ticks
  • add classified degraded/quarantine diagnostics for corrupt, WAL/sidecar, disk I/O, and busy/locked board DB failures, with retry after the same DB fingerprint sits in quarantine
  • expose board dispatch health and fleet-wide active worker diagnostics through the Kanban dashboard API

Verification

  • python -m py_compile gateway/run.py hermes_cli/kanban_db.py plugins/kanban/dashboard/plugin_api.py tests/gateway/test_kanban_dispatch_quarantine.py tests/hermes_cli/test_kanban_db.py tests/plugins/test_kanban_worker_runs.py
  • env -u HERMES_KANBAN_DB -u HERMES_KANBAN_BOARD -u HERMES_KANBAN_REQUIRE_REVIEW_BEFORE_DONE -u HERMES_KANBAN_MERGE_CAPTAIN_PROFILE -u HERMES_KANBAN_WORKSPACES_ROOT python -m pytest tests/gateway/test_kanban_dispatch_quarantine.py tests/plugins/test_kanban_worker_runs.py tests/hermes_cli/test_kanban_notify.py -q -o 'addopts=' (31 passed)
  • env -u HERMES_KANBAN_DB -u HERMES_KANBAN_BOARD -u HERMES_KANBAN_REQUIRE_REVIEW_BEFORE_DONE -u HERMES_KANBAN_MERGE_CAPTAIN_PROFILE -u HERMES_KANBAN_WORKSPACES_ROOT python -m pytest tests/plugins/test_kanban_dashboard_plugin.py -q -o 'addopts=' (101 passed, 1 warning)
  • env -u HERMES_KANBAN_DB -u HERMES_KANBAN_BOARD -u HERMES_KANBAN_REQUIRE_REVIEW_BEFORE_DONE -u HERMES_KANBAN_MERGE_CAPTAIN_PROFILE -u HERMES_KANBAN_WORKSPACES_ROOT python -m pytest tests/hermes_cli/test_kanban_db.py::test_connect_rejects_tls_record_in_sqlite_header tests/hermes_cli/test_kanban_db.py::test_init_db_refuses_corrupt_existing_file tests/hermes_cli/test_kanban_db.py::test_connect_refuses_corrupt_existing_file tests/hermes_cli/test_kanban_db.py::test_locked_healthy_db_does_not_classify_as_corrupt tests/hermes_cli/test_kanban_db.py::test_connect_falls_back_to_delete_on_locking_protocol tests/hermes_cli/test_kanban_db.py::test_add_column_if_missing_is_idempotent_on_race -q -o 'addopts=' (6 passed)
  • live diagnostic smoke: simulated bad-board disk I/O failure and healthy default tick; /api/plugins/kanban/boards returned bad-board.dispatch_status.state=quarantined with last_error.kind=disk_io, while default.dispatch_status.state=healthy

Notes

  • Unknown non-SQLite/programming exceptions still go through normal exception logging; only classified board-local SQLite failure modes get degraded/quarantine handling.
  • Busy/locked DB failures degrade one tick but do not quarantine, so normal transient lock contention keeps retrying.

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins comp/gateway Gateway runner, session dispatch, delivery labels May 28, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the board-local diagnostic work. The core corrupt-board isolation is already present on current main (gateway/kanban_watchers.py:943-1055, from d37574775 and c94ad8981), but the disk-I/O/WAL diagnostics remain a distinct enhancement.

Problems

  • Blocking: gateway/run.py:5861 delegates to _dispatch_kanban_board_once, but that helper only forwards board, max_spawn, max_in_progress, stale_timeout_seconds, and failure_limit (gateway/run.py:1320-1328). It drops the existing default_assignee and max_in_progress_per_profile forwarding that current main retains at gateway/kanban_watchers.py:1016-1025. This regresses configured default assignment and per-profile concurrency caps.

Suggested changes

  • Preserve and test both configuration values through the extracted helper.
  • Salvage the watcher change onto gateway/kanban_watchers.py; current main moved this watcher there in 1c68f6f81.

Automated hermes-sweeper review.

Comment thread gateway/run.py
# and issue #21378. `_dispatch_kanban_board_once` also handles
# board-local SQLite quarantine so one bad board cannot break
# gateway-embedded dispatch for the others.
return _dispatch_kanban_board_once(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this helper call no longer forwards default_assignee or max_in_progress_per_profile. Current main passes both to dispatch_once; preserve them in _dispatch_kanban_board_once and add a regression test so configured default assignment and per-profile concurrency limits do not silently revert.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants