Skip to content

fix(gateway): quarantine corrupt kanban board DB to stop traceback spam (#26479) - #26753

Closed
Tranquil-Flow wants to merge 1 commit into
NousResearch:mainfrom
Tranquil-Flow:fix/26479-kanban-corrupt-board-quiet
Closed

fix(gateway): quarantine corrupt kanban board DB to stop traceback spam (#26479)#26753
Tranquil-Flow wants to merge 1 commit into
NousResearch:mainfrom
Tranquil-Flow:fix/26479-kanban-corrupt-board-quiet

Conversation

@Tranquil-Flow

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #26479. The embedded kanban dispatcher's tick used `except Exception: logger.exception(...)`, so a corrupt `/kanban.db` produced a full traceback every `dispatch_interval_seconds` indefinitely.

Root cause

`apply_wal_with_fallback` only catches `sqlite3.OperationalError` (the WAL-on-NFS case), so `sqlite3.DatabaseError("file is not a database")` from the `PRAGMA journal_mode=WAL` on a non-SQLite file propagates up through `_kb.connect(board=slug)` into the tick. The tick's catch-all then logs a fresh traceback every interval.

Fix

Catch `sqlite3.DatabaseError` specifically in `_tick_once_for_board`, de-duplicate by file signature: keep a `dict[slug, (mtime, size)]` on the runner; log one WARNING per signature, suppress until the file changes, then retry once. A successful tick clears quarantine so future corruption logs again.

Related Issue

Fixes #26479

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature
  • 🔒 Security fix
  • 📝 Documentation update
  • ♻️ Refactor
  • 🧪 Tests only

Scope notes (what was deliberately deferred)

  • No automatic quarantine-and-rename or DB recreation. That's a destructive recovery decision; this PR limits itself to silence and surface area. The warning tells the operator to act.
  • Other boards keep ticking — `continue` past the bad slug; ready-queue health probes (`_ready_nonempty`) are unchanged.
  • No top-level `import sqlite3` — local import in the closure keeps the patch in one tight block.

How was this tested?

New regression test `test_gateway_dispatcher_quarantines_corrupt_kanban_db` in `tests/hermes_cli/test_kanban_core_functionality.py` drives the real `_kanban_dispatcher_watcher` across 3 ticks against a non-SQLite file, asserts exactly one WARNING and zero tracebacks. Then mutates the file (different bytes → different mtime+size) and asserts a fresh WARNING fires on the next run.

Results: 4/4 dispatcher-watcher tests pass; `test_kanban_notify.py` regression suite: 10/10.

Diff: `gateway/run.py +42/-1`, one new test (+106).

Checklist

  • I have read the contributing guidelines
  • My code follows the existing code style
  • I have added tests that prove my fix is effective
  • All new and existing tests pass locally

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have labels May 16, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #26490 which fixes the same issue (#26479 — corrupt kanban board DB causes repeated tracebacks). Both PRs add corruption detection and suppress repeated logging.

…am (NousResearch#26479)

When the embedded kanban dispatcher hits a corrupt default-board DB
(non-SQLite bytes, truncated WAL recovery), sqlite3 raises
DatabaseError from `PRAGMA journal_mode=WAL` on every connect(). The
previous `except Exception: logger.exception(...)` fired a full
traceback once per dispatch interval forever — noisy logs, no recovery
path, and the loop kept trying anyway.

Catch sqlite3.DatabaseError specifically and dedupe by file signature:
log one WARNING per (mtime, size) of the DB file, suppress further
warnings until the operator changes the file. When the file changes
the next tick retries — succeeds (clears quarantine) or logs once
again. Other boards keep ticking, ready-queue health probes are
untouched, and a successful tick clears quarantine for that board.

State lives on the runner via getattr() lazy-init, same pattern as
`_kanban_sub_fail_counts` in the notifier watcher.
@Tranquil-Flow
Tranquil-Flow force-pushed the fix/26479-kanban-corrupt-board-quiet branch from 2335fff to 5b2c8f9 Compare May 16, 2026 04:45
@Tranquil-Flow

Copy link
Copy Markdown
Contributor Author

Closing as graceful supersession by #26490 (@aqilaziz, opened 2026-05-15, ~1 day earlier). Both PRs detect corrupt kanban board DB and suppress traceback spam.

Mechanisms differ slightly — @aqilaziz disables dispatch for the affected board until the DB file changes or gateway restarts; ours quarantines via a separate state path with broader test coverage (+237/-2 vs +152/-8). End-user outcome is equivalent for the issue. Going with theirs as the earlier-filed; happy to port any salvageable test patterns into #26490 if useful @aqilaziz.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway kanban dispatcher should handle corrupted default board DB without repeated tracebacks

2 participants