fix(gateway): quarantine corrupt kanban board DB to stop traceback spam (#26479) - #26753
Closed
Tranquil-Flow wants to merge 1 commit into
Closed
fix(gateway): quarantine corrupt kanban board DB to stop traceback spam (#26479)#26753Tranquil-Flow wants to merge 1 commit into
Tranquil-Flow wants to merge 1 commit into
Conversation
Collaborator
…am (NousResearch#26479) When the embedded kanban dispatcher hits a corrupt default-board DB (non-SQLite bytes, truncated WAL recovery), sqlite3 raises DatabaseError from `PRAGMA journal_mode=WAL` on every connect(). The previous `except Exception: logger.exception(...)` fired a full traceback once per dispatch interval forever — noisy logs, no recovery path, and the loop kept trying anyway. Catch sqlite3.DatabaseError specifically and dedupe by file signature: log one WARNING per (mtime, size) of the DB file, suppress further warnings until the operator changes the file. When the file changes the next tick retries — succeeds (clears quarantine) or logs once again. Other boards keep ticking, ready-queue health probes are untouched, and a successful tick clears quarantine for that board. State lives on the runner via getattr() lazy-init, same pattern as `_kanban_sub_fail_counts` in the notifier watcher.
Tranquil-Flow
force-pushed
the
fix/26479-kanban-corrupt-board-quiet
branch
from
May 16, 2026 04:45
2335fff to
5b2c8f9
Compare
Contributor
Author
|
Closing as graceful supersession by #26490 (@aqilaziz, opened 2026-05-15, ~1 day earlier). Both PRs detect corrupt kanban board DB and suppress traceback spam. Mechanisms differ slightly — @aqilaziz disables dispatch for the affected board until the DB file changes or gateway restarts; ours quarantines via a separate state path with broader test coverage (+237/-2 vs +152/-8). End-user outcome is equivalent for the issue. Going with theirs as the earlier-filed; happy to port any salvageable test patterns into #26490 if useful @aqilaziz. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #26479. The embedded kanban dispatcher's tick used `except Exception: logger.exception(...)`, so a corrupt `/kanban.db` produced a full traceback every `dispatch_interval_seconds` indefinitely.
Root cause
`apply_wal_with_fallback` only catches `sqlite3.OperationalError` (the WAL-on-NFS case), so `sqlite3.DatabaseError("file is not a database")` from the `PRAGMA journal_mode=WAL` on a non-SQLite file propagates up through `_kb.connect(board=slug)` into the tick. The tick's catch-all then logs a fresh traceback every interval.
Fix
Catch `sqlite3.DatabaseError` specifically in `_tick_once_for_board`, de-duplicate by file signature: keep a `dict[slug, (mtime, size)]` on the runner; log one WARNING per signature, suppress until the file changes, then retry once. A successful tick clears quarantine so future corruption logs again.
Related Issue
Fixes #26479
Type of Change
Scope notes (what was deliberately deferred)
How was this tested?
New regression test `test_gateway_dispatcher_quarantines_corrupt_kanban_db` in `tests/hermes_cli/test_kanban_core_functionality.py` drives the real `_kanban_dispatcher_watcher` across 3 ticks against a non-SQLite file, asserts exactly one WARNING and zero tracebacks. Then mutates the file (different bytes → different mtime+size) and asserts a fresh WARNING fires on the next run.
Results: 4/4 dispatcher-watcher tests pass; `test_kanban_notify.py` regression suite: 10/10.
Diff: `gateway/run.py +42/-1`, one new test (+106).
Checklist