Skip to content

fix(kanban): tolerate races on legacy-DB column migration - #21874

Closed
li0near wants to merge 1 commit into
NousResearch:mainfrom
li0near:fix/kanban-migration-duplicate-column
Closed

fix(kanban): tolerate races on legacy-DB column migration#21874
li0near wants to merge 1 commit into
NousResearch:mainfrom
li0near:fix/kanban-migration-duplicate-column

Conversation

@li0near

@li0near li0near commented May 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a benign-but-noisy traceback emitted on every gateway start against a kanban DB created before the max_retries column was added:

sqlite3.OperationalError: duplicate column name: max_retries
  File "hermes_cli/kanban_db.py", line 1034, in _migrate_add_optional_columns
    conn.execute("ALTER TABLE tasks ADD COLUMN max_retries INTEGER")
ERROR gateway.run: kanban dispatcher: tick failed on board default

Root cause

_migrate_add_optional_columns reads PRAGMA table_info(tasks) once into a cols set, then issues a series of ALTER TABLE … ADD COLUMN. Two callers can race on the same DB file and both pass the pre-ALTER snapshot check before either ALTER lands; the loser raises duplicate column name. SQLite has no ADD COLUMN IF NOT EXISTS, so the snapshot guard is not enough.

The race fires today inside the embedded gateway dispatcher in _tick_once_for_board (gateway/run.py), which calls _kb.connect(board=slug) (runs the migration) immediately followed by _kb.init_db(board=slug) — and init_db deliberately busts the per-process _INITIALIZED_PATHS cache and re-runs the migration. It can also fire if a second process opens the same DB during gateway startup.

The error is harmless (by the time it fires, the column is already present and the next dispatcher tick succeeds), but the traceback is logged on every gateway start against a legacy DB and obscures real failures.

Fix

  1. Make the migration idempotent against itself. New _safe_add_column helper wraps every optional-column ALTER and swallows exactly sqlite3.OperationalError: duplicate column name, re-raising anything else. Applied to all 13 optional-column ALTERs on tasks and the one on task_events.
  2. Drop the redundant init_db() call in _tick_once_for_board. connect() already runs the schema + migration on first open per process; the explicit init_db() only existed to guarantee re-migration after a cache bust, which is no longer needed now that the helper is race-tolerant. Removes one source of the race entirely. The dropped call was already wrapped in except Exception: pass, so no callers depended on its result.

Test Plan

New tests in tests/hermes_cli/test_kanban_db.py:

  • test_migration_is_idempotent_against_itself — builds a legacy DB shape (pre-migration tasks schema), runs four concurrent migrators on it, asserts (a) no errors and (b) every expected column was actually added.
  • test_safe_add_column_reraises_unrelated_errors — confirms the helper does NOT mask other OperationalError cases (e.g. ALTER on a missing table) so real schema bugs still surface.

Test results

Regression test fails on unpatched upstream/main (proves the test catches the bug, not just passes vacuously):

$ git checkout upstream/main -- hermes_cli/kanban_db.py gateway/run.py
$ pytest tests/hermes_cli/test_kanban_db.py::test_migration_is_idempotent_against_itself \
        tests/hermes_cli/test_kanban_db.py::test_safe_add_column_reraises_unrelated_errors

FAILED test_migration_is_idempotent_against_itself
  AssertionError: assert [OperationalError('duplicate column name: tenant'), ...] == []
FAILED test_safe_add_column_reraises_unrelated_errors
  (helper does not exist on upstream/main)

The duplicate column name: tenant error is the same class of failure as the max_retries traceback in the bug report — the new test captures the general race, not just one column.

With the patch applied — full kanban-db suite passes:

$ pytest tests/hermes_cli/test_kanban_db.py -q
62 passed in 7.12s

Broader kanban surface — 378/379 pass:

$ pytest tests/hermes_cli/test_kanban_*.py tests/tools/test_kanban_tools.py -q
1 failed, 378 passed in 12.57s

The one failure is tests/hermes_cli/test_kanban_boards.py::TestCLI::test_boards_create_and_switch. I confirmed it reproduces on plain upstream/main with my changes reverted, so it is pre-existing and unrelated to this PR.

End-to-end: verified on a real legacy kanban.db that originally produced the traceback in the bug report. After the patch, the column is present, the dispatcher tick runs cleanly, and gateway.log is silent on restart.

Risk

Low.

  • _safe_add_column only suppresses the exact "duplicate column name" substring and re-raises everything else; the migration logic is otherwise unchanged. Regression-tested.
  • The dropped init_db() in _tick_once_for_board was already inside a bare except Exception: pass, so no caller could have depended on its return value or side effects beyond running the migration that connect() had already run on first open. Outer try/except in the dispatcher is unchanged.

Symptom: every gateway start against a kanban DB created before the
`max_retries` column logged a benign but noisy traceback:

    sqlite3.OperationalError: duplicate column name: max_retries
    ...
    ERROR gateway.run: kanban dispatcher: tick failed on board default

Root cause: `_migrate_add_optional_columns` reads
`PRAGMA table_info(tasks)` once into `cols`, then issues a series of
`ALTER TABLE … ADD COLUMN`. Two callers can race on the same DB file
and both pass the pre-ALTER `cols` snapshot check before either ALTER
lands; the loser raises `duplicate column name`. The race actually
fires today inside the embedded gateway dispatcher, which calls
`_kb.connect(board=slug)` (runs migration) immediately followed by
`_kb.init_db(board=slug)` (deliberately busts the per-process cache
and re-runs the migration). It can also fire if a second process opens
the same DB during gateway startup. SQLite has no
`ADD COLUMN IF NOT EXISTS`, so the snapshot guard is not enough.

Fixes:
1. Wrap every optional-column ALTER in a new `_safe_add_column` helper
   that swallows exactly `duplicate column name` and re-raises anything
   else. Migration becomes idempotent against itself for any caller.
2. Drop the redundant `init_db()` call in
   `_tick_once_for_board` (`gateway/run.py`). `connect()` already
   runs the schema + migration on first open per process; the explicit
   `init_db()` only existed to guarantee re-migration after a cache
   bust, which we no longer need now that the helper is race-tolerant.

Tests:
- `test_migration_is_idempotent_against_itself` builds a legacy DB
  shape and runs four concurrent migrators; without the fix at least
  one would raise. Asserts no errors and that every expected column
  was actually added (proves the helper didn't silently drop ALTERs).
- `test_safe_add_column_reraises_unrelated_errors` confirms the
  helper does not mask other `OperationalError` cases (bad SQL,
  missing table) so real schema bugs still surface.

Verified end-to-end on a real legacy `/opt/data/kanban.db` upgraded
through this code path: the column is present, the dispatcher tick
runs cleanly, no traceback in gateway.log.
@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #22994 (rebase) — your gateway-side double-init fix shipped on main with your authorship preserved as the committing author. Salvage widened the fix to the second redundant call site (the notifier watcher had the same pattern) and added regression tests so the bug can't silently come back. AUTHOR_MAP entry added so you'll show up in release notes. Thanks @li0near!
#22994

@teknium1 teknium1 closed this May 10, 2026
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have labels May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants