Skip to content

fix(kanban): wrap ALTER TABLE ADD COLUMN with race-safe helper - #21628

Closed
baocin wants to merge 1 commit into
NousResearch:mainfrom
baocin:fix/kanban-safe-add-column
Closed

fix(kanban): wrap ALTER TABLE ADD COLUMN with race-safe helper#21628
baocin wants to merge 1 commit into
NousResearch:mainfrom
baocin:fix/kanban-safe-add-column

Conversation

@baocin

@baocin baocin commented May 8, 2026

Copy link
Copy Markdown
Contributor

The kanban DB migration in _migrate_add_optional_columns reads a single
PRAGMA table_info snapshot at entry, then issues multiple ALTER TABLE
ADD COLUMN statements guarded by that snapshot. When two gateway
processes restart concurrently (e.g. --replace takeover + cron job
tick), the PRAGMA can be stale by the time a given ALTER TABLE
executes, causing:

sqlite3.OperationalError: duplicate column name: <name>

This has been hit in production on the consecutive_failures column.
The same race will fire for any new column added to the migration in
the future.

Add _safe_add_column() which tries the ALTER TABLE and silently
ignores duplicate column name — another process already added it.

Changes:

  • New helper: _safe_add_column(conn, table, col_def) with
    try/except for OperationalError, only re-raising non-duplicate errors.
  • All 12 ALTER TABLE ADD COLUMN calls in _migrate_add_optional_columns
    now go through _safe_add_column.
  • The task_events.run_id ALTER TABLE also uses the helper.
  • Updated the NOTE docstring to reflect the new guard strategy.

The kanban DB migration in _migrate_add_optional_columns reads a single
PRAGMA table_info snapshot at entry, then issues multiple ALTER TABLE
ADD COLUMN statements guarded by that snapshot. When two gateway
processes restart concurrently (e.g. --replace takeover + cron job
tick), the PRAGMA can be stale by the time a given ALTER TABLE
executes, causing:

  sqlite3.OperationalError: duplicate column name: <name>

Add _safe_add_column() which tries the ALTER TABLE and silently
ignores 'duplicate column name' — another process already did it.
Wraps every ADD COLUMN call in the migration so the pattern is safe
regardless of which columns are added in the future.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins labels May 8, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #21529 (same fix: wrap ALTER TABLE ADD COLUMN in try/except for duplicate column error). Also competes with #21021. Fixes #21503 and #21374.

@teknium1

Copy link
Copy Markdown
Contributor

Closing as redundant — the same race-safe migration helper landed on main via PR #22627 (commit 7869838, by @wesleysimplicio) on May 9. The current _add_column_if_missing() does exactly what your _safe_add_column() does: tries the ALTER, catches OperationalError, swallows only "duplicate column name", re-raises everything else.

We also followed it up with PR #22994 which removed the redundant double-init pattern in the gateway watchers (connect() followed by init_db()), which was the upstream cause of the cross-process race firing in the first place. Together those two fixes should fully address the production incident you cited.

Thanks for the clean writeup @baocin — the diagnosis was spot-on.

References:

@teknium1 teknium1 closed this May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants