fix(kanban): probe integrity_check in read-only immutable mode to avoid WAL checkpoint race

[zhangtaibo]

Summary

_guard_existing_db_is_healthy runs PRAGMA integrity_check on every new process's first connect(). Previously the probe opened the DB in read/write mode, which under SQLite WAL mode opens the -wal/-shm sidecars and can trigger a WAL checkpoint. Under a worker stampede this races the gateway (the active writer) and corrupts indexes.

This PR switches the probe to read-only immutable=1 URI mode so SQLite skips WAL/SHM entirely — no sidecars opened, no locks taken, no checkpoint triggered.

Root cause

Recurring wrong # of entries in index idx_events_* corruption, nightly at cron-burst moments (02:00). Pattern observed across ~11 incidents:

1. 02:00 cron spawns N worker subprocesses + web-ui spawns hermes kanban watch children.
2. Each new process hits _guard_existing_db_is_healthy on first connect() (_INITIALIZED_PATHS is per-process, so the cache does not help across processes).
3. Each probe opens the DB r/w → opens -wal/-shm (O_RDWR|O_CREAT) → may trigger a WAL checkpoint.
4. N concurrent checkpoints race the gateway dispatcher (the single active writer) → WAL inconsistency → index corruption.
5. Recovery cascade: corruption detected → backup + REINDEX by N processes simultaneously → re-corrupts, generates 5–10 backups/minute.

Fix

Add _sqlite_connect_readonly(path) that opens via file:{path}?immutable=1 URI. SQLite treats the file as immutable: it does not open -wal/-shm, does not take a WAL lock, does not checkpoint.

_guard_existing_db_is_healthy now uses the readonly probe. Structural integrity is fully verifiable from the main DB file alone (the post-checkpoint stable state). WAL contents are data, not structure; any deferred WAL-resident corruption surfaces on the next checkpoint by the gateway (the single writer) rather than racing it.

Verification (strace)

	Files opened	Writes
Before (r/w probe)	kanban.db + kanban.db-wal + kanban.db-shm (latter two O_RDWR|O_CREAT)	fsync/fdatasync on WAL
After (readonly probe)	kanban.db only	none

• 10 concurrent readonly probes complete cleanly; -wal/-shm never created.
• Intentionally corrupted DB still detected: sqlite refused to open file: database disk image is malformed.
• Full connect() → query → write flow unchanged (writes still go through the normal r/w _sqlite_connect).

Relationship to existing issues

• Closes [Bug] Kanban DB (kanban.db) index corruption under concurrent multi-process access in WAL mode #34385 (recurring index corruption under concurrent multi-process WAL access) — this PR addresses the probe-checkpoint-vs-writer race.
• Complements bug: kanban DB corruption under high concurrent-worker load -- workers need per-write serialization #53819's writer-serialization proposal (file-scoped fcntl.flock on writes) — that targets a different race (multi-writer contention). Both can coexist; this PR is the smaller, more surgical fix for the probe-triggered races we observe nightly.
• Does not require the dispatcher hardening from Dispatcher resilience: deterministic spawn-crash loop, transient SQLite I/O latches dispatch off, archived-parent silently promotes children #30417 / [Bug]: Kanban: rapid worker spawn-crash loop (sub-2s/crash) corrupts board SQLite B-tree before failure_limit trips #30896 — those address spawn-crash loops; this addresses the probe race alone.

Test plan

• python3 -c "import ast; ast.parse(open('hermes_cli/kanban_db.py').read())" — syntax OK
• strace: readonly probe opens 1 file, no write/fsync/fdatasync syscalls
• strace: r/w probe (before) opened 3 files including O_RDWR|O_CREAT on -wal/-shm
• 10-process concurrent readonly probe stress — no WAL/SHM artifacts
• Malformed DB still raises KanbanDbCorruptError with backup
• CI: existing kanban_db test suite passes (no new failures expected — probe result is identical ok/malformed either way)

🤖 Generated with Claude Code

[sergeikabuldzhanov]

Two concerns from testing this on a live deployment (WAL, multi-process worker fleet), plus a data point.

1. The probe race this PR targets may not exist: probes are already serialized by _cross_process_init_lock.

_guard_existing_db_is_healthy is only reachable from connect() inside the with _cross_process_init_lock(path): block — an flock on kanban.db.init.lock whose docstring says "Serialize first-connect WAL/schema/integrity setup across processes." So under a worker stampede the probes run one at a time, not concurrently. What remains is a single probe-triggered checkpoint racing the gateway writer — and checkpoint-vs-writer is a core supported WAL operation that SQLite serializes internally with the CHECKPOINTER/WRITER locks. On a local filesystem that pairing doesn't corrupt indexes; if it did, WAL itself would be broken. The nightly idx_events_* corruption likely has a different producer.

2. immutable=1 on a live DB can misreport a healthy DB as corrupt — recreating the exact backup-cascade symptom.

Immutable mode takes no read locks and ignores the WAL entirely. But connect() sets PRAGMA wal_autocheckpoint=100, so the active writer checkpoints into the main DB file constantly. A lock-free probe reading the main file mid-checkpoint can observe torn/half-written pages → integrity_check returns "malformed" → KanbanDbCorruptError + .corrupt.*.bak + quarantine, on a perfectly healthy DB. That's a new false-positive source in the exact code path this PR is trying to calm down. The SQLite docs are explicit that immutable=1 on a file that is being written yields undefined results — "database corruption may result" on reads.

Alternative that keeps the PR's goal (probe never writes the live DB/sidecars) without the torn-read hazard: open mode=ro (honors locks, reads committed WAL content), Connection.backup() into a private temp file next to the DB, run integrity_check on the snapshot. We've been running that variant under a stress test — 300 probes against a writer churning commits + passive checkpoints — with zero false corruption verdicts and no WAL/SHM artifacts; the same harness makes a lock-free main-file read observably inconsistent.

Data point on the symptom this PR cites: we hit the recurring-corruption + backup-cascade shape too (251 .bak files in one incident). Root cause there was real table b-tree damage (2nd reference to page 5486 on the task_comments table root), verified via a locked .backup snapshot — not a probe artifact, and probe mode (r/w vs ro) couldn't have prevented it. The cascade volume came from every worker spawn re-probing the same genuinely-corrupt file. A complementary mitigation worth considering: rate-limit/skip re-backup when a corrupt verdict for the same content hash already exists (the content-addressed .bak naming already dedupes the copies; the repeated probe+raise loop is what floods logs and spawns).

[sergeikabuldzhanov]

Follow-up to my earlier comment — after running the snapshot-probe variant in production for a day, I think both that variant and this PR are patching the symptom at the wrong level. The deeper issue: integrity checking is a global, periodic concern implemented as a per-process startup ritual.

Every process that ever touches a board — dispatcher, each spawned worker, every CLI invocation, each dashboard — runs a full-DB integrity_check on first connect, serialized through the unbounded first-connect flock, and each is empowered to declare corruption, write a .bak, and quarantine. Consequences:

• Amplification, not protection. When corruption is real, N processes independently re-probe the same corrupt bytes. We measured this: one genuinely corrupt board (real b-tree damage, 2nd reference to page 5486) × ~12 workers respawning = 251 .corrupt.*.bak files and a backup/quarantine stampede. The content-addressed naming dedupes copies of identical bytes, but the live WAL keeps advancing between probes, so hashes differ and backups multiply anyway.
• No prevention. Corruption is produced in the write path (or below SQLite — kill-timing/fsync/memory-pressure). A startup probe can't stop it, and a worker whose DB goes bad mid-run gets SQLITE_CORRUPT from ordinary queries regardless.
• Redundant by construction for workers. A dispatcher-spawned worker's board was just verified by the dispatcher that spawned it. HERMES_KANBAN_TASK is already in the worker's env, so the skip is one line.

Where the pieces belong:

1. Full integrity check → the board owner, once. The gateway dispatcher already holds a singleton per-board lock; probe there at startup + on a timer. Any probe flavor is cheap when exactly one process runs it.
2. Worker/CLI first-connect → the existing 100-byte header check only (keeps catching the Interrupted OpenAI/httpx request thread survives across turns and writes TLS record bytes to unrelated file descriptors on delayed close #29507 TLS-overwrite shape). Skip the full probe when HERMES_KANBAN_TASK is set.
3. Backup/quarantine decision → owner-only. Real corruption then yields one .bak and one quarantine instead of a fleet doing forensics on the same bytes.

We're running (2) + the snapshot probe as a local patch now — happy to PR it if maintainers prefer that shape over immutable=1. Point (1) is a slightly bigger refactor (move the probe into the dispatcher tick) but is what actually removes the stampede class rather than making each stampeding probe safer.

[sergeikabuldzhanov]

Correction to my first comment, from completed forensics on our incident: I claimed the first-connect probes are strictly serialized by _cross_process_init_lock. That's wrong under contention — the lock acquisition is bounded (~10s) and then falls through to probing without the lock (kanban init lock ... not acquired within 10s — proceeding, hermes_cli.kanban_db). We observed 8 such timeouts during a dispatcher burst (9 workers spawned in one tick), producing 236 corrupt verdicts in two minutes. So a probe stampede IS real — the PR's motivation stands on that point, and I retract that part of my critique.

The torn-read objection to immutable=1 stands, and our forensics sharpened it. Full timeline from 251 content-addressed quarantine snapshots:

1. Corruption was born in the WAL view during normal writes (first bad verdict 12:08; that snapshot's main file integrity-checks ok today — only the WAL view was inconsistent, and a row written 6s before a probe was already missing from its index).
2. A checkpoint materialized it into the main file ~20 min later (first main-file-corrupt snapshot 12:28:58).
3. The probe stampede (lock fall-through above) turned it into a 251-backup cascade.
4. Continued live writes on corrupt index structures deepened it from index-count damage to table b-tree damage (2nd reference to page N) over the next 20 min.

Notably: no OOM kills, no process crashes, no gateway restarts in the onset window — but the host was observably stalled (trivial SQLite count queries exceeded 10s) with the gateway service cgroup (daemon + its spawned worker subprocesses and their build toolchains — the daemon itself idles at ~60MB) at 13.5G RSS + 11.8G swap in a 16G WSL2 VM. The initial WAL-page producer is undecidable from our evidence because of a capturable gap worth fixing regardless of probe mode: _backup_corrupt_db swallows WAL/SHM sidecar copy failures silently (except OSError: pass), so the two earliest — most diagnostic — backups have no WAL sidecars.

Concrete suggestions that fall out, orthogonal to probe mode:

• On init-lock timeout, skip the probe (a holder is already probing) instead of proceeding locklessly — one-line fix that kills the stampede class outright.
• Log sidecar backup failures loudly and retry once — the WAL bytes are the difference between a root-caused incident and an undecidable one.

[sergeikabuldzhanov]

Follow-up: the architectural form of the feedback above is now a standalone proposal — #62009 (maintenance operations owner-only, data reads/writes unchanged). It narrows this PR's question rather than competing with it: with a single owner probing on a schedule, whichever probe mode this PR settles on runs in one process instead of every connecting process.

[teknium1]

Thanks for narrowing the first-connect probe path. The contention premise remains on current main: _cross_process_init_lock deliberately proceeds without its cross-process lock after its bounded timeout (hermes_cli/kanban_db.py:1288-1292, 1387-1394), and connect() then calls the full guard (hermes_cli/kanban_db.py:1738-1745).

Problems

• immutable=1 is unsafe for this live board. The proposed URI at hermes_cli/kanban_db.py:1354 tells SQLite to skip locking and change detection; SQLite documents that if such a file changes, reads can return incorrect results or SQLITE_CORRUPT (SQLite URI documentation). Normal connections continue to enable WAL and write the board (hermes_cli/kanban_db.py:1722-1731, 1758-1770).
• The guard currently intentionally opens read/write so SQLite can recover/checkpoint a healthy WAL or hot journal before declaring corruption (hermes_cli/kanban_db.py:1623-1635). Replacing it with a probe that intentionally excludes WAL/SHM can miss a WAL-resident integrity failure.
• The PR changes only production code; it adds no concurrent-writer/WAL probe regression test.

Suggested changes

• Keep a WAL-consistent probe rather than immutable=1, and add a real multiprocess writer/probe regression.
• Consider the owner-only maintenance direction in #62009 so the bounded-lock fall-through cannot create a fleet of independent probes and quarantines.

Automated hermes-sweeper review.

[teknium1]

immutable=1 asserts this file cannot change and disables SQLite locking/change detection, but this guard runs against an actively written WAL board. SQLite documents incorrect results or SQLITE_CORRUPT if an immutable-marked file changes. Please do not use this mode for the live integrity decision; preserve a WAL-consistent view and cover it with a concurrent writer/probe regression.

[teknium1]

Closing: immutable=1 on a live-WAL database is documented-unsafe (reads can return SQLITE_CORRUPT or stale data). The underlying idea — don't checkpoint from the probe — is right and is handled differently now that #68654 moved checkpointing to the dispatcher tick. A mode=ro (non-immutable) probe PR would be welcome.