fix(state): PASSIVE not TRUNCATE for all state.db checkpoints (#45383)

[lkz-de]

fix(state): PASSIVE not TRUNCATE for all state.db checkpoints (#45383)

Problem

SessionDB.close() ran PRAGMA wal_checkpoint(TRUNCATE). Every cron run_agent
opens and closes its own transient SessionDB, so on a busy fleet TRUNCATE — a
full WAL reset — fired many times an hour, racing the gateway's long-lived writer
on a large (multi-GB, ~500k-page) WAL database and tearing hot B-tree pages.

This is structurally the corruption this module's own periodic checkpoint was
already switched to PASSIVE to avoid (issue #45383 — see the _try_wal_checkpoint
docstring). But close() and two manual-maintenance paths kept using TRUNCATE.

Evidence

Diagnosed as the root cause of three state.db B-tree corruptions in Aug 2026:

• Damage localized to the hottest-written pages — gateway_routing and the
  sessions indexes — across all incidents. Random storage faults would not
  repeat the same victim set.
• Corrupt pages were whole zero-filled, still live, and off the freelist
  (secure_delete=OFF, so app deletes never zero pages) — the signature of
  zero-filled WAL-frame reads over hot page slots during a checkpoint/reset
  race, not disk or application SQL.
• Corruptions occurred with no crash/OOM and only after WAL was enabled —
  impossible for a plain SQL bug, and inconsistent with fsync/host faults (with
  no crash, the page cache returns correct data regardless of fsync honesty).

Fix

Route every checkpoint on the shared state.db through PASSIVE:

• close() (hermes_state.py)
• pre-VACUUM in vacuum() (hermes_state.py)
• post-optimize-storage (hermes_state_search.py)

PASSIVE never resets/truncates the WAL and never takes the exclusive checkpoint
lock, so a transient closer cannot lose a race with the live writer. The WAL is
instead bounded by journal_size_limit and the writer's natural
post-checkpoint reset.

TRUNCATE legitimately belongs on a sole-opener/quiescent connection (offline
maintenance); this PR does not try to detect that — PASSIVE is the safe
default. A future refinement could gate TRUNCATE on has_live_connection().

Relationship to existing PRs / issues

• fix(session): use PASSIVE wal checkpoint on close and pre-VACUUM #80346 (@webtecnica) independently proposes the same TRUNCATE→PASSIVE
  conversion for close() and the pre-VACUUM checkpoint, and closes close() still uses TRUNCATE checkpoint — #45383 recurs on the shutdown path #80255
  (the open issue tracking those two sites). Credit to that PR for identifying
  them first. This PR overlaps on those two sites and additionally converts the
  third remaining TRUNCATE on the shared state.db — the post-
  optimize-storage checkpoint in hermes_state_search.py — and adds the
  production corruption evidence above.
• fix(state): PASSIVE periodic WAL checkpoint to stop B-tree corruption (#45383) #64607 already converted the periodic _try_wal_checkpoint() to PASSIVE;
  fix(state): use PASSIVE WAL checkpoint in close() and pre-VACUUM paths (#45383) #72546/fix(state): use PASSIVE WAL checkpoint in close() and pre-VACUUM paths (#45383) #72549 were earlier closed attempts in this area.
• Maintainers should pick whichever vehicle is easier to land: if fix(session): use PASSIVE wal checkpoint on close and pre-VACUUM #80346 is
  preferred, I'm happy to reduce this PR to the hermes_state_search.py delta
  as a follow-up; if this PR is preferred, it covers all three sites against
  current main.

Also addresses #80255.

Companion config (not in this diff)

Because close() no longer truncates, deployments should bound the WAL: set
database.journal_size_limit (e.g. 64 MB). The plumbing already exists in
apply_database_pragmas.

Testing

• uv run --with pytest --with pytest-asyncio python -m pytest tests/test_wal_checkpoint_strategy.py tests/test_hermes_state.py -q -o 'addopts=' — 226 passed.

[webtecnica]

Thanks @lkz-de for the superset — our #80346 covered close + pre-VACUUM but missed the post-optimize-storage site, and your production corruption evidence is the decisive argument for PASSIVE-everywhere. Closed #80346 in favor of this one; the full TRUNCATE→PASSIVE conversion + WAL-race protection is exactly what #80255 needed. Happy to help with the merge or any follow-up.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

fix(state): PASSIVE not TRUNCATE for all state.db checkpoints (#45383)

The corruption rationale is convincing and the TRUNCATE-everywhere removal is consistent. Observations:

1. WAL bound depends on config — the updated docstrings state the WAL "is bounded by journal_size_limit", but that PRAGMA is only applied when the operator sets database.journal_size_limit in config (see _apply_database_pragmas, default None → not applied; SQLite's default limit is effectively unbounded). With TRUNCATE gone from every path, a long-lived gateway's WAL now stays at its high-water mark until a natural writer reset occurs (which requires no other attached readers). Consider either defaulting journal_size_limit in DEFAULT_CONFIG, or explicitly documenting in the PR/changelog that operators who care about WAL file size must set it.
2. close() going PASSIVE means the transient per-cron-run connections no longer shrink the WAL at all — an accepted tradeoff given the race, but a real disk-footprint regression for cron-heavy setups. A changelog note for operators watching state.db-wal size would be worth it.
3. Tests are thorough: the TrackingConnection verifies PASSIVE-before-VACUUM ordering, PASSIVE-after-optimize-VACUUM ordering, and that close() never TRUNCATEs; the existing read-only no-checkpoint guard still holds. The doc-comment in tests/test_wal_checkpoint_strategy.py was updated consistently with the new policy.