fix(dashboard): stop checkpoints from read polling

[joelbrilliant]

Summary

• open dashboard session read paths with SQLite read-only connections
• skip wal_checkpoint(TRUNCATE) when closing read-only SessionDB connections
• preserve writer close checkpointing, WAL mode, FTS and trigram search, and macOS durability safeguards

Root cause

The desktop dashboard Sessions page polls /api/sessions every five seconds. That GET path created a writable SessionDB, then SessionDB.close() unconditionally requested a truncate checkpoint.

When another process had pending WAL frames, the read-only dashboard poll forced those frames through a checkpoint even though canonical session data had not changed. This matched the repeatable WAL truncation and disk-write bursts observed from the always-running dashboard backend.

Impact

Read-only desktop polling no longer requests SQLite checkpoints. Dashboard mutation endpoints remain explicitly writable, while writer shutdown retains its existing truncate checkpoint and durability behaviour.

Applicability

This fix is not tied to one operator, database size, profile name, install path, tunnel or service manager. It applies to any Hermes dashboard reading a WAL-backed state.db. The default store and named profile stores resolve through Hermes paths, while the regression tests run against isolated temporary homes and disposable databases.

Verification

• scripts/run_tests.sh tests/test_web_server_sessiondb_eventloop.py tests/test_hermes_state.py tests/hermes_cli/test_web_server.py tests/hermes_cli/test_web_server_session_search.py tests/gateway/test_session_api.py
  • 858 passed after the unrelated macOS PTY environment test passed on retry
  • the CI regression file passed 2 of 2 tests directly
• ruff check .
• repeated public GET reproduction against a disposable WAL database
  • HTTP statuses: 200, 200, 200
  • WAL bytes: 840512 before and 840512 after
  • data_version: 2 before and 2 after
  • canonical counts: (1, 1) before and (1, 1) after

No WAL, synchronous, FTS, checkpoint full-fsync, or macOS corruption safeguards are disabled.

---

Update 2026-07-20 - pushed 9b246615d: self-healing for the read-only open path. Stale-schema stores (predating sessions.archived), zero-byte store files, and racing first-load polls now trigger one writable healing open through the existing init/reconcile/repair machinery, then reopen read-only. The happy path stays read-only end to end with no checkpoint.

Testing for the update:

• Platform: macOS (Darwin 25.5.0, arm64), SQLite 3.53.0. Local interpreter is Python 3.14.4, above the declared requires-python <3.14; the 4 failures below are environment-related and reproduce identically on the base commit, so they are unrelated to this change. Windows/Linux not exercised locally; nothing in the change is platform-specific (stdlib threading.Lock, Path.stat, sqlite error-message classification already used elsewhere in hermes_state).
• python3 -m pytest tests/hermes_cli/test_web_server.py tests/hermes_cli/test_web_server_session_search.py tests/test_web_server_sessiondb_eventloop.py -q: 460 passed, 4 failed (TestPtyWebSocket x3, TestNewEndpoints::test_blueprint_instantiate_creates_job; identical failures on the base commit in this environment).
• python3 -m pytest tests/test_hermes_state.py -q: 385 passed.
• The three new hardening tests were first run against the unhardened helper and all fail there (stale-schema store 500s with no such column: archived via list_sessions_rich; zero-byte store and concurrent fresh-store reads 500 with no such table: sessions), then pass with the hardened helper.
• test_get_sessions_poll_preserves_pending_wal passes unchanged: across three read polls the -wal byte size, PRAGMA data_version, and row counts are identical, which also proves the healing path stays cold on the happy path (a healing open would checkpoint on close and fail the byte check).

Overlap with #60885: that PR guards SessionDB.close() so read-only connections never request a WAL checkpoint, which this PR also carries in its hermes_state.py hunk. Beyond the shared close() guard, this PR classifies every _open_session_db_for_profile call site with an explicit read_only access mode, converts the dashboard read endpoints (/api/sessions, stats, search, messages, export, analytics, cron runs) to genuine mode=ro opens with self-healing for fresh/stale/racing stores, and adds a WAL-preservation regression test. The two changes are compatible in either order: if #60885 lands first, this PR's close() hunk merges down to a near no-op; if this one lands first, #60885 can drop its overlapping close() change and keep whatever scope remains. No sequencing requirement from this side, maintainers' choice.

[alt-glitch]

This was generated by AI during triage.

Related to #60885: both prevent read-only dashboard activity from checkpointing WAL, while this PR also classifies the dashboard's SessionDB read and mutation endpoints explicitly. The overlapping but broader scope needs a maintainer choice.

[joelbrilliant]

Thanks, agreed that the two PRs overlap on the read-only close guard.

The runtime distinction here is that /api/status was already read-only on current main, and manual status requests did not independently reproduce the burst. The repeatable trigger was the Sessions page's five-second /api/sessions poll. That path still used the writable helper, then closed the connection after every request. With pending WAL frames, it produced the measured checkpoint and truncate burst.

This PR adds a public-route WAL regression and makes access mode explicit across the session dashboard helpers, while mutation paths assert read_only=False. It does not include #60885's MCP startup changes.

If #60885 lands first, the endpoint access-mode classification and public /api/sessions regression remain the non-overlapping part here. Maintainers can choose the preferred sequencing without losing that coverage.

[joelbrilliant]

Controlled runtime verification is complete on macOS using this branch with the installed dashboard assets.

Before restart, the registered dashboard process wrote 19,677,184 bytes in 30 seconds. Four write intervals were nonzero, and the largest one-second burst was 6,569,984 bytes. The zero-length WAL mtime changed ten times.

After restart, a five-minute sample issued 61 remote /api/sessions polls through Cloudflare and the API proxy:

• 61 HTTP 200 responses
• 0 remote errors
• 0 write bursts at or above 1 MiB
• 131,072-byte largest one-second write
• 2,072,576 total process bytes written over five minutes
• WAL stayed at zero bytes with one mtime change

The remote dashboard session list remained available throughout the sample. The recurring checkpoint and truncate write pattern did not recur.

[joelbrilliant]

Pushed 9b246615d, hardening the read-only open path in _open_session_db_for_profile against three edge cases found in an internal adversarial review:

• Stale-schema stores: read-only opens skip _reconcile_columns(), so a store predating sessions.archived 500'd on every dashboard read until something opened it writable. A failed schema probe now triggers one writable healing open through the existing init/reconcile/repair machinery, then reopens read-only; a second failure propagates unchanged.
• Bootstrap race: concurrent first-load polls raced sqlite file creation, and the losers opened mode=ro against a partially initialised store. The one-time bootstrap is now serialised under a module lock with a re-check inside it.
• Zero-byte state.db (crashed first boot) passed the exists() guard and 500'd permanently; it is now treated as missing and bootstrapped.

The happy path stays read-only end to end, no writable open and no checkpoint, which test_get_sessions_poll_preserves_pending_wal still proves: WAL byte size and data_version are unchanged across polls (a healing open would checkpoint on close and fail the byte check). The three new regression tests fail against the unhardened helper and pass with it.

PR body updated with the test matrix and a cross-reference to #60885.

[teknium1]

Thanks for the WAL-preservation work. The core premise remains live: current GET /api/sessions opens a writable DB at hermes_cli/web_server.py:4770, while SessionDB.close() performs wal_checkpoint(TRUNCATE) at hermes_state.py:2229.

Problems

• The read-only conversion at diff RIGHT line 4245 conflicts with current auto-archive. GET /api/sessions invokes _maybe_auto_archive_for_profile at hermes_cli/web_server.py:4775; that path writes session/archive metadata through hermes_state.py:7892-7899. The existing contract is covered by tests/hermes_cli/test_web_server.py:2229-2265.
• The proposed schema probe does not include sessions.pinned. Current listing requests pinned backfill at hermes_cli/web_server.py:4801, and its query requires s.pinned at hermes_state.py:4956.

Suggested changes

• Run enabled auto-archive through a dedicated writable maintenance open before reopening read-only for the list response.
• Extend stale-schema healing and tests for the current pinned column.

This is an automated hermes-sweeper review.

[joelbrilliant]

Thanks, both findings were valid. I rebased onto current main and pushed b8249b40e.

Auto-archive now runs through a dedicated writable maintenance connection, which closes before the list query opens read-only. The stale-store probe now covers both sessions.archived and sessions.pinned, including a legacy missing-pinned regression.

The focused state, dashboard, search and event-loop suite is 254 passed, with ruff clean. I also added the repository’s requires_wal gate after CI correctly exercised its SQLite runtime without WAL.

[joelbrilliant]

The shared Vercel fixture fix has now landed on main via #74576 as 8eb06e75b, so I closed my duplicate #74586.

I rebased this PR onto the corrected main and pushed b30d6d9a7. CI run 30514242061 is fully green. The earlier WAL-specific failure was fixed with the requires_wal marker and did not recur.

[kshitijk4poor]

Merged via #76895 — thank you @joelbrilliant. Both your commits were cherry-picked, so you remain the author in git history.

This fixed two real bugs we verified side-by-side before merging: dashboard read polling truncating another writer's pending WAL to 0 bytes on every poll (your read_only connections + checkpoint-skip-on-RO-close fix it exactly), and read-only FTS search silently returning nothing (your capability probing restores both the FTS5 and CJK trigram paths). Your test suite — WAL preservation, auto-archive maintenance writer, fresh/zero-byte stores, stale-schema healing, 8-thread concurrent first-load, and the AST access-mode invariant — all held up.

The salvage added one fix commit on top: the new RO FTS probe could raise DatabaseError on a malformed store and leak a tracked connection (blocking the forensic backup in the writable heal that follows); fixed with close-then-reraise mirroring your own _open_probed cleanup, plus a regression test. Everything else is your work verbatim.