fix(dashboard): heal named-profile session stores behind the schema contract (OOF-76)

[shannonsands]

Problem (OOF-76)

Named-profile session stores (<home>/profiles/<name>/state.db) drift behind the dashboard's read surface silently: idle profiles' gateways never open them writable, so _reconcile_columns() never runs. Once a store misses newer read-surface tables/columns (system_prompts, sessions.last_activity_at/system_prompt_hash, ...), every cross-profile dashboard session route 500s forever — until a redeploy happens to force a writable open somewhere. Two production instances were forensically confirmed serving permanent 500s this way (one store stuck at schema v23) while readiness stayed green.

The false-green had a second cause worth calling out: the application schema_version integer legitimately lags SCHEMA_VERSION on FTS5-unavailable runtimes even when every regular table has converged, so any drift check keyed on that integer is wrong by design. The fix diffs the actual table contract instead.

Fix

hermes_state_common.py — new SCHEMA_SQL-derived contract:

• _state_schema_contract() builds a {table: {columns}} map by executing SCHEMA_SQL against an in-memory database (same technique as SessionSchemaMixin._parse_schema_columns), cached per process.
• state_schema_mismatches(conn) diffs a live store's regular tables against it. FTS virtual tables and the schema_version integer are deliberately out of scope. A store with zero tables is uninitialised (bootstrap's job), not stale; once any contract table exists, the full contract is required. Extra tables/columns are tolerated (reconcile only ever ADDs).

hermes_cli/web_server.py — shared heal-capable read-only opener _open_session_db_at_path():

• Staleness detected two ways: a fast hot-surface read probe (extended to the newest dashboard columns) plus the full contract diff, cached on SQLite's schema cookie (PRAGMA schema_version — the engine's DDL counter) so converged stores pay one PRAGMA read per open.
• On staleness: one serialized, idempotent writable reconcile per (store identity, stale cookie) state. Concurrent stale reads produce exactly one DDL heal. Deterministic heal failures (disk full, unfixable DDL) are never re-run as DDL on every poll; transient lock/busy contention un-marks itself and stays retryable.
• Store identity is (path, st_dev, st_ino) so a restore/recovery replacing the file in place gets fresh heal + validation state; a second drift later in the process lifetime gets its own heal attempt (keyed on the observed cookie) instead of 500ing until restart.
• Reopen after heal keeps full probing: an unhealable store fails visible at the route, never silently serves a degraded read surface, and never loops DDL.

hermes_cli/web_routers/profiles.py — both cross-profile aggregation routes (/api/profiles/sessions, /api/profiles/sessions/sidebar) now go through the shared opener instead of raw read-only SessionDB opens (the primary 500 path in the incident).

gateway/readiness.py — schema-aware probes:

• state_db probe now degrades on a readable-but-drifted default store (was readability-only — the exact false-green).
• New profile_state_dbs probe: bounded scan of named-profile stores (count cap and 1s wall-clock budget — each sqlite connect carries a 1s busy timeout), aggregate counts only (never profile names/paths), capped window rotates across polls so fleets larger than the cap get eventual coverage, truncation reported explicitly.
• The profile check is advisory: visible in checks but excluded from the restart-driving overall verdict. Restarting cannot heal an idle profile store (only the dashboard's heal-capable read path can), so folding it into the rollup would invite an unhealable-signal restart loop (OOF-39 class). Public /api/status keeps its default-store-only storage probe for the same reason.

Tests

• tests/state/test_state_schema_contract.py (new): contract contents, FTS exclusion, caching; converged/empty/foreign-only stores; lagging schema_version integer is NOT drift (the OOF-76 false-positive); missing table/column reporting; extra tables/columns tolerated.
• tests/gateway/test_readiness.py: drift degrades state_db; foreign-only store is drift; uninitialised stores stay ok; profile probe aggregate counts + no name leakage; bounded scan + truncation flag; window rotation reaches a stale store past the cap; advisory rollup (drifted profile store ⇒ overall still ok, check degraded); disk probe pinned (host sits above the 90% threshold — pre-existing environmental failure now mooted).
• tests/hermes_cli/test_web_server.py: idle named-profile heal for missing column AND missing table; sidebar route heal; repeated drift in one process heals again; 6 concurrent stale reads → exactly one writable reconcile, all 200; deterministic heal failure attempted once, transient lock failure retried and healed.

Validation

• Targeted: 167 passed / 1 skipped across web-server + readiness + contract suites; ruff clean.
• Full tests/gateway/: branch 10 failed / 4857 passed vs baseline (origin/main @ ff3793f) 12 failed / 4849 passed — zero branch-only failures; the branch additionally fixes the environmental readiness failure.
• Broad web/state/hermes_state comparison: failure sets identical to baseline (pre-existing test_dashboard_param_clamps + test_write_lock_patience flakes on both sides).

Fixes OOF-76.