fix(readiness): surface unrepaired state.db corruption on the state_db probe (OOF-106) - #87052
shannonsands wants to merge 2 commits into
Conversation
…b probe (OOF-106) The state_db readiness probe only read sqlite_master (page 1), so page-level corruption in the sessions table b-tree kept /api/status green — in the OOF-106 incident a page-corrupt state.db reported "ok" for 10+ days while sessions silently failed to persist and repair attempts churned in the background. Two additions, both bounded and read-only: * Consult the persistent repair-attempt ledger (NousResearch#86747's state.db.repair-attempts.json): when its fingerprint still matches the current file bytes, automatic repair has already failed on this exact database and the probe reports "degraded / unrepaired corruption" without opening the DB at all. Import-light (reads the sidecar JSON directly); malformed/stale ledgers read as "no signal". * Deepen the read probe one step past page 1: fetch a single row from the sessions table (SELECT * so the table b-tree, not an index, is walked). Still O(1) pages, still read-only, catches root-page damage of the table every session write depends on. Tests: ledger-match degrades, stale ledger ignored, garbage ledger ignored, and a real zeroed-root-page corruption fixture that passes the schema-only probe but fails the deepened one. Complements NousResearch#86747 (bounded repair loop + backup dedup), which covers the repair/backup side of OOF-106 but leaves the readiness surface falsely green. Supersedes the readiness part of NousResearch#82940.
fix(readiness): surface unrepaired state.db corruption on the state_db probe (OOF-106) Good fix closing a real false-green gap, with strong tests (ledger match / stale / malformed / zeroed root page). Points:
|
…l writer Review follow-up (NousResearch#87052): the probe hand-parses the sidecar ledger that hermes_state writes — an unenforced cross-module contract. New parity test drives hermes_state._record_repair_outcome() directly and asserts the probe degrades on a recorded failure and recovers when the writer clears the ledger, so any schema drift (filename, fingerprint format, failed_attempts key) fails CI instead of silently re-opening the false-green gap. Also: docstring note on size:mtime_ns granularity (coarse-mtime filesystems can hold a stale match until the next write — pessimistic, never falsely green) and a stray blank line in the test file.
|
Thanks — addressed in 1. Ledger contract (taken, highest value): added 2. 3. Fingerprint granularity (taken): docstring now notes the coarse-mtime false-match case and why it's acceptable — the failure mode is a briefly pessimistic signal until the next successful write, never a false green. 4. Blank lines (taken): fixed. |
Problem
The
state_dbreadiness probe only readssqlite_master(page 1). Page-level corruption in thesessionstable b-tree sails past it, so/api/statusstays green while session persistence is broken. In the OOF-106 incident (ashriel-fox-cloud-5148), a page-corruptstate.dbreported "ok" for 10+ days while repair attempts churned in the background and accumulated 505 malformed backups.#86747 fixed the repair/backup side (bounded attempt ledger + backup dedup/retention) — this PR closes the remaining gap: the readiness surface.
Changes (both bounded, read-only)
Consult the persistent repair-attempt ledger (
state.db.repair-attempts.jsonfrom state.db corruption cascade: repair path loops forever, accumulating 89GB of dead backups #86747): when its fingerprint (size:mtime_ns) still matches the current file bytes, automatic repair has already failed on this exact database and nothing has changed since. The probe reportsdegraded / "unrepaired corruption"without opening the DB. Import-light — reads the sidecar JSON directly, nohermes_stateimport. Malformed/stale/absent ledgers read as "no signal" (never degrade a healthy DB, never crash the probe).Deepen the read probe one step past page 1: fetch a single row from
sessions(SELECT *deliberately, so the table b-tree is walked rather than an index). Still O(1) pages, still read-only underPRAGMA query_only, catches root-page damage of the one table every session write depends on. Guarded for pre-schema databases.Tests
degraded / unrepaired corruptionokok, no crashsessionsroot page (schema page intact — passes the old probe, fails the new one) →degradedtests/gateway/test_readiness.py: 5 passed + 1 pre-existing failure (test_collect_runtime_readiness_reports_healthy_local_runtime) confirmed failing identically on clean main.ruffclean.Relationship to #82940
Supersedes the readiness portion of #82940, rebuilt against #86747's ledger format. The rest of #82940 (backup dedup, retention cap, persistent repair guard) was independently landed by #86747, so #82940 is being closed.
Linear: OOF-106