fix(state): bound the state.db repair loop and stop 89GB dead-backup accumulation - #86867
Merged
Merged
Conversation
…ulation (#86747) A corruption class the repair strategies cannot heal (b-tree page damage) failed repair_state_db_schema on every process start, forever: _claim_repair_attempt's in-memory set only bounds one process, so each restart re-ran the full surgery AND took a fresh ~900MB forensic backup of the same damaged bytes — 105 attempts / 89GB of dead state.db.malformed-backup-* files over 11 days in the reporting install. Three bounded behaviors, all sidecar-file based (no schema changes): 1. Persistent attempt ledger (<db>.repair-attempts.json): after 3 failed repair passes against the same file fingerprint (size + mtime_ns), repair_state_db_schema refuses with a terminal, actionable error (restore a backup / `sqlite3 state.db ".recover"` / delete the ledger to force a retry) instead of re-running surgery. Success clears the ledger; a replaced or restored file re-keys it and gets fresh attempts. Missing/corrupt ledger fails open (never blocks a first repair). 2. Backup dedupe: _backup_db_file reuses the newest existing forensic backup when it is byte-identical to the damaged file (size+mtime match, preserved by copy2) instead of copying another ~900MB. 3. Retention cap: only the 3 newest malformed-backup copies (plus sidecars) are kept; older ones are pruned after each new backup. Also fixes a same-second timestamp collision that silently overwrote an earlier forensic copy. Tests cover ledger accumulation, terminal refusal (surgery not called, no new backup), budget reset on file change, success-clears-ledger, corrupt-ledger tolerance, dedupe, distinct-state backups, retention prune incl. sidecars, and the end-to-end one-backup invariant. Fixes #86747
Contributor
૮ >ﻌ< ა ci reviewran on bb88a4c — fix(state): bound the state.db repair loop and stop dead-bac
|
This was referenced Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #86747 — the state.db repair loop ran forever on unrepairable corruption, accumulating 89 GB of identical dead forensic backups (105 failed attempts over 11 days in @jermynyee's install).
Root cause
_claim_repair_attemptis an in-memory set — it bounds the repair to one attempt per process, but every restart got a fresh set. B-tree page corruption is beyond all ofrepair_state_db_schema's strategies (FTS rebuild, sqlite_master dedupe, FTS drop+VACUUM), so each restart re-ran the full surgery and took a fresh ~900 MBstate.db.malformed-backup-*copy of the same damaged bytes.What this PR does (the reporter's P0 + P1-terminal-error, sidecar-file based, no schema changes)
<db>.repair-attempts.json): after 3 failed repair passes against the same file fingerprint (size + mtime_ns),repair_state_db_schemarefuses with a terminal, actionable error — restore a backup, salvage viasqlite3 state.db ".recover", or delete the ledger to force a retry — instead of re-running surgery. A successful repair clears the ledger; a replaced/restored file re-keys it and gets a fresh budget. Missing or corrupt ledger fails open, so a first repair is never blocked._backup_db_filereuses the newest existing forensic backup when it's identical to the damaged file (size + mtime match, preserved bycopy2) — the single highest-impact change per the report (stops ~90% of the waste).-wal/-shmsidecars) are kept; older ones are pruned. Also fixes a same-second timestamp collision that could silently overwrite an earlier forensic copy.Not included (left for follow-up discussion): flipping
updates.pre_update_backupdefault totrue(the reporter's P2 — a behavior/default change worth its own review) and aVACUUM INTOlast-ditch strategy.Tests (sabotage-run verified: suite fails against origin/main, green with the fix)
Ledger accumulation, terminal refusal (surgery provably not called, no new backup taken), budget reset on file change, success-clears-ledger, corrupt-ledger tolerance, backup dedupe, distinct-state backups, retention prune incl. sidecars, and the end-to-end invariant that N failed repairs on the same file leave exactly one backup. Existing
tests/test_state_db_malformed_repair.pyandtests/test_state_db_notadb_selfheal.pystay green.Credit: excellent forensic report with exact line numbers and a prioritized fix list by @jermynyee.
Fixes #86747
Infographic