fix(dashboard): return 503, not 500, for a deleted WAL or replaced state.db - #121428
Merged
kshitijk4poor merged 5 commits intoSep 24, 2026
Merged
kshitijk4poor merged 5 commits into
kshitijk4poor merged 5 commits into
Conversation
… 503 in corrupt_store_as_status (NousResearch#110054) Analytics routes open SessionDB under corrupt_store_as_status, which only caught sqlite3.DatabaseError. DeletedWalGenerationError and StateDbReplacedError subclass RuntimeError, so a retired WAL generation escaped the guard and every dashboard poll turned into a 500 storm. Catch StateDbReplacedError alongside the malformed-image case and return the same structured 503 payload with detail.error='deleted_wal' / 'state_db_replaced'. Hunk applied from PR NousResearch#110054 commit 41e2b12 (web-router mapping only; the holder-termination path from that PR is left to the maintainer).
…placed 503 payload The contributor's 503 messages told the user to "click Recover or run `hermes doctor --fix`". Main's deleted_wal/replaced explainers (agent/turn_explainers.py, hermes_state_errors.py) say the opposite: running `doctor --fix` while a holder process is alive repairs the wrong generation in place, and the Desktop Recover button from NousResearch#110054 was not taken. Hoist the two payloads into module constants next to CORRUPT_STORE_DETAIL and reuse the explainer guidance (quit every Hermes process, run `hermes doctor`, do NOT run `--fix`); the log line likewise points at plain `hermes doctor`. Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
One test in the existing analytics corrupt-store file: both RuntimeError subclasses become a structured 503 with the right detail.error and path, the message never nudges `doctor --fix` except as a negation, and a busy/locked sqlite3.OperationalError still propagates. Replaces the PR's test_corrupt_store_as_status_handles_deleted_wal_and_replaced_errors, which asserted the removed `--fix` wording. Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
…r bucket The deleted_wal/replaced payload selection re-derived the type-ordered bucket table that hermes_state_errors._PERSISTENCE_CAUSE_BY_TYPE already owns; a future bucket could silently diverge from the dashboard mapping. Look the payload up by cause bucket instead (default: the corrupt payload, which also covers an FTS-scoped malformed image). Proven behaviour-equivalent for every exception that passes the guard (both replaced-family types and subclasses, malformed sqlite errors incl. StateDbCorruptError and fts_index-scoped ones); busy/locked and unrelated errors still propagate. Also: one `state_db_*` naming scheme for the `error` codes (`deleted_wal` -> `state_db_deleted_wal`; no consumer keys on it — web/src/lib/api.ts only branches on the auth codes), drop the redundant sqlite3.DatabaseError isinstance that is_malformed_db_error already performs, and make the test assert the invariant (every `--fix` mention is negated) rather than the exact wording.
…api/sessions too GET /api/sessions and _resolve_session_id caught only sqlite3.DatabaseError, so StateDbReplacedError / DeletedWalGenerationError (RuntimeError family) fell through to the bare `except Exception` -> 500 "Internal server error" — the same mis-mapping NousResearch#110054 fixed for the analytics reads. Route that exception family through the existing corrupt_store_as_status so the sessions list and detail routes return the same structured 503 payload; the sqlite arms (busy -> 503, malformed -> latch + 503) are untouched, which is why the mapping is added as a sibling arm instead of wrapping the read (that would skip note_storage_error for a malformed image). The kept invariant test now also exercises both sessions.py paths; it fails without this change (raw DeletedWalGenerationError escapes the resolver, the route returns 500).
kshitijk4poor
enabled auto-merge (rebase)
September 24, 2026 11:09
alt-glitch
added
type/bug
This was referenced Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
state.db's WAL has been deleted or the database file was replaced underneath a running process, the dashboard now returns a 503 with recovery guidance instead of a bare 500. This covers the analytics routes,GET /api/sessionsand session lookup. It is the slim, still-missing part of #110054, salvaged from #110073 by @JoaoMarcos44.Why
#110054:
DeletedWalGenerationErrorandStateDbReplacedErrorsubclassRuntimeError, notsqlite3.DatabaseError.corrupt_store_as_status(analytics) and the sessions router only mapped malformed-sqlite errors to 503, so these two escaped as 500s. On main, a probe against/api/analytics/usagereturns 500 for both.Changes
edafec95d0(@JoaoMarcos44):corrupt_store_as_statusalso catchesStateDbReplacedError(and itsDeletedWalGenerationErrorsubclass) and returns 503.c2ed1cff96: the payload no longer tells users to "run hermes doctor --fix". Main's guard text says not to run--fixwhile holders are live, so every mention of--fixis now negated.5c26b1c908: an invariant test covering both error types, which also asserts that no--fixinstruction is left un-negated.768555b7a7: the payload is keyed by the existingclassify_persistence_error()bucket rather than a newisinstanceladder. Codes are now uniformlystate_db_corrupt/state_db_deleted_wal/state_db_replaced; no frontend code keys on them.2c63a9bd23:GET /api/sessionsand_resolve_session_idroute the same errors throughcorrupt_store_as_status. That leaves the malformed-sqlite latch untouched.Validation
analytics.router): on main both errors return 500; on this branch both return 503 with the right code and only negated--fixtext. The controls are identical to main: healthy → 200, malformed → 503state_db_corrupt, locked → propagates, unrelatedRuntimeError→ 500./simplify-codereviewers, fold, final deep review with mutation + 3 reviewers. All PASS / no material findings.Not in this PR
--fixwhile holders are live") and had four unresolved review findings from @andrexibiza.hermes doctorstops sending users with a live gateway into--fixon a large WAL (#110054 item 3) #113055, hermes doctor names the processes holding a retired WAL generation instead of a green state.db (#110054) #116297, fix(sessions): storage maintenance refuses while a writer holds state.db; retired-WAL guard tells users what to do (#110054) #117687, state.db residuals: serve stands down under a live gateway, recovery survives the first resume, import refuses a held deleted database #117746) or is a Desktop degraded-storage decision. So this PR references Pain cluster: after the deleted-WAL guard fires there is no in-product recovery — Desktop users restart/ask-the-agent/run doctor --fix and make it worse (4 Discord threads, 13 issues this week) #110054 without closing it.Credit
Refs #110054