state.db FTS damage no longer kills turns; doctor/repair/recover handle real incident shapes (#97794, #88587, #100227, #103840, #106667, #102240, #98050, #103647, #96591, #105887; salvage #97843 #88604 #56824 #91413 #102808 #103657 #106890 #103321) - #108130
Merged
Merged
Conversation
Contributor
૮ >ﻌ< ა ci reviewran on 58767f6 — test(web): count only the dashboard's own warning in the cor
|
alt-glitch
added
type/bug
`sqlite3 state.db .recover` re-emits the FTS5 shadow tables (messages_fts_data, _idx, _docsize, _config, _content) as ordinary tables but cannot re-emit the CREATE VIRTUAL TABLE row. The next SessionDB open ran the FTS DDL in _ensure_fts_schema and died with "fts5: error creating shadow table messages_fts_data: table 'messages_fts_data' already exists", so a recovered database was unusable until someone hand-dropped the shadows. _init_fts now runs _drop_orphan_fts_shadow_tables before any FTS DDL. It is per-family and exact-name scoped: a family's shadows are dropped only when its own vtable row is absent from sqlite_master (type='table' AND sql LIKE 'CREATE VIRTUAL TABLE%'), so a healthy messages_fts_trigram survives a base-family repair untouched. A repaired base/trigram family is then treated like a missing-trigger repair and rebuilt from the canonical messages table under the cross-process rebuild admission; the shadows are derived index state, nothing is lost. Live repro: real `sqlite3 x.db .recover | sqlite3 y.db` on sqlite 3.50.4 keeps the vtable rows (the shell emits CREATE VIRTUAL TABLE), so the deterministic fixture removes the vtable row via writable_schema leaving the shadows behind: BEFORE OperationalError on open; AFTER opens, fts_enabled, MATCH returns every message, trigram sqlite_master rowids unchanged. Salvaged from PR #56824 (intent applied onto the current hermes_state_fts / hermes_state_schema siblings). The ownership-safety point (never touch a live family's shadows) was raised by @ggoldani in #103868 / #103840. Refs #103840 Refs #56815 Co-authored-by: ggoldani <ggoldani@users.noreply.github.com>
…l salvage
`hermes sessions recover --allow-partial` walks damaged tables by rowid range
and falls back to an exact-rowid lookup for the last cell of a broken range.
A phantom row produced by page damage (a `sessions` row with a NULL
`started_at`, an `async_delegations` row with a NULL `state`) reads fine from
the source but violates the destination's NOT NULL constraint, and the
resulting sqlite3.IntegrityError escaped the exact-lookup boundary and aborted
the whole recovery — losing every healthy row behind it.
Catch IntegrityError at that boundary, count it under
`destination_rejected_rows` and record the rowid as a skipped singleton
("destination constraint rejected row: ...") so the run completes, verifies,
and the report shows exactly which rows were dropped.
Live repro (real fixture: source schema's NOT NULL relaxed via
writable_schema, phantom `sessions` row inserted): BEFORE
sqlite3.IntegrityError "NOT NULL constraint failed: sessions.started_at" at
session_recovery.py recover_exact_rowid; AFTER status=partial copied=3
destination_rejected_rows=1, verified=True. Reporter @i8ei confirmed the same
patch recovers the field database (22 sessions / 2,248 messages) in #102240.
Salvaged from PR #91413 (rebased onto the _RowidRangeSalvage refactor; test
reduced to one invariant).
Refs #102240
Reported-by: i8ei
…INT64 domain When the leftmost (or rightmost) leaf of a table b-tree is damaged, the edge probe `SELECT rowid ... ORDER BY rowid ASC LIMIT 1` walks the table tree and raises, and _salvage_rowid_bounds fell back to INT64_MIN. The gallop from the surviving edge cannot cap that side either (every probe crosses the damaged leaf), so bisection burned the entire 10,000-query budget moving the bound inward by a few thousand rowids out of 9.2e18 and the table was lost — a 4-row gateway_routing table in #98050, sessions + session_model_usage in #100313. `SELECT min(rowid), max(rowid)` is answered by the planner from any covering index (every Hermes table has at least the PRIMARY KEY autoindex) without touching the damaged leaf, which is exactly what the reporter verified by hand. Ask it for the missing edge(s) first; only when it fails too does the domain fallback + gallop run as before. Reported under `aggregate_edges` so recovery.json still shows how the bound was obtained. Live repro (real fixture: leftmost `sessions` leaf cell count overwritten, 400 rows): BEFORE bounds low=-9223372036854775808 copied=0 range_queries=10000 query_limit_reached=True status=failed; AFTER low=1 high=400 copied=391 range_queries=40 status=partial (only the damaged leaf's rows are lost). Refs #98050 Refs #100313 Reported-by: Ace-Kelly Corroborated-by: Proff506
…ename The demote enumeration (name LIKE 'messages_fts_%') sweeps the messages_fts_cjk vtable and its shadow tables into the fts_v22_trash_* renames. Renaming the cjk vtable cascades to its shadow tables and breaks the vtable constructor chain, so 'hermes sessions optimize-storage' aborts with 'vtable constructor failed: messages_fts_cjk' on every DB that carries both a legacy inline FTS layout and an established cjk index (#103647). The cjk family is an independent v23+ index, not part of the demoted legacy layout: skip it in the enumeration.
Review feedback on #103657: the sibling clauses in the same statement declare ESCAPE, and the trash enumeration two blocks up escapes its underscores via replace. Use the same escaped ESCAPE form so every underscore in the statement is a literal match instead of a single-character wildcard. Behavior on the fixed schema is unchanged (same enumeration split); this is consistency plus defense against future lookalike table names. Also reword the exclusion comment to the verified failure mechanism: fts5 xRename renames the whole shadow family in one step, so sweeping the cjk vtable aborts the loop on the next shadow entry and drags the _config table (read by the vtable constructor) into the trash family.
…nd_found lane A SIGKILL mid-write (OOM killer, #106667) can leave state.db with a garbage page-1 header. SQLite refuses the file outright ("file is not a database", SQLITE_NOTADB) and the sqlite3 shell's .recover opens the file like any other client, so the lost_and_found lane failed with the same rc=26 twice although every data page after the header survived. The #106587 quarantine now preserves such a file as state.db.notadb-<ts>-<pid>.bak; this makes that preserved file recoverable with `hermes sessions recover --source <bak> --allow-partial`. When both .recover attempts fail with "not a database", zero the 100-byte header of the lane's private snapshot copy and rerun them. .recover trips only on the magic check and infers page size and layout from the pages themselves, so a zeroed header is enough; a spliced donor header (the PR's original mechanism) instead advertises a database size / freelist that contradicts the file and yields "database disk image is malformed" on a direct open — verified live on a 139-page fixture, which also showed the zeroed header recovers 60/60 sessions and 300/300 messages whether the damage covers 100 bytes or the whole first page. The user's file is never written; the report carries `sqlite3_cli.header_zeroed` and a warning about the WAL boundary. Live repro (sqlite3 shell 3.53.1 on PATH, header overwritten with random bytes): BEFORE "page-level .recover salvage failed: ... file is not a database (26)"; AFTER "Recovered 60 sessions and 300 messages", source md5 unchanged. Salvaged from PR #102808 (intent; trimmed from 302 to ~40 source LOC by dropping the donor-header/page-size sweep and the redundant preopen probe — the shell's own refusal is the detector). Independent review on the PR by @strzhao. Refs #106667 Refs #106587 Reported-by: TaoMasterCoder Cross-referenced-by: kshitijk4poor
… too
optimize_fts() and vacuum() already refuse to run against a quarantined
handle (_db_corrupt / _db_replaced / _db_wal_generation_lost): both would
rewrite index/file pages in place, turning contained, diagnosable
corruption into an amplified one. rebuild_fts() never got the same guard,
despite being the more destructive of the two ("discards and recreates the
index data entirely", per its own docstring, vs. optimize_fts's segment
merge).
It's also independently reachable outside _execute_write's own quarantine
check: gateway/session_transcript.py's _rebuild_fts_once() calls
db.rebuild_fts() directly from the FTS-corruption transcript-retry path,
with no quarantine check of its own (only a WAL split-brain / foreign-holder
check, a different concern). A quarantined handle hitting that retry path
would run a full FTS rebuild — and commit it — on a corrupt, replaced, or
split-WAL-generation file.
Add the same self._raise_if_db_corrupt()/self._raise_if_db_replaced() pair
optimize_fts() already has, at the top of rebuild_fts(), before it enters
the cross-process rebuild admission.
…file damage classify_persistence_error bucketed every _DB_CORRUPTION_MARKERS hit as "corrupt", so an error SQLite itself scoped to the FTS5 index layer (SQLITE_CORRUPT_VTAB, or an `fts5: corrupt structure record for table "messages_fts"` report) that escaped the write path — the detach in _enter_fts_fail_open refused (generation/lock check), or a read/search path with no fail-open at all — reached the turn boundary and the gateway startup notice as structural corruption: the turn ended with `.recover` / restore-backup advice on a file whose canonical tables were provably healthy. One provenance rule, hermes_state_errors.is_fts_scoped_corruption_error, now feeds both the write-repair gate (SessionDB._is_fts_write_corruption_error delegates to it, so the gateway transcript retry inherits it) and the classifier: a known result code outranks prose (only SQLITE_CORRUPT_VTAB is FTS-scoped; bare SQLITE_CORRUPT/NOTADB and any contradictory code fail closed), and without a code the text must both carry a corruption marker and name a messages_fts* object. The new "fts_index" cause renders index-scoped guidance (doctor --fix / restart, do not run recovery) in the turn explainer and the home-channel notice. The structural fail-close is untouched: bare malformed / not-a-database still quarantine and still classify "corrupt". Salvaged from PR #97843 (SulthanZahran1), trimmed: the quick_check-backed "corrupt_unconfirmed" tier is dropped — on a live handle that just observed an unscoped SQLITE_CORRUPT, PRAGMA quick_check on a damaged shadow b-tree raises rather than reports on 3.53.1, so the probe could never downgrade the exact shape it was built for, and a verdict that softens quarantine guidance on prose alone weakens the fail-close. #97841 (Finn763) reached the same fts_index cause via text markers only; its LIKE-degradation intent already lives in _search_messages_impl (_fts_stale). Fixes #97794 Co-authored-by: finn763 <165816600+finn763@users.noreply.github.com>
…t to sessions recover `hermes doctor` reported every write-health-probe failure as "state.db FTS write corruption" and `--fix` ran the FTS repair ladder — rebuild, REINDEX, sqlite_master surgery + VACUUM — on the damaged file. When the damage is structural (canonical tables/indexes), none of those rungs can fix it, each one writes to the torn file in place, and the operator is then told to "restore from the backup copy beside state.db": a `.malformed-backup` that is a snapshot of the same corrupt image. `hermes sessions recover`, the tool that actually rebuilds canonical rows into a fresh file, was never mentioned (#88587; the 1.7 GB field incident lost days to it). Discriminate before mutating. hermes_state_repair.integrity_damage_is_structural maps `PRAGMA integrity_check` output onto the file: a `Tree N` id resolved through sqlite_master.rootpage, an index named in `row N missing from index X`, or a `Freelist:` line is structural unless the object is a Hermes-owned messages_fts* table/shadow (full-matched, so a user lookalike such as archive_fts_data is never swept into the rebuildable set). state_db_has_structural_damage runs it read-only on a fresh connection; an integrity_check that RAISES under the walk (torn root page) is structural too — no FTS-only fixture does that while sessions/messages read cleanly. doctor's state check consults it first: structural damage becomes a manual issue naming `hermes [-p <profile>] sessions recover --source <this db> --inspect-only` (profile pinned, #105887) and explicitly warning off the .malformed-backup; nothing is mutated and no backup is written. FTS-only damage keeps the existing in-place repair path. Verified against real fixtures: a torn `sessions` root page (before: "FTS write corruption", --fix wrote a 1:1 malformed-backup and failed; after: structural, recover guidance, no writes) and the 16-byte DEADBEEF messages_fts_data stomp (still repaired in place via rebuild_fts). Salvaged from PR #88604 (liuhao1024) onto the split doctor_state.py; the classifier lives beside the repair ladder in hermes_state_repair so the ladder itself can consult it next. Fixes #88587
The recovery commands rendered on structural corruption — the turn explainer's `session_persistence_failed`/corrupt body, the gateway's home-channel state.db warning, and hermes_state_repair._persistent_repair_exhausted_error — already interpolate the active profile's state.db path, but every `hermes ...` verb in them was bare. A bare `hermes` follows the sticky `active_profile` file, so an operator running the pasted `hermes doctor --fix` (or `hermes sessions recover` with a relative source) from a named-profile incident could inspect or repair a different profile's database (#105887). hermes_constants.profile_cli_selector() renders `-p <name> ` for a named profile home (default home and custom roots outside the profile tree render nothing: the default is what a bare `hermes` already means, and a custom root is only reachable via HERMES_HOME). Every command in the three guidance sites now carries it, and the new `fts_index` guidance inherits the same interpolation. Live check with HERMES_HOME=<root>/profiles/research and active_profile=other: before `1. Run \`hermes doctor --fix\`` (targets "other"); after `1. Run \`hermes -p research doctor --fix\`` and `hermes -p research sessions recover --source <root>/profiles/research/state.db --inspect-only`. Refs #105887 Reported-by: Cuttingwater
…IntegrityError
`_db_opens_cleanly` drove one probe row through the messages_fts* triggers
and rolled back. FTS5 only buffers that row in an in-memory segment until
commit, so the probe never wrote to `<fts>_idx`/`_data` and could not hit a
stale `messages_fts_trigram_idx` row waiting at the next segid — the class
where PRAGMA integrity_check, the FTS5 integrity-check command and MATCH all
report clean while every committed append fails with
`IntegrityError: constraint failed`. The probe also caught only
OperationalError; IntegrityError is a DatabaseError sibling, so even a
colliding probe would have escaped and been reported as healthy.
Now the probe issues `INSERT INTO <fts>(<fts>) VALUES('flush')` for every
FTS family inside the rolled-back transaction (capability / not-built errors
stay benign), catches sqlite3.DatabaseError, and always rolls back in a
finally. `hermes doctor` and `hermes sessions repair --check-only` surface
the corruption and `repair_state_db_schema` heals it via the FTS rebuild
strategy (verified with a real stale-segid fixture).
Refs #100227
Reported-by: #100227
…ealthy store `hermes sessions repair --check-only` printed the corruption reason and exited 0, so scripts and the console wrapper gating on the status read a broken state.db as healthy. Return 1 from the CLI handler (main.py already sys.exits a truthy return) and from the console handler, where `_capture_output` turns the status into a ConsoleCommandError carrying the printed reason. Salvaged from PR #103321 (the check-only reporting part only; the probe rewrite and connection-tracking changes were not taken). Refs #63386.
… of a traceback per poll
The dashboard polls /api/analytics/usage and /api/analytics/models every
few seconds. When state.db is malformed the read raised straight through
the handler, so uvicorn logged a full traceback at ERROR on every poll —
one fleet host wrote ~520K identical journal entries in 24 h.
Wrap both analytics handlers in `corrupt_store_as_status`: a corrupt-image
sqlite3.DatabaseError (is_malformed_db_error) becomes a 503 with an
explicit `state_db_corrupt` payload pointing at `hermes doctor`, and the
warning is gated per store path via `{path: monotonic}` (>=300 s), then
debug. Busy/locked and every other error propagate unchanged, and the file
is never renamed or quarantined from the dashboard — repair stays with
`hermes doctor` / `hermes sessions repair`. `_session_db_path_for_profile`
is split out of `_open_session_db_for_profile` so the router can name the
store without opening it.
Refs #96591
Reported-by: #96591
…e poll test hermes_state emits a once-per-process SQLite-version advisory on CI's linked 3.50.4, which caplog captured as a second WARNING. Scope the assertion to the hermes_cli.web_server logger the router actually writes to.
teknium1
force-pushed
the
fix/statedb-repair
branch
from
September 11, 2026 13:28
a4461b7 to
58767f6
Compare
This was referenced Sep 11, 2026
This was referenced Sep 11, 2026
This was referenced Sep 11, 2026
gabrielcosi
pushed a commit
to gabrielcosi/home-ops
that referenced
this pull request
Sep 12, 2026
…9.7 ➔ v2026.9.11) (#754) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/gabrielcosi/hermes-agent](https://github.com/NousResearch/hermes-agent) | patch | `v2026.9.7` → `v2026.9.11` | --- ### Release Notes <details> <summary>NousResearch/hermes-agent (ghcr.io/gabrielcosi/hermes-agent)</summary> ### [`v2026.9.11`](https://github.com/NousResearch/hermes-agent/releases/tag/v2026.9.11): Hermes Agent v0.21.2 (v2026.9.11) [Compare Source](NousResearch/hermes-agent@v2026.9.7...v2026.9.11) ##### Hermes Agent v0.21.2 (v2026.9.11) — The state.db Patch Release **Release Date:** September 11, 2026 > Patch release. v0.21.0 shipped a large rewrite of the session store's connection handling, and for some installs it made `state.db` fragile: second writers cancelling each other's locks, healthy databases reported as corrupt, one bad row killing `sessions list`. This release closes that class and rolls up everything else that landed on `main` in the four days since v0.21.1. ##### About this release Measured at commit `04dd80a977f40b05e5b2054111747af07a61886a`, the window since v0.21.1 contains **947 non-merge commits** across **1,869 changed files** (+182,504 / −15,564) and **312 merged PRs**. **140 contributors** appear in commits, co-author trailers, or salvage credits. ##### ✨ Highlights ##### state.db reliability campaign (six PRs, 44 issues closed) If your `state.db` broke after 0.21.0, this is the release for you. Six PRs fix the root causes rather than the symptoms: - **No more second writers.** Profile gateways wrote hosted-room state into the *root* `state.db` every 5 seconds; the dashboard opened a writable handle on startup; cron's lifecycle guard did a raw `open()` on a live database (which cancels the gateway's POSIX locks — the classic "how to corrupt SQLite" recipe); `doctor --fix` would checkpoint under a live holder. All four are gone: hosted rooms live in `shared-state.db`, the dashboard opens read-only first, the guard goes through the tracked connection registry, and `doctor --fix` refuses a checkpoint it can't prove is safe. ([#​108076](NousResearch/hermes-agent#108076) — salvage [#​103489](NousResearch/hermes-agent#103489) [@​RikETS](https://github.com/RikETS), [#​102682](NousResearch/hermes-agent#102682) [@​JoaoMarcos44](https://github.com/JoaoMarcos44), [#​108012](NousResearch/hermes-agent#108012) [@​Halldrix](https://github.com/Halldrix), [#​105428](NousResearch/hermes-agent#105428) [@​TaoMasterCoder](https://github.com/TaoMasterCoder)) - **Healthy WAL databases stop wedging.** OpenZFS `(deleted)` dentries and a `close()` racing an `append_message` both produced a sticky `DeletedWalGenerationError` on a perfectly good store; the read pool was handed out under an unconfirmed journal mode; a transient `disk I/O error` on WSL2 killed `get_session` on the first attempt; and a "state.db locked" banner was broadcast after the lock had already cleared. ([#​108082](NousResearch/hermes-agent#108082) — salvage [#​107411](NousResearch/hermes-agent#107411) [@​chelsealong](https://github.com/chelsealong), [#​105578](NousResearch/hermes-agent#105578) [@​ca-shrimp](https://github.com/ca-shrimp), [#​105711](NousResearch/hermes-agent#105711) [@​gaoanze888](https://github.com/gaoanze888), [#​106958](NousResearch/hermes-agent#106958) [@​nikkoxgonzales](https://github.com/nikkoxgonzales); co-authored [@​QDung210](https://github.com/QDung210), [@​fangliquanflq](https://github.com/fangliquanflq), [@​Sahilvishnaliya](https://github.com/Sahilvishnaliya)) - **FTS damage no longer kills your turn.** An error scoped to the full-text-search index was classified as whole-file corruption and fail-closed the conversation. It's now `fts_index`: search degrades, the index rebuilds later, the transcript store is untouched. Same PR: doctor names structural damage honestly instead of "FTS write corruption", the FTS write probe catches the stale-index shape that passed every check while every write failed, `.recover` output no longer fails startup on orphan FTS5 shadow tables, header-zeroed databases recover instead of being refused, and the dashboard analytics poller returns a 503 instead of 520K tracebacks a day. ([#​108130](NousResearch/hermes-agent#108130) — salvage [#​97843](NousResearch/hermes-agent#97843) [@​SulthanZahran1](https://github.com/SulthanZahran1) + [#​97841](NousResearch/hermes-agent#97841) [@​Finn763](https://github.com/Finn763), [#​88604](NousResearch/hermes-agent#88604) [#​56824](NousResearch/hermes-agent#56824) [#​103657](NousResearch/hermes-agent#103657) [@​liuhao1024](https://github.com/liuhao1024), [#​106890](NousResearch/hermes-agent#106890) [@​nftpoetrist](https://github.com/nftpoetrist), [#​103321](NousResearch/hermes-agent#103321) [@​jangomango76](https://github.com/jangomango76), [#​91413](NousResearch/hermes-agent#91413) [@​leegunwoo98](https://github.com/leegunwoo98), [#​102808](NousResearch/hermes-agent#102808) [@​TaoMasterCoder](https://github.com/TaoMasterCoder)) - **One corrupt row no longer kills `sessions list`, export, or insights.** A TEXT timestamp or a `1e30` epoch used to crash the whole listing; malformed marker JSON crashed `json_extract`; more than 999 ids crashed bulk delete/prune. One `coerce_epoch()` helper on every reader (bad rows render `?` with a WARNING naming the session), a `json_valid` guard, IN-list chunking, and batched export hydration. ([#​108086](NousResearch/hermes-agent#108086) — salvage [#​106071](NousResearch/hermes-agent#106071) [@​Xipong](https://github.com/Xipong), [#​101726](NousResearch/hermes-agent#101726) [@​efe-arv](https://github.com/efe-arv), [#​94701](NousResearch/hermes-agent#94701) [@​liuhao1024](https://github.com/liuhao1024), [#​102679](NousResearch/hermes-agent#102679) [@​mssteuer](https://github.com/mssteuer), [#​100658](NousResearch/hermes-agent#100658) [@​Mi55ed](https://github.com/Mi55ed)) - **Sessions never bind to or read another profile's database.** The Desktop launch backend could pin itself to the wrong profile's `state.db` under a HERMES\_HOME override race; `session_search` by bare ID silently scanned every profile and returned someone else's transcript; recovery guidance pointed at the wrong file; profile delete kept a handle open (WinError 32). ([#​108074](NousResearch/hermes-agent#108074) — salvage [#​102534](NousResearch/hermes-agent#102534) [@​HexLab98](https://github.com/HexLab98), [#​106975](NousResearch/hermes-agent#106975) [@​Sora-bluesky](https://github.com/Sora-bluesky)) - **Opening state.db no longer takes the write lock when nothing needs writing.** A one-shot `hermes` process opening the store behind a busy gateway stalled 4–20 s and then failed with "database is locked". Now 0.01 s. ([#​108067](NousResearch/hermes-agent#108067) — salvage [#​106751](NousResearch/hermes-agent#106751) [@​kshitijk4poor](https://github.com/kshitijk4poor), [#​101881](NousResearch/hermes-agent#101881) [@​jonpol01](https://github.com/jonpol01)) Also in the window from the same subsystem: a fresh `state.db` no longer publishes FTS tables before owning the rebuild lock ([#​106311](NousResearch/hermes-agent#106311)), a handle that lost its WAL generation no longer checkpoints stale frames at shutdown ([#​106315](NousResearch/hermes-agent#106315), [#​106840](NousResearch/hermes-agent#106840)), a clobbered first page is quarantined with its WAL instead of opened destructively ([#​106587](NousResearch/hermes-agent#106587)), WAL setup leaves an unverifiable database untouched ([#​106568](NousResearch/hermes-agent#106568)), and quarantined handles refuse VACUUM/FTS optimize ([#​106343](NousResearch/hermes-agent#106343), [#​106349](NousResearch/hermes-agent#106349)). Most of these salvaged community diagnoses by [@​kshitijk4poor](https://github.com/kshitijk4poor). ##### Multi-profile isolation hardening A cluster of fixes for installs running several profiles under one gateway (multiplex): secondary-profile bots no longer inherit the default profile's allow-lists ([#​107616](NousResearch/hermes-agent#107616)), adapters no longer send credentials to the default profile's host ([#​107617](NousResearch/hermes-agent#107617)), stdio MCP servers no longer receive the default profile's vault secrets ([#​107630](NousResearch/hermes-agent#107630)), `MEDIA:` delivery can no longer attach another profile's `.env` / `auth.json` / `state.db` ([#​107609](NousResearch/hermes-agent#107609)), Feishu drive callbacks and `/p/<profile>/` webhook replies stay on the routed profile ([#​107620](NousResearch/hermes-agent#107620), [#​107626](NousResearch/hermes-agent#107626)), and secondary profiles no longer get a sibling's Nous bearer from per-process memos ([#​107611](NousResearch/hermes-agent#107611)). ##### Desktop backend spawn storms are over Bot Mode used to spawn or dial one backend per profile on launch and on every roster tick, hovering the Bots roster spawned a backend per row, and profile switches could spawn a duplicate primary. ([#​108069](NousResearch/hermes-agent#108069), [#​108107](NousResearch/hermes-agent#108107), [#​108118](NousResearch/hermes-agent#108118), [#​108134](NousResearch/hermes-agent#108134), [#​107969](NousResearch/hermes-agent#107969), [#​108112](NousResearch/hermes-agent#108112) — salvage [#​102512](NousResearch/hermes-agent#102512), [#​103634](NousResearch/hermes-agent#103634), [#​103399](NousResearch/hermes-agent#103399), [#​107997](NousResearch/hermes-agent#107997) and others by [@​kshitijk4poor](https://github.com/kshitijk4poor)) ##### Password-blind credential vault The agent can now sign in, pay, and fill addresses from 1Password, Bitwarden, or the local Hermes vault without ever seeing a secret; two-factor codes come from a saved authenticator key or are asked for in the user's UI ([#​106480](NousResearch/hermes-agent#106480), [#​107585](NousResearch/hermes-agent#107585)). Private git plugins install with the user's stored credentials ([#​106981](NousResearch/hermes-agent#106981)). ##### Plugin catalog and one Plugins page A curated, SHA-pinned plugin index with CLI, admission CI, docs and dashboard ([#​69446](NousResearch/hermes-agent#69446)); Desktop gets one Plugins page owning agent + desktop plugins, install, catalog and per-commit pinning ([#​107212](NousResearch/hermes-agent#107212), [#​107314](NousResearch/hermes-agent#107314), [#​107321](NousResearch/hermes-agent#107321)); Radio ships as an opt-in SDK plugin ([#​107072](NousResearch/hermes-agent#107072)). ##### Nous free tier and guided first launch Free inference and connectors out of the box with one command to sign in ([#​105258](NousResearch/hermes-agent#105258), [#​105260](NousResearch/hermes-agent#105260)), `/login` from a chat ([#​105261](NousResearch/hermes-agent#105261)), connector tools (Gmail, Linear, Notion, ...) searchable through `tool_search` ([#​106842](NousResearch/hermes-agent#106842)), and a guided first launch behind `HERMES_GUEST_ONBOARDING=1` ([#​107697](NousResearch/hermes-agent#107697), [#​107958](NousResearch/hermes-agent#107958), [#​107985](NousResearch/hermes-agent#107985), [#​108211](NousResearch/hermes-agent#108211)). ##### 🐛 Notable Bug Fixes **Gateway & platforms** - A bare `display:` key in config.yaml no longer crashes every gateway turn ([#​106305](NousResearch/hermes-agent#106305)); a queued-lane final refused by the platform is recorded and redelivered ([#​106316](NousResearch/hermes-agent#106316)); a stalled WebSocket send no longer blocks every later event ([#​106581](NousResearch/hermes-agent#106581)); the first turn no longer waits on the Python toolchain probe ([#​106556](NousResearch/hermes-agent#106556)). - Telegram bots must @​mention when `bots_require_mention` is on, breaking bot-to-bot loops ([#​106534](NousResearch/hermes-agent#106534)); Matrix renders LaTeX ([#​106515](NousResearch/hermes-agent#106515)); Signal renders markdown tables ([#​106538](NousResearch/hermes-agent#106538)); WhatsApp replies to view-once messages keep their quote ([#​106541](NousResearch/hermes-agent#106541)); media-only replies report SUCCESS everywhere ([#​106557](NousResearch/hermes-agent#106557)). **Providers & routing** - `/model` and auxiliary auto never bill a provider you didn't select ([#​107366](NousResearch/hermes-agent#107366)); never auto-switch to a provider you have no credentials for ([#​107281](NousResearch/hermes-agent#107281)); Bedrock Claude/Converse/Mantle models survive `/model`, fallback and restore ([#​107621](NousResearch/hermes-agent#107621), [#​107658](NousResearch/hermes-agent#107658)); Bedrock Guardrails enforced ([#​107815](NousResearch/hermes-agent#107815)). - Codex: patch-budget image 400 shrinks and retries ([#​106525](NousResearch/hermes-agent#106525)); unentitled primary + fallback no longer oscillate ([#​106549](NousResearch/hermes-agent#106549)); Azure Foundry replayed-reasoning rejection classified and pruned ([#​106718](NousResearch/hermes-agent#106718) [@​erosika](https://github.com/erosika)). MCP OAuth refresh no longer erases the refresh token ([#​106185](NousResearch/hermes-agent#106185)). Anthropic clients send exactly one credential ([#​107978](NousResearch/hermes-agent#107978)). - DeepSeek V4.1 Flash on Nous Portal and OpenRouter pickers ([#​107489](NousResearch/hermes-agent#107489)); GPT Image 2.5 via OpenAI and FAL ([#​105988](NousResearch/hermes-agent#105988)); Opus 5 / Fable 5.1 on the native Anthropic picker ([#​106636](NousResearch/hermes-agent#106636) [@​xxxigm](https://github.com/xxxigm)). **Agent loop & compression** - One blocked periodic callback no longer stalls lease refresh ([#​106308](NousResearch/hermes-agent#106308)); a mid-turn `/steer` is persisted as its own user row ([#​106317](NousResearch/hermes-agent#106317), [#​106344](NousResearch/hermes-agent#106344)); local-inference memory-ceiling rejections back off instead of compressing history ([#​106307](NousResearch/hermes-agent#106307)); context-overflow after partial streaming ends the turn ([#​106567](NousResearch/hermes-agent#106567)); length continuation stops when the prompt filled the window ([#​106571](NousResearch/hermes-agent#106571)); compression no longer times out silently on aux retries ([#​106866](NousResearch/hermes-agent#106866)); `model_thresholds` keys can be provider-scoped ([#​108061](NousResearch/hermes-agent#108061)). - Surface switch (Desktop↔TUI) no longer rebuilds the system prompt and busts the prompt cache ([#​105844](NousResearch/hermes-agent#105844)); CLI keeps the `api_content` sidecar so the cache survives an early persist ([#​105842](NousResearch/hermes-agent#105842)). **CLI, TUI & Desktop** - `hermes -z --resume` continues the session ([#​106313](NousResearch/hermes-agent#106313)); Shift+letter and Cmd+Shift+Z work on extended-key terminals ([#​90674](NousResearch/hermes-agent#90674) [@​francip](https://github.com/francip), [#​105493](NousResearch/hermes-agent#105493)); `browser_exec` timeout kills the whole process tree ([#​106589](NousResearch/hermes-agent#106589)); update checks poll the GitHub API once a day instead of git-fetching every 30 min ([#​107648](NousResearch/hermes-agent#107648)); `hermes update` names the real cause and can't hang on a stalled fetch ([#​108053](NousResearch/hermes-agent#108053)). - Desktop: UI language survives the update relaunch ([#​106476](NousResearch/hermes-agent#106476)), expired OAuth grants get a one-click re-sign-in ([#​106965](NousResearch/hermes-agent#106965)), HUD mode shows the transcript again and always gives the window back ([#​107491](NousResearch/hermes-agent#107491), [#​107423](NousResearch/hermes-agent#107423)), the backend exits when its Desktop parent dies ([#​107977](NousResearch/hermes-agent#107977)), Windows updates stop reporting false failures ([#​106175](NousResearch/hermes-agent#106175), [#​107183](NousResearch/hermes-agent#107183)), WSLg renders on the Windows GPU ([#​106528](NousResearch/hermes-agent#106528)), Telegram quick setup with QR ported from the dashboard ([#​107242](NousResearch/hermes-agent#107242)), and \~60 more Desktop fixes largely from [@​OutThisLife](https://github.com/OutThisLife) and [@​kshitijk4poor](https://github.com/kshitijk4poor). **Cron & Kanban** - An off-tick "run now" no longer cancels the next scheduled run ([#​106306](NousResearch/hermes-agent#106306)); a killed manual run no longer blocks the next one for 5 minutes ([#​106733](NousResearch/hermes-agent#106733)); a one-shot changed to recurring keeps firing ([#​106532](NousResearch/hermes-agent#106532)); unpinned jobs run on their creation-snapshot model ([#​106499](NousResearch/hermes-agent#106499)); `--clone-all` no longer copies cron jobs ([#​106478](NousResearch/hermes-agent#106478)); `kanban promote` refuses undone parents ([#​106550](NousResearch/hermes-agent#106550)); `kanban_request_review` rejects unknown reviewer profiles ([#​106547](NousResearch/hermes-agent#106547)). **Tools & memory** - A stdio MCP server dying mid-call no longer replays the tool call ([#​106546](NousResearch/hermes-agent#106546)); a skills-only background review can no longer delete memory entries ([#​106310](NousResearch/hermes-agent#106310)); mem0 memory no longer drops long turns ([#​106542](NousResearch/hermes-agent#106542)); `tool_search` returns nothing rather than five tools sharing one word ([#​106676](NousResearch/hermes-agent#106676)); remote NOPASSWD sudo no longer prompts ([#​107939](NousResearch/hermes-agent#107939)); RSS and Reddit reading no longer activate by default ([#​105873](NousResearch/hermes-agent#105873)). **Housekeeping** - `config.yaml` backups live in one bounded `backups/config/` dir ([#​106388](NousResearch/hermes-agent#106388)); `hermes backup` keeps the newest 3 zips ([#​106455](NousResearch/hermes-agent#106455)); `hermes setup --reset` backs up the real config ([#​106453](NousResearch/hermes-agent#106453)); `debug share` retention shrunk to 1 day on the dpaste fallback ([#​106531](NousResearch/hermes-agent#106531)). ##### 👥 Contributors Thank you to the **140 contributors** whose commits, co-author trailers, and salvaged PRs landed in this window. **state.db campaign — salvaged PR authors:** [@​RikETS](https://github.com/RikETS), [@​JoaoMarcos44](https://github.com/JoaoMarcos44), [@​Halldrix](https://github.com/Halldrix), [@​TaoMasterCoder](https://github.com/TaoMasterCoder), [@​chelsealong](https://github.com/chelsealong), [@​ca-shrimp](https://github.com/ca-shrimp), [@​gaoanze888](https://github.com/gaoanze888), [@​nikkoxgonzales](https://github.com/nikkoxgonzales), [@​QDung210](https://github.com/QDung210), [@​fangliquanflq](https://github.com/fangliquanflq), [@​Sahilvishnaliya](https://github.com/Sahilvishnaliya), [@​kshitijk4poor](https://github.com/kshitijk4poor), [@​jonpol01](https://github.com/jonpol01), [@​HexLab98](https://github.com/HexLab98), [@​Sora-bluesky](https://github.com/Sora-bluesky), [@​Xipong](https://github.com/Xipong), [@​efe-arv](https://github.com/efe-arv), [@​liuhao1024](https://github.com/liuhao1024), [@​mssteuer](https://github.com/mssteuer), [@​Mi55ed](https://github.com/Mi55ed), [@​SulthanZahran1](https://github.com/SulthanZahran1), [@​Finn763](https://github.com/Finn763), [@​nftpoetrist](https://github.com/nftpoetrist), [@​jangomango76](https://github.com/jangomango76), [@​leegunwoo98](https://github.com/leegunwoo98), [@​ggoldani](https://github.com/ggoldani). **state.db campaign — issue reporters** (the forensics in these threads were often better than the fixes): [@​thedigitalcarpenterdad](https://github.com/thedigitalcarpenterdad), [@​Rroven](https://github.com/Rroven), [@​aoeman84](https://github.com/aoeman84), [@​StephanRosin](https://github.com/StephanRosin), [@​rubensandrade-sketch](https://github.com/rubensandrade-sketch), [@​wanliqin](https://github.com/wanliqin), [@​chenzheshushi-commits](https://github.com/chenzheshushi-commits), [@​CarlosReyesPena](https://github.com/CarlosReyesPena), [@​revazone](https://github.com/revazone), [@​reservassai-art](https://github.com/reservassai-art), [@​Cuttingwater](https://github.com/Cuttingwater), [@​soroush5](https://github.com/soroush5), [@​e-shizz](https://github.com/e-shizz), [@​shobhit-87labs](https://github.com/shobhit-87labs), [@​shivanathd](https://github.com/shivanathd), [@​hoelzl](https://github.com/hoelzl), [@​i8ei](https://github.com/i8ei), [@​Ace-Kelly](https://github.com/Ace-Kelly), [@​YinsenWANG](https://github.com/YinsenWANG), [@​zbabiarz](https://github.com/zbabiarz), [@​Sravanjangam](https://github.com/Sravanjangam), [@​0gl20shk0sbt36](https://github.com/0gl20shk0sbt36), [@​RChina](https://github.com/RChina), [@​bronder](https://github.com/bronder), [@​ccwssy](https://github.com/ccwssy), [@​bottenbenny](https://github.com/bottenbenny), and [@​Hitman117890](https://github.com/Hitman117890) whose Discord report kicked the campaign off. **Everyone in the window (alphabetical):** [@​0genlab](https://github.com/0genlab), [@​0xalydev](https://github.com/0xalydev), [@​100yenadmin](https://github.com/100yenadmin), [@​1052326311](https://github.com/1052326311), [@​686f6c61](https://github.com/686f6c61), [@​69k4xmdfm2-blip](https://github.com/69k4xmdfm2-blip), [@​abundantbeing](https://github.com/abundantbeing), [@​Adolanium](https://github.com/Adolanium), [@​Ahmett101](https://github.com/Ahmett101), [@​albert748](https://github.com/albert748), [@​AlexxRussell](https://github.com/AlexxRussell), [@​alt-glitch](https://github.com/alt-glitch), [@​auroracapital](https://github.com/auroracapital), [@​austinpickett](https://github.com/austinpickett), [@​babatorik](https://github.com/babatorik), [@​Bartok9](https://github.com/Bartok9), [@​benbarclay](https://github.com/benbarclay), [@​bennybuoy](https://github.com/bennybuoy), [@​brian717](https://github.com/brian717), [@​briandevans](https://github.com/briandevans), [@​buihongduc132](https://github.com/buihongduc132), [@​ca-shrimp](https://github.com/ca-shrimp), [@​cervantesh](https://github.com/cervantesh), [@​Cesar-Azeredo](https://github.com/Cesar-Azeredo), [@​ChanPark03](https://github.com/ChanPark03), [@​chelsealong](https://github.com/chelsealong), [@​ckomma](https://github.com/ckomma), [@​ClintonEmok](https://github.com/ClintonEmok), [@​crazyief](https://github.com/crazyief), [@​ctaylor86](https://github.com/ctaylor86), [@​dalzio](https://github.com/dalzio), [@​DavidMetcalfe](https://github.com/DavidMetcalfe), [@​Drexuxux](https://github.com/Drexuxux), [@​edosulai](https://github.com/edosulai), [@​efe-arv](https://github.com/efe-arv), [@​emozilla](https://github.com/emozilla), [@​ericmaddox](https://github.com/ericmaddox), [@​erosika](https://github.com/erosika), [@​ethernet8023](https://github.com/ethernet8023), [@​everm1nd](https://github.com/everm1nd), [@​FalconOrtiz](https://github.com/FalconOrtiz), [@​fangliquanflq](https://github.com/fangliquanflq), [@​Finn763](https://github.com/Finn763), [@​FirmamentalSpring](https://github.com/FirmamentalSpring), [@​francip](https://github.com/francip), [@​g3org3yo](https://github.com/g3org3yo), [@​gaoanze888](https://github.com/gaoanze888), [@​ggoldani](https://github.com/ggoldani), [@​Halldrix](https://github.com/Halldrix), [@​haydster7](https://github.com/haydster7), [@​hbizi](https://github.com/hbizi), [@​helix4u](https://github.com/helix4u), [@​HexLab98](https://github.com/HexLab98), [@​huklaa](https://github.com/huklaa), [@​IAvecilla](https://github.com/IAvecilla), [@​infinitycrew39](https://github.com/infinitycrew39), [@​jahfaliabdulrahman-dev](https://github.com/jahfaliabdulrahman-dev), [@​jangomango76](https://github.com/jangomango76), [@​JoaoMarcos44](https://github.com/JoaoMarcos44), [@​jonpol01](https://github.com/jonpol01), [@​jwilson411](https://github.com/jwilson411), [@​KeyArgo](https://github.com/KeyArgo), [@​kokhlo](https://github.com/kokhlo), [@​KoNit-K](https://github.com/KoNit-K), [@​kshitijk4poor](https://github.com/kshitijk4poor), [@​kyssta-exe](https://github.com/kyssta-exe), [@​leegunwoo98](https://github.com/leegunwoo98), [@​lesterlxt](https://github.com/lesterlxt), [@​liuhao1024](https://github.com/liuhao1024), [@​Mabolla](https://github.com/Mabolla), [@​manuelschipper](https://github.com/manuelschipper), [@​MaxFreedomPollard](https://github.com/MaxFreedomPollard), [@​mearls0501](https://github.com/mearls0501), [@​mengyuyuan](https://github.com/mengyuyuan), [@​Mi55ed](https://github.com/Mi55ed), [@​MiseHinoha](https://github.com/MiseHinoha), [@​mjshorty](https://github.com/mjshorty), [@​mkrb84](https://github.com/mkrb84), [@​moisesvalero](https://github.com/moisesvalero), [@​moken627-hub](https://github.com/moken627-hub), [@​mssteuer](https://github.com/mssteuer), [@​nateEc](https://github.com/nateEc), [@​nftpoetrist](https://github.com/nftpoetrist), [@​nickseelert](https://github.com/nickseelert), [@​nikkoxgonzales](https://github.com/nikkoxgonzales), [@​notwitcheer](https://github.com/notwitcheer), [@​onuraycicek](https://github.com/onuraycicek), [@​outdog-hwh](https://github.com/outdog-hwh), [@​OutThisLife](https://github.com/OutThisLife), [@​philmossman](https://github.com/philmossman), [@​phuongvm](https://github.com/phuongvm), [@​pierrenode](https://github.com/pierrenode), [@​portavales](https://github.com/portavales), [@​PRATHAMESH75](https://github.com/PRATHAMESH75), [@​QDung210](https://github.com/QDung210), [@​rewbs](https://github.com/rewbs), [@​RikETS](https://github.com/RikETS), [@​romanovzky](https://github.com/romanovzky), [@​ryantuc](https://github.com/ryantuc), [@​Sahilvishnaliya](https://github.com/Sahilvishnaliya), [@​salch-cred](https://github.com/salch-cred), [@​sgarrand](https://github.com/sgarrand), [@​shannonsands](https://github.com/shannonsands), [@​simpolism](https://github.com/simpolism), [@​Solitud1nem](https://github.com/Solitud1nem), [@​somewheresy](https://github.com/somewheresy), [@​Sora-bluesky](https://github.com/Sora-bluesky), [@​sprmn24](https://github.com/sprmn24), [@​squevo](https://github.com/squevo), [@​StellarisW](https://github.com/StellarisW), [@​Stoltemberg](https://github.com/Stoltemberg), [@​SulthanZahran1](https://github.com/SulthanZahran1), [@​Svector-anu](https://github.com/Svector-anu), [@​szicely](https://github.com/szicely), [@​TaoMasterCoder](https://github.com/TaoMasterCoder), [@​teknium1](https://github.com/teknium1), [@​ten82e](https://github.com/ten82e), [@​thedavidweng](https://github.com/thedavidweng), [@​tkaufmann](https://github.com/tkaufmann), [@​Totoro-qaq](https://github.com/Totoro-qaq), [@​Tranquil-Flow](https://github.com/Tranquil-Flow), [@​tuancookiez-hub](https://github.com/tuancookiez-hub), [@​TurgutKural](https://github.com/TurgutKural), [@​ugoenyioha](https://github.com/ugoenyioha), [@​unsupportedpastels](https://github.com/unsupportedpastels), [@​victor-kyriazakos](https://github.com/victor-kyriazakos), [@​webtecnica](https://github.com/webtecnica), [@​wliu-dev](https://github.com/wliu-dev), [@​wukangcheng1994](https://github.com/wukangcheng1994), [@​Xipong](https://github.com/Xipong), [@​Xixiartemis](https://github.com/Xixiartemis), [@​xkam7ar](https://github.com/xkam7ar), [@​xxxigm](https://github.com/xxxigm), [@​yavarb](https://github.com/yavarb), [@​yoniebans](https://github.com/yoniebans), [@​Youssef](https://github.com/Youssef), [@​yoyodine-industries](https://github.com/yoyodine-industries), [@​yuanchenglu](https://github.com/yuanchenglu), [@​YuhGuan](https://github.com/YuhGuan), [@​Zeus-Deus](https://github.com/Zeus-Deus). Also: Youssef. ##### Updating - Existing install: `hermes update` - Fresh install: `curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash` - Managed deployments should update through their deployment tooling using the new tag. - If your `state.db` was already damaged by 0.21.0/0.21.1: run `hermes doctor` first; it now names structural vs index damage correctly and points at `hermes sessions recover --inspect-only` (profile-pinned) when a rebuild isn't enough. **Full Changelog:** [v2026.9.7...v2026.9.11](NousResearch/hermes-agent@v2026.9.7...v2026.9.11) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/Berlin) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC42NS4wIiwidXBkYXRlZEluVmVyIjoiNDQuNjUuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==--> Reviewed-on: https://git.xcd.dev/gabrielcosi/home-ops/pulls/754
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A damaged FTS5 index no longer kills turns or gets mislabelled as whole-file corruption, and the recovery lane (
hermes doctor,sessions repair,sessions recover,optimize-storage) now handles the shapes real incident databases actually arrive in: orphan shadow tables after.recover, a torn page-1 header, phantom NOT NULL rows, a damaged rowid edge, a CJK index family, and a segment-collision write failure that the health probe used to miss.Changes (13 commits, contributor authorship kept)
Fail-open for FTS, fail-closed for structure
hermes_state_errors.is_fts_scoped_corruption_error— result-code-first provenance rule: an error raised from themessages_fts*shadow tables classifies as new persistence causefts_index, nevercorrupt; bareSQLITE_CORRUPT/SQLITE_NOTADBstayscorrupt. Turn explainer and gateway home-channel notice show "search degraded, canonical transcript intact" instead of.recover/restore advice. (salvage fix(state): fail open on FTS5-index-only corruption instead of killing turns #97843 @SulthanZahran1, Co-authored-by @Finn763 fix(state): don't fail-close turns on FTS-index-only errors #97841; [Bug]: Healthy state.db fail-closes turns as 'structural corruption' when only the FTS5 index layer errors (false alarm, dropped messages, destructive recovery advice) #97794)rebuild_fts()refuses a quarantined corrupt/replaced handle likevacuum()/optimize_fts()already did. (salvage fix(state): quarantine a corrupt/replaced handle out of rebuild_fts() too #106890 @nftpoetrist)hermes doctorresolves damaged tree ids throughsqlite_master.rootpage: structural damage is reported as "canonical tables/indexes damaged, not the FTS index" and routes tohermes sessions recover --inspect-only;--fixno longer writes a 1:1 copy of the damage as a "backup" and tells you to restore it. (salvage fix(doctor): name structural state.db corruption honestly #88604 @liuhao1024; doctor reports structural state.db corruption as "FTS write corruption"; --fix cannot repair it and recommends restoring a corrupt backup #88587)hermes_constants.profile_cli_selector()(hermes -p <profile> doctor --fix, correct--sourcepath) in the explainer, gateway notice and repair-exhausted error. (Corrupt-session recovery guidance can target the wrong profile database #105887)Detection
_db_opens_cleanlywrite probe now issuesINSERT INTO <fts>(<fts>) VALUES('flush')per FTS family inside the rolled-back transaction and catchessqlite3.DatabaseError(anIntegrityErrorfrom a stale_idxrow at the next segid was slipping pastexcept OperationalError). doctor /sessions repair --check-only/repair_state_db_schemanow see and heal the [Bug]: _db_opens_cleanly / hermes doctor miss segment-dependent FTS trigram corruption; sessions repair says 'no repair needed' while every message write fails #100227 class.sessions repair --check-onlyexits 1 on an unhealthy store. (salvage fix(state): preserve repair locks and report unhealthy checks (#63386) #103321 @jangomango76, exit-code part only)/api/analytics/usage|modelson a malformed state.db: one WARNING per path per ≥300 s, HTTP 503{"error":"state_db_corrupt"}instead of a traceback per poll; busy/locked still propagate; the file is never renamed. (web_server _get_usage: malformed state.db causes unthrottled traceback storm (520K journal entries/24h) #96591)Recovery lane
sqlite_master) are dropped per family onSessionDBopen, then that family rebuilds under the normal admission gate; a healthy sibling family is never touched. (salvage fix(state): repair orphan FTS5 shadow tables on SessionDB open #56824 @liuhao1024, Co-authored-by @ggoldani fix(state): self-heal orphan FTS5 shadow tables (views+triggers, ownership-safe) #103868; [Bug]: state.db recovery via.recoverresurrects orphan FTS5 shadow tables — gateway fails startup with "fts5: error creating shadow table messages_fts_data: table already exists" #103840)sessions recover --allow-partialskips rows the destination schema rejects (destination_rejected_rows) instead of aborting. (salvage fix(cli): skip invalid rows during partial session recovery #91413 @leegunwoo98; fix(recovery): hermes sessions recover --allow-partial aborts on damaged phantom rows violating NOT NULL constraints #102240)_salvage_rowid_boundsseeds a damaged edge fromSELECT min(rowid), max(rowid)(covering PK index) before falling back to the INT64 domain — 40 range queries instead of exhausting the 10,000 budget. (sessions recover --allow-partial: damaged low edge of a 4-row table exhausts the salvage budget over the full 64-bit rowid domain, though min(rowid)/max(rowid) answer the bounds in one query #98050, reported @Ace-Kelly, corroborated @Proff506)file is not a database) snapshots: the lost_and_found lane zeroes the 100-byte header on the snapshot copy and reruns.recover. (salvage fix(sessions): salvage header-damaged state.db in the lost_and_found lane #102808 @TaoMasterCoder, trimmed 302→~40 LOC; [Bug]: state.db page-1 header damage (OOM kill mid-write) is unrecoverable —sessions recover --allow-partialrefuses the file outright #106667). Mechanism changed vs the PR: the donor-header splice was verified wrong for direct opens (in-header page count/freelist contradict the file →malformed);.recoverignores the header, so a zeroed header recovers every row for both 100-byte and whole-page-1 garbage.optimize-storagelegacy demote excludes themessages_fts_cjkfamily fromLIKE 'messages_fts_%'(xRename cascade into the cjk vtable). (salvage fix(state): exclude the cjk index family from the legacy FTS demote rename #103657 @liuhao1024, 2 commits; sessions optimize-storage fails on CJK-extension DBs: demote rename hits messages_fts_cjk shadows without the tokenizer loaded (vtable constructor failed) #103647)Live repro (real corruption fixtures, no mocked detectors)
DatabaseError code=267,classify → corrupt, turn fail-closes with.recoveradvicecause=fts_index, canonicalmessagesintact, turn continuessessionsroot page--fixwrites.malformed-backup-*copy of the damage, failssessions recover --inspect-only; no backup written; DEADBEEF FTS stomp still →rebuild_ftsHERMES_HOME=<root>/profiles/research, active_profile=other1. Run hermes doctor --fixhermes -p research doctor --fix/... sessions recover --source <root>/profiles/research/state.db --inspect-onlyrebuild_fts()on quarantined handlevacuum()messages_fts_trigram_idxrow at next segidintegrity_check=ok,_db_opens_cleanly→None, check-only "opens cleanly", doctor healthy, real append →IntegrityError constraint failed_db_opens_cleanly→'fts5 write probe failed: constraint failed'; check-only ✗ rc 1; doctor ⚠;repair_state_db_schemaheals, appends workstate_db_corrupt, 1 WARNING, 0 tracebacks (12 lines)OperationalError: fts5: error creating shadow table messages_fts_data: table already existson openfts_enabled=True, MATCH 5/5, trigram family rowids unchangedpage-level .recover salvage failed: file is not a database (26)Recovered 60 sessions and 300 messages, source md5 unchangedsessionsrow, NULLstarted_atIntegrityError: NOT NULL constraint failed: sessions.started_at, abortstatus=partial copied=3 destination_rejected_rows=1 verified=Truesessionsleaf cell-count\xff\xff, 400 rowslow=-9223372036854775808 copied=0 range_queries=10000 status=failedlow=1 high=400 copied=391 range_queries=40 status=partialfts5_cjk.so,optimize-storageno such table: messages_fts_cjk_datain_demote_legacy_fts_to_trash._stagefts_v22_trash_messages_fts_cjk*Root cause
Every corruption-shaped error was bucketed as whole-file damage, and the recovery lane assumed an openable file with a consistent schema — so FTS-only damage killed turns while doctor misnamed it, and the tools meant to salvage real incident databases refused the exact files those incidents produce.
Verified already fixed on main (no change here)
hermes_state_holderskeys on(st_dev, st_ino)and parses argv (06d7b77b1cb); live harness with a real second-filesystemstate.dbandgrep hermes-agent/journalctl -u hermes-agent/ ssh-wrapper argvs →holders=[]. fix(state): identify state.db holders by dev+inode, not path string #96011 and fix(state): keep unrelated processes from blocking FTS recovery #92419 are carried in that commit as Co-authored-by.060cd7f9bb5. fix(state): a permanent state.db holder no longer blocks the FTS rebuild forever (#106393, salvage #106559) #107981's remaining delta (rebuild anyway after futility) proceeds past uninspectable / deleted-WAL holders (itspid > 0guard ignores the sentinel prefixeslive_writer_holds_dbfails closed on) — not taken.Design calls made in this PR (flagging for review)
corrupt_unconfirmedquick_check tier: on 3.53.1quick_checkraises rather than reports on a torn shadow b-tree, so the probe could never downgrade the shape it targets, and softening quarantine guidance by probe weakens the fail-close.sessions repairheals trigram corruption), fix(state): scope uninspectable-holder fallback to the instance's own HERMES_HOME #105428 (argv-substring identity inference, fail-open case).Fixes #97794, #88587, #105887, #100227, #96591, #103840, #106667, #102240, #98050, #103647
Refs #106393, #96009, #92401, #56815, #106212, #86062
Infographic