state.db no longer gets a second writer from profile gateways, the dashboard, the lifecycle guard or doctor --fix (#102120 #107688 #102589 #107440; salvage #103489 #102682 #108012 #105428) - #108076
Merged
Conversation
…gateways must not write master state.db gateway.hosted_rooms.default_db_path() resolved profile gateways (~/.hermes/profiles/<name>/) to the ROOT state.db, making every profile gateway process a long-lived writer on the master session store. Across a 6-gateway fleet this was the recurring multi-writer corruption vector (WAL/FTS collisions during bot-gateway restart storms, observed 2026-09-03 with four state.db corruptions in five weeks fleet-wide; upstream refs #100896, #100313, #102827). Move hosted_room* tables to a dedicated root shared-state.db so profile gateways never open state.db writable. Single-gateway installs are unaffected (same file semantics, different filename); multi-profile fleets stop sharing the master store. Verified live in production: deployed 2026-09-04, zero I/O-error / reopen-race events across 9h+ of active multi-gateway load (previously ~1 failure per 2-3 min). Co-authored-by: Rik <rik.sedd@gmail.com>
…e.db (#103339) doctor --fix ran a raw sqlite3.connect + wal_checkpoint(PASSIVE) with no holder guard. Under a running gateway that second connection joins the live WAL and its close-time handling is the second-writer corruption class #103339 tracks. Route the checkpoint through live_writer_holds_db and skip with an actionable finding when the database is held. [Salvage note: the original PR also flipped the repair probe's DatabaseError branch to fail closed; that hunk is dropped here because it refuses every header-destroyed DB (the exact class repair exists for) — the probe's own connect raises before the EXCLUSIVE/BEGIN statements ever run.]
The dashboard's `_eager_reconcile_own_session_db` did an unconditional writable `acquire()` at every startup. When the gateway shares that state.db the dashboard became a second long-lived writable SessionDB owner: a close-time WAL checkpoint plus a possible FTS rebuild in `_init_fts`, the two-writer vector behind the corruption reports in #107688 and #100896 ("5 live SessionDB handles" precursor, gateway + dashboard both holding the WAL). Route the startup reconcile through `_open_session_db_at_path(..., read_only=True)`, which already bootstraps a missing store and heals a stale/malformed schema through exactly ONE writable open before reopening read-only. A healthy store now gets zero writable opens from the dashboard while the #79531/#80037 "bring schema current before the first poll" contract is kept (existing heal test unchanged). Live repro (healthy store, count writable SessionDB.__init__ calls from the startup worker): before=1 after=0. Reported-by: #107688, #100896 (@kokhlo diagnosis) Refs #107688 #100896
`hermes profile delete` leaves `profiles/.deleted/<name>` behind. `HostedRoomService.local_profiles()` fed every subdirectory name to the roster, so `.deleted` failed `validate_roster`'s identifier check and `plan_next_task` raised on every cycle for every room until the directory was removed by hand (#106847, bug 2). Skip dot-dirs and tombstoned profiles, using the same `named_profile_is_deleted` predicate `hermes_cli.profiles` uses to list live profiles. Refs #106847
…b; label sidecars Invariant test for the #103489 salvage: a profile gateway and the root gateway resolve the same hosted-room coordination file at the install root, and that file is never the root SessionDB (`state.db`). Every profile gateway starts the hosted-room worker unconditionally, so the old mapping made each profile process a long-lived writer on the master session store — the simultaneous-restart corruption in #102120 and the main offender in the #103339 / #103490 fleet reports. Also: WAL-fallback labels say `shared-state.db`, and test fakes take the store path from `default_db_path()` instead of hardcoding `state.db`. Reported-by: #102120 #103339 (@RChina) #103490 Refs #102120 #103339 #103490
… HERMES_HOME Field evidence (2026-09-07, production host): the host runs two independent Hermes instances - a main gateway (user ubuntu, HERMES_HOME=/home/ubuntu/.hermes) and a demo gateway (user demo, HERMES_HOME=/home/demo/.hermes). Because the demo process is owned by another user, its /proc/<pid>/fd table is unreadable and foreign_state_db_holders() falls back to cmdline + _looks_like_hermes(). The demo argv matches Hermes patterns exactly, so it was flagged as an uninspectable holder of the MAIN instance's state.db even though lsof proved 0 open handles on it. Result: _recover_stale_fts was deferred 42 times across 6 gateway restarts, the fts_stale breadcrumb never cleared, and FTS self-repair stayed permanently disabled. #92419 removed substring false positives (journalctl/grep mentioning hermes); a genuine second instance with a DIFFERENT HERMES_HOME was still misjudged. Add _argv_scoped_to_other_home(): when the argv of an uninspectable Hermes process proves it lives under a different /.hermes home (or a state.db sidecar under a different parent) AND no token references our state.db, sidecars, or home, do not count it as our holder. Applied to all three uninspectable branches (holder + two descriptor paths). Ambiguous argv without absolute-path tokens remains fail-closed, preserving the conservative intent. References #92401
Keep the two invariant tests from #105428: a genuine second instance whose argv is scoped to another HERMES_HOME is not a holder of our state.db; argv that names our state.db stays flagged. The helper-shape tests were change-detectors on `_argv_scoped_to_other_home` internals. The salvaged commit is re-authored to TaoMasterCoder's GitHub noreply identity: the original `devops@77hub.com` is a shared org address (misconfigured local git, not malice). Refs #107440
Contributor
૮ >ﻌ< ა ci reviewran on a88ab21 — test(state): trim holder-scope suite to the two behaviour co
|
9 tasks
alt-glitch
added
type/bug
andrexibiza
reviewed
Sep 11, 2026
andrexibiza
left a comment
andrexibiza
left a comment
Contributor
There was a problem hiding this comment.
Reviewed exact head a88ab21e0213079b9f6216af5674103b27adc4c2 against live main@cbd03e6e4ca143c1d5c2db881320afb85783c30b, including the full 15-file / 9-commit carrier, exact-head Actions, the canonical #102120 / #103339 defect reports, the live source/overlap carriers #103489 / #102682 / #108012 / #105428 / #102153 / #102176 / #102596, and the closed historical structural-gate work in #103362.
There is a lot of strong consolidation here. The lifecycle-guard change is the right primitive: offline_file_access holds the live-connection exclusion across the entire raw open/read/close sequence rather than doing a stale precheck. The dashboard change correctly removes the healthy-store writable open from a view process. The holder-scope salvage keeps ambiguous argv fail-closed while ending the cross-HERMES_HOME false positive. And moving hosted-room tables away from the master session store is a meaningful ownership reduction: profile gateways stop being writers on state.db merely because Group Chat exists.
I have two blockers on the current carrier, both at boundaries where the PR currently claims more closure than the implementation provides.
P1 — the hosted-room move isolates state.db, but it does not close #102120's shared-database race
The new regression explicitly pins:
assert from_profile == from_root, "one coordination file per install"
assert from_root.name != "state.db"That proves the useful isolation invariant, but it also proves that every profile gateway still converges on one shared SQLite file (shared-state.db). #102120's reproduced failure is not only “the wrong filename was state.db”; it is simultaneous multi-profile startup causing every hosted-room worker to open/init/prune the same SQLite database at the same time, with the failure occurring in the shared _connect() / journal-mode path. Renaming the target preserves that concurrency topology. The blast radius is much better — master sessions are no longer the victim — but Group Chat's own coordination DB can still hit the same first-open/prune race.
That means Fixes #102120 and “superseded by the routing fix: #102153 / #102176” are not supported yet. #103489 / RikETS owns the valuable isolation repair being salvaged here. #102153 (jwilson411: first-open serialization) and #102176 (Sahilvishnaliya: prune serialization) remain complementary work against the still-shared coordination store; they are not duplicates of the filename move.
Please either:
- keep #102120 open (
Refs, notFixes) and describe #102153/#102176 as complementary follow-up; or - close the actual shared-store race in this carrier and add a deterministic multi-profile simultaneous-start witness against
shared-state.dbproving initialization/prune cannot corrupt or strand the store.
I would not flatten the source histories: preserve RikETS's isolation work separately from the contributors who worked on serialization.
P1 — doctor --fix still consumes an expired / fail-open quiescence proof
The new doctor path asks live_writer_holds_db() for a boolean and then, after that helper has restored locking_mode=NORMAL and closed its probe, opens a different connection and executes wal_checkpoint(PASSIVE). A writer can legally enter in that gap. The lifecycle-guard half of this very PR correctly explains why “check, then raw I/O” is a race; doctor currently repeats the same shape across a process boundary.
There is a second problem in the same authority contract: the new comment says True means “held OR unprovable”, but the helper still returns False for non-lock OperationalError, plain DatabaseError, and other exceptions. The salvage note deliberately dropped #108012's global DatabaseError -> True change because repair needs different semantics for header-destroyed databases. That is a good reason not to overload the existing repair probe — but it also means False is not currently proof that doctor may perform structural work.
The fix should be a scoped structural/quiescence authority for the checkpoint, not another preflight: acquire/prove ownership atomically, prevent a new writer from becoming authorized, open + checkpoint + close while that authority remains live, then release it. Only a fully successful proof should authorize the effect; an unprovable state should skip the checkpoint. Add a deterministic adversarial test that pauses after the initial quiet observation, admits a writer before the checkpoint effect, and proves doctor performs zero checkpoint/fixed mutation. Keep the quiet-DB positive control.
This is the same proof-lifetime invariant that #103339 asks for. Closed-unmerged #103362 is useful historical design evidence for holding structural authority through the operation, but it had other correctness problems and should not be treated as a wholesale solution. Live #108012 is directly overlapping on this doctor seam and should be reconciled rather than silently superseded.
Interlocks / merge order / attribution
- #103489 (RikETS): directly salvaged and valuable; owns hosted-room isolation from master
state.db. - #102682 (JoaoMarcos44): directly salvaged; its
offline_file_accessscope is the correct check+use closure for the lifecycle reader. - #108012 (Halldrix): directly overlapping live doctor carrier; this PR salvages only part of it, so its remaining fail-closed/proof-lifetime discussion still matters.
- #105428 (TaoMasterCoder): directly salvaged holder-instance scoping; attribution is preserved in the commit series.
- #102153 / #102176: complementary hosted-room shared-store serialization, not superseded by changing
state.db→shared-state.db. - #102596 (chelsealong): overlapping lifecycle-guard report/fix direction; the #102682 salvage here is the stronger scoped implementation because it removes the check/use race.
- #103362: closed-unmerged historical structural-gate provenance only; useful for the invariant, not a replacement carrier.
Exact-object CI / integration state
Exact head is green: CI 34583634033, Docker 34583633371, and Nix 34583633316 all completed successfully. I also inspected the CI matrix; the blocking lint/test/attribution/supply-chain paths are green on a88ab21.
The surviving history is 1/9 commits with exact hosted workflow proof: the first eight exact SHAs (31eee30, 3797d3a, 35abf19, 322c96a, 2ba185b, 84df8dd, 704bb6f, c20ce64) currently have no PR workflow runs associated with those objects. Do not erase the source authorship to manufacture a cleaner graph, and do not add proof-only commits; the surviving objects need exact-SHA evidence under the repo's normal validation mechanism.
The branch is also 9 commits ahead / 8 behind live main from merge base f8456248d94eb76a1528968f061ed397c9e26db5. Current main touched hermes_cli/web_server_lifecycle.py in that intervening range, so the final integrated current-main object needs its own exact CI/Docker/Nix proof even though GitHub currently reports the PR mergeable.
This is substantial, useful consolidation. The ownership direction is right and the contributor salvage is unusually careful. The remaining work is to keep the same discipline at the two places that still cross an irreversible boundary: a shared coordination store needs serialization even after isolation, and a checkpoint needs authority that lives as long as the checkpoint itself. 🚀
| conn = sqlite3.connect(str(state_db_path)) | ||
| conn.execute("PRAGMA wal_checkpoint(PASSIVE)") | ||
| conn.close() | ||
| with contextlib.closing(sqlite3.connect(str(state_db_path))) as conn: |
Contributor
There was a problem hiding this comment.
P1 — this effect is no longer covered by the quiescence proof above. live_writer_holds_db() restores NORMAL/rolls back/closes its exclusive probe before returning False; this then creates a new connection. A gateway writer can enter between those two events, so wal_checkpoint(PASSIVE) can still run as the second live connection in the #103339 failure shape. Also, the helper currently returns False for non-lock OperationalError, DatabaseError, and other probe failures, so False is not equivalent to “quiet proven.” Please carry a scoped cross-process structural/quiescence authority through connect → checkpoint → close, and make unprovable probe outcomes refuse the effect. The regression needs the adversarial interleaving: quiet observation → writer admission attempt → checkpoint must not execute.
Contributor
|
@andrexibiza, @teknium1 — confirming the P1-2 overlap from our side, per the reconciliation call. Live #108012 (head One deliberate alignment with the salvage note: we dropped the preflight Neighbor suites green locally (49 passed, 3 environmental skips; no hosted checks on the fork head yet). Available to reconcile in either direction — rebase onto this carrier once it lands, or contribute the guard-held hunk here — whichever ordering you prefer. |
This was referenced Sep 11, 2026
This was referenced Sep 11, 2026
gabrielcosi
pushed a commit
to gabrielcosi/home-ops
that referenced
this pull request
Sep 12, 2026
…9.7 ➔ v2026.9.11) (#754) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/gabrielcosi/hermes-agent](https://github.com/NousResearch/hermes-agent) | patch | `v2026.9.7` → `v2026.9.11` | --- ### Release Notes <details> <summary>NousResearch/hermes-agent (ghcr.io/gabrielcosi/hermes-agent)</summary> ### [`v2026.9.11`](https://github.com/NousResearch/hermes-agent/releases/tag/v2026.9.11): Hermes Agent v0.21.2 (v2026.9.11) [Compare Source](NousResearch/hermes-agent@v2026.9.7...v2026.9.11) ##### Hermes Agent v0.21.2 (v2026.9.11) — The state.db Patch Release **Release Date:** September 11, 2026 > Patch release. v0.21.0 shipped a large rewrite of the session store's connection handling, and for some installs it made `state.db` fragile: second writers cancelling each other's locks, healthy databases reported as corrupt, one bad row killing `sessions list`. This release closes that class and rolls up everything else that landed on `main` in the four days since v0.21.1. ##### About this release Measured at commit `04dd80a977f40b05e5b2054111747af07a61886a`, the window since v0.21.1 contains **947 non-merge commits** across **1,869 changed files** (+182,504 / −15,564) and **312 merged PRs**. **140 contributors** appear in commits, co-author trailers, or salvage credits. ##### ✨ Highlights ##### state.db reliability campaign (six PRs, 44 issues closed) If your `state.db` broke after 0.21.0, this is the release for you. Six PRs fix the root causes rather than the symptoms: - **No more second writers.** Profile gateways wrote hosted-room state into the *root* `state.db` every 5 seconds; the dashboard opened a writable handle on startup; cron's lifecycle guard did a raw `open()` on a live database (which cancels the gateway's POSIX locks — the classic "how to corrupt SQLite" recipe); `doctor --fix` would checkpoint under a live holder. All four are gone: hosted rooms live in `shared-state.db`, the dashboard opens read-only first, the guard goes through the tracked connection registry, and `doctor --fix` refuses a checkpoint it can't prove is safe. ([#​108076](NousResearch/hermes-agent#108076) — salvage [#​103489](NousResearch/hermes-agent#103489) [@​RikETS](https://github.com/RikETS), [#​102682](NousResearch/hermes-agent#102682) [@​JoaoMarcos44](https://github.com/JoaoMarcos44), [#​108012](NousResearch/hermes-agent#108012) [@​Halldrix](https://github.com/Halldrix), [#​105428](NousResearch/hermes-agent#105428) [@​TaoMasterCoder](https://github.com/TaoMasterCoder)) - **Healthy WAL databases stop wedging.** OpenZFS `(deleted)` dentries and a `close()` racing an `append_message` both produced a sticky `DeletedWalGenerationError` on a perfectly good store; the read pool was handed out under an unconfirmed journal mode; a transient `disk I/O error` on WSL2 killed `get_session` on the first attempt; and a "state.db locked" banner was broadcast after the lock had already cleared. ([#​108082](NousResearch/hermes-agent#108082) — salvage [#​107411](NousResearch/hermes-agent#107411) [@​chelsealong](https://github.com/chelsealong), [#​105578](NousResearch/hermes-agent#105578) [@​ca-shrimp](https://github.com/ca-shrimp), [#​105711](NousResearch/hermes-agent#105711) [@​gaoanze888](https://github.com/gaoanze888), [#​106958](NousResearch/hermes-agent#106958) [@​nikkoxgonzales](https://github.com/nikkoxgonzales); co-authored [@​QDung210](https://github.com/QDung210), [@​fangliquanflq](https://github.com/fangliquanflq), [@​Sahilvishnaliya](https://github.com/Sahilvishnaliya)) - **FTS damage no longer kills your turn.** An error scoped to the full-text-search index was classified as whole-file corruption and fail-closed the conversation. It's now `fts_index`: search degrades, the index rebuilds later, the transcript store is untouched. Same PR: doctor names structural damage honestly instead of "FTS write corruption", the FTS write probe catches the stale-index shape that passed every check while every write failed, `.recover` output no longer fails startup on orphan FTS5 shadow tables, header-zeroed databases recover instead of being refused, and the dashboard analytics poller returns a 503 instead of 520K tracebacks a day. ([#​108130](NousResearch/hermes-agent#108130) — salvage [#​97843](NousResearch/hermes-agent#97843) [@​SulthanZahran1](https://github.com/SulthanZahran1) + [#​97841](NousResearch/hermes-agent#97841) [@​Finn763](https://github.com/Finn763), [#​88604](NousResearch/hermes-agent#88604) [#​56824](NousResearch/hermes-agent#56824) [#​103657](NousResearch/hermes-agent#103657) [@​liuhao1024](https://github.com/liuhao1024), [#​106890](NousResearch/hermes-agent#106890) [@​nftpoetrist](https://github.com/nftpoetrist), [#​103321](NousResearch/hermes-agent#103321) [@​jangomango76](https://github.com/jangomango76), [#​91413](NousResearch/hermes-agent#91413) [@​leegunwoo98](https://github.com/leegunwoo98), [#​102808](NousResearch/hermes-agent#102808) [@​TaoMasterCoder](https://github.com/TaoMasterCoder)) - **One corrupt row no longer kills `sessions list`, export, or insights.** A TEXT timestamp or a `1e30` epoch used to crash the whole listing; malformed marker JSON crashed `json_extract`; more than 999 ids crashed bulk delete/prune. One `coerce_epoch()` helper on every reader (bad rows render `?` with a WARNING naming the session), a `json_valid` guard, IN-list chunking, and batched export hydration. ([#​108086](NousResearch/hermes-agent#108086) — salvage [#​106071](NousResearch/hermes-agent#106071) [@​Xipong](https://github.com/Xipong), [#​101726](NousResearch/hermes-agent#101726) [@​efe-arv](https://github.com/efe-arv), [#​94701](NousResearch/hermes-agent#94701) [@​liuhao1024](https://github.com/liuhao1024), [#​102679](NousResearch/hermes-agent#102679) [@​mssteuer](https://github.com/mssteuer), [#​100658](NousResearch/hermes-agent#100658) [@​Mi55ed](https://github.com/Mi55ed)) - **Sessions never bind to or read another profile's database.** The Desktop launch backend could pin itself to the wrong profile's `state.db` under a HERMES\_HOME override race; `session_search` by bare ID silently scanned every profile and returned someone else's transcript; recovery guidance pointed at the wrong file; profile delete kept a handle open (WinError 32). ([#​108074](NousResearch/hermes-agent#108074) — salvage [#​102534](NousResearch/hermes-agent#102534) [@​HexLab98](https://github.com/HexLab98), [#​106975](NousResearch/hermes-agent#106975) [@​Sora-bluesky](https://github.com/Sora-bluesky)) - **Opening state.db no longer takes the write lock when nothing needs writing.** A one-shot `hermes` process opening the store behind a busy gateway stalled 4–20 s and then failed with "database is locked". Now 0.01 s. ([#​108067](NousResearch/hermes-agent#108067) — salvage [#​106751](NousResearch/hermes-agent#106751) [@​kshitijk4poor](https://github.com/kshitijk4poor), [#​101881](NousResearch/hermes-agent#101881) [@​jonpol01](https://github.com/jonpol01)) Also in the window from the same subsystem: a fresh `state.db` no longer publishes FTS tables before owning the rebuild lock ([#​106311](NousResearch/hermes-agent#106311)), a handle that lost its WAL generation no longer checkpoints stale frames at shutdown ([#​106315](NousResearch/hermes-agent#106315), [#​106840](NousResearch/hermes-agent#106840)), a clobbered first page is quarantined with its WAL instead of opened destructively ([#​106587](NousResearch/hermes-agent#106587)), WAL setup leaves an unverifiable database untouched ([#​106568](NousResearch/hermes-agent#106568)), and quarantined handles refuse VACUUM/FTS optimize ([#​106343](NousResearch/hermes-agent#106343), [#​106349](NousResearch/hermes-agent#106349)). Most of these salvaged community diagnoses by [@​kshitijk4poor](https://github.com/kshitijk4poor). ##### Multi-profile isolation hardening A cluster of fixes for installs running several profiles under one gateway (multiplex): secondary-profile bots no longer inherit the default profile's allow-lists ([#​107616](NousResearch/hermes-agent#107616)), adapters no longer send credentials to the default profile's host ([#​107617](NousResearch/hermes-agent#107617)), stdio MCP servers no longer receive the default profile's vault secrets ([#​107630](NousResearch/hermes-agent#107630)), `MEDIA:` delivery can no longer attach another profile's `.env` / `auth.json` / `state.db` ([#​107609](NousResearch/hermes-agent#107609)), Feishu drive callbacks and `/p/<profile>/` webhook replies stay on the routed profile ([#​107620](NousResearch/hermes-agent#107620), [#​107626](NousResearch/hermes-agent#107626)), and secondary profiles no longer get a sibling's Nous bearer from per-process memos ([#​107611](NousResearch/hermes-agent#107611)). ##### Desktop backend spawn storms are over Bot Mode used to spawn or dial one backend per profile on launch and on every roster tick, hovering the Bots roster spawned a backend per row, and profile switches could spawn a duplicate primary. ([#​108069](NousResearch/hermes-agent#108069), [#​108107](NousResearch/hermes-agent#108107), [#​108118](NousResearch/hermes-agent#108118), [#​108134](NousResearch/hermes-agent#108134), [#​107969](NousResearch/hermes-agent#107969), [#​108112](NousResearch/hermes-agent#108112) — salvage [#​102512](NousResearch/hermes-agent#102512), [#​103634](NousResearch/hermes-agent#103634), [#​103399](NousResearch/hermes-agent#103399), [#​107997](NousResearch/hermes-agent#107997) and others by [@​kshitijk4poor](https://github.com/kshitijk4poor)) ##### Password-blind credential vault The agent can now sign in, pay, and fill addresses from 1Password, Bitwarden, or the local Hermes vault without ever seeing a secret; two-factor codes come from a saved authenticator key or are asked for in the user's UI ([#​106480](NousResearch/hermes-agent#106480), [#​107585](NousResearch/hermes-agent#107585)). Private git plugins install with the user's stored credentials ([#​106981](NousResearch/hermes-agent#106981)). ##### Plugin catalog and one Plugins page A curated, SHA-pinned plugin index with CLI, admission CI, docs and dashboard ([#​69446](NousResearch/hermes-agent#69446)); Desktop gets one Plugins page owning agent + desktop plugins, install, catalog and per-commit pinning ([#​107212](NousResearch/hermes-agent#107212), [#​107314](NousResearch/hermes-agent#107314), [#​107321](NousResearch/hermes-agent#107321)); Radio ships as an opt-in SDK plugin ([#​107072](NousResearch/hermes-agent#107072)). ##### Nous free tier and guided first launch Free inference and connectors out of the box with one command to sign in ([#​105258](NousResearch/hermes-agent#105258), [#​105260](NousResearch/hermes-agent#105260)), `/login` from a chat ([#​105261](NousResearch/hermes-agent#105261)), connector tools (Gmail, Linear, Notion, ...) searchable through `tool_search` ([#​106842](NousResearch/hermes-agent#106842)), and a guided first launch behind `HERMES_GUEST_ONBOARDING=1` ([#​107697](NousResearch/hermes-agent#107697), [#​107958](NousResearch/hermes-agent#107958), [#​107985](NousResearch/hermes-agent#107985), [#​108211](NousResearch/hermes-agent#108211)). ##### 🐛 Notable Bug Fixes **Gateway & platforms** - A bare `display:` key in config.yaml no longer crashes every gateway turn ([#​106305](NousResearch/hermes-agent#106305)); a queued-lane final refused by the platform is recorded and redelivered ([#​106316](NousResearch/hermes-agent#106316)); a stalled WebSocket send no longer blocks every later event ([#​106581](NousResearch/hermes-agent#106581)); the first turn no longer waits on the Python toolchain probe ([#​106556](NousResearch/hermes-agent#106556)). - Telegram bots must @​mention when `bots_require_mention` is on, breaking bot-to-bot loops ([#​106534](NousResearch/hermes-agent#106534)); Matrix renders LaTeX ([#​106515](NousResearch/hermes-agent#106515)); Signal renders markdown tables ([#​106538](NousResearch/hermes-agent#106538)); WhatsApp replies to view-once messages keep their quote ([#​106541](NousResearch/hermes-agent#106541)); media-only replies report SUCCESS everywhere ([#​106557](NousResearch/hermes-agent#106557)). **Providers & routing** - `/model` and auxiliary auto never bill a provider you didn't select ([#​107366](NousResearch/hermes-agent#107366)); never auto-switch to a provider you have no credentials for ([#​107281](NousResearch/hermes-agent#107281)); Bedrock Claude/Converse/Mantle models survive `/model`, fallback and restore ([#​107621](NousResearch/hermes-agent#107621), [#​107658](NousResearch/hermes-agent#107658)); Bedrock Guardrails enforced ([#​107815](NousResearch/hermes-agent#107815)). - Codex: patch-budget image 400 shrinks and retries ([#​106525](NousResearch/hermes-agent#106525)); unentitled primary + fallback no longer oscillate ([#​106549](NousResearch/hermes-agent#106549)); Azure Foundry replayed-reasoning rejection classified and pruned ([#​106718](NousResearch/hermes-agent#106718) [@​erosika](https://github.com/erosika)). MCP OAuth refresh no longer erases the refresh token ([#​106185](NousResearch/hermes-agent#106185)). Anthropic clients send exactly one credential ([#​107978](NousResearch/hermes-agent#107978)). - DeepSeek V4.1 Flash on Nous Portal and OpenRouter pickers ([#​107489](NousResearch/hermes-agent#107489)); GPT Image 2.5 via OpenAI and FAL ([#​105988](NousResearch/hermes-agent#105988)); Opus 5 / Fable 5.1 on the native Anthropic picker ([#​106636](NousResearch/hermes-agent#106636) [@​xxxigm](https://github.com/xxxigm)). **Agent loop & compression** - One blocked periodic callback no longer stalls lease refresh ([#​106308](NousResearch/hermes-agent#106308)); a mid-turn `/steer` is persisted as its own user row ([#​106317](NousResearch/hermes-agent#106317), [#​106344](NousResearch/hermes-agent#106344)); local-inference memory-ceiling rejections back off instead of compressing history ([#​106307](NousResearch/hermes-agent#106307)); context-overflow after partial streaming ends the turn ([#​106567](NousResearch/hermes-agent#106567)); length continuation stops when the prompt filled the window ([#​106571](NousResearch/hermes-agent#106571)); compression no longer times out silently on aux retries ([#​106866](NousResearch/hermes-agent#106866)); `model_thresholds` keys can be provider-scoped ([#​108061](NousResearch/hermes-agent#108061)). - Surface switch (Desktop↔TUI) no longer rebuilds the system prompt and busts the prompt cache ([#​105844](NousResearch/hermes-agent#105844)); CLI keeps the `api_content` sidecar so the cache survives an early persist ([#​105842](NousResearch/hermes-agent#105842)). **CLI, TUI & Desktop** - `hermes -z --resume` continues the session ([#​106313](NousResearch/hermes-agent#106313)); Shift+letter and Cmd+Shift+Z work on extended-key terminals ([#​90674](NousResearch/hermes-agent#90674) [@​francip](https://github.com/francip), [#​105493](NousResearch/hermes-agent#105493)); `browser_exec` timeout kills the whole process tree ([#​106589](NousResearch/hermes-agent#106589)); update checks poll the GitHub API once a day instead of git-fetching every 30 min ([#​107648](NousResearch/hermes-agent#107648)); `hermes update` names the real cause and can't hang on a stalled fetch ([#​108053](NousResearch/hermes-agent#108053)). - Desktop: UI language survives the update relaunch ([#​106476](NousResearch/hermes-agent#106476)), expired OAuth grants get a one-click re-sign-in ([#​106965](NousResearch/hermes-agent#106965)), HUD mode shows the transcript again and always gives the window back ([#​107491](NousResearch/hermes-agent#107491), [#​107423](NousResearch/hermes-agent#107423)), the backend exits when its Desktop parent dies ([#​107977](NousResearch/hermes-agent#107977)), Windows updates stop reporting false failures ([#​106175](NousResearch/hermes-agent#106175), [#​107183](NousResearch/hermes-agent#107183)), WSLg renders on the Windows GPU ([#​106528](NousResearch/hermes-agent#106528)), Telegram quick setup with QR ported from the dashboard ([#​107242](NousResearch/hermes-agent#107242)), and \~60 more Desktop fixes largely from [@​OutThisLife](https://github.com/OutThisLife) and [@​kshitijk4poor](https://github.com/kshitijk4poor). **Cron & Kanban** - An off-tick "run now" no longer cancels the next scheduled run ([#​106306](NousResearch/hermes-agent#106306)); a killed manual run no longer blocks the next one for 5 minutes ([#​106733](NousResearch/hermes-agent#106733)); a one-shot changed to recurring keeps firing ([#​106532](NousResearch/hermes-agent#106532)); unpinned jobs run on their creation-snapshot model ([#​106499](NousResearch/hermes-agent#106499)); `--clone-all` no longer copies cron jobs ([#​106478](NousResearch/hermes-agent#106478)); `kanban promote` refuses undone parents ([#​106550](NousResearch/hermes-agent#106550)); `kanban_request_review` rejects unknown reviewer profiles ([#​106547](NousResearch/hermes-agent#106547)). **Tools & memory** - A stdio MCP server dying mid-call no longer replays the tool call ([#​106546](NousResearch/hermes-agent#106546)); a skills-only background review can no longer delete memory entries ([#​106310](NousResearch/hermes-agent#106310)); mem0 memory no longer drops long turns ([#​106542](NousResearch/hermes-agent#106542)); `tool_search` returns nothing rather than five tools sharing one word ([#​106676](NousResearch/hermes-agent#106676)); remote NOPASSWD sudo no longer prompts ([#​107939](NousResearch/hermes-agent#107939)); RSS and Reddit reading no longer activate by default ([#​105873](NousResearch/hermes-agent#105873)). **Housekeeping** - `config.yaml` backups live in one bounded `backups/config/` dir ([#​106388](NousResearch/hermes-agent#106388)); `hermes backup` keeps the newest 3 zips ([#​106455](NousResearch/hermes-agent#106455)); `hermes setup --reset` backs up the real config ([#​106453](NousResearch/hermes-agent#106453)); `debug share` retention shrunk to 1 day on the dpaste fallback ([#​106531](NousResearch/hermes-agent#106531)). ##### 👥 Contributors Thank you to the **140 contributors** whose commits, co-author trailers, and salvaged PRs landed in this window. **state.db campaign — salvaged PR authors:** [@​RikETS](https://github.com/RikETS), [@​JoaoMarcos44](https://github.com/JoaoMarcos44), [@​Halldrix](https://github.com/Halldrix), [@​TaoMasterCoder](https://github.com/TaoMasterCoder), [@​chelsealong](https://github.com/chelsealong), [@​ca-shrimp](https://github.com/ca-shrimp), [@​gaoanze888](https://github.com/gaoanze888), [@​nikkoxgonzales](https://github.com/nikkoxgonzales), [@​QDung210](https://github.com/QDung210), [@​fangliquanflq](https://github.com/fangliquanflq), [@​Sahilvishnaliya](https://github.com/Sahilvishnaliya), [@​kshitijk4poor](https://github.com/kshitijk4poor), [@​jonpol01](https://github.com/jonpol01), [@​HexLab98](https://github.com/HexLab98), [@​Sora-bluesky](https://github.com/Sora-bluesky), [@​Xipong](https://github.com/Xipong), [@​efe-arv](https://github.com/efe-arv), [@​liuhao1024](https://github.com/liuhao1024), [@​mssteuer](https://github.com/mssteuer), [@​Mi55ed](https://github.com/Mi55ed), [@​SulthanZahran1](https://github.com/SulthanZahran1), [@​Finn763](https://github.com/Finn763), [@​nftpoetrist](https://github.com/nftpoetrist), [@​jangomango76](https://github.com/jangomango76), [@​leegunwoo98](https://github.com/leegunwoo98), [@​ggoldani](https://github.com/ggoldani). **state.db campaign — issue reporters** (the forensics in these threads were often better than the fixes): [@​thedigitalcarpenterdad](https://github.com/thedigitalcarpenterdad), [@​Rroven](https://github.com/Rroven), [@​aoeman84](https://github.com/aoeman84), [@​StephanRosin](https://github.com/StephanRosin), [@​rubensandrade-sketch](https://github.com/rubensandrade-sketch), [@​wanliqin](https://github.com/wanliqin), [@​chenzheshushi-commits](https://github.com/chenzheshushi-commits), [@​CarlosReyesPena](https://github.com/CarlosReyesPena), [@​revazone](https://github.com/revazone), [@​reservassai-art](https://github.com/reservassai-art), [@​Cuttingwater](https://github.com/Cuttingwater), [@​soroush5](https://github.com/soroush5), [@​e-shizz](https://github.com/e-shizz), [@​shobhit-87labs](https://github.com/shobhit-87labs), [@​shivanathd](https://github.com/shivanathd), [@​hoelzl](https://github.com/hoelzl), [@​i8ei](https://github.com/i8ei), [@​Ace-Kelly](https://github.com/Ace-Kelly), [@​YinsenWANG](https://github.com/YinsenWANG), [@​zbabiarz](https://github.com/zbabiarz), [@​Sravanjangam](https://github.com/Sravanjangam), [@​0gl20shk0sbt36](https://github.com/0gl20shk0sbt36), [@​RChina](https://github.com/RChina), [@​bronder](https://github.com/bronder), [@​ccwssy](https://github.com/ccwssy), [@​bottenbenny](https://github.com/bottenbenny), and [@​Hitman117890](https://github.com/Hitman117890) whose Discord report kicked the campaign off. **Everyone in the window (alphabetical):** [@​0genlab](https://github.com/0genlab), [@​0xalydev](https://github.com/0xalydev), [@​100yenadmin](https://github.com/100yenadmin), [@​1052326311](https://github.com/1052326311), [@​686f6c61](https://github.com/686f6c61), [@​69k4xmdfm2-blip](https://github.com/69k4xmdfm2-blip), [@​abundantbeing](https://github.com/abundantbeing), [@​Adolanium](https://github.com/Adolanium), [@​Ahmett101](https://github.com/Ahmett101), [@​albert748](https://github.com/albert748), [@​AlexxRussell](https://github.com/AlexxRussell), [@​alt-glitch](https://github.com/alt-glitch), [@​auroracapital](https://github.com/auroracapital), [@​austinpickett](https://github.com/austinpickett), [@​babatorik](https://github.com/babatorik), [@​Bartok9](https://github.com/Bartok9), [@​benbarclay](https://github.com/benbarclay), [@​bennybuoy](https://github.com/bennybuoy), [@​brian717](https://github.com/brian717), [@​briandevans](https://github.com/briandevans), [@​buihongduc132](https://github.com/buihongduc132), [@​ca-shrimp](https://github.com/ca-shrimp), [@​cervantesh](https://github.com/cervantesh), [@​Cesar-Azeredo](https://github.com/Cesar-Azeredo), [@​ChanPark03](https://github.com/ChanPark03), [@​chelsealong](https://github.com/chelsealong), [@​ckomma](https://github.com/ckomma), [@​ClintonEmok](https://github.com/ClintonEmok), [@​crazyief](https://github.com/crazyief), [@​ctaylor86](https://github.com/ctaylor86), [@​dalzio](https://github.com/dalzio), [@​DavidMetcalfe](https://github.com/DavidMetcalfe), [@​Drexuxux](https://github.com/Drexuxux), [@​edosulai](https://github.com/edosulai), [@​efe-arv](https://github.com/efe-arv), [@​emozilla](https://github.com/emozilla), [@​ericmaddox](https://github.com/ericmaddox), [@​erosika](https://github.com/erosika), [@​ethernet8023](https://github.com/ethernet8023), [@​everm1nd](https://github.com/everm1nd), [@​FalconOrtiz](https://github.com/FalconOrtiz), [@​fangliquanflq](https://github.com/fangliquanflq), [@​Finn763](https://github.com/Finn763), [@​FirmamentalSpring](https://github.com/FirmamentalSpring), [@​francip](https://github.com/francip), [@​g3org3yo](https://github.com/g3org3yo), [@​gaoanze888](https://github.com/gaoanze888), [@​ggoldani](https://github.com/ggoldani), [@​Halldrix](https://github.com/Halldrix), [@​haydster7](https://github.com/haydster7), [@​hbizi](https://github.com/hbizi), [@​helix4u](https://github.com/helix4u), [@​HexLab98](https://github.com/HexLab98), [@​huklaa](https://github.com/huklaa), [@​IAvecilla](https://github.com/IAvecilla), [@​infinitycrew39](https://github.com/infinitycrew39), [@​jahfaliabdulrahman-dev](https://github.com/jahfaliabdulrahman-dev), [@​jangomango76](https://github.com/jangomango76), [@​JoaoMarcos44](https://github.com/JoaoMarcos44), [@​jonpol01](https://github.com/jonpol01), [@​jwilson411](https://github.com/jwilson411), [@​KeyArgo](https://github.com/KeyArgo), [@​kokhlo](https://github.com/kokhlo), [@​KoNit-K](https://github.com/KoNit-K), [@​kshitijk4poor](https://github.com/kshitijk4poor), [@​kyssta-exe](https://github.com/kyssta-exe), [@​leegunwoo98](https://github.com/leegunwoo98), [@​lesterlxt](https://github.com/lesterlxt), [@​liuhao1024](https://github.com/liuhao1024), [@​Mabolla](https://github.com/Mabolla), [@​manuelschipper](https://github.com/manuelschipper), [@​MaxFreedomPollard](https://github.com/MaxFreedomPollard), [@​mearls0501](https://github.com/mearls0501), [@​mengyuyuan](https://github.com/mengyuyuan), [@​Mi55ed](https://github.com/Mi55ed), [@​MiseHinoha](https://github.com/MiseHinoha), [@​mjshorty](https://github.com/mjshorty), [@​mkrb84](https://github.com/mkrb84), [@​moisesvalero](https://github.com/moisesvalero), [@​moken627-hub](https://github.com/moken627-hub), [@​mssteuer](https://github.com/mssteuer), [@​nateEc](https://github.com/nateEc), [@​nftpoetrist](https://github.com/nftpoetrist), [@​nickseelert](https://github.com/nickseelert), [@​nikkoxgonzales](https://github.com/nikkoxgonzales), [@​notwitcheer](https://github.com/notwitcheer), [@​onuraycicek](https://github.com/onuraycicek), [@​outdog-hwh](https://github.com/outdog-hwh), [@​OutThisLife](https://github.com/OutThisLife), [@​philmossman](https://github.com/philmossman), [@​phuongvm](https://github.com/phuongvm), [@​pierrenode](https://github.com/pierrenode), [@​portavales](https://github.com/portavales), [@​PRATHAMESH75](https://github.com/PRATHAMESH75), [@​QDung210](https://github.com/QDung210), [@​rewbs](https://github.com/rewbs), [@​RikETS](https://github.com/RikETS), [@​romanovzky](https://github.com/romanovzky), [@​ryantuc](https://github.com/ryantuc), [@​Sahilvishnaliya](https://github.com/Sahilvishnaliya), [@​salch-cred](https://github.com/salch-cred), [@​sgarrand](https://github.com/sgarrand), [@​shannonsands](https://github.com/shannonsands), [@​simpolism](https://github.com/simpolism), [@​Solitud1nem](https://github.com/Solitud1nem), [@​somewheresy](https://github.com/somewheresy), [@​Sora-bluesky](https://github.com/Sora-bluesky), [@​sprmn24](https://github.com/sprmn24), [@​squevo](https://github.com/squevo), [@​StellarisW](https://github.com/StellarisW), [@​Stoltemberg](https://github.com/Stoltemberg), [@​SulthanZahran1](https://github.com/SulthanZahran1), [@​Svector-anu](https://github.com/Svector-anu), [@​szicely](https://github.com/szicely), [@​TaoMasterCoder](https://github.com/TaoMasterCoder), [@​teknium1](https://github.com/teknium1), [@​ten82e](https://github.com/ten82e), [@​thedavidweng](https://github.com/thedavidweng), [@​tkaufmann](https://github.com/tkaufmann), [@​Totoro-qaq](https://github.com/Totoro-qaq), [@​Tranquil-Flow](https://github.com/Tranquil-Flow), [@​tuancookiez-hub](https://github.com/tuancookiez-hub), [@​TurgutKural](https://github.com/TurgutKural), [@​ugoenyioha](https://github.com/ugoenyioha), [@​unsupportedpastels](https://github.com/unsupportedpastels), [@​victor-kyriazakos](https://github.com/victor-kyriazakos), [@​webtecnica](https://github.com/webtecnica), [@​wliu-dev](https://github.com/wliu-dev), [@​wukangcheng1994](https://github.com/wukangcheng1994), [@​Xipong](https://github.com/Xipong), [@​Xixiartemis](https://github.com/Xixiartemis), [@​xkam7ar](https://github.com/xkam7ar), [@​xxxigm](https://github.com/xxxigm), [@​yavarb](https://github.com/yavarb), [@​yoniebans](https://github.com/yoniebans), [@​Youssef](https://github.com/Youssef), [@​yoyodine-industries](https://github.com/yoyodine-industries), [@​yuanchenglu](https://github.com/yuanchenglu), [@​YuhGuan](https://github.com/YuhGuan), [@​Zeus-Deus](https://github.com/Zeus-Deus). Also: Youssef. ##### Updating - Existing install: `hermes update` - Fresh install: `curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash` - Managed deployments should update through their deployment tooling using the new tag. - If your `state.db` was already damaged by 0.21.0/0.21.1: run `hermes doctor` first; it now names structural vs index damage correctly and points at `hermes sessions recover --inspect-only` (profile-pinned) when a rebuild isn't enough. **Full Changelog:** [v2026.9.7...v2026.9.11](NousResearch/hermes-agent@v2026.9.7...v2026.9.11) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/Berlin) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC42NS4wIiwidXBkYXRlZEluVmVyIjoiNDQuNjUuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==--> Reviewed-on: https://git.xcd.dev/gabrielcosi/home-ops/pulls/754
This was referenced Sep 14, 2026
Closed
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Profile gateways, the dashboard, the lifecycle guard and
doctor --fixno longer become a second writer (or lock-cancelling raw reader) on a livestate.db.What changed
gateway/hosted_rooms.py::default_db_path, salvaged from fix(gateway): isolate hosted-room state in shared-state.db — profile gateways must not write master state.db #103489 by @RikETS): the cross-profile Group Chat coordination store is<install root>/shared-state.db, not the root gateway'sstate.db. Every profile gateway starts the hosted-room worker unconditionally, so the old mapping made each profile process a long-lived writer on the master session store — the simultaneous multi-profile restart corruption in hosted_room_worker corrupts shared state.db on simultaneous multi-profile gateway restart (e.g. viahermes update) #102120 and the main offender in the [Bug] Second writer viadoctor --fix/repair_state_db_schema/ hosted_rooms still corrupts live-WAL state.db — upstream guards are fail-open; proposing a lazy flock single-writer gate (field-verified) #103339 / Fleet-scale state.db amplification: #94736-class races scale with gateway count — observations from a 6-gateway production install #103490 fleet reports. WAL labels and test fakes follow the resolver.tui_gateway/hosted_room_service.py::local_profiles): skips theprofiles/.deleted/tombstone dir (plan_next_taskraised every cycle for every room afterhermes profile delete, Bot Mode: hosted room turns silently refused when gateway + Desktop serve both run room workers (stale bot_room active-session slots); profile delete tombstone breaks local_profiles() #106847 bug 2).hermes_cli/web_server_lifecycle.py::_eager_reconcile_own_session_db): startup schema reconcile goes through_open_session_db_at_path(read_only=True). A healthy store gets zero writable opens from the dashboard; the stale/malformed-schema heal (Cross-platform update regression: existing sessions disappear when state.db migration is skipped #79531/Session list API fails with "no such column: s.last_read_at" after update until first new session triggers migration #80037 contract) still runs through exactly one writable open. Removes the gateway+dashboard two-writer topology in Dashboard does an unconditional writable SessionDB open at startup, exposing the documented concurrent-FTS-rebuild corruption vector #107688 / [Bug] state.db corruption x4 in 5 weeks (gateway+dashboard multi-writer WAL) — "5 live SessionDB handles" warning fires 7 min before onset; journal_mode=delete containment #100896.cron/lifecycle_guard.py::_read_referenced_script, salvaged from fix(cron): serialize lifecycle script reads with SQLite locks #102682 by @JoaoMarcos44): the rawos.open/read/closeof any path-like token is wrapped inoffline_file_access, the tracked live-connection registry. A command that merely mentions the gateway's ownstate.dbno longer cancels the gateway's POSIX locks (cron/lifecycle_guard raw-opens state.db when a command mentions its path, dropping the gateway's POSIX locks -> WAL split-brain #102589).hermes_cli/doctor_state.py::_state_db_wal, salvaged from fix(state): refuse checkpoint and repair on unprovable live-writer state (#103339) #108012 by @Halldrix): the rawwal_checkpoint(PASSIVE)is gated onlive_writer_holds_db; under a running gateway it skips with an actionable finding instead of joining the live WAL as a second connection ([Bug] Second writer viadoctor --fix/repair_state_db_schema/ hosted_rooms still corrupts live-WAL state.db — upstream guards are fail-open; proposing a lazy flock single-writer gate (field-verified) #103339 trigger 2). The PR's second hunk (repair probe fails closed onDatabaseError) was dropped: it refused every header-destroyed DB — the class repair exists for — and the existing repair-lock ownership check already covers that path.hermes_state_holders.py, salvaged from fix(state): scope uninspectable-holder fallback to the instance's own HERMES_HOME #105428 by @TaoMasterCoder, re-authored to their GitHub noreply — the original author email is a shared org address): a genuine second Hermes instance whose argv is provably scoped to a differentHERMES_HOMEis no longer counted as an uninspectable holder of ourstate.db, so stale-FTS repair is not deferred forever ([Bug]: uninspectable-holder fallback counts a genuine second Hermes instance under a different HERMES_HOME as a state.db holder, permanently deferring stale-FTS repair #107440). Ambiguous argv stays fail-closed. Test suite trimmed to the two behaviour contracts.Live repro (real SQLite, temp
HERMES_HOME,sys.path[0]= tree under test)origin/main.hermes/state.db(writes master).hermes/shared-state.dbSessionDBopens on a healthy store/proc/locks)unsafe=True)doctor --fixcheckpointed under a live foreign-process holderrepair_state_db_schemaon a header-destroyed file (regression guard)Root cause
Several long-lived processes (profile gateways via hosted rooms, the dashboard) held their own writable connection to one
state.db, and two maintenance paths (lifecycle guard, doctor checkpoint) opened the live file outside the tracked registry — the file had many owners, and any of them closing/checkpointing under the real owner is the WAL split-brain / lock-cancellation corruption class.Already on main, not redone
#94736 self-heal reopen (#102827), read-pool isolation (0e9b57d / e8c4156), live-connection guard registry (
hermes_state_holders,offline_file_access), API-serverSessionDBcache throughhermes_state_registry.acquire(aad74f2 — #100682's premise no longer holds), doctor_session_counttry/finally (2a9c5d8 — #100836's premise no longer holds).Not in this PR
bot_roomactive-session slots outliving the driver lease) — a session-reaper/lease design change, not a writer-topology fix.session_db_recovery.pyrejection-path shape; needs a fresh look against main.state.db: rooms are re-created inshared-state.dbon first use; no migration of in-flight rooms.Fixes #102120
Fixes #107688
Fixes #102589
Fixes #107440
Refs #103339 #103490 #100896 #106847 #100682 #100836 #96027
Salvaged with authorship: #103489 (@RikETS), #102682 (@JoaoMarcos44), #108012 (@Halldrix), #105428 (@TaoMasterCoder). Superseded by the routing fix: #102153 (@jwilson411), #102237 (@salch-cred), #102176. Supersedes #102596 (@chelsealong, check-then-act race).