fix(gateway): run state.db auto-prune on the housekeeping cadence, not just at startup - #79
Conversation
… at startup The sessions auto-maintenance sweep (archive + prune ended sessions + throttled VACUUM) only ever ran at process construction. A gateway that stays resident for weeks under launchd/systemd therefore never pruned again, so ~/.hermes/state.db grew without bound even with sessions.auto_prune enabled — the 2026-08-15 disk-pressure incident found a 3GB state.db of ~97% real rows on exactly that profile. - Extract the construction-time maintenance block into GatewayRunner._run_state_db_maintenance_once(): same behavior, but config is re-read on every call so operators can flip sessions.auto_prune on a live gateway without restarting it. - Add a supervised _state_db_maintenance_watcher spawned from start(): hourly tick, each tick delegates to the SessionDB maybe_* helpers, which self-throttle via state_meta (min_interval_hours, default 24h; VACUUM additionally via min_vacuum_interval_days) — so the tick is a cheap point-read almost every time, the real work runs at most once per configured interval, and the throttle is shared across every hermes process. The sweep runs in a worker thread (asyncio.to_thread) because prune takes SessionDB's write lock and a due VACUUM can hold it for seconds per 100MB. - Unit tests for the extracted pass (config forwarding, gates, never-raises) and the watcher (re-runs on cadence, stops cleanly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WNiHNUntm1Sa9ob2KdCz49
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da0723319d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| min_vacuum_interval_days=int( | ||
| _sess_cfg.get("min_vacuum_interval_days", 30) | ||
| ), | ||
| vacuum=bool(_sess_cfg.get("vacuum_after_prune", True)), |
There was a problem hiding this comment.
When sessions.auto_prune is enabled, the default vacuum_after_prune: true now lets an hourly watcher invoke SessionDB.vacuum() while the gateway is serving traffic. That method explicitly requires callers to ensure no writers are active and holds an exclusive SQLite lock for the complete rewrite; on the multi-GB database motivating this change, it can outlast the 60-second transcript-write retry budget and cause active turns to fail persistence. Running it through to_thread keeps the event loop responsive but does not prevent database starvation, so periodic passes should prune without VACUUM or coordinate an idle/drained window while retaining VACUUM for startup. The added tests mock _db and therefore do not exercise this locking path against a real temporary store.
AGENTS.md reference: AGENTS.md:L84-L87
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
All three findings adopted in 83e0f35 — the review was right that the housekeeping lane was the existing scheduler to extend, not duplicate. The separate asyncio watcher is gone; the hourly prune now lives in _start_gateway_housekeeping's session-maintenance branch (the former AUTO_ARCHIVE_EVERY branch, so archive keeps exactly one scheduler), always with vacuum=False on live ticks — VACUUM stays owned by the pre-serve construction pass — and the branch iterates profiles_to_serve(multiplex=…) under _profile_runtime_scope so every multiplexed profile's state.db gets its own config-gated sweep. Tests rewritten accordingly (test_state_db_periodic_maintenance.py).
Generated by Claude Code
| await asyncio.sleep(self._STATE_DB_MAINTENANCE_TICK_SECS) | ||
| if not self._running: | ||
| return | ||
| await asyncio.to_thread(self._run_state_db_maintenance_once) |
There was a problem hiding this comment.
In a gateway with gateway.multiplex_profiles enabled, this periodic call only operates on self._session_db, which was constructed for the process's active/default HERMES_HOME, and _run_state_db_maintenance_once() also reloads only that profile's config. Secondary-profile turns run under their own profile runtime scopes and persist to separate profiles/<name>/state.db files, so even when those profiles enable sessions.auto_prune, their databases still receive no recurring maintenance and retain the same unbounded-growth failure this change is intended to fix. Iterate the served profile homes and run each profile's configured sweep against its own SessionDB.
AGENTS.md reference: AGENTS.md:L54-L57
Useful? React with 👍 / 👎.
| if _sess_cfg.get("auto_archive", False): | ||
| self._session_db._db.maybe_auto_archive( |
There was a problem hiding this comment.
The gateway already runs maybe_auto_archive() hourly from _start_gateway_housekeeping via its AUTO_ARCHIVE_EVERY branch, but the new hourly watcher calls it again here. Because the state_meta check and archive operation are not one atomic transaction, coincident due ticks can both select the same stale rows and repeat the full per-session update sweep, doubling write contention and logging duplicate archive counts on large stores. Keep auto-archive in the existing housekeeping path and limit this new periodic path to the previously missing prune operation, rather than maintaining two independent schedulers for the same chore.
AGENTS.md reference: AGENTS.md:L75-L79
Useful? React with 👍 / 👎.
Addresses the Codex review on the periodic-maintenance change: - One scheduler, not two: the new hourly prune now lives in _start_gateway_housekeeping's existing session-maintenance branch (formerly the AUTO_ARCHIVE_EVERY branch) instead of a second supervised asyncio watcher that double-scheduled maybe_auto_archive against the housekeeping tick. The duplicate watcher is removed. - No VACUUM on live ticks: the housekeeping sweep always passes vacuum=False. SessionDB.vacuum() holds an exclusive lock for a full DB rewrite and is only safe before traffic is served, so VACUUM stays owned by the construction-time pass (_run_state_db_maintenance_once, which runs pre-serve and keeps honoring vacuum_after_prune). - Multiplexed profiles are swept too: the branch iterates profiles_to_serve(multiplex=gateway.multiplex_profiles) and runs each profile's archive + prune under its own _profile_runtime_scope, so every profiles/<name>/state.db gets recurring retention gated by that profile's own sessions config. The multiplex flag is threaded from runner.config at the housekeeping thread launch site. - Tests rewritten around the housekeeping branch: live ticks prune with vacuum=False, multiplex sweeps mint one profile-scoped SessionDB per home and close it, disabled profiles open no DB, and the startup pass still forwards vacuum config. File renamed to test_state_db_periodic_maintenance.py to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WNiHNUntm1Sa9ob2KdCz49
pai-scaffolde
changed the title
… (#81) PR #64 was replayed onto main with the wrong merge base (its parent branch's TIP rather than the true fork point). Files main had gained after that branch forked therefore looked like deletions, so merging #64 silently reverted #79: * gateway/run.py — _run_state_db_maintenance_once() refactor undone * hermes_cli/config_defaults.py — housekeeping comment reverted * tests/gateway/test_state_db_periodic_maintenance.py — deleted This re-applies #79 verbatim onto current main. The LSP work from #63 and #64 (including sessions.max_clients) is untouched. Audited: #63 and #65 match their original diffstats exactly; #64 was the only bad replay. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2 participants
What does this PR do?
~/.hermes/state.dbalready has a full row-level retention path —SessionDB.prune_sessions()deletes ended sessions'messagesrows plus unreferencedsystem_prompts, andmaybe_auto_prune_and_vacuum()wraps it with state_meta throttling and a VACUUM — but it only ever ran at process construction (CLI startup, gateway__init__). A gateway resident for weeks under launchd/systemd never pruned again, so state.db grew without bound even withsessions.auto_pruneenabled. The 2026-08-15 disk-pressure remediation found a 3GB state.db of ~97% real rows (auto_vacuum=0, freelist only ~155MB) on exactly that profile.This PR gives the existing pruner a recurring cadence, wired into the gateway's existing hourly housekeeping lane (revised per Codex review — the first cut used a separate supervised watcher, which double-scheduled auto-archive and risked live VACUUMs):
_start_gateway_housekeepingsession-maintenance branch — the formerAUTO_ARCHIVE_EVERYbranch now runs archive and prune, hourly, per served profile. Each profile is swept under its own_profile_runtime_scopeviaprofiles_to_serve(multiplex=gateway.multiplex_profiles), so a multiplexed gateway'sprofiles/<name>/state.dbfiles each get retention gated by their ownsessionsconfig. Themaybe_*helpers self-throttle viastate_meta(min_interval_hours, default 24h), so the hourly tick is a cheap point-read almost every time. Live ticks always passvacuum=False—SessionDB.vacuum()holds an exclusive lock for a full DB rewrite and is only safe before traffic is served.GatewayRunner._run_state_db_maintenance_once()— the construction-time maintenance block, extracted. Behavior unchanged; it remains the only pass that may VACUUM (it runs pre-serve and honorsvacuum_after_prune/min_vacuum_interval_days).runner.configat the housekeeping thread launch site.Defaults are unchanged:
sessions.auto_prunestays opt-in (false), retention stays 90d. Both sweeps remain never-raise.Related Issue
Fixes the recurrence path of the 2026-08-15 disk-pressure incident (state.db unbounded growth on long-lived gateways). No pre-existing issue number.
Type of Change
Changes Made
gateway/run.py— extended the housekeeping session-maintenance branch (archive + prune, per served profile,vacuum=False); threadedmultiplexinto_start_gateway_housekeeping; extracted_run_state_db_maintenance_once()for the startup pass.hermes_cli/config_defaults.py—sessions.auto_prunecomment documents the recurring housekeeping cadence.tests/gateway/test_state_db_periodic_maintenance.py— new: startup pass forwards config (vacuum allowed pre-serve); housekeeping tick prunes withvacuum=False; multiplex sweeps mint one profile-scopedSessionDBper home and close it; disabled profiles open no DB; never-raises on missing DB or broken config loader.How to Test
pytest tests/gateway/test_state_db_periodic_maintenance.py tests/test_session_vacuum_config.py tests/gateway/test_async_session_db.py -q→ 14 passed.sessions.auto_prune: truein config.yaml, start the gateway, and observestate.db auto-maintenance: pruned N session(s)in agent.log once permin_interval_hourswithout restarting the gateway.last_auto_prunegate).Checklist
Code
fix(gateway):)tests/gateway/test_state_db_periodic_maintenance.py,tests/test_session_vacuum_config.py,tests/gateway/test_async_session_db.py, housekeeping-related suite); full-suite run not performed in this environmentDocumentation & Housekeeping
cli-config.yaml.example— N/A (no new/changed config keys)CONTRIBUTING.md/AGENTS.md— N/A🤖 Generated with Claude Code
https://claude.ai/code/session_01WNiHNUntm1Sa9ob2KdCz49