perf: reduce agent latency across compression, state, and turn hot paths

[Cemontero0417]

Summary

• WAL checkpoint frequency reduced 50→10 writes to keep SQLite query times fast as sessions grow
• Context compression pre-warming: summary generation starts in a background thread when context hits 80% of threshold, eliminating the 2–10s freeze every ~10 turns
• FTS index backfill deferred to a daemon thread for databases ≥500 messages, preventing multi-second startup freezes after schema upgrades
• Memory manager replaces per-session ThreadPoolExecutor(max_workers=1) instances with a shared pool (max_workers=4), cutting thread count from N×1 to 4 under concurrent gateway load while preserving per-session write ordering
• Tool-output pruning switched from O(n) upfront deep-copy to lazy shallow-list copy
• Nous rate guard skips the atomic file write during 429 storms when existing state already covers the same reset window (< 5s delta)
• Skills manifest walk cached for 30s so repeated snapshot validations do not stat every skill file on every gateway request
• Skill scaffolding parsing memoized with lru_cache(maxsize=4) to avoid re-parsing the same message across prefetch and sync paths
• Iteration budget read-only properties drop their lock — GIL guarantees atomic int reads in CPython

Test plan

• All existing tests pass (443 passed, 5 skipped)
• Compression pre-warm triggers at 80% context threshold and joins faster than cold generation
• WAL file stays small after extended sessions
• FTS search works correctly after schema migration on a large database
• Nous 429 recovery unaffected by write deduplication

🤖 Generated with Claude Code

[tonydwb]

Code Review Summary

Verdict: COMMENT — high surface area, human review recommended

This PR touches 8+ files across core modules (context_compressor, conversation_loop, memory_manager, prompt_builder, hermes_state, nous_rate_guard, iteration_budget) with multiple independent optimizations. Each change is individually sound, but the breadth warrants a human pass before merge.

Individual Changes (all appear correct):

• Pre-warm summary generation (context_compressor): background thread starts summary at 80% threshold. Fingerprint-based cache avoids redundant work. 180s timeout on join.
• Shallow list copy in _prune_old_tool_results: O(k) instead of O(n) deep copy — correct since dicts are copied on write.
• IterationBudget lock removal: plain int read is atomic under CPython GIL — correct.
• Global memory sync executor: shared ThreadPoolExecutor(max_workers=4) replaces per-session pools. Per-session write ordering via _write_serializer lock. flush_pending uses concurrent.futures.wait.
• LRU cache on _strip_skill_scaffolding: memoizes repeated calls with same text.
• Manifest cache TTL (30s): avoids repeated O(skill_files) stat walks on every gateway request.
• Rate guard dedup: skips filesystem write if existing state file covers same reset time within 5s.
• FTS backfill deferral: background thread for large DBs (>=500 messages), inline for small. Properly handles trigger repair path.
• WAL checkpoint frequency: 50 -> 10 writes between checkpoints.

Suggestion

Consider splitting this into 2-3 smaller PRs (e.g., compression prewarm + memory executor refactor, FTS backfill + WAL checkpoint, iteration_budget + rate_guard). Easier to bisect if any regression surfaces.

---

Reviewed by Hermes Agent

[teknium1]

Thanks for investigating several real hot paths. Current main still has the synchronous FTS rebuild (hermes_state.py:1672-1678) and 50-write checkpoint cadence (hermes_state.py:933-934), but this bundle has correctness regressions that need rework.

Problems

• agent/context_compressor.py:973-975 runs _generate_summary concurrently without a session-generation fence. Current reset/end paths clear mutable compression state (agent/context_compressor.py:726-785), so a running prewarm can complete into a later session.
• agent/memory_manager.py:61-64 changes background memory work to stdlib ThreadPoolExecutor. Current main deliberately uses DaemonThreadPoolExecutor (agent/memory_manager.py:654-662); its implementation documents why standard executor workers can block interpreter exit (tools/daemon_pool.py:3-16).
• hermes_state.py:1424-1437 starts FTS backfill after initialization returns. That permits searches while historical rows are still absent, unlike the current synchronous rebuild at hermes_state.py:1672-1678.

Suggested changes

• Fence prewarm results by session/window generation and test reset while work is running.
• Retain daemon-safe shared workers and add shutdown coverage for a wedged provider.
• Preserve immediate FTS-search correctness, with a >=500-message migration regression test.

This is an automated hermes-sweeper review.

[teknium1]

This runs _generate_summary against mutable compressor state without a session-generation fence. on_session_reset() and on_session_end() reset that state, but a running Future cannot be cancelled; tag the work with a session/window generation and discard a late result after reset.

[teknium1]

Please retain daemon-safe workers here. Current main uses DaemonThreadPoolExecutor specifically because stdlib ThreadPoolExecutor workers are joined by the interpreter's atexit hook; this can reintroduce hangs when a provider is blocked on I/O.

[teknium1]

Starting the backfill here returns a SessionDB with FTS enabled before old rows are indexed. An immediate search can omit historical messages; preserve a readiness/fallback contract and add a >=500-message migration test that searches immediately after open.

[kshitijk4poor]

Closing after a hunk-by-hunk adjudication against current main — thank you @chris-montero for a wide-ranging and genuinely perf-literate PR. The verdict is that each piece is either superseded, deliberately designed differently on main, or unsafe as written; the full table lives in our triage notes, summary here:

1. Compression prewarm: review found the background thread mutates self._previous_summary while compress() can be reading it — a real race. Prewarm territory is also now owned by feat: opt-in prompt-cache prewarm — pay the ~20s first-message cost while the user is still typing (TUI/desktop) #73017 (opt-in prompt-cache prewarm with byte-parity guarantees); any prewarm work should build there.
2. _prune_old_tool_results shallow copy: the compressor has been substantially reworked since your base; the copy-semantics win isn't separable anymore.
3. IterationBudget lock removal: main deliberately keeps all accessors locked — the mixed read-unlocked/write-locked discipline is fragile under free-threaded Python for a ~50ns win.
4. MemoryManager shared executor: main solved the same cost with a lazily-created per-instance DaemonThreadPoolExecutor (+ shutdown guard) — daemon workers so a wedged provider can never block interpreter exit.
5. _strip_skill_scaffolding lru_cache: still novel, but a micro-win whose surrounding callers were reworked — not worth a salvage vehicle alone.
6. record_nous_rate_limit dedup: the function only runs on rate-limit events, not per-turn — the write it dedups is rare.
7. prompt_builder manifest TTL: main's manifest is already keyed on (st_mtime_ns, st_size) — mtime-keyed invalidation without a TTL.
8. FTS backfill/checkpoint: this area was restructured by a series of recent merges (fix(state): do not stamp empty FTS after interrupted optimize-storage demote (salvage #72717) #76832, fix(dashboard): preserve maintenance writes on read polling (salvage #67903) #76895, fix(state): throttle repeated VACUUM rewrites (salvage #67351) #76839 among others).

Several of these ideas were independently proven right by where main ended up — the instinct was correct even where the implementation is now moot. If you want to pursue the lru_cache micro-win or a compute-only prewarm on top of #73017, fresh focused PRs against current main would be welcome.