fix(state): route session-resume reads through the WAL read-only connection - #73539
fix(state): route session-resume reads through the WAL read-only connection#73539pierrenode wants to merge 1 commit into
Conversation
…ection get_messages_as_conversation, get_resume_conversations, and get_ancestor_display_prefix still took self._lock — the same global choke point the read-path split (WAL per-thread read-only connections) was meant to remove from every recall/browse read. These three are the hottest reads in the file: every session resume across the gateway, CLI, and ACP adapter goes through one of them, so a resume racing a burst of concurrent-session writer flushes still convoys behind them exactly like the fixed paths used to. _session_lineage_root_to_tip (the lineage walk shared by all three, plus get_conversation_root) had its own independent self._lock use and needed the same conversion — without it the outer functions still blocked on the very first line. Verified empirically: a reader thread calling all three functions while another thread holds self._lock blocked for the writer's full hold duration before the fix, and returned immediately after (SQLite 3.50.4 in this dev venv falls back to journal_mode=DELETE per the WAL-reset-bug guard, so the requires_wal-marked regression test is exercised via a local WAL-forced script instead; it still runs and passes on any runtime where WAL is actually active).
teknium1
left a comment
There was a problem hiding this comment.
Thanks for finding the remaining transcript-read locks. The three converted methods still take self._lock on current main (hermes_state.py:6363, :6544, :6598), and _read_ctx() is the established WAL-safe mechanism (hermes_state.py:2061).
Problems
- Normal resume still blocks before these reads:
resolve_resume_session_id()takesself._lockathermes_state.py:6287and callsget_compression_tip(), which locks athermes_state.py:4950. Current TUI resume calls the resolver attui_gateway/methods_session.py:364beforeget_resume_conversations()at:496; CLI and gateway have the same resolver step. - The added test covers the three converted reads but not that resolver/compression-tip preamble, so it cannot establish the stated no-convoy contract for a normal resume.
Suggested changes
- Convert the read-only queries in
resolve_resume_session_id()andget_compression_tip()to_read_ctx()as well, retaining its DELETE-mode fallback. - Add a WAL-gated writer-lock regression case for resolving a compression continuation, then cover the full resume sequence.
Automated hermes-sweeper review.
|
|
||
| acquired = db._lock.acquire() | ||
| try: | ||
| done = {} |
There was a problem hiding this comment.
This proves the three converted methods no longer take the writer lock, but normal CLI/gateway/TUI resume first calls resolve_resume_session_id(). That helper and its get_compression_tip() preamble still lock on current main, so please add a compression-continuation case that covers the full resume read sequence.
|
Merged via #77803 — thank you @pierrenode. Your commit was cherry-picked, so you remain the author in git history. Nicely scoped fix: main's |
Summary
get_messages_as_conversation,get_resume_conversations, andget_ancestor_display_prefixstill acquiredself._lockbefore reading, even though the recent WAL read-path split (per-thread read-only connections via_read_ctx()) was meant to remove exactly this choke point from every recall/browse read. These three are arguably the hottest reads in the file — every session resume across the gateway, CLI, and ACP adapter goes through one of them — so a resume racing a burst of concurrent-session writer flushes still convoys behind them the same way the already-fixed paths used to (measured convoy on the original fix: a 0.23s query stretching to 112s under 6-8 concurrent turns)._session_lineage_root_to_tip— the lineage-walk helper shared by all three functions above, plusget_conversation_root— had its own independentself._lockuse. Converting only the three outer functions is not enough: they all call this helper as their first step, so it needed the same conversion or the block would just move one frame down.with self._lock: self._conn.execute(...)→with self._read_ctx() as conn: conn.execute(...)pattern already applied toget_session,get_messages,get_messages_around,get_anchored_view, andsearch_messages.Test plan
test_session_resume_reads_do_not_take_writer_locktotests/test_session_db_read_path_split.py, mirroring the existingtest_reads_do_not_take_writer_lockpattern (holdsself._lockin the main thread, asserts the reader thread completes without blocking).journal_mode=DELETEand everyrequires_wal-marked test — including the new one — skips here, same as its siblings. Verified the fix empirically instead: a standalone script that forcesjournal_mode=WAL+_wal_active=Trueon a tempSessionDB, holdsself._lockin the main thread, and calls all three functions from a second thread.get_messages_as_conversation, then again onget_resume_conversationsvia the un-converted_session_lineage_root_to_tip).uv run --frozen --extra dev python -m pytest tests/test_hermes_state.py tests/test_session_db_read_path_split.py tests/cli/test_cli_resume_command.py tests/cli/test_resume_display.py tests/tui_gateway/test_protocol.py tests/test_tui_gateway_server.py tests/hermes_cli/test_context_switch_guard.py -q— 1119 passed, 7 skipped (WAL-gated), no regressions.ruff check hermes_state.py tests/test_session_db_read_path_split.py— clean.