feat(state): indexed session metadata search over title/id/display_name (row_id substrate + Unicode/trigram lanes) - #91341
Skywind5487 wants to merge 13 commits into
Conversation
…search (#128) Route the Desktop /api/sessions/search metadata lane through the shared list_sessions_rich(search_query=...) seam and extend that seam to match gateway display_name, so a session whose stored title or peer/chat name matches the query but whose message body does not is found and rendered with its stored title. - hermes_state.list_sessions_rich: search_query lane now matches title, gateway display_name, and id (raw + punctuation-compacted), preserving literal %/_ escaping and compression-chain membership. - hermes_cli/web_routers/sessions.py GET /api/sessions/search: insert a whole-store metadata-discovery pass (list_sessions_rich search_query) between the exact-id and message-content lanes, deduped by lineage root; stored title already survives via get_session_rich_row hydration. - Desktop: SessionSearchResult gains optional title; searchResultToSession renders it instead of hard-coding title: null. - Tests: behavior-level RED (stored-title-only session surfaces with its title) + display_name listing RED (raw, compact, literal wildcards).
… + update title contract (#128) Code-review findings from the first slice: - Extract _session_result_entry helper shared by the ID lane and metadata lane of GET /api/sessions/search, eliminating the Duplicated Code and Data Clumps smells (same row→preview→payload shape repeated across lanes). - Fix started_at/session_started key mismatch: ID lane and metadata lane both hydrate started_at from get_session_rich_row but the metadata row also carries session_started; the helper now accepts both keys so either path feeds add_lineage_result correctly. - Remove dead max(safe_limit * 4, safe_limit) → safe_limit * 4. - Update SessionSearchResult.title? JSDoc to match the actual contract: the server hydrates title for ALL search lanes via get_session_rich_row, not just metadata/ID hits.
- Extract _session_result_entry shared by ID + metadata lanes, eliminating Duplicated Code and Data Clumps smells. - Accept both started_at (real DB/content lane) and session_started (test fake) in the helper; normalise the test fake to started_at for parity. - Remove dead max(safe_limit * 4, safe_limit) → safe_limit * 4. - Fix SessionSearchResult.title JSDoc to match actual server contract (all lanes hydrate title via get_session_rich_row, not just metadata).
e4ac3ad's refactor broke GET /api/sessions/search: the metadata lane (list_sessions_rich search_query) was nested inside the id-match branch so stored-title-only sessions never surfaced, an undefined _lineage_row_payload helper was referenced, and the loop body lost its indentation. Restore the independent metadata lane (title/id/display_name discovery), keep the Desktop session_started contract, and re-pin the key mismatch in the test.
…play_name Named sessions.row_id (copy-swap-rename migration preserving exact legacy rowids incl. deleted-row holes) plus a raw (title, id, display_name) external-content sessions_fts with a resumable fts_session_rebuild_* lifecycle (seed H/P, chunked backfill, finish clears markers). Empty DBs seed no markers; historical rows are backfilled by the chunk engine, rows committing after the claim are live-indexed by gated triggers.
…ycle Optional sessions_fts_cjk (cjk_unicode61) over raw (title, id, display_name) keyed by named row_id, with its own fts_session_cjk_* marker pair and stale key so tokenizer availability never gates the complete Unicode index. Mirrors the message-CJK self-heal: tokenizer-less hosts drop the triggers and breadcrumb, capable hosts seed H/P over populated DBs and serve only after the backfill clears the markers.
…/display_name sessions_fts_trigram reads a derived compact projection (compact(title), RAW id, compact(display_name)) through the sessions_fts_trigram_src VIEW so punctuation-compacted infix queries match at index speed while sessions stays canonical. Own fts_session_trigram_* marker pair keeps P target-specific; the compact separator policy is defined once and shared by the SQL VIEW and the Python query helper. Also narrows the Unicode-lane marker helper in the #128 unicode tests to fts_session_rebuild_* so sibling-lane markers don't leak in.
…KE fallback Metadata discovery now routes through a classifier (lone-CJK -> like, CJK -> cjk+unicode union, explicit token syntax -> unicode, plain 3+ char literal -> trigram, else like) into the session-metadata FTS lanes, with a bounded canonical LIKE fallback (literal %/_/\\ escaping, raw + compact fields) used only on zero-result or unavailable routes. list_sessions_rich(search_query=...) consumes the router candidate-first: FTS hits narrow the compression chain to the resulting row_ids; the previous LIKE lane remains as the fallback.
…ane helper The Unicode / CJK / trigram session-metadata lanes each carried a near-verbatim rebuild_step / rebuild_status / seed / clear and a same-shape schema ensure, differing only by marker prefix, table and source. Introduce a single _SessionFtsLane identity (common.py) plus parameterized _fts_session_lane_step/ _status/_seed/_clear and _ensure_session_optional_lane helpers; the public per-lane methods stay as thin wrappers. Also drop the dead _sessions_fts_available flag (the Unicode lane's availability is implicit via the rebuild-gap check) and stop writing it in SessionDB.__init__ / the Unicode ensure.
…orage The three session-metadata lanes seeded H/P markers on any DB that already had sessions at open (the upgrade path), but nothing ever ran the chunk backfill: optimize_fts_storage only drove the message-FTS steps, so the Unicode lane's rebuild gap never closed (every query fell to the whole-store LIKE lane) and the optional CJK/trigram lanes never became available. Add a Phase 1c that runs each lane's step to completion inside the same throttled backfill loop, and pin the upgrade path with a regression test.
…ed status The list_sessions_rich LIKE fallback kept a divergent compact policy (broad \\W strip + inline REPLACE) from the router/trigram canonical policy; both now use the single compact_session_metadata_text / _session_metadata_compact_sql helpers, fixing a punctuation recall gap on the fallback path. The three per-lane FTS candidate queries collapse onto one _fts_metadata_lane_match helper, and MetadataCandidateResult.status (read only by tests) is dropped — an empty row_ids tuple already means zero.
…metadata lane match The lane MATCH in _fts_metadata_lane_match selected \ ow_id\, but the external-content session FTS tables (sessions_fts / _cjk / _trigram) expose their implicit \ owid\ — \ ow_id\ is only the name of the content-side rowid column. Every lane MATCH raised OperationalError, was caught, and silently fell back to the bounded LIKE lane, so the indexed metadata substrate never served through the candidate router. The green tests did not catch it because they assert results, which the LIKE fallback reproduces identically. Fix: SELECT rowid AS row_id. Add route-asserting regressions so a routed query must actually serve via the trigram/unicode lane, not the fallback.
alt-glitch
added
type/feature
…e the row_id prefix The sessions.row_id migration is self-healing (idempotent on open, no-op when row_id already exists), so it does not need a schema-version bump — reverting SCHEMA_VERSION 27->26 keeps the version-gated migration chain unchanged and fixes the change-detector test that pinned 26. The migration inserts row_id as the FIRST sessions column, which broke the lost-and-found page-level salvage lane: its 'columns are only ever appended' invariant assumed an older record is a strict prefix of the current column order. A current-layout salvaged row now leads with a NULL row_id alias (INTEGER PRIMARY KEY stores NULL in the record) followed by the logical id + source. Teach the mapper to strip that leading NULL (guarding against message rows, whose third cell is a role, not a source) and to drop row_id from the prefix column map so id lands in the id column.
Nice details elsewhere: the pre-drop {id: row_id} identity verification, dropping sessions_defaults index 0 and shifting NOT-NULL substitutes when stripping the rowid prefix, reusing the message-FTS stale-breadcrumb pattern for optional CJK lanes, and keeping id raw while only title/display_name go through the compact transform all show real care. |
3 participants
What does this PR do?
Find a conversation by what you remember about it — stored title, logical session id, or gateway display name — including Unicode/CJK and infix-style metadata queries, without regressing existing message-content search.
Today
list_sessions_rich(search_query=...)(hermes_state.py:8646) matches onlytitleandidwith a%...%LIKE scan: an O(n) full-table scan that (a) cannot use any index, (b) does not match gatewaydisplay_nameat all, and (c) has no CJK/trigram lane. This PR replaces that with candidate-first routing over three FTS5 external-content metadata lanes, keeping a bounded literal-safe LIKE fallback for zero-result / unavailable routes.Indexed substrate. Three external-content FTS5 lanes over
(title, id, display_name):sessions_fts, always on);sessions_fts_trigram, infix/punctuation-normalized);sessions_fts_cjk, served only when a CJK tokenizer is available).A classifier routes each query (lone-CJK → LIKE, CJK → cjk+unicode union, explicit FTS token syntax → unicode, plain 3+ char literal → trigram, else LIKE). The bounded canonical LIKE fallback (literal
%/_/\escaping, raw + compact fields) runs only on zero-result / unavailable routes — never an unbounded scan.list_sessions_rich(search_query=...)consumes the router candidate-first.sessions.row_idmigration (please read before reviewing). This is not generic schema cleanup. External-content FTS needs a stable integer document identity; the oldid TEXT PRIMARY KEYonly had a hidden rowid, which is not durable application-owned identity and can be renumbered byVACUUM(desynchronizing FTS from canonical rows). The resumable rebuild also needs post-capture rows to never reuse a deleted id below the captured high-water. The migration namesrow_id INTEGER PRIMARY KEY AUTOINCREMENT(keepingid TEXT NOT NULL UNIQUEas the public identity) and preserves every surviving legacy hidden rowid exactly, including deleted-row gaps. Cost: ≤1.35s one-time at 100k sessions, transactional (create/copy/drop/rename + index recreate). Rollback = restore the previoussessionstable; crash-safety = the swap is one transaction (reopen after a crash resumes from the H/P progress marker, never serves a partial index). The alternative — keeping the hidden rowid — is not viable: hidden rowids are renumbered byVACUUM, so the FTS→canonical mapping can silently point at the wrong session, and post-capture rows could reuse a deleted id below the high-water, breaking the rebuild ownership invariant.Why this approach is right (trade-off). The current LIKE path is O(n): measured 4.5ms at 1k sessions but 200–460ms at 10k–100k — a perceptible CLI/UI freeze and a blocking gateway query at realistic install sizes. The indexed route is sub-ms to ~7ms regardless of scale. The trigram lane is what makes infix / punctuation-normalized / display_name fast; CJK is kept as an optional lane because it gives no speedup on hosts without a CJK tokenizer (it degrades to the bounded LIKE fallback) — it is a capability-presence feature, not a performance feature, so it must not gate the core.
Related Issue
This is the upstream contribution of the fork's Session Metadata Search work. Related fork issues: Skywind5487#128 (implementation), #140 (benchmark), #141 (upstream gate). No upstream issue exists for this exact contract; the closest open upstream PRs are #71912 (display_name search), #89553 (Desktop stored-title), #87636 (Desktop/web fuzzy search), #67381 (title substring in
search_messages), #75496 (CLI sessions list redesign) — see the Overlap section below.Type of Change
Changes Made
hermes_state_common.py— session-metadata FTS DDL:sessions_fts(raw Unicode external-content),sessions_fts_cjk(optional),sessions_fts_trigram(compact projection viasessions_fts_trigram_srcVIEW), shared compact-separator policy, per-lane H/P rebuild markers.hermes_state_schema.py—sessions.row_idmigration (copy-swap-rename preserving exact legacy rowids incl. deleted-row holes), lane ensure/seed on open, optional-lane stale handling.hermes_state_search.py— candidate router (_classify_metadata_query,_metadata_candidate_row_ids), per-lane MATCH helpers, bounded literal-safe LIKE fallback, resumable lane backfill driven fromoptimize_fts_storage.hermes_state.py—list_sessions_rich(search_query=...)consumes the router candidate-first.hermes_cli/web_routers/sessions.py— web session search route uses the metadata lane (stored-title-only sessions surface).apps/desktop/src/app/chat/sidebar/index.tsx,apps/desktop/src/types/hermes.ts— Desktop result propagation for stored title/origin metadata.tests/test_session_metadata_fts.py,tests/test_session_metadata_cjk_fts.py,tests/test_session_metadata_trigram_fts.py,tests/test_session_metadata_picker_routing.py,tests/hermes_cli/test_web_server_session_search.py,tests/hermes_cli/test_session_listing.py.How to Test
scripts/run_tests.sh tests/test_session_metadata_fts.py tests/test_session_metadata_cjk_fts.py tests/test_session_metadata_trigram_fts.py tests/test_session_metadata_picker_routing.py tests/hermes_cli/test_web_server_session_search.py -q→ 40 passed.scripts/run_tests.sh tests/test_hermes_state.py tests/hermes_cli/test_session_listing.py tests/tools/test_session_search.py -q→ 299 passed, 2 skipped.list_sessions_rich(order_by_last_active=True, search_query="report")returns the stored-title session via the trigram lane;search_query="finance"matches a gatewaydisplay_name;search_query="an94"matchesAN-94(punctuation-normalized);search_query="zzzznope"returns[]via the bounded fallback (no unbounded scan).id TEXT PRIMARY KEYDB migrates torow_idpreserving exact rowids (incl. deleted-row holes); reopen is idempotent;optimize_fts_storage()drives the session lanes to completion.Tested platform: Windows 10, Python 3.11.11, SQLite 3.47.1 (trigram tokenizer available; CJK tokenizer NOT available — the default install; the CJK capable path is covered by tests using a locally built tokenizer). Cross-platform: CJK/trigram lanes are optional and degrade to the bounded LIKE fallback on hosts without the tokenizer; the Unicode lane is the always-on substrate; no platform-specific code.
Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests pass — targeted CI-parity wrapper run (40 + 299 green); full-suite wrapper pending before Ready-for-ReviewDocumentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/ABenchmark evidence (#140)
Synthetic scale sweep (fresh DBs, live-trigger inserts, median of 25 warm calls):
Real production DB (7,268 sessions / 231,513 messages, 1.67GB, schema 25):
Space vs time (real DB): raw metadata text (title+id+display_name) = 0.30MB → FTS index = 2.68MB (9.05x text→FTS expansion, but 0.2% of the 1.67GB DB). The entire session-metadata index costs 2.68MB and buys 16–64x faster metadata search, including CJK titles and gateway display_names the current LIKE path cannot match at all. One-time upgrade backfill (throttled): 7.9s@10k / 83s@100k;
row_idmigration ≤1.35s@100k, transactional.Bug found during benchmarking: the lane MATCH selected a non-existent
row_idcolumn on the FTS5 external-content tables (they exposerowid), so every routed query silently fell back to the LIKE lane and the indexed substrate never served — while tests stayed green because they assert results, which the fallback reproduces. Fixed (SELECT rowid AS row_id) with route-asserting regressions (test_*_route_serves_via_fts_lane).Overlap with open upstream PRs
list_sessions_rich(the slow path at scale); Sandboxing is not a threat model #128's lane feeds it sub-ms candidates. They compose, not conflict: feat(desktop+web): fuzzy session/skills search with match-field UI #87636 owns UI/rank, Sandboxing is not a threat model #128 owns the indexed backend substrate.search_messages): COEXIST (layering) — different seam (search_messagesvslist_sessions_rich); contracts are orthogonal.main.Screenshots / Logs
N/A (backend + tests; manual exercise output in How to Test).