Skip to content

fix(holographic): robust FTS5 retrieval for operator characters and partial-match queries - #44040

Closed
alfranli123 wants to merge 1 commit into
NousResearch:mainfrom
alfranli123:fix/holographic-fts5-query-robustness
Closed

fix(holographic): robust FTS5 retrieval for operator characters and partial-match queries#44040
alfranli123 wants to merge 1 commit into
NousResearch:mainfrom
alfranli123:fix/holographic-fts5-query-robustness

Conversation

@alfranli123

Copy link
Copy Markdown

FTS5 MATCH raises syntax errors on natural-language queries containing operator characters such as / or - (e.g. 90/180, gpt-5.5), and implicit AND semantics return zero rows for multi-word queries unless every token matches. In both cases fact_store search silently returns no results, which reads as memory loss to the agent.

This adds a two-stage fallback in FactRetriever._fts_candidates:

  1. On FTS5 syntax error, retry with each whitespace token quoted as a phrase (neutralizes FTS5 operator parsing while preserving matching).
  2. If AND semantics return zero rows for a multi-word query, retry with OR over the quoted tokens so partial matches degrade gracefully instead of failing closed.

Measured on a live 2,133-fact store: keyword recall eval went 8/14 to 14/14 (MRR 1.0), latency unchanged (4-61 ms per query). Both fallbacks only run when the primary query path fails or returns nothing, so well-formed queries are untouched.

Related: filed #44037 for a separate fd-recycle corruption issue observed on the same provider.

…nd partial-match queries

FTS5 MATCH raises syntax errors on natural queries containing '/' or '-'
(e.g. "90/180", "gpt-5.5"), and the implicit AND semantics return zero
rows for multi-word queries unless every token matches. Both cases made
fact_store search silently return no results.

Two-stage fallback in _fts_candidates:
1. On FTS5 syntax error, retry with each token quoted as a phrase.
2. On zero rows from AND semantics, retry with OR over quoted tokens.

Measured on a 2133-fact store: recall pass rate 8/14 -> 14/14 (MRR 1.0)
on a keyword eval suite, no regression on latency (4-61 ms).
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have tool/memory Memory tool and memory providers comp/plugins Plugin system and bundled plugins duplicate This issue or pull request already exists labels Jun 11, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #14033 — same holographic FTS5-sanitize fix (neutralize operator chars + OR fallback for partial matches). Part of the long-standing FTS5-sanitize cluster; related to open #11333, #43490, #42268.

@liuhao1024

Copy link
Copy Markdown
Contributor

Verification: Reviewed the diff — improves FTS5 resilience for queries containing operators/punctuation. The retry logic is well-structured:

  1. First attempt: original query (AND semantics, unquoted)
  2. On failure: retry with each token double-quoted (neutralizes FTS5 syntax like /, -, .)
  3. On empty results: retry with OR semantics over quoted tokens (graceful degradation for multi-word queries)

The sanitization tok.replace('"', " ").strip() correctly handles embedded quotes. The if not sanitized or sanitized == query guard prevents infinite retry loops.

One edge case to consider: if the original query is a single token like "90/180", the AND→OR retry is skipped (len(tokens) > 1), which is correct — single-token OR is identical to single-token AND.

LGTM — the three-tier fallback (AND raw → AND quoted → OR quoted) covers the common failure modes without over-matching.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused FTS5 resilience work. This is already implemented on current main with a stronger shared sanitizer, so this PR is redundant.

  • Automated hermes-sweeper review verified FactRetriever._fts_candidates uses _sanitize_fts_query() before MATCH at plugins/memory/holographic/retrieval.py:503.
  • The sanitizer strips FTS5 operator characters, phrase-quotes terms, and OR-joins content tokens at plugins/memory/holographic/retrieval.py:585-619 (commit cb6d6d46ab6b20b173c8215a1f066b53847e9ee5).
  • The direct MemoryStore.search_facts sibling path shares the sanitizer at plugins/memory/holographic/store.py:250-256.
  • Regression coverage is present in tests/plugins/memory/test_holographic_retrieval.py:58-112; the implementation first shipped in v2026.7.1.

This also aligns with the duplicate relationship noted in the discussion.

@teknium1 teknium1 closed this Jul 14, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins duplicate This issue or pull request already exists P3 Low — cosmetic, nice to have sweeper:implemented-on-main Sweeper: behavior already present on current main tool/memory Memory tool and memory providers type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants