Skip to content

fix(searcher): probe HNSW divergence in CLI search to avoid silent SIGBUS - #1967

Open
trek-e wants to merge 1 commit into
MemPalace:developfrom
trek-e:fix/cli-search-divergence-probe
Open

fix(searcher): probe HNSW divergence in CLI search to avoid silent SIGBUS#1967
trek-e wants to merge 1 commit into
MemPalace:developfrom
trek-e:fix/cli-search-divergence-probe

Conversation

@trek-e

@trek-e trek-e commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the HNSW divergence gate to the CLI search() path that the MCP path has had since #1222.

Bug: mempalace search on a palace whose HNSW segment has diverged from SQLite segfaults ChromaDB's Rust bindings — exit 139 (SIGBUS) at chromadb/api/rust.py:_query, zero stdout, zero stderr. The MCP path avoids this via the module-level _vector_disabled flag (set when hnsw_capacity_status reports divergence → route to BM25-only SQLite fallback). The CLI search() path was missing the same gate, so it called col.query() straight into the diverged segment.

Fix: before _warn_if_legacy_metric(col) and col.query(), probe hnsw_capacity_status. If diverged, route to a new _print_search_results_bm25_only() helper that:

  • Emits a clear NOTICE pointing at mempalace repair
  • Renders BM25-only results in the same shape as the vector path (so users get lexical matches in the format they expect)

The probe is exception-safe: any failure falls through to False so the probe itself can never be the thing that crashes search.

How to test

uv run pytest tests/test_searcher.py -v                # 47 pass, includes 2 new
uv run pytest tests/test_searcher.py::TestSearchCLI -v # focused

Two new regression tests in TestSearchCLI:

  • test_search_routes_to_bm25_when_hnsw_diverged — verifies col.query() is NOT called and the BM25 fallback fires when the probe reports divergence. Reproduces the bug pattern (without the actual segfault) by mocking hnsw_capacity_status to return {"diverged": True}.
  • test_search_proceeds_to_vector_when_hnsw_healthy — paired guard that the probe does not short-circuit on a healthy palace (prevents a regression where the gate accidentally always fires).

Checklist

  • Tests pass (uv run pytest tests/test_searcher.py tests/test_hnsw_capacity.py tests/test_backends.py → 168 passed)
  • No hardcoded paths
  • Linter passes on changed files (uvx --from "ruff==0.15.14" ruff check mempalace/searcher.py tests/test_searcher.py → All checks passed; ruff format --check → 2 files already formatted)

Scope

Intentionally narrow — one probe + one fallback printer in searcher.py, plus the two regression tests. No changes to hnsw_capacity_status (the detection probe, owned by d70433), no changes to _bm25_only_via_sqlite (the existing fallback used by the MCP path), no changes to the MCP server's _vector_disabled mechanism.

Part of epic #1963 (concurrent-writer HNSW corruption cluster). This is a Tier-1 tactical guardrail — it doesn't prevent divergence, but it stops the CLI from silently crashing when divergence exists.

🤖 This PR was prepared with AI assistance. The fix design and implementation are mine; the AI pair helped with test scaffolding and PR logistics.


Note on test-windows CI

The job failed with tests/test_daemon.py::test_worker_overrides_client_palace_path - DaemonError: timed out. This is a pre-existing Windows timing flake, not a regression from this PR:

The four green test jobs (Linux 3.9/3.11/3.13, macOS) cover the actual change in this PR.

…GBUS

The MCP path gates vector queries against HNSW divergence via the
module-level ``_vector_disabled`` flag (MemPalace#1222): when
``hnsw_capacity_status`` reports a diverged segment, MCP search tools
route to the BM25-only SQLite fallback. The CLI ``search()`` path was
missing the gate, so ``mempalace search`` on a diverged palace called
``col.query()`` straight into ChromaDB's Rust bindings and segfaulted
(exit 139 / SIGBUS at ``chromadb/api/rust.py:_query``) with zero
diagnostic output.

Add the same gate to the CLI path: before ``_warn_if_legacy_metric``
and ``col.query()``, probe ``hnsw_capacity_status``; if diverged,
route to a BM25-only fallback printer that emits a clear NOTICE
pointing at ``mempalace repair`` and renders results in the same shape
as the vector path. The probe is exception-safe — any failure falls
through to the vector path so the probe itself can never be the thing
that crashes search.

Reproduces deterministically on a ~1M-drawer palace after HNSW
compaction fails mid-write (see epic MemPalace#1963 for the broader
concurrent-writer corruption cluster this belongs to).

Adds two regression tests:
- ``test_search_routes_to_bm25_when_hnsw_diverged`` — verifies
  ``col.query()`` is NOT called and the BM25 fallback fires when the
  probe reports divergence.
- ``test_search_proceeds_to_vector_when_hnsw_healthy`` — paired guard
  that the probe does not short-circuit on a healthy palace.

🤖 This commit was prepared with AI assistance. The fix design and
implementation are mine; the AI pair helped with the test scaffolding.
@igorls

igorls commented Aug 15, 2026

Copy link
Copy Markdown
Member

Thanks for this contribution, and apologies for the slow turnaround.

develop has moved a fair way since this was opened and the branch no longer merges cleanly. If you're still interested in landing it, could you rebase onto current develop? Once it merges cleanly and CI is green I'll get it reviewed for the 3.8.0 cycle.

If you'd rather not pick it back up, no problem at all — just say so and I'll close it out, and thanks either way for taking the time to send it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants