techempower-org · jphein · Sep 4, 2026 · Sep 4, 2026 · Sep 4, 2026 · Sep 4, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -57,7 +57,7 @@ TechEmpower's production fork of [MemPalace/mempalace](https://github.com/MemPal
 
 ```bash
 source .venv/bin/activate
-python -m pytest tests/ -q              # 6788 tests (benchmarks deselected)
+python -m pytest tests/ -q              # 6794 tests (benchmarks deselected)
 mempalace status                         # check palace state
 mempalace search "query"                 # test search
 python -m mempalace.mcp_server           # run MCP server standalone

diff --git a/README.md b/README.md
@@ -27,7 +27,7 @@
 
 ## What this is
 
-A verbatim-first local AI memory system. This fork tracks `upstream/develop` through the post-v3.9.0 sync (2026-09-01, commit `e8098348`) and runs in production on a **618K+ drawer Postgres + pgvector + Apache AGE palace** behind [palace-daemon](https://github.com/techempower-org/palace-daemon). It carries fork-ahead commits that compose with — not replace — bensig's release direction; the v3.3.5 release (2026-05-10) includes our co-authored `_get_collection` retry-once via upstream #1377. 6788 tests pass on `main`.
+A verbatim-first local AI memory system. This fork tracks `upstream/develop` through the post-v3.9.0 sync (2026-09-01, commit `e8098348`) and runs in production on a **618K+ drawer Postgres + pgvector + Apache AGE palace** behind [palace-daemon](https://github.com/techempower-org/palace-daemon). It carries fork-ahead commits that compose with — not replace — bensig's release direction; the v3.3.5 release (2026-05-10) includes our co-authored `_get_collection` retry-once via upstream #1377. 6794 tests pass on `main`.
 
 The fork's architectural thinking — the four-layer memory model, the [verbatim-vs-derivative thesis](docs/research/verbatim-vs-derivative-axis.md), design principles, and the two-memory-layer pairing with Auto Dream — lives in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md). The new things here are *what we've learned*, not just what we've fixed.
 

diff --git a/mempalace/backends/postgres.py b/mempalace/backends/postgres.py
@@ -776,8 +776,50 @@ def _get_conn(self):
         if self._conn is None or self._conn.closed:
             self._conn = psycopg2.connect(self.dsn)
             self._conn.autocommit = True
+            self._apply_session_settings(self._conn)
         return self._conn
 
+    def _apply_session_settings(self, conn) -> None:
+        """Per-connection GUCs the kNN path depends on.
+
+        ``hnsw.iterative_scan`` (pgvector >= 0.8): a filtered kNN
+        (``WHERE wing = %s ORDER BY embedding <=> q LIMIT k``) is planned as
+        an HNSW index scan *then* a filter. The index hands back its
+        ``ef_search`` nearest rows palace-wide and the filter discards the
+        ones outside the wing — so a wing that isn't in the global top-N
+        gets ZERO rows, no matter how good its best match is. Measured
+        2026-09-03 on a 757K-drawer palace: a 13K-drawer wing returned 0 for a
+        well-formed prose query (``Rows Removed by Filter: 47``) while the
+        same query unscoped returned 20 and an exact scan returned 5.
+        ``relaxed_order`` makes the index keep scanning until LIMIT rows
+        survive the filter. Session-scoped, so it must be set on every new
+        connection (autocommit means no ``SET LOCAL``). Tolerated on older
+        pgvector (unknown GUC) — logged once, never fatal.
+        MEMPALACE_PG_HNSW_ITERATIVE_SCAN=off disables; any other value is
+        passed through (``relaxed_order`` | ``strict_order``).
+        """
+        mode = os.environ.get("MEMPALACE_PG_HNSW_ITERATIVE_SCAN", "relaxed_order").strip()
+        if not mode or mode.lower() == "off":
+            return
+        if mode not in ("relaxed_order", "strict_order"):
+            logger.warning(
+                "MEMPALACE_PG_HNSW_ITERATIVE_SCAN=%r not in (relaxed_order, strict_order); "
+                "using relaxed_order",
+                mode,
+            )
+            mode = "relaxed_order"
+        try:
+            cur = conn.cursor()
+            cur.execute("SET hnsw.iterative_scan = " + mode)
+        except Exception as exc:  # noqa: BLE001 — pgvector < 0.8 has no such GUC
+            if not getattr(self, "_iterative_scan_warned", False):
+                logger.info(
+                    "postgres: hnsw.iterative_scan unavailable (%s); filtered kNN may "
+                    "under-return on wing-scoped queries — upgrade pgvector to >= 0.8",
+                    str(exc).splitlines()[0][:120],
+                )
+                self._iterative_scan_warned = True
+
     def _detect_extensions(self, *, create: bool = False) -> None:
         if self._vec_type:
             return

diff --git a/mempalace/cli.py b/mempalace/cli.py
@@ -577,6 +577,8 @@ def _print_search_header(
         print(f"  Scope has: {data['available_in_scope']} drawers matching filter")
     for w in warnings:
         print(f"  ! {w}")
+    if data.get("fallback"):
+        print(f"  ! fallback: {data['fallback']}")
     print(f"  via palace-daemon @ {_daemon_url()}")
     print(f"{'=' * 60}\n")
 
@@ -2479,6 +2481,54 @@ def cmd_daemon(args):
         sys.exit(1)
 
 
+def _daemon_search_auto(args, n_results: int, tags):
+    """``--mode auto``: BM25-fast first, hybrid fallback when fast under-shoots.
+
+    BM25-fast is a pure keyword search, so paraphrased or synonym-only
+    queries get 0 hits even when the palace has the content. When fast
+    succeeded but under-shot the requested limit, retry through hybrid
+    (vector + BM25 + AGE graph) and prefer those results when they cover
+    more ground; the user sees a "hybrid (auto-fallback from bm25-fast)"
+    banner so the rescue is visible. The fallback only fires when fast
+    returned a real response — ``None`` (daemon /search/fast 404 or
+    unreachable) is returned as-is so the caller routes to the MCP
+    envelope, which already runs hybrid-shaped. See #283.
+
+    When BOTH arms come back empty, say so in the source banner and carry
+    hybrid's ``warnings``: a bare "0 results" under a bm25-fast banner read
+    as "fast route, no fallback" and sent a fleet session chasing a misroute
+    that wasn't one — the real cause was a filtered-kNN under-return the
+    daemon HAD warned about (2026-09-03).
+
+    Extracted from cmd_search (ruff C901).
+    """
+    if args.room or tags:
+        return None
+    data = _daemon_search_fast(args.query, n_results, wing=args.wing)
+    if data is None:
+        return None
+    bm25_hits = data.get("results") or []
+    if len(bm25_hits) >= n_results:
+        return data
+    try:
+        fb = _daemon_search_hybrid(args.query, n_results, wing=args.wing, room=args.room)
+    except DaemonError:
+        fb = None
+    fb_hits = (fb.get("results") or []) if fb else []
+    if fb_hits and len(fb_hits) > len(bm25_hits):
+        fb["source"] = "hybrid (auto-fallback from bm25-fast)"
+        return fb
+    if fb is not None and not bm25_hits:
+        # ``source`` stays "bm25-fast" — it is a stable machine-read field
+        # (hooks and tests key on it). The fact that hybrid was tried and
+        # also came back empty goes in its own field, and hybrid's warnings
+        # are carried so the header shows the daemon's diagnostic.
+        data["fallback"] = "hybrid also returned 0 hits"
+        if fb.get("warnings"):
+            data["warnings"] = list(fb.get("warnings") or [])
+    return data
+
+
 def cmd_search(args):
     fmt = _resolve_search_format(args)
     want_json = fmt == "json"
@@ -2502,34 +2552,7 @@ def cmd_search(args):
             elif search_mode == "fast":
                 data = _daemon_search_fast(args.query, n_results, wing=args.wing)
             elif search_mode == "auto":
-                if not args.room and not tags:
-                    data = _daemon_search_fast(args.query, n_results, wing=args.wing)
-                    # auto-with-fallback: BM25-fast is a pure keyword search,
-                    # so paraphrased or synonym-only queries get 0 hits even
-                    # when the palace has the content. When fast succeeded
-                    # but under-shot the requested limit, retry through
-                    # hybrid (vector + BM25 + AGE graph) and prefer those
-                    # results when they cover more ground. The user sees a
-                    # "hybrid (auto-fallback from bm25-fast)" source banner
-                    # so the rescue is visible. We only fire the fallback
-                    # when fast returned a real response — if it returned
-                    # ``None`` (daemon /search/fast 404 or unreachable),
-                    # the existing ``data is None`` branch below routes to
-                    # the MCP envelope, which already runs hybrid-shaped.
-                    # See techempower-org/mempalace#283.
-                    if data is not None:
-                        bm25_hits = data.get("results") or []
-                        if len(bm25_hits) < n_results:
-                            try:
-                                fb = _daemon_search_hybrid(
-                                    args.query, n_results, wing=args.wing, room=args.room
-                                )
-                            except DaemonError:
-                                fb = None
-                            fb_hits = (fb.get("results") or []) if fb else []
-                            if fb_hits and len(fb_hits) > len(bm25_hits):
-                                fb["source"] = "hybrid (auto-fallback from bm25-fast)"
-                                data = fb
+                data = _daemon_search_auto(args, n_results, tags)
 
             if data is None:
                 data = _call_daemon_tool("mempalace_search", arguments)

diff --git a/tests/test_cli_daemon.py b/tests/test_cli_daemon.py
@@ -412,6 +412,36 @@ def test_no_fallback_when_bm25_meets_limit(self):
         m_hyb.assert_not_called()
         assert fast_result["source"] == "bm25-fast"
 
+    def test_both_empty_reports_fallback_and_carries_hybrid_warnings(self):
+        """Both arms empty: ``source`` stays the stable "bm25-fast", the fact
+        that hybrid was tried goes in ``fallback``, and hybrid's warnings are
+        carried — a bare "0 results" under a bm25-fast banner read as "no
+        fallback happened" and hid the daemon's filtered-kNN diagnostic
+        (fleet incident 2026-09-03)."""
+        from mempalace import cli
+
+        fast_result = {"results": [], "query": "q", "source": "bm25-fast"}
+        hybrid_result = {
+            "results": [],
+            "source": "hybrid",
+            "warnings": [
+                "13014 drawers in scope; vector ranked only 0 within the distance threshold"
+            ],
+        }
+
+        env = {"PALACE_DAEMON_URL": "http://daemon.example:8085"}
+        with (
+            patch.dict("os.environ", env, clear=True),
+            patch("mempalace.cli._daemon_search_fast", return_value=fast_result),
+            patch("mempalace.cli._daemon_search_hybrid", return_value=hybrid_result) as m_hyb,
+        ):
+            cli.cmd_search(self._args())
+
+        m_hyb.assert_called_once()
+        assert fast_result["source"] == "bm25-fast"
+        assert "hybrid" in fast_result["fallback"] and "0" in fast_result["fallback"]
+        assert fast_result["warnings"] == hybrid_result["warnings"]
+
     def test_keeps_bm25_when_hybrid_also_empty(self):
         """When the fallback retry also returns nothing, keep BM25's empty
         result rather than discarding it — the user gets the original

diff --git a/tests/test_postgres_iterative_scan.py b/tests/test_postgres_iterative_scan.py
@@ -0,0 +1,117 @@
+"""pgvector filtered-kNN under-return: the backend must enable hnsw.iterative_scan.
+
+Measured 2026-09-03 on the production palace (pgvector 0.8.2, 757K drawers): a
+wing-scoped prose query returned 0 rows because the HNSW scan hands back its
+global nearest ~ef_search rows and the wing filter discards them all
+(EXPLAIN: ``Rows Removed by Filter: 47``). The same query unscoped returned 20;
+an exact scan returned 5; ``SET hnsw.iterative_scan = relaxed_order`` made the
+index scan return 5. The GUC is session-scoped, so it must be issued on every
+new connection.
+"""
+
+import logging
+from unittest.mock import patch
+
+
+from mempalace.backends import postgres as pg
+
+
+class _FakeCursor:
+    def __init__(self, conn):
+        self.conn = conn
+
+    def execute(self, sql, *a):
+        self.conn.executed.append(str(sql))
+        if self.conn.fail_on_set and "hnsw.iterative_scan" in str(sql):
+            raise RuntimeError('unrecognized configuration parameter "hnsw.iterative_scan"')
+
+    def fetchall(self):
+        return []
+
+
+class _FakeConn:
+    def __init__(self, fail_on_set=False):
+        self.executed = []
+        self.autocommit = None
+        self.closed = 0
+        self.fail_on_set = fail_on_set
+
+    def cursor(self):
+        return _FakeCursor(self)
+
+
+class _FakeMod:
+    def __init__(self, conn):
+        self._conn = conn
+
+    def connect(self, dsn):
+        return self._conn
+
+
+def _owner():
+    return next(
+        c
+        for c in vars(pg).values()
+        if isinstance(c, type) and hasattr(c, "_apply_session_settings")
+    )
+
+
+def _backend(conn):
+    b = _owner().__new__(_owner())
+    b.dsn = "postgresql://fake/db"
+    b._conn = None
+    return b
+
+
+def test_new_connection_enables_iterative_scan(monkeypatch):
+    monkeypatch.delenv("MEMPALACE_PG_HNSW_ITERATIVE_SCAN", raising=False)
+    conn = _FakeConn()
+    with patch.object(pg, "_load_psycopg2", return_value=(_FakeMod(conn), None)):
+        b = _backend(conn)
+        assert b._get_conn() is conn
+    sets = [s for s in conn.executed if "hnsw.iterative_scan" in s]
+    assert sets == ["SET hnsw.iterative_scan = relaxed_order"]
+    assert conn.autocommit is True
+
+
+def test_reused_connection_does_not_reissue(monkeypatch):
+    monkeypatch.delenv("MEMPALACE_PG_HNSW_ITERATIVE_SCAN", raising=False)
+    conn = _FakeConn()
+    with patch.object(pg, "_load_psycopg2", return_value=(_FakeMod(conn), None)):
+        b = _backend(conn)
+        b._get_conn()
+        b._get_conn()
+    assert sum("hnsw.iterative_scan" in s for s in conn.executed) == 1
+
+
+def test_env_off_disables(monkeypatch):
+    monkeypatch.setenv("MEMPALACE_PG_HNSW_ITERATIVE_SCAN", "off")
+    conn = _FakeConn()
+    with patch.object(pg, "_load_psycopg2", return_value=(_FakeMod(conn), None)):
+        _backend(conn)._get_conn()
+    assert not any("hnsw.iterative_scan" in s for s in conn.executed)
+
+
+def test_strict_order_passthrough_and_bad_value_falls_back(monkeypatch):
+    conn = _FakeConn()
+    monkeypatch.setenv("MEMPALACE_PG_HNSW_ITERATIVE_SCAN", "strict_order")
+    with patch.object(pg, "_load_psycopg2", return_value=(_FakeMod(conn), None)):
+        _backend(conn)._get_conn()
+    assert "SET hnsw.iterative_scan = strict_order" in conn.executed
+    conn2 = _FakeConn()
+    monkeypatch.setenv("MEMPALACE_PG_HNSW_ITERATIVE_SCAN", "bogus; DROP TABLE x")
+    with patch.object(pg, "_load_psycopg2", return_value=(_FakeMod(conn2), None)):
+        _backend(conn2)._get_conn()
+    assert "SET hnsw.iterative_scan = relaxed_order" in conn2.executed, "never pass through junk"
+
+
+def test_old_pgvector_without_guc_is_tolerated(monkeypatch, caplog):
+    monkeypatch.delenv("MEMPALACE_PG_HNSW_ITERATIVE_SCAN", raising=False)
+    conn = _FakeConn(fail_on_set=True)
+    with patch.object(pg, "_load_psycopg2", return_value=(_FakeMod(conn), None)):
+        b = _backend(conn)
+        with caplog.at_level(logging.INFO, logger="mempalace.postgres"):
+            assert b._get_conn() is conn  # no raise
+            b._conn = None  # force a second connect → warn only once
+            b._get_conn()
+    assert sum("iterative_scan unavailable" in r.message for r in caplog.records) == 1
diff --git a/website/public/llms-full.txt b/website/public/llms-full.txt
@@ -46,7 +46,7 @@ Files included, in order:
 
 ## What this is
 
-A verbatim-first local AI memory system. This fork tracks `upstream/develop` through the post-v3.9.0 sync (2026-09-01, commit `e8098348`) and runs in production on a **618K+ drawer Postgres + pgvector + Apache AGE palace** behind [palace-daemon](https://github.com/techempower-org/palace-daemon). It carries fork-ahead commits that compose with — not replace — bensig's release direction; the v3.3.5 release (2026-05-10) includes our co-authored `_get_collection` retry-once via upstream #1377. 6788 tests pass on `main`.
+A verbatim-first local AI memory system. This fork tracks `upstream/develop` through the post-v3.9.0 sync (2026-09-01, commit `e8098348`) and runs in production on a **618K+ drawer Postgres + pgvector + Apache AGE palace** behind [palace-daemon](https://github.com/techempower-org/palace-daemon). It carries fork-ahead commits that compose with — not replace — bensig's release direction; the v3.3.5 release (2026-05-10) includes our co-authored `_get_collection` retry-once via upstream #1377. 6794 tests pass on `main`.
 
 The fork's architectural thinking — the four-layer memory model, the [verbatim-vs-derivative thesis](docs/research/verbatim-vs-derivative-axis.md), design principles, and the two-memory-layer pairing with Auto Dream — lives in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md). The new things here are *what we've learned*, not just what we've fixed.
 
@@ -659,7 +659,7 @@ TechEmpower's production fork of [MemPalace/mempalace](https://github.com/MemPal
 
 ```bash
 source .venv/bin/activate
-python -m pytest tests/ -q              # 6788 tests (benchmarks deselected)
+python -m pytest tests/ -q              # 6794 tests (benchmarks deselected)
 mempalace status                         # check palace state
 mempalace search "query"                 # test search
 python -m mempalace.mcp_server           # run MCP server standalone