diff --git a/CHANGELOG.md b/CHANGELOG.md index 692c260de4..c180741e1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -106,6 +106,7 @@ Large palaces get fast and stay small: both storage backends lost their palace-w - **`mempalace init` no longer tracebacks on a directory it cannot enter.** `_parse_gradle`'s `is_file()` gate sat in front of the `try` that the parser's own `except OSError` provides, so a manifest under a directory with `r` but no `x` raised `PermissionError` out of a call that used to answer "no manifest name". The gate moved inside that `try`, and `_collect_manifest_names` stats through `os.path.isfile`, which reports rather than raises. (#2221) - **`split` no longer blocks on a FIFO at its own output name, nor writes through a broken link.** The type gate in `main` covers the files the glob listed; `split_file` builds its output names itself, so a pre-existing named pipe at one of them wedged `write_text` in the kernel waiting for a reader. The check asks about the link itself rather than its target, because a dangling symlink at an output name reads as "nothing there" and `write_text` would create the target — landing a chunk outside the output directory. Output names that are anything but a regular file are now skipped with a `SKIP` line. (#2221) - **The FTS5 auto-heal checks the content table before it rebuilds from it.** `PRAGMA quick_check`'s isolated `malformed inverted index for FTS5 table` says the inverted index and `embedding_fulltext_search_content` disagree, not which of them is wrong, and damaging the content table produces that same wording on SQLite 3.45.1, 3.47.1 and 3.51.2 alike. The heal rebuilt from that table regardless and reported "rebuilt from intact content", so a damaged content table cost the palace its lexical reach — 12 of 30 drawers stopped answering `lexical_search` for a word `embedding_metadata` still held — permanently on the `mine` path, where nothing re-files afterwards. Chroma writes every document twice, into `embedding_metadata` under `chroma:document` and into the FTS5 table at `rowid = embeddings.id`, so the shadow copy has an authority: the heal now checks it against that table, restores the rows that disagree and rebuilds, all in one transaction under the mine lock. Rows the authority cannot speak for keep their content and are named in the output, and a check that cannot conclude declines the rebuild instead of guessing. (#2278) +- **Layer 1 wake-up fetches the most recent drawers instead of an arbitrary scan window.** `BaseCollection.get_recent()` returns the newest N by `filed_at`; pgvector overrides it with an `ORDER BY ... DESC LIMIT n` pushdown (capability token `supports_recency_order`), and backends without pushdown keep the previous scan-and-sort behavior. On palaces larger than the 2,000-drawer scan cap, wake-up no longer leads with the oldest backfill. Ordering is on the stored `filed_at` text, the same comparison Layer 1 has used since #1630. (#1630) - **Checkpoints summarize the conversation, not the harness.** Text the tooling injects with `role: "user"` (slash-command expansion, local-command blocks, task notifications, skill preambles, pasted-image placeholders, interruption records) no longer becomes the checkpoint's `recent:` line. The match is anchored to the opening of the message, so a genuine message that quotes one of those wrappers anywhere after the first character is still summarized. (#2170) --- diff --git a/mempalace/backends/base.py b/mempalace/backends/base.py index 62645206e6..202d481d05 100644 --- a/mempalace/backends/base.py +++ b/mempalace/backends/base.py @@ -354,6 +354,40 @@ class LexicalResult: hits: list[LexicalHit] +def recency_sort_key(meta: Optional[dict], order_field: str = "filed_at") -> tuple[int, str]: + """Sort key for newest-first ordering on an ISO-8601 metadata field. + + Returns ``(1, value)`` for a usable timestamp string and ``(0, "")`` + otherwise, so that with ``reverse=True`` records missing the field sort + last instead of raising on a str/None comparison. Backends that implement + :meth:`BaseCollection.get_recent` with a local sort MUST use this key so + every backend orders identically. + + This compares the timestamps as *text*, which is chronological only while + every value shares one offset representation. It is not a new assumption: + Layer 1 has sorted ``filed_at`` as text since #1630 and this key just + names the behaviour. It is also not currently true of ``filed_at`` -- + ``diary_ingest`` writes ``datetime.now(timezone.utc).isoformat()`` + (``...+00:00``) while every other writer uses ``datetime.now().isoformat()`` + (naive local), so on a host that is not on UTC the two sort against each + other skewed by the local offset. Standardising the writers is a separate + change; it needs a migration for palaces that already hold both forms. + """ + value = (meta or {}).get(order_field) + if not isinstance(value, str) or not value: + return (0, "") + return (1, value) + + +def _recency_order(metadatas: list[dict], order_field: str) -> list[int]: + """Indices into ``metadatas``, newest first, missing timestamps last.""" + return sorted( + range(len(metadatas)), + key=lambda i: recency_sort_key(metadatas[i], order_field), + reverse=True, + ) + + # --------------------------------------------------------------------------- # Collection contract # --------------------------------------------------------------------------- @@ -528,6 +562,98 @@ def get_all_metadata(self, where: Optional[dict] = None) -> list[dict]: offset += len(batch_meta) return all_meta + def get_recent( + self, + *, + limit: int, + where: Optional[dict] = None, + order_field: str = "filed_at", + include: Optional[list[str]] = None, + ) -> GetResult: + """Return up to ``limit`` records, newest first by ``order_field``. + + ``order_field`` names a metadata key holding an ISO-8601 timestamp + string (``filed_at`` for drawers). Ordering is descending on that + string; see :func:`recency_sort_key` for what text ordering promises. + Records whose value is missing, empty, or not a string sort last. + + The default implementation pages through :meth:`get` in storage order + up to ``limit`` records and sorts that window locally. That is exact + when the collection holds no more than ``limit`` records matching + ``where``, and *approximate* above that: the window is whatever the + backend hands back first, so the genuinely newest records can fall + outside it (issue #1630's known limitation for Layer 1 wake-up). + Backends able to push the ordering into storage MUST override this and + advertise the ``supports_recency_order`` capability token. What that + token promises, exactly: + + * The returned records really are the top ``limit`` under the ordering + above, at any collection size, **whenever the backend can also + evaluate ``where`` in storage** (including ``where=None``). + * It says nothing about whether that text ordering matches wall-clock + order. That is a property of what the writers store, not of the + backend. + * A filter the backend cannot push into storage has to be evaluated + record by record, so a backend MAY bound how far it walks and return + fewer than ``limit`` records rather than read the whole collection. + A backend that bounds it MUST document the bound on its override. + pgvector does; see :meth:`PgVectorCollection._scroll_recent_local`. + + Callers that need the guarantee should check the token rather than + assume it, and should read it as covering the filters the backend can + push down. The default is always available so no backend breaks. + + ``include`` follows the same contract as :meth:`get`: projections the + caller did not ask for come back empty. ``metadatas`` is fetched + regardless because the sort reads ``order_field`` from it, but it is + only *returned* when requested. + """ + if limit <= 0: + return GetResult.empty() + include = ["documents", "metadatas"] if include is None else list(include) + want_documents = "documents" in include + want_metadatas = "metadatas" in include + # The local sort needs order_field, so metadatas always come back from + # the backend even when the caller projected them out of the result. + fetch_include = include if want_metadatas else [*include, "metadatas"] + + ids: list[str] = [] + documents: list[str] = [] + metadatas: list[dict] = [] + offset = 0 + fetched = 0 + page_size = min(500, limit) + while fetched < limit: + kwargs: dict = {"include": fetch_include, "limit": page_size, "offset": offset} + if where: + kwargs["where"] = where + batch = self.get(**kwargs) + batch_ids = list(batch.get("ids") or []) + batch_docs = list(batch.get("documents") or []) + batch_metas = list(batch.get("metadatas") or []) + page_len = max(len(batch_ids), len(batch_docs), len(batch_metas)) + if not page_len: + break + # Pad the projections the caller did not request so the three + # lists stay index-aligned for the sort below. + ids.extend(batch_ids or [""] * page_len) + documents.extend(batch_docs or [""] * page_len) + metadatas.extend(batch_metas or [{}] * page_len) + offset += page_len + fetched += page_len + if page_len < page_size: + break + + n = min(len(ids), len(documents), len(metadatas)) + ids, documents, metadatas = ids[:n], documents[:n], metadatas[:n] + order = _recency_order(metadatas, order_field)[:limit] + return GetResult( + ids=[ids[i] for i in order], + documents=[documents[i] for i in order] if want_documents else [], + metadatas=[metadatas[i] for i in order] if want_metadatas else [], + embeddings=None, + ) + def facet_counts( self, field: str, diff --git a/mempalace/backends/embedding_wrapper.py b/mempalace/backends/embedding_wrapper.py index dd8dcd3333..f52662f846 100644 --- a/mempalace/backends/embedding_wrapper.py +++ b/mempalace/backends/embedding_wrapper.py @@ -167,6 +167,21 @@ def health(self): def lexical_search(self, *, query: str, n_results: int = 10, where: Optional[dict] = None): return self._inner.lexical_search(query=query, n_results=n_results, where=where) + def get_recent( + self, + *, + limit: int, + where: Optional[dict] = None, + order_field: str = "filed_at", + include: Optional[list[str]] = None, + ): + # Concrete on ``BaseCollection`` (the scan-and-sort default), so MRO + # would resolve it here and shadow a backend that pushes the ordering + # into storage. Forward explicitly. + return self._inner.get_recent( + limit=limit, where=where, order_field=order_field, include=include + ) + def facet_counts( self, field: str, where: Optional[dict] = None, limit: int = 1000 ) -> dict[str, int]: diff --git a/mempalace/backends/pgvector.py b/mempalace/backends/pgvector.py index 0c18a855df..32f74cb14e 100644 --- a/mempalace/backends/pgvector.py +++ b/mempalace/backends/pgvector.py @@ -72,6 +72,16 @@ {"$eq", "$ne", "$in", "$nin", "$and", "$or", "$contains", "$gt", "$gte", "$lt", "$lte"} ) _PUSHDOWN_OPERATORS = frozenset({"$eq", "$ne", "$in", "$nin", "$and"}) +# Bounds for the local-post-filter branch of ``get_recent``. The pushdown +# branch needs none — SQL does ORDER BY ... LIMIT n and returns exactly n +# rows. The post-filter branch cannot push the predicate, so it walks the +# table newest-first in pages and stops as soon as ``limit`` rows match. +# The page size trades round trips against rows on the wire; the row cap +# bounds the pathological case (a filter that matches nothing in a large +# table) so one call can never walk unboundedly. +_RECENT_SCAN_PAGE_MIN = 500 +_RECENT_SCAN_PAGE_MAX = 5000 +_RECENT_SCAN_ROW_CAP = 50_000 def _utcnow() -> str: @@ -678,6 +688,7 @@ def scroll_rows( with_document: bool = True, limit: Optional[int] = None, offset: Optional[int] = None, + order_field: Optional[str] = None, ) -> list[dict]: qi = _quote_identifier(table) params: list = [] @@ -695,7 +706,21 @@ def scroll_rows( # primary key gives OFFSET a stable order (an unordered scan may skip # or repeat rows across pages); callers that scroll the whole table # pass neither bound, leaving their SQL unchanged. - if limit is not None or offset: + if order_field is not None: + # Newest-first on an ISO-8601 metadata field. ISO-8601 sorts + # chronologically as text, so ``metadata->>field DESC`` needs no + # timestamp cast (a cast would also fail hard on one malformed + # value). NULLS LAST keeps records without the field at the end; + # ``id`` breaks ties so the order is total and stable. + params.append(order_field) + sql += " ORDER BY metadata->>%s DESC NULLS LAST, id" + if limit is not None: + params.append(int(limit)) + sql += " LIMIT %s" + if offset: + params.append(int(offset)) + sql += " OFFSET %s" + elif limit is not None or offset: sql += " ORDER BY id" if limit is not None: params.append(int(limit)) @@ -893,6 +918,7 @@ def _scroll( with_document=True, limit=None, offset=None, + order_field=None, ) -> list[dict]: self._ensure_open() if not self._table_exists(): @@ -906,6 +932,7 @@ def _scroll( with_document=with_document, limit=limit, offset=offset, + order_field=order_field, ) def get_all_metadata(self, where=None) -> list[dict]: @@ -1194,6 +1221,152 @@ def get( embeddings=[row["embedding"] or [] for row in rows] if spec.embeddings else None, ) + def _scroll_recent_local(self, *, where, limit, order_field, with_embedding, with_document): + """Newest-first paged scan with the filter applied in Python. + + Used when ``where`` is not exactly expressible as ``metadata @> ...`` + ($or, comparisons, ...). The predicate cannot ride along, but the + *ordering* still can, so instead of dragging the whole table across + the wire to keep ``limit`` rows we walk it newest-first one SQL page + at a time and stop at the first page that completes the answer. On + the common shape (a filter most rows match) that is a single page. + + Page stability: ``ORDER BY metadata->>field DESC NULLS LAST, id`` is a + total order because ``id`` is the primary key, so OFFSET paging is + well defined — the same guarantee the existing ``ORDER BY id`` paging + in :meth:`_PgVectorClient.scroll_rows` relies on. Concurrent writes + can still shift rows across a page boundary: an insert of a newer row + pushes one row down and would hand it back twice, which the ``seen`` + set drops, and a delete pulls one row up and can skip it. That skip + window is inherent to OFFSET paging and is unchanged from the + pre-existing paged ``get``; a keyset cursor would close it and is a + separate change. + + Bounded, not exhaustive: the walk stops after ``_RECENT_SCAN_ROW_CAP`` + rows, so a filter that matches almost nothing in a huge table returns + fewer than ``limit`` rows rather than reading the table. Because the + walk is newest-first, what it does return is still the newest matching + rows within the newest ``_RECENT_SCAN_ROW_CAP`` records — a much + tighter approximation than the base class's storage-order window, but + an approximation, unlike the pushdown branch which is exact at any + table size. + """ + # ``limit`` is ``int`` in the contract; the ``None`` the caller's + # guard tolerates means "no bound", which on this branch is the cap. + target = _RECENT_SCAN_ROW_CAP if limit is None else int(limit) + page_size = max(_RECENT_SCAN_PAGE_MIN, min(target, _RECENT_SCAN_PAGE_MAX)) + matched: list[dict] = [] + seen: set[str] = set() + offset = 0 + scanned = 0 + while len(matched) < target and scanned < _RECENT_SCAN_ROW_CAP: + want = min(page_size, _RECENT_SCAN_ROW_CAP - scanned) + page = self._scroll( + where=None, + with_embedding=with_embedding, + with_document=with_document, + limit=want, + offset=offset or None, + order_field=order_field, + ) + if not page: + break + scanned += len(page) + offset += len(page) + for row in page: + if row["id"] in seen: + continue + seen.add(row["id"]) + if _matches_where(row["metadata"], where): + matched.append(row) + if len(matched) >= target: + break + if len(page) < want: + break # short page — end of table + return matched[:target] + + def get_recent(self, *, limit, where=None, order_field="filed_at", include=None): + """Newest-first fetch with the ordering pushed into SQL. + + The base implementation scans a window in storage order and sorts it + locally, so on a collection larger than ``limit`` the genuinely newest + records can be missing from the window entirely (#1630). Postgres can + do the whole thing: ``ORDER BY metadata->>'filed_at' DESC ... LIMIT n`` + picks the true top ``limit`` under that ordering at any table size, + which is why this backend advertises ``supports_recency_order``. + + Filters that ``metadata @> ...`` cannot express exactly ($or, + comparisons, ...) keep the local post-filter contract that ``get`` + uses, but they do *not* fetch the table to do it: the ordering is + still pushed into SQL and :meth:`_scroll_recent_local` walks the + result newest-first a page at a time, stopping as soon as ``limit`` + rows match. + + **This is the one case where ``supports_recency_order`` is weaker than + it sounds.** That walk is capped, so a non-pushdown filter matching + very little in a very large table returns fewer than ``limit`` records + rather than reading the table. The token covers the filters this + backend can push down, which is every filter Layer 1 uses (``None`` or + ``{"wing": ...}``) and everything built from ``$eq``/``$ne``/``$in``/ + ``$nin``/``$and``. See :meth:`_scroll_recent_local` for the bound and + for what the capped answer still guarantees. + + ``limit=None`` is outside the contract (the signature is ``int``). The + pushdown branch treats it as unbounded; the local branch cannot, and + treats it as the scan cap. + + Ordering is on the JSON *text* of ``order_field``, matching the text + ordering :func:`recency_sort_key` already applies in Layer 1. See that + function for what text ordering does and does not promise; this method + inherits those limits rather than introducing them. At least three + places where the SQL order and ``recency_sort_key`` differ, none of + them reachable through anything that writes ``filed_at``: + + * a JSON value that is not a string sorts by its text form here but + sorts last there; + * an empty string sorts above SQL NULL here but ties with a missing + key there; + * both ``metadata->>%s`` and the ``id`` tiebreak sort under the + database collation, while ``recency_sort_key`` sorts by Python + codepoint. Under a collation such as ``en_US.UTF-8`` punctuation is + weighted differently, so the two can disagree on timestamps that + differ only in punctuation (``+00:00`` against ``Z``). The test + double emulates the ordering in Python and so cannot catch this. + """ + if limit is not None and limit <= 0: + return GetResult.empty() + _validate_where(where) + spec = _IncludeSpec.resolve(include, default_distances=False) + if _requires_local_filter(where): + rows = self._scroll_recent_local( + where=where, + limit=limit, + order_field=order_field, + with_embedding=spec.embeddings, + # The post-filter reads only ``metadata``, and this branch can + # scan far more rows than it returns, so project the document + # text out unless the caller actually asked for it (#1840's + # wire-byte win). The pushdown branch below is left alone: it + # fetches ``limit`` rows, so the projection is worth little + # there and not worth changing that path's SQL for. (It would + # still pay off for ``limit=None``, which is outside the + # contract; fold it in when that path grows a real caller.) + with_document=spec.documents, + ) + else: + rows = self._scroll( + where=where, + with_embedding=spec.embeddings, + limit=limit, + order_field=order_field, + ) + return GetResult( + ids=[row["id"] for row in rows], + documents=[row["document"] for row in rows] if spec.documents else [], + metadatas=[row["metadata"] for row in rows] if spec.metadatas else [], + embeddings=[row["embedding"] or [] for row in rows] if spec.embeddings else None, + ) + def delete(self, *, ids=None, where=None): _validate_where(where) if not self._table_exists(): @@ -1336,6 +1509,7 @@ class PgVectorBackend(BaseBackend): "supports_metadata_filters", "supports_lexical_search", "supports_metadata_facets", + "supports_recency_order", "supports_namespace_isolation", "supports_server_side_indexes", "server_mode", diff --git a/mempalace/layers.py b/mempalace/layers.py index c258a7ec3e..a1dd71a396 100644 --- a/mempalace/layers.py +++ b/mempalace/layers.py @@ -87,28 +87,47 @@ class Layer1: MAX_DRAWERS = 15 # at most 15 moments in wake-up MAX_CHARS = 3200 # hard cap on total L1 text (~800 tokens) - MAX_SCAN = 2000 # don't scan more than this for L1 generation + MAX_SCAN = 2000 # size of the candidate window pulled for L1 generation def __init__(self, palace_path: str = None, wing: str = None): cfg = MempalaceConfig() self.palace_path = palace_path or cfg.palace_path self.wing = wing - def generate(self) -> str: - """Pull top drawers from ChromaDB and format as compact L1 text.""" - try: - col = _get_collection(self.palace_path, create=False) - except Exception: - return "## L1 — No palace found. Run: mempalace mine " + def _fetch_candidates(self, col) -> tuple[list, list]: + """Fetch the L1 candidate window: the MAX_SCAN most recently filed drawers. - # Fetch all drawers in batches to avoid SQLite variable limit (~999) + Uses the backend's ``get_recent`` capability, which pushes + ``ORDER BY filed_at DESC LIMIT n`` into storage where the backend can + (pgvector today) and otherwise falls back to the scan-then-sort default + in ``BaseCollection``. That default is what this method used to do + inline, so backends without pushdown behave exactly as before. + + Third-party collections predating ``get_recent`` (or any backend error) + degrade to the inline paged scan below rather than failing wake-up. + """ + where = {"wing": self.wing} if self.wing else None + getter = getattr(col, "get_recent", None) + if getter is not None: + try: + result = getter( + limit=self.MAX_SCAN, + where=where, + order_field="filed_at", + include=["documents", "metadatas"], + ) + return list(result.documents or []), list(result.metadatas or []) + except Exception: + pass # capability missing or backend hiccup — page it manually + + # Fetch in batches to avoid SQLite variable limit (~999) _BATCH = 500 docs, metas = [], [] offset = 0 while True: kwargs = {"include": ["documents", "metadatas"], "limit": _BATCH, "offset": offset} - if self.wing: - kwargs["where"] = {"wing": self.wing} + if where: + kwargs["where"] = where try: batch = col.get(**kwargs) except Exception: @@ -122,6 +141,16 @@ def generate(self) -> str: offset += len(batch_docs) if len(batch_docs) < _BATCH or len(docs) >= self.MAX_SCAN: break + return docs, metas + + def generate(self) -> str: + """Pull top drawers from the palace and format as compact L1 text.""" + try: + col = _get_collection(self.palace_path, create=False) + except Exception: + return "## L1 — No palace found. Run: mempalace mine " + + docs, metas = self._fetch_candidates(col) if not docs: return "## L1 — No memories yet." @@ -136,6 +165,10 @@ def generate(self) -> str: # newest first. This keeps importance as the primary key for the day a # scoring pass populates it, while making the "recent filing" half of # the promise true today with data we already have. + # The candidate window this sorts is now the MAX_SCAN *most recently + # filed* drawers rather than the first MAX_SCAN the backend happened to + # hand back, so on a palace larger than MAX_SCAN the newest drawers are + # actually in the running (#1630's known limitation). scored = [] for doc, meta in zip(docs, metas): meta = meta or {} diff --git a/tests/test_backends.py b/tests/test_backends.py index 37c477e0fc..4f0615d7e0 100644 --- a/tests/test_backends.py +++ b/tests/test_backends.py @@ -671,6 +671,165 @@ def test_base_collection_update_default_rejects_mismatched_lengths(): BaseCollection.update(collection, ids=["1", "2"], metadatas=[{"k": 9}]) +class _PagedCollection: + """Minimal collection exposing only ``get`` with real limit/offset paging.""" + + def __init__(self, records): + self._records = records + self.calls = [] + + def get(self, *, ids=None, where=None, limit=None, offset=None, include=None, **kwargs): + self.calls.append({"where": where, "limit": limit, "offset": offset, "include": include}) + rows = self._records + if where: + rows = [r for r in rows if all(r[1].get(k) == v for k, v in where.items())] + start = offset or 0 + end = start + (limit if limit is not None else len(rows)) + page = rows[start:end] + return GetResult( + ids=[r[0] for r in page], + documents=[r[2] for r in page], + metadatas=[r[1] for r in page], + ) + + +def _recent(collection, **kwargs): + from mempalace.backends.base import BaseCollection + + return BaseCollection.get_recent(collection, **kwargs) + + +def test_base_get_recent_default_sorts_window_newest_first(): + """The ABC default scans a window and sorts it locally by filed_at.""" + col = _PagedCollection( + [ + ("a", {"filed_at": "2024-01-01T00:00:00Z"}, "oldest"), + ("b", {"filed_at": "2026-08-06T00:00:00Z"}, "newest"), + ("c", {"filed_at": "2025-05-05T00:00:00Z"}, "middle"), + ] + ) + page = _recent(col, limit=10) + assert page.ids == ["b", "c", "a"] + assert page.documents == ["newest", "middle", "oldest"] + + +def test_base_get_recent_default_sorts_missing_field_last(): + col = _PagedCollection( + [ + ("a", {}, "undated"), + ("b", {"filed_at": ""}, "empty"), + ("c", {"filed_at": 20260806}, "not-a-string"), + ("d", {"filed_at": "2025-01-01T00:00:00Z"}, "dated"), + ] + ) + assert _recent(col, limit=10).ids[0] == "d" + assert set(_recent(col, limit=10).ids[1:]) == {"a", "b", "c"} + + +def test_base_get_recent_default_window_is_capped_at_limit(): + """The default is approximate above ``limit``: it only sees the first window. + + This is exactly the scan-order limitation documented on #1630 — a backend + that can push ORDER BY into storage overrides the method to fix it. + """ + records = [("old%d" % i, {"filed_at": "2020-01-01T00:00:00Z"}, "old") for i in range(1200)] + records.append(("newest", {"filed_at": "2026-08-06T00:00:00Z"}, "the newest drawer")) + col = _PagedCollection(records) + + page = _recent(col, limit=1000) + + assert len(page.ids) == 1000 + assert "newest" not in page.ids + # Paged in 500-record batches rather than one giant fetch. + assert [c["limit"] for c in col.calls] == [500, 500] + assert [c["offset"] for c in col.calls] == [0, 500] + + +def test_base_get_recent_default_passes_where_and_zero_limit(): + col = _PagedCollection( + [ + ("a", {"wing": "x", "filed_at": "2024-01-01T00:00:00Z"}, "x drawer"), + ("b", {"wing": "y", "filed_at": "2026-01-01T00:00:00Z"}, "y drawer"), + ] + ) + page = _recent(col, limit=10, where={"wing": "x"}) + assert page.ids == ["a"] + assert col.calls[0]["where"] == {"wing": "x"} + + assert _recent(col, limit=0).ids == [] + + +def test_base_get_recent_default_honours_include_projection(): + """Unrequested projections come back empty, as they do from ``get``. + + ``metadatas`` is fetched from the backend regardless because the local + sort reads ``order_field`` out of it, but it is only returned when the + caller asked for it. Without that, a backend without recency pushdown + would answer ``include=["metadatas"]`` with a list of padding strings + while pgvector answers with ``[]``. + """ + + class _ProjectingCollection: + """Honours ``include`` the way the real backends do.""" + + def __init__(self): + self.calls = [] + + def get(self, *, include=None, limit=None, offset=None, **kwargs): + self.calls.append(list(include or [])) + if offset: + return GetResult(ids=[], documents=[], metadatas=[]) + keys = set(include or []) + return GetResult( + ids=["a", "b"], + documents=["older", "newer"] if "documents" in keys else [], + metadatas=( + [ + {"filed_at": "2024-01-01T00:00:00Z"}, + {"filed_at": "2026-01-01T00:00:00Z"}, + ] + if "metadatas" in keys + else [] + ), + ) + + col = _ProjectingCollection() + page = _recent(col, limit=5, include=["metadatas"]) + assert page.ids == ["b", "a"] + assert page.documents == [] + assert page.metadatas == [ + {"filed_at": "2026-01-01T00:00:00Z"}, + {"filed_at": "2024-01-01T00:00:00Z"}, + ] + + col = _ProjectingCollection() + page = _recent(col, limit=5, include=["documents"]) + # metadatas are fetched anyway so the sort has order_field to read... + assert "metadatas" in col.calls[0] + # ...which is why the newest document leads, but they are not returned. + assert page.documents == ["newer", "older"] + assert page.metadatas == [] + + +def test_base_get_recent_default_accepts_dict_shaped_get(): + """Collections still returning Chroma-shaped dicts page correctly.""" + + class _DictCollection: + def get(self, **kwargs): + if kwargs.get("offset"): + return {"ids": [], "documents": [], "metadatas": []} + return { + "ids": ["a", "b"], + "documents": ["older", "newer"], + "metadatas": [ + {"filed_at": "2024-01-01T00:00:00Z"}, + {"filed_at": "2026-01-01T00:00:00Z"}, + ], + } + + assert _recent(_DictCollection(), limit=5).documents == ["newer", "older"] + + def test_chroma_backend_accepts_palace_ref_kwarg(tmp_path): palace_path = tmp_path / "palace" backend = ChromaBackend() diff --git a/tests/test_layers.py b/tests/test_layers.py index 56263a3b9a..e569e99b13 100644 --- a/tests/test_layers.py +++ b/tests/test_layers.py @@ -3,6 +3,7 @@ import os from unittest.mock import MagicMock, patch +from mempalace.backends.base import BaseCollection, GetResult from mempalace.layers import Layer0, Layer1, Layer2, Layer3, MemoryStack @@ -71,13 +72,26 @@ def test_layer0_default_path(): def _mock_chromadb_for_layer(docs, metas, monkeypatch=None): - """Return a mock collection whose get() returns docs/metas.""" + """Return a mock collection whose get() returns docs/metas. + + ``get_recent`` is bound to the ``BaseCollection`` default, so the double + behaves like a backend that has the capability but no storage-side + ordering: it pages through ``get`` and sorts the window locally. + """ mock_col = MagicMock() # First batch returns data, second batch returns empty (end of pagination) mock_col.get.side_effect = [ {"documents": docs, "metadatas": metas}, {"documents": [], "metadatas": []}, ] + mock_col.get_recent = lambda **kwargs: BaseCollection.get_recent(mock_col, **kwargs) + return mock_col + + +def _mock_legacy_collection(): + """A collection double predating ``get_recent`` (third-party backend).""" + mock_col = MagicMock() + del mock_col.get_recent return mock_col @@ -114,7 +128,7 @@ def test_layer1_generates_essential_story(): def test_layer1_empty_palace(): - mock_col = MagicMock() + mock_col = _mock_legacy_collection() mock_col.get.return_value = {"documents": [], "metadatas": []} with ( patch("mempalace.layers.MempalaceConfig") as mock_cfg, @@ -224,7 +238,7 @@ def test_layer1_breaks_importance_ties_by_filed_at_recency(): def test_layer1_batch_exception_breaks(): """If col.get raises on a batch, loop breaks gracefully.""" - mock_col = MagicMock() + mock_col = _mock_legacy_collection() mock_col.get.side_effect = [ {"documents": ["doc1"], "metadatas": [{"room": "r"}]}, RuntimeError("batch error"), @@ -240,6 +254,198 @@ def test_layer1_batch_exception_breaks(): assert "ESSENTIAL STORY" in result +# ── Layer1 — recency fetch (capable backend vs scan fallback) ─────────── + + +def test_layer1_uses_backend_recency_capability(): + """A backend with recency pushdown is asked for the newest window, not a scan.""" + calls = {} + + mock_col = MagicMock() + + def fake_get_recent(*, limit, where=None, order_field="filed_at", include=None): + calls["limit"] = limit + calls["where"] = where + calls["order_field"] = order_field + return GetResult( + ids=["b", "a"], + documents=["The newest memory we filed today.", "An older memory from last year."], + metadatas=[ + {"room": "moments", "filed_at": "2026-03-01T00:00:00Z"}, + {"room": "moments", "filed_at": "2026-01-01T00:00:00Z"}, + ], + ) + + mock_col.get_recent = fake_get_recent + + with ( + patch("mempalace.layers.MempalaceConfig") as mock_cfg, + patch("mempalace.layers._get_collection", return_value=mock_col), + ): + mock_cfg.return_value.palace_path = "/fake" + result = Layer1(palace_path="/fake").generate() + + assert calls["limit"] == Layer1.MAX_SCAN + assert calls["order_field"] == "filed_at" + assert calls["where"] is None + # The capability answered, so the paging scan never ran. + mock_col.get.assert_not_called() + assert result.index("The newest memory") < result.index("An older memory") + + +def test_layer1_recency_capability_receives_wing_filter(): + captured = {} + + mock_col = MagicMock() + + def fake_get_recent(*, limit, where=None, order_field="filed_at", include=None): + captured["where"] = where + return GetResult( + ids=["a"], + documents=["A wing-scoped memory from the project."], + metadatas=[{"room": "r"}], + ) + + mock_col.get_recent = fake_get_recent + + with ( + patch("mempalace.layers.MempalaceConfig") as mock_cfg, + patch("mempalace.layers._get_collection", return_value=mock_col), + ): + mock_cfg.return_value.palace_path = "/fake" + Layer1(palace_path="/fake", wing="my_project").generate() + + assert captured["where"] == {"wing": "my_project"} + + +def test_layer1_falls_back_to_scan_when_capability_missing(): + """Collections predating get_recent still wake up via the paged scan.""" + mock_col = _mock_legacy_collection() + mock_col.get.side_effect = [ + { + "documents": ["Legacy memory from a collection with no capability."], + "metadatas": [{"room": "r"}], + }, + {"documents": [], "metadatas": []}, + ] + with ( + patch("mempalace.layers.MempalaceConfig") as mock_cfg, + patch("mempalace.layers._get_collection", return_value=mock_col), + ): + mock_cfg.return_value.palace_path = "/fake" + result = Layer1(palace_path="/fake").generate() + + assert "Legacy memory" in result + assert mock_col.get.called + + +def test_layer1_falls_back_to_scan_when_capability_raises(): + """A backend error inside get_recent degrades to the scan, not to an empty L1.""" + mock_col = MagicMock() + mock_col.get_recent.side_effect = RuntimeError("server said no") + mock_col.get.side_effect = [ + { + "documents": ["Scanned memory recovered after the backend errored."], + "metadatas": [{"room": "r"}], + }, + {"documents": [], "metadatas": []}, + ] + with ( + patch("mempalace.layers.MempalaceConfig") as mock_cfg, + patch("mempalace.layers._get_collection", return_value=mock_col), + ): + mock_cfg.return_value.palace_path = "/fake" + result = Layer1(palace_path="/fake").generate() + + assert "Scanned memory" in result + + +def _oversized_palace(): + """A palace larger than MAX_SCAN whose newest drawer is filed last. + + Storage order is oldest-first, so the newest drawer sits beyond the + MAX_SCAN window a scan-and-sort fetch can see (#1630 known limitation). + """ + total = Layer1.MAX_SCAN + 5 + docs = [f"Backfill drawer {i} from the original mine." for i in range(total - 1)] + docs.append("The newest session: we shipped the recency fetch and verified it.") + metas = [{"room": "r", "filed_at": f"2020-01-0{i % 9 + 1}T00:00:00Z"} for i in range(total - 1)] + metas.append({"room": "r", "filed_at": "2026-08-06T00:00:00Z"}) + return docs, metas + + +class _StorageOrderCollection(BaseCollection): + """Collection with no recency pushdown — inherits the BaseCollection default.""" + + def __init__(self, docs, metas): + self._docs = docs + self._metas = metas + + def add(self, **kwargs): ... + + def upsert(self, **kwargs): ... + + def query(self, **kwargs): ... + + def delete(self, **kwargs): ... + + def count(self): + return len(self._docs) + + def get(self, *, limit=None, offset=None, **kwargs): + start = offset or 0 + end = start + (limit if limit is not None else len(self._docs)) + return GetResult( + ids=[str(i) for i in range(start, min(end, len(self._docs)))], + documents=self._docs[start:end], + metadatas=self._metas[start:end], + ) + + +class _RecencyOrderCollection(_StorageOrderCollection): + """Collection that pushes the ordering into storage, like pgvector does.""" + + def get_recent(self, *, limit, where=None, order_field="filed_at", include=None): + order = sorted( + range(len(self._docs)), + key=lambda i: self._metas[i].get(order_field, ""), + reverse=True, + )[:limit] + return GetResult( + ids=[str(i) for i in order], + documents=[self._docs[i] for i in order], + metadatas=[self._metas[i] for i in order], + ) + + +def _generate_l1(col): + with ( + patch("mempalace.layers.MempalaceConfig") as mock_cfg, + patch("mempalace.layers._get_collection", return_value=col), + ): + mock_cfg.return_value.palace_path = "/fake" + return Layer1(palace_path="/fake").generate() + + +def test_layer1_capable_backend_surfaces_newest_beyond_scan_window(): + """With pushdown, the newest drawer leads wake-up even past MAX_SCAN rows.""" + docs, metas = _oversized_palace() + result = _generate_l1(_RecencyOrderCollection(docs, metas)) + assert "The newest session" in result + + +def test_layer1_scan_fallback_is_capped_at_max_scan(): + """Without pushdown the window is still MAX_SCAN rows — the documented limit.""" + docs, metas = _oversized_palace() + col = _StorageOrderCollection(docs, metas) + with patch("mempalace.layers.MempalaceConfig") as mock_cfg: + mock_cfg.return_value.palace_path = "/fake" + fetched_docs, _ = Layer1(palace_path="/fake")._fetch_candidates(col) + assert len(fetched_docs) == Layer1.MAX_SCAN + # The drawer filed beyond the window is exactly what a capable backend fixes. + assert "The newest session: we shipped the recency fetch and verified it." not in fetched_docs + + # ── Layer2 — mocked chromadb ──────────────────────────────────────────── diff --git a/tests/test_pgvector_backend.py b/tests/test_pgvector_backend.py index 1ed51ce3e2..e52f7f08fe 100644 --- a/tests/test_pgvector_backend.py +++ b/tests/test_pgvector_backend.py @@ -16,7 +16,9 @@ PalaceRef, available_backends, ) +from mempalace.backends import pgvector as pgvector_module from mempalace.backends.base import UnsupportedCapabilityError +from mempalace.backends.base import recency_sort_key as _recency_sort_key from mempalace.backends.pgvector import ( PgVectorBackend, _PgVectorClient, @@ -104,12 +106,33 @@ def scroll_rows( with_document=True, limit=None, offset=None, + order_field=None, ): self.scroll_calls.append( - {"where": where, "limit": limit, "offset": offset, "with_document": with_document} + { + "where": where, + "limit": limit, + "offset": offset, + "with_document": with_document, + "order_field": order_field, + } ) rows = self._filtered(table, where) - if limit is not None or offset: + if order_field is not None: + # Mirror the real backend: ORDER BY metadata->>field DESC NULLS + # LAST, id — then LIMIT/OFFSET. Sorting by id first and then + # stable-sorting by the recency key reproduces that tiebreak. + rows = sorted(rows, key=lambda row: row["id"]) + rows = sorted( + rows, + key=lambda row: _recency_sort_key(row.get("metadata") or {}, order_field), + reverse=True, + ) + if offset: + rows = rows[offset:] + if limit is not None: + rows = rows[:limit] + elif limit is not None or offset: # Mirror the real backend: ORDER BY id, then LIMIT/OFFSET. rows = sorted(rows, key=lambda row: row["id"]) if offset: @@ -418,7 +441,9 @@ def test_pgvector_get_unfiltered_page_pushes_limit_offset(tmp_path, fake_pgvecto # An unfiltered page is pushed to SQL as LIMIT/OFFSET instead of fetching # the whole table and slicing in Python (the O(rows x pages) path). - assert client.scroll_calls == [{"where": None, "limit": 2, "offset": 1, "with_document": True}] + assert client.scroll_calls == [ + {"where": None, "limit": 2, "offset": 1, "with_document": True, "order_field": None} + ] # ORDER BY id, then OFFSET 1 LIMIT 2 -> b, c. assert page.ids == ["b", "c"] @@ -439,7 +464,13 @@ def test_pgvector_get_filtered_page_stays_on_full_scan(tmp_path, fake_pgvector): # A filtered get keeps the full-scan path (no LIMIT/OFFSET pushed) so the # exact _matches_where re-filter runs before pagination. assert client.scroll_calls == [ - {"where": {"wing": "x"}, "limit": None, "offset": None, "with_document": True} + { + "where": {"wing": "x"}, + "limit": None, + "offset": None, + "with_document": True, + "order_field": None, + } ] assert page.ids == ["c"] @@ -458,7 +489,7 @@ def test_pgvector_get_offset_only_and_limit_only_push(tmp_path, fake_pgvector): client.scroll_calls.clear() page = col.get(offset=2, include=["metadatas"]) assert client.scroll_calls == [ - {"where": None, "limit": None, "offset": 2, "with_document": True} + {"where": None, "limit": None, "offset": 2, "with_document": True, "order_field": None} ] assert page.ids == ["c", "d"] @@ -466,7 +497,7 @@ def test_pgvector_get_offset_only_and_limit_only_push(tmp_path, fake_pgvector): client.scroll_calls.clear() page = col.get(limit=2, include=["metadatas"]) assert client.scroll_calls == [ - {"where": None, "limit": 2, "offset": None, "with_document": True} + {"where": None, "limit": 2, "offset": None, "with_document": True, "order_field": None} ] assert page.ids == ["a", "b"] @@ -486,7 +517,7 @@ def test_pgvector_get_negative_bounds_use_python_slice(tmp_path, fake_pgvector): # through to the unchanged full-scan + Python-slice path. page = col.get(offset=-1, include=["metadatas"]) assert client.scroll_calls == [ - {"where": None, "limit": None, "offset": None, "with_document": True} + {"where": None, "limit": None, "offset": None, "with_document": True, "order_field": None} ] assert page.ids == ["c"] @@ -537,7 +568,7 @@ def test_pgvector_get_all_metadata_skips_document_column(tmp_path, fake_pgvector # Exactly one scroll, with_document=False (no document text on the wire). assert client.scroll_calls == [ - {"where": None, "limit": None, "offset": None, "with_document": False} + {"where": None, "limit": None, "offset": None, "with_document": False, "order_field": None} ] # Returns just the metadata dicts (full set, any order — sort by wing+room for stability). metas_sorted = sorted(metas, key=lambda m: (m["wing"], m["room"])) @@ -571,11 +602,302 @@ def test_pgvector_get_all_metadata_filtered_uses_fast_path(tmp_path, fake_pgvect # Exactly one scroll with with_document=False — pushdown forwards the # equality filter to SQL; no document text on the wire. assert client.scroll_calls == [ - {"where": {"wing": "x"}, "limit": None, "offset": None, "with_document": False} + { + "where": {"wing": "x"}, + "limit": None, + "offset": None, + "with_document": False, + "order_field": None, + } ] assert sorted(metas, key=lambda m: m["wing"]) == [{"wing": "x"}, {"wing": "x"}] +def _fill_for_recency(col): + col.add( + ids=["old", "new", "middle", "undated"], + documents=["oldest drawer", "newest drawer", "middle drawer", "undated drawer"], + metadatas=[ + {"wing": "x", "filed_at": "2024-01-01T00:00:00Z"}, + {"wing": "y", "filed_at": "2026-08-06T00:00:00Z"}, + {"wing": "x", "filed_at": "2025-05-05T00:00:00Z"}, + {"wing": "x"}, + ], + embeddings=[[1, 0], [0, 1], [0.5, 0.5], [0.2, 0.8]], + ) + + +def test_pgvector_advertises_recency_order_capability(): + assert "supports_recency_order" in PgVectorBackend.capabilities + + +def test_pgvector_get_recent_orders_newest_first_in_sql(tmp_path, fake_pgvector): + """The ordering is pushed into the scan, so no full table comes back.""" + _backend, col = _collection(tmp_path) + _fill_for_recency(col) + client = fake_pgvector.instances[0] + client.scroll_calls.clear() + + page = col.get_recent(limit=2) + + assert page.ids == ["new", "middle"] + assert page.documents == ["newest drawer", "middle drawer"] + # One scan, ordered and limited by the database — not a full fetch + slice. + assert client.scroll_calls == [ + { + "where": None, + "limit": 2, + "offset": None, + "with_document": True, + "order_field": "filed_at", + } + ] + + +def test_pgvector_get_recent_sorts_missing_field_last(tmp_path, fake_pgvector): + _backend, col = _collection(tmp_path) + _fill_for_recency(col) + assert col.get_recent(limit=10).ids == ["new", "middle", "old", "undated"] + + +def test_pgvector_get_recent_pushes_down_equality_filter(tmp_path, fake_pgvector): + _backend, col = _collection(tmp_path) + _fill_for_recency(col) + client = fake_pgvector.instances[0] + client.scroll_calls.clear() + + page = col.get_recent(limit=10, where={"wing": "x"}) + + assert page.ids == ["middle", "old", "undated"] + assert client.scroll_calls == [ + { + "where": {"wing": "x"}, + "limit": 10, + "offset": None, + "with_document": True, + "order_field": "filed_at", + } + ] + + +def test_pgvector_get_recent_local_filter_still_orders(tmp_path, fake_pgvector): + """A filter pgvector cannot push exactly falls back to the local post-filter.""" + _backend, col = _collection(tmp_path) + _fill_for_recency(col) + client = fake_pgvector.instances[0] + client.scroll_calls.clear() + + page = col.get_recent(limit=2, where={"$or": [{"wing": "x"}, {"wing": "y"}]}) + + assert page.ids == ["new", "middle"] + # The predicate cannot ride along, but the ORDER BY and the LIMIT still + # do: one bounded page, not a fetch of the whole table. + assert client.scroll_calls == [ + { + "where": None, + "limit": 500, + "offset": None, + "with_document": True, + "order_field": "filed_at", + } + ] + + +def test_pgvector_get_recent_local_filter_does_not_fetch_whole_table(tmp_path, fake_pgvector): + """Reviewer scenario from #2168: 800 rows, an $or filter, limit=5. + + The predicate is not pushdown-safe, so it is evaluated in Python — but the + scan that feeds it must still be bounded. Before the fix this issued one + LIMIT-less scroll and dragged all 800 rows over the wire to keep 5. + """ + _backend, col = _collection(tmp_path) + rows = 800 + col.add( + ids=[f"d{i:04d}" for i in range(rows)], + documents=[f"drawer {i}" for i in range(rows)], + # Every row matches the $or, so the first page already answers it. + metadatas=[ + {"wing": "w1" if i % 2 else "w2", "filed_at": f"2026-01-01T00:00:{i % 60:02d}Z"} + for i in range(rows) + ], + embeddings=[[1, 0]] * rows, + ) + client = fake_pgvector.instances[0] + client.scroll_calls.clear() + + page = col.get_recent(limit=5, where={"$or": [{"wing": "w1"}, {"wing": "w2"}]}) + + assert len(page.ids) == 5 + # Every scroll carries a SQL LIMIT, and the total rows requested is a + # small multiple of the answer rather than the whole table. + assert client.scroll_calls, "expected at least one scroll" + assert all(call["limit"] is not None for call in client.scroll_calls) + assert sum(call["limit"] for call in client.scroll_calls) < rows + # One page suffices when the filter is not selective. + assert len(client.scroll_calls) == 1 + + +def test_pgvector_get_recent_local_filter_pages_until_enough_match(tmp_path, fake_pgvector): + """A selective filter walks further, still newest-first and still bounded.""" + _backend, col = _collection(tmp_path) + rows = 1200 + # Only the 3 oldest rows match, so the walk has to reach the far end. + col.add( + ids=[f"d{i:04d}" for i in range(rows)], + documents=[f"drawer {i}" for i in range(rows)], + metadatas=[ + {"wing": "hit" if i < 3 else "miss", "filed_at": f"2026-01-01T00:00:00.{i:06d}Z"} + for i in range(rows) + ], + embeddings=[[1, 0]] * rows, + ) + client = fake_pgvector.instances[0] + client.scroll_calls.clear() + + page = col.get_recent(limit=5, where={"$or": [{"wing": "hit"}, {"wing": "nobody"}]}) + + # Newest-first among the matches: d0002 was filed after d0001 after d0000. + assert page.ids == ["d0002", "d0001", "d0000"] + # Paged, every page bounded, and OFFSET advances rather than re-reading. + assert len(client.scroll_calls) == 3 + assert [call["limit"] for call in client.scroll_calls] == [500, 500, 500] + assert [call["offset"] for call in client.scroll_calls] == [None, 500, 1000] + + +def test_pgvector_get_recent_local_filter_caps_pathological_scan( + tmp_path, fake_pgvector, monkeypatch +): + """A filter matching nothing stops at the row cap instead of walking on.""" + monkeypatch.setattr(pgvector_module, "_RECENT_SCAN_ROW_CAP", 20) + monkeypatch.setattr(pgvector_module, "_RECENT_SCAN_PAGE_MIN", 10) + _backend, col = _collection(tmp_path) + rows = 100 + col.add( + ids=[f"d{i:04d}" for i in range(rows)], + documents=[f"drawer {i}" for i in range(rows)], + metadatas=[ + {"wing": "miss", "filed_at": f"2026-01-01T00:00:{i % 60:02d}Z"} for i in range(rows) + ], + embeddings=[[1, 0]] * rows, + ) + client = fake_pgvector.instances[0] + client.scroll_calls.clear() + + page = col.get_recent(limit=5, where={"$or": [{"wing": "nope"}, {"wing": "nada"}]}) + + assert page.ids == [] + # Two pages of 10 == the cap, then it stops. Not 100 rows. + assert sum(call["limit"] for call in client.scroll_calls) == 20 + + +def test_pgvector_get_recent_local_filter_dedupes_rows_shifted_by_a_write( + tmp_path, fake_pgvector, monkeypatch +): + """A row pushed across a page boundary by a concurrent insert is not returned twice. + + OFFSET paging is only stable while the table is. Inserting a row that + sorts newer than the page boundary shifts everything below it down by one, + so the next OFFSET re-serves the last row of the previous page. The ``id`` + dedupe is what keeps that out of the result. + """ + # Force pages smaller than the limit so there is a boundary to shift across. + monkeypatch.setattr(pgvector_module, "_RECENT_SCAN_PAGE_MIN", 2) + monkeypatch.setattr(pgvector_module, "_RECENT_SCAN_PAGE_MAX", 2) + _backend, col = _collection(tmp_path) + col.add( + ids=[f"d{i}" for i in range(6)], + documents=[f"drawer {i}" for i in range(6)], + metadatas=[{"wing": "hit", "filed_at": f"2026-01-0{i + 1}T00:00:00Z"} for i in range(6)], + embeddings=[[1, 0]] * 6, + ) + client = fake_pgvector.instances[0] + + real_scroll_rows = client.scroll_rows + inserted = {"done": False} + + def _scroll_and_insert(table, **kwargs): + rows = real_scroll_rows(table, **kwargs) + if not inserted["done"]: + inserted["done"] = True + # Newest of all, so every page below shifts down by one row. + col.add( + ids=["intruder"], + documents=["filed mid-scan"], + metadatas=[{"wing": "hit", "filed_at": "2026-12-31T00:00:00Z"}], + embeddings=[[1, 0]], + ) + return rows + + monkeypatch.setattr(client, "scroll_rows", _scroll_and_insert) + + page = col.get_recent(limit=6, where={"$or": [{"wing": "hit"}, {"wing": "nobody"}]}) + + assert len(page.ids) == len(set(page.ids)), page.ids + # Still newest-first over whatever the shifting scan managed to see. + assert page.ids[0] == "d5" + + +def test_pgvector_get_recent_local_filter_projects_out_document(tmp_path, fake_pgvector): + """The post-filter reads only metadata, so unrequested documents stay off the wire.""" + _backend, col = _collection(tmp_path) + _fill_for_recency(col) + client = fake_pgvector.instances[0] + client.scroll_calls.clear() + + page = col.get_recent( + limit=2, where={"$or": [{"wing": "x"}, {"wing": "y"}]}, include=["metadatas"] + ) + + assert page.ids == ["new", "middle"] + assert page.documents == [] + assert all(call["with_document"] is False for call in client.scroll_calls) + + +def test_pgvector_get_recent_honours_include_and_zero_limit(tmp_path, fake_pgvector): + _backend, col = _collection(tmp_path) + _fill_for_recency(col) + page = col.get_recent(limit=1, include=["metadatas"]) + assert page.ids == ["new"] + assert page.documents == [] + assert page.metadatas == [{"wing": "y", "filed_at": "2026-08-06T00:00:00Z"}] + assert col.get_recent(limit=0).ids == [] + + +def test_pgvector_get_recent_custom_order_field(tmp_path, fake_pgvector): + _backend, col = _collection(tmp_path) + col.add( + ids=["a", "b"], + documents=["written first", "written second"], + metadatas=[ + {"filed_at": "2026-01-01T00:00:00Z", "authored_at": "2020-01-01T00:00:00Z"}, + {"filed_at": "2025-01-01T00:00:00Z", "authored_at": "2024-01-01T00:00:00Z"}, + ], + embeddings=[[1, 0], [0, 1]], + ) + assert col.get_recent(limit=2, order_field="authored_at").ids == ["b", "a"] + + +def test_pgvector_scroll_rows_sql_orders_by_metadata_field(): + """The generated SQL orders on the metadata key, with NULLS LAST and an id tiebreak.""" + captured = {} + + class _Recorder(_PgVectorClient): + def __init__(self): # no connection + self._config = None + + def _execute(self, sql, params=None, *, fetch=False, many=False): + captured["sql"] = sql + captured["params"] = params + return [] + + _Recorder().scroll_rows("tbl", limit=5, order_field="filed_at") + + assert "ORDER BY metadata->>%s DESC NULLS LAST, id" in captured["sql"] + assert "LIMIT %s" in captured["sql"] + # Positional binding order matches the SQL text order: order key, then limit. + assert captured["params"] == ["filed_at", 5] + + def test_pgvector_delete_by_where_pushdown_and_local(tmp_path, fake_pgvector): _backend, col = _collection(tmp_path) col.add( diff --git a/website/concepts/memory-stack.md b/website/concepts/memory-stack.md index 3442d48fef..5473360131 100644 --- a/website/concepts/memory-stack.md +++ b/website/concepts/memory-stack.md @@ -31,12 +31,18 @@ Project: A journaling app that helps people process emotions. Auto-generated from the highest-importance drawers in the palace. Groups by room, picks the top moments, and keeps the output bounded. The generation process: -1. Reads all drawers from ChromaDB -2. Scores each by importance/emotional weight +1. Fetches a candidate window of the 2,000 most recently filed drawers +2. Scores each by importance/emotional weight, breaking ties by filing time 3. Takes the top 15 moments 4. Groups by room for readability 5. Truncates to fit within 3,200 characters +Step 1 asks the storage backend for the newest drawers directly when it can +push the ordering into storage (pgvector does; it advertises +`supports_recency_order`). Backends without that capability page through the +collection in storage order and sort the window locally, so on a palace larger +than the window the newest drawers may not all be in the running. + ``` ## L1 — ESSENTIAL STORY