fix(layers): fetch the most recent drawers for L1 wake-up

[ATKabli]

What does this PR do?

#1630 fixed the L1 wake-up ordering by adding filed_at as a secondary sort key, and documented the part it could not fix:

The fetch still scans up to MAX_SCAN=2000 drawers in collection (insertion) order, so on a very large unscoped wing the most-recent drawers may fall outside the cap before sorting... A SQL-side most-recent-N by filed_at fetch would fix the unscoped case but is a larger change left for a follow-up.

This is that follow-up. Above the 2,000-drawer cap the sort was correct and the input was not: Layer1.generate scored whatever the backend handed back first, which in practice is the oldest slice of the palace. On a 149k-drawer palace, wake-up permanently opened on the first files ever mined and never on this week's sessions, no matter how the sort was written.

A backend capability rather than a pgvector special case, so every backend keeps working and capable backends get correctness at scale:

• BaseCollection.get_recent(limit, where, order_field, include) returns up to limit records newest-first on an ISO-8601 metadata field. The ABC default pages through get() and sorts the window locally, which is exactly what Layer1 did inline, so any backend that does not override it behaves as it did before this PR. The docstring states that the default is exact when no more than limit records match where, and approximate above that.
• PgVectorCollection.get_recent overrides it with ORDER BY metadata->>%s DESC NULLS LAST, id pushed into the scan, and PgVectorBackend advertises supports_recency_order. That token means the backend really does return the top limit under the stored text ordering, at any table size, for every filter it can evaluate in storage, which is every filter L1 uses ($eq, $ne, $in, $nin, $and). It says nothing about whether that text ordering matches wall-clock order, which is a property of what the writers store, and filters the backend cannot push down are bounded by a documented scan cap. A backend that bounds its non-pushdown walk has to document the bound.
• No timestamp cast. The ordering is on the stored text, the same comparison Layer 1 has used since fix(layers): order L1 wake-up by recency (filed_at) so it surfaces the latest moments #1630, and a cast would fail hard on a single malformed value. See the honest caveat on filed_at below.
• Filters pgvector cannot push down exactly keep the local post-filter contract, but no longer fetch the table to honour it. The ordering is still pushed into SQL and the branch walks the result newest-first a page at a time, stopping as soon as limit rows match, capped at 50,000 rows scanned.
• EmbeddingCollection.get_recent forwards explicitly. Without the forwarder, MRO resolves the concrete ABC default on the wrapper and shadows the inner backend's pushdown. The existing invariant test test_wrapper_forwards_all_concrete_basecollection_methods catches exactly this and did.
• Layer1._fetch_candidates uses the capability and falls back to the previous paged scan when a collection predates get_recent or the backend errors, so a plugin backend or a transient failure degrades wake-up instead of emptying it.
• recency_sort_key is shared so every local sort orders identically: records with a missing, empty, or non-string timestamp sort last instead of raising on a str/None comparison.

L1's own ranking semantics do not change. importance stays the primary key and filed_at the tiebreak. Only the candidate window changes, from "the first 2,000 rows the backend hands back" to "the 2,000 most recently filed".

Measured on the 149k-drawer palace this was written for: wake-up leads with the newest sessions and renders in 0.72s.

The bounded scan, and what it costs (second commit)

@mvalentsev is right, and the second commit is the answer to his review. get_recent pushed the ORDER BY into SQL but passed limit=None to _scroll whenever _requires_local_filter(where) was true, so $or, $contains and every comparison operator dragged the whole table across the wire to keep limit rows. His 800-row case issued exactly one LIMIT-less scroll. That is now a regression test asserting on scroll_calls.

His second observation is also correct: L1 never reaches that path. Layer1._fetch_candidates passes {"wing": ...} or None, both pushdown-safe, so wake-up always took the exact LIMIT n branch. It was a latent bug in the public method, not in the feature this PR is about, and my own test test_pgvector_get_recent_local_filter_still_orders was asserting "limit": None as expected, which is how it got through.

Three things about the fix that are better heard from me than found:

• The 50,000-row cap is a real trade, not a free win. A filter matching almost nothing in a huge table now returns fewer than limit rows rather than reading the table. The unbounded version was genuinely exact there. The approximation is much tighter than the base class default's, because the walk is newest-first, so what comes back is the newest matching rows within the newest 50,000 records rather than the newest within an arbitrary storage-order window. The pushdown branch keeps no cap and stays exact. supports_recency_order was rewritten to stop reading as a blanket promise.
• OFFSET paging under concurrent writes. ORDER BY metadata->>field DESC NULLS LAST, id is a total order because id is the primary key, so OFFSET paging is well defined, the same guarantee the existing ORDER BY id paging relies on. Rows are deduped by id, so a concurrent insert that shifts a row across a page boundary cannot return it twice. A concurrent delete can still skip one row; that is inherent to OFFSET paging and unchanged from the pre-existing paged get. A keyset cursor would close it, needs plumbing through scroll_rows, and is deliberately not smuggled in here.
• Two things elsewhere in the diff were wrong, found while chasing this. BaseCollection.get_recent returned padding for projections the caller excluded, so include=["metadatas"] answered documents=["", "", ...] where pgvector answers []; worse, include=["documents"] meant metadatas never came back, every sort key collapsed to (0, ""), and the sort silently did nothing while the method claimed newest-first. Fixed, with a test. include projections the caller excluded now come back empty from the base default too, matching get and pgvector.

Honest caveat: filed_at is not written in one offset form today

The pgvector docstring used to claim filed_at is always UTC. It is not. Ten of eleven production writers use datetime.now().isoformat() (naive local): miner.py:1393,1607, convo_miner.py:157,576, format_miner.py:573,639, closet_llm.py:342, sweeper.py:280, mcp_server.py:2749 (add_drawer), mcp_server.py:3712 (diary_write). One, diary_ingest.py:174 (+191, 302), uses datetime.now(timezone.utc).isoformat() and writes ...+00:00. Both land in the same collection through the same accessor with no wing separation, and the split runs inside one feature: diary_write is naive, diary_ingest is UTC.

So text order between those two groups is skewed by the host's UTC offset, and the sign flips with the hemisphere:

mined (naive local, TZ=Asia/Riyadh): 2026-08-08T14:12:20.804044
diary (utc +00:00)                 : 2026-08-08T11:12:20.804241+00:00
same instant, text sort says mined is NEWER: True

Asia/Riyadh        mined > diary lexically = True
America/New_York   mined > diary lexically = False
UTC                mined > diary lexically = False

This predates the PR. layers.py has compared filed_at as raw text since #1630, and mcp_server.py::_filed_at_in_window parses with fromisoformat then strips tzinfo, absorbing the same skew. No ORDER BY filed_at existed in SQL before this PR.

But this PR changes the failure mode, which is why it is in the body rather than buried. Before, the mis-ordering only affected ranking within a storage-order window. With the pushdown, ORDER BY ... LIMIT n decides membership of the candidate window, so on a palace above MAX_SCAN=2000 the skew can push drawers out of the L1 candidate set entirely rather than merely mis-rank them inside it. Bounded by the host's offset, but a new consequence of an old bug. Standardising the writers on UTC needs a migration for palaces already holding both forms; happy to open that separately if you agree that is the right split.

One further caveat that no test in this repo can catch: metadata->>%s and the id tiebreak both sort under the database collation, while recency_sort_key sorts by Python codepoint. Under en_US.UTF-8/ICU those disagree on strings differing only in punctuation, which is exactly the axis the two filed_at forms differ on, and the in-repo test double emulates ordering in Python. Named in the docstring.

How to test

pytest tests/ -q --ignore=tests/benchmarks     4320 passed, 31 skipped
ruff format --check .                          212 files already formatted
ruff check .                                   All checks passed!

Baseline origin/develop@906b918 (develop after the 3.7.1 sync) in the same environment: 4295 passed, 31 skipped, so +25 tests and no existing test deleted. ruff 0.16.1 (the repo pin), Python 3.12.13. One existing test, test_pgvector_get_recent_local_filter_still_orders, had its assertion corrected from "limit": None to "limit": 500, which is the whole point of the second commit.

Reviewer's scenario, run against the unfixed source:

>       assert all(call["limit"] is not None for call in client.scroll_calls)
E       assert False
FAILED tests/test_pgvector_backend.py::test_pgvector_get_recent_local_filter_does_not_fetch_whole_table

One scroll, limit=None, 800 rows fetched to return 5. After: a single LIMIT 500 scroll.

The base-class projection defect, against the unfixed base:

>       assert page.documents == []
E       AssertionError: assert ['', ''] == []

New tests, all six of which fail on the parent commit:

• test_pgvector_get_recent_local_filter_does_not_fetch_whole_table — his 800-row $or case; asserts every scroll carries a SQL LIMIT and total rows requested stays under the table size
• test_pgvector_get_recent_local_filter_pages_until_enough_match — selective filter, 1,200 rows, 3 matches at the far end; asserts three pages, each bounded, OFFSET advancing None → 500 → 1000, order still newest-first
• test_pgvector_get_recent_local_filter_caps_pathological_scan — filter matching nothing, cap monkeypatched to 20; stops at 20 of 100 rows
• test_pgvector_get_recent_local_filter_dedupes_rows_shifted_by_a_write — a newer row inserted mid-scan shifts the page boundary; asserts no duplicate. Proven to bite: with the seen check stubbed out the result is ['d5','d4','d4','d3','d2','d1']
• test_pgvector_get_recent_local_filter_projects_out_document — with_document=False on the local branch
• test_base_get_recent_default_honours_include_projection — unrequested projections empty, metadatas fetched anyway so the sort works

Plus the original coverage: base default (ordering, missing/empty/non-string timestamps sorting last, the window cap with 500-record paging asserted, where passthrough, zero limit, Chroma-shaped dict get()); pgvector (capability token, SQL text and bound-parameter order, LIMIT pushdown, exact-filter pushdown, include projection, zero limit, custom order_field); Layer1 (capability used with limit=MAX_SCAN and no scan issued, wing filter forwarded as where, fallback when get_recent is absent, fallback when it raises, and a capable backend surfacing a drawer filed beyond the scan window).

Differential harness for the paged path. The paged branch was checked against a brute-force reference (sort all rows by (present, filed_at) desc then id asc, filter, take limit) over 1,900 randomised cases: row counts 0 to 1,001 straddling every page boundary, six filter shapes, limit 1 to 1,000, page sizes 1 to 500, caps 1 to 50,000, duplicate filed_at forcing the id tiebreak, missing and empty-string filed_at, and a mode where matches exist only at the far end. Zero divergences. The cap-boundary arithmetic (want = min(page_size, cap - scanned)) was separately confirmed not to fire the short-page break spuriously at cap=97 page=10, cap=13 page=5, cap=7 page=3.

Landing order with #2169. The two branches were merged in a scratch worktree. mempalace/layers.py auto-merges with no conflict. The only textual conflicts are additive and both resolve by union: the shared import block at the top of tests/test_layers.py, and the ## [Unreleased] anchor in CHANGELOG.md. The composed tree was verified green at the base where the compose was run (3903 passed, 31 skipped, exactly 3850 + 25 + 28, ruff clean), so nothing is lost or duplicated by combining them; both branches now sit rebased on the same post-3.7.1 develop (906b918) and the conflict shape is unchanged. Whichever lands second needs that two-hunk rebase and nothing more.

Checklist

• Tests pass (python -m pytest tests/ -v) — 4320 passed, 31 skipped, against 4295 on develop in the same env
• No hardcoded paths
• Linter passes (ruff check .) — clean on ruff 0.16.1, along with ruff format --check .

Docs: website/concepts/memory-stack.md L1 generation steps, and a CHANGELOG entry under Unreleased. No new dependencies.

[mvalentsev]

PgVectorCollection.get_recent passes limit=None to _scroll when the filter is not pushdown-safe, so the whole table crosses the wire to return limit rows.

With 800 rows and where={"$or": [{"wing": "w1"}, {"wing": "w2"}]}, returning 5 rows fetched all 800 and the SQL carried no LIMIT. The pushdown-safe {"wing": "w1"} fetched 5.

L1 does not hit this, since _fetch_candidates only ever passes {"wing": ...} or None, both pushdown-safe. The method is public though.

See also #660, which also replaces L1's candidate fetch.

[ATKabli]

You're right, and thanks for the precise repro, that made it a five-minute confirmation instead of an argument.

get_recent pushed the ORDER BY into SQL but passed limit=None to _scroll whenever _requires_local_filter(where) was true, so $or, $contains and every comparison operator dragged the whole table across the wire to keep limit rows. Your 800-row case issued exactly one LIMIT-less scroll. I've turned that into a regression test asserting on scroll_calls the way the neighbouring tests do; it fails on the previous head of this branch with assert all(call["limit"] is not None ...).

Your second observation is also correct and I want to be explicit about it rather than let it look like I'm minimising the first: L1 never reaches this path. Layer1._fetch_candidates passes {"wing": ...} or None, both pushdown-safe, so wake-up always took the exact LIMIT n branch. This was a latent bug in the public method, not in the feature the PR is about. It also means my own test test_pgvector_get_recent_local_filter_still_orders was asserting "limit": None as expected, which is how it got through.

The fix is in a separate commit so you can read just the delta. The predicate still can't ride along, but the ordering can, so instead of one unbounded scan the local-filter branch now walks the table newest-first one SQL page at a time and stops at the first page that completes the answer. On your case that's a single LIMIT 500 scroll. The exact-pushdown path is byte-for-byte unchanged.

Three things I'd rather you hear from me than find:

• OFFSET paging under concurrent writes. ORDER BY metadata->>field DESC NULLS LAST, id is total because id is the PK, so paging is well defined, the same guarantee the existing ORDER BY id paging relies on. Rows are deduped by id, so a concurrent insert that shifts a row across a boundary can't return it twice. A concurrent delete can still skip one row; that's inherent to OFFSET paging and unchanged from the pre-existing paged get. A keyset cursor would close it and is a separate change, not something I want to smuggle in here.
• The walk is capped at 50,000 rows, and that costs something the old code had. A filter matching almost nothing in a huge table now returns fewer than limit rows rather than reading the table. The unbounded version was genuinely exact there, so this is a trade, not a free win. It's a much tighter approximation than the base class's, since the walk is newest-first, but it is one. Consequently I've rewritten what supports_recency_order claims instead of leaving it reading as a blanket promise: it covers the filters the backend can evaluate in storage (every filter L1 uses, plus anything built from $eq/$ne/$in/$nin/$and), and a backend that bounds its non-pushdown walk has to document the bound. The old wording would have been a promise this code no longer keeps.
• Two things I got wrong elsewhere in the diff, found while chasing this. The base-class default was returning padding for projections the caller excluded, so include=["metadatas"] answered with a list of empty strings for documents where pgvector answers []; worse, include=["documents"] meant metadatas never came back, every sort key collapsed to the same value, and the sort silently did nothing. Fixed, with a test. And I claimed in the docstring that filed_at is always UTC. It isn't: diary_ingest writes datetime.now(timezone.utc).isoformat() and the other ten writers use datetime.now().isoformat(), so on a host off UTC the two sort against each other skewed by the local offset. That predates this PR (L1 has compared filed_at as text since fix(layers): order L1 wake-up by recency (filed_at) so it surfaces the latest moments #1630), but the pushdown changes the failure mode from mis-ranking inside the window to mis-deciding membership of it, so it belongs in the record. Standardising the writers needs a migration for palaces already holding both forms; happy to open that separately if you agree it's the right split.

One more caveat I'd rather write down than leave implicit: metadata->>%s and the id tiebreak both sort under the database collation, while recency_sort_key sorts by Python codepoint. Under en_US.UTF-8/ICU those can disagree on strings differing only in punctuation, which is exactly the axis the two filed_at forms differ on, and the in-repo test double emulates ordering in Python so no test can catch it. Named in the docstring.

Rebased on current develop. Full suite 3875 passed / 31 skipped against 3850 on develop in the same environment; ruff 0.16.1 clean. The paged branch is also checked against a brute-force reference over 1,900 randomised cases (row counts straddling every page boundary, six filter shapes, duplicate and missing filed_at, caps from 1 to 50,000) with zero divergences.

[ATKabli]

The fix described above is pushed as the second commit. The non-pushdown-safe path now walks LIMIT'd pages newest-first (500–5000 per page), dedupes by id, and stops at limit matches or a 50,000-row cap — never a LIMIT-less scroll. Your 800-row $or scenario is a regression test asserting on scroll_calls: one call, bounded limit, well under the table size. The capability docstring was rewritten so supports_recency_order no longer reads as a blanket promise, and the base-class include projection bug found while chasing this is fixed with its own test.

Rebased onto 906b918 (post-3.7.1 develop): 4320 passed, 31 skipped, ruff clean. The PR body is updated to match, including an honest section on what the 50k cap trades away.