perf(backends): O(N) streaming scan() primitive for full-corpus iteration (#1797)

[trek-e]

Stacked on MemPalace#1664 — base is the perf/1657-read-path-o1 branch, so this diff shows only the scan() work and requires MemPalace#1664 to merge first. After MemPalace#1664 lands in develop, this retargets to develop for upstream review.

Closes MemPalace#1797.

What

Adds BaseCollection.scan(*, where, include, limit) -> Iterator[(id, document, metadata)] — the single linear traversal the rebuild / dedup / migrate / wake-up paths share — and routes the offset-paginated callers through it.

Offset pagination was O(N²): ChromaDB's offset is scan-and-discard at the SQLite layer, so page k re-skips k·B rows. On a 623K-drawer palace the last page re-skipped ~622K rows.

• base.py: scan() default paginates get() (correct; backends override).
• chroma.py: _sql_scan() streams embeddings LEFT JOIN embedding_metadata ORDER BY e.id in one cursor — each drawer's rows are contiguous, so it yields with O(1) accumulator memory, no offset re-skip, and no HNSW load (reads chroma.sqlite3 by path, like extract_via_sqlite). The LEFT JOIN surfaces drawers with no user metadata; chroma:document is excluded from the join unless documents are requested. A single {key: value} where is pushed to SQL; richer filters fall back to the base scan. SQLite errors during streaming propagate — a rebuild fails loud, never silently truncates.
• Rewired: repair._extract_drawers, dedup.get_source_groups, migrate._iter_collection_items, palace.prefetch_mined_set / bulk_check_mined, layers.Layer1.generate (MAX_SCAN → limit). EmbeddingCollection forwards scan(). dedup_source_group is unchanged (by-id fetch, not a scan).

Testing

New tests/test_scan.py (10 tests): parity (SQL override == get()), metadata-only excludes documents, documents included, where filter, limit, empty collection, complex-where fallback, drawer-with-no-metadata still yielded, base-default parity, wrapper delegation. Full suite: 2826 passed, 5 skipped; ruff clean.

Builds on the SqliteExactRetriever / base.py aggregation seam from MemPalace#1664.