Skip to content

feat(api): add ParadeDB pg_search as Citus-compatible BM25 backend - #1755

Merged
nicoloboschi merged 3 commits into
mainfrom
feat/paradedb-pg-search-bm25
May 26, 2026
Merged

feat(api): add ParadeDB pg_search as Citus-compatible BM25 backend#1755
nicoloboschi merged 3 commits into
mainfrom
feat/paradedb-pg-search-bm25

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Summary

Adds a fourth value (pg_search) for HINDSIGHT_API_TEXT_SEARCH_EXTENSION alongside the existing native, vchord, and pg_textsearch. ParadeDB pg_search is the only true-BM25 backend that works on a Citus distributed Postgres cluster — both vchord_bm25 and Timescale pg_textsearch are Citus-incompatible — so this unblocks horizontally scaled deployments. Closes #1754.

The change slots into the existing pluggable design: every backend-specific branch lives behind if config.text_search_extension == ..., so this PR just adds a fourth arm wherever the existing three live. No new abstraction is introduced.

Notes on the SQL shape

  • The retrieval arm builds the @@@ predicate as paradedb.boolean(should => ARRAY[paradedb.match('text', \$4), paradedb.match('context', \$4), paradedb.match('text_signals', \$4)]) rather than the simpler id @@@ \$4. The @@@ operator on the key_field requires field-qualified terms (text:foo), and the boolean fan-out keeps multi-field coverage without query-string interpolation.
  • The runtime reconciler (ensure_text_search_extension) disambiguates pg_textsearch vs pg_search by inspecting pg_indexes.indexdef for the key_field reloption — both extensions register a bm25 access method on a text column, so the access-method check alone is ambiguous.
  • INSERT path is a no-op for pg_search (same as pg_textsearch); the BM25 index reads base columns directly.

What ships

  • Config validation accepts pg_search
  • engine/sql/postgresql.py build_bm25_arm / prepare_bm25_text branches
  • Alembic migration arms for memory_units, learnings, reflections, and the text_signals follow-up migration
  • Runtime reconciler arm in migrations.py (incl. pg_textsearch vs pg_search disambiguation)
  • Docker compose example based on paradedb/paradedb:latest-pg17
  • Docs in configuration.md and retrieval.md
  • Unit tests for the new SQL shape

Test plan

  • uv run pytest tests/test_db_abstraction.py tests/test_migration_shape.py tests/test_admin_backup_restore.py — 151 passed
  • ./scripts/hooks/lint.sh — passes
  • uv run ty check hindsight_api/ — passes
  • End-to-end validation against the docker-compose example:
    • Built the ParadeDB image and ran migrations to head
    • Verified the BM25 index DDL is exactly USING bm25 (id, text, context, text_signals) WITH (key_field=id)
    • Retained four memories via POST /memories
    • Recall with query "BM25 ranking ParadeDB" returned the ParadeDB row at rank Fix dependencies and docker #1, bank isolation respected
  • Reviewer to spot-check the boolean-fan-out query shape — happy to switch to a simpler text @@@ \$4 form scoped to a single column if that's preferred
  • Optional follow-up (not in this PR): bring up a Citus coordinator + worker with pg_search, distribute memory_units on bank_id, confirm recall still works

@nicoloboschi
nicoloboschi force-pushed the feat/paradedb-pg-search-bm25 branch from 353f2a1 to f4a5e6f Compare May 26, 2026 12:43
Adds a fourth value (`pg_search`) for `HINDSIGHT_API_TEXT_SEARCH_EXTENSION`
alongside the existing `native`, `vchord`, and `pg_textsearch`. ParadeDB
pg_search is the only true-BM25 backend that works on a Citus distributed
Postgres cluster, so this unblocks horizontally scaled deployments.

The retrieval arm builds the @@@ predicate via paradedb.boolean(should =>
ARRAY[paradedb.match('text', $4), ...]) since @@@ on the key_field requires
field-qualified terms; this preserves multi-field coverage (text + context
+ text_signals) without needing query string interpolation.

Includes a docker-compose example under docker/docker-compose/pg_search/
based on the official paradedb/paradedb:latest-pg17 image.

Closes #1754
…ector comment

- n9i0 (learnings + pinned_reflections) validation now permits 'pgroonga',
  treating it as native at this migration stage. ensure_text_search_extension()
  at startup converts the reflections table (renamed from pinned_reflections in
  p1k2l3m4n5o6) to pgroonga structures; the learnings table is dropped in the
  same later migration so its transient native column never reaches steady state.
  Without this, pgroonga users hit ValueError on a fresh install.

- consolidator.py single-observation INSERT: the previous comment claimed
  search_vector was GENERATED ALWAYS, but migration p4q5r6s7t8u9 dropped that
  expression. Updated to reflect current behavior and flag the resulting gap
  for native (observations land with NULL search_vector and are not BM25-
  searchable until reflected/re-ingested) so a follow-up can address it.
@nicoloboschi
nicoloboschi force-pushed the feat/paradedb-pg-search-bm25 branch from 68a69f5 to 5f9d5ec Compare May 26, 2026 14:06
Rebasing onto main pulled in hindsight-docs/ changes from #1704
(Codex OAuth embeddings) and #1538 (pgroonga). Re-run the
generate-docs-skill.sh generator so the cached
skills/hindsight-docs/references/developer/configuration.md mirror
matches the current developer docs and verify-generated-files passes.
@nicoloboschi
nicoloboschi merged commit 4cd260b into main May 26, 2026
140 of 142 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ParadeDB support for BM25 search

1 participant