feat(api): add ParadeDB pg_search as Citus-compatible BM25 backend - #1755
Merged
Conversation
nicoloboschi
force-pushed
the
feat/paradedb-pg-search-bm25
branch
from
May 26, 2026 12:43
353f2a1 to
f4a5e6f
Compare
Adds a fourth value (`pg_search`) for `HINDSIGHT_API_TEXT_SEARCH_EXTENSION`
alongside the existing `native`, `vchord`, and `pg_textsearch`. ParadeDB
pg_search is the only true-BM25 backend that works on a Citus distributed
Postgres cluster, so this unblocks horizontally scaled deployments.
The retrieval arm builds the @@@ predicate via paradedb.boolean(should =>
ARRAY[paradedb.match('text', $4), ...]) since @@@ on the key_field requires
field-qualified terms; this preserves multi-field coverage (text + context
+ text_signals) without needing query string interpolation.
Includes a docker-compose example under docker/docker-compose/pg_search/
based on the official paradedb/paradedb:latest-pg17 image.
Closes #1754
…ector comment - n9i0 (learnings + pinned_reflections) validation now permits 'pgroonga', treating it as native at this migration stage. ensure_text_search_extension() at startup converts the reflections table (renamed from pinned_reflections in p1k2l3m4n5o6) to pgroonga structures; the learnings table is dropped in the same later migration so its transient native column never reaches steady state. Without this, pgroonga users hit ValueError on a fresh install. - consolidator.py single-observation INSERT: the previous comment claimed search_vector was GENERATED ALWAYS, but migration p4q5r6s7t8u9 dropped that expression. Updated to reflect current behavior and flag the resulting gap for native (observations land with NULL search_vector and are not BM25- searchable until reflected/re-ingested) so a follow-up can address it.
nicoloboschi
force-pushed
the
feat/paradedb-pg-search-bm25
branch
from
May 26, 2026 14:06
68a69f5 to
5f9d5ec
Compare
Rebasing onto main pulled in hindsight-docs/ changes from #1704 (Codex OAuth embeddings) and #1538 (pgroonga). Re-run the generate-docs-skill.sh generator so the cached skills/hindsight-docs/references/developer/configuration.md mirror matches the current developer docs and verify-generated-files passes.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a fourth value (
pg_search) forHINDSIGHT_API_TEXT_SEARCH_EXTENSIONalongside the existingnative,vchord, andpg_textsearch. ParadeDB pg_search is the only true-BM25 backend that works on a Citus distributed Postgres cluster — bothvchord_bm25and Timescalepg_textsearchare Citus-incompatible — so this unblocks horizontally scaled deployments. Closes #1754.The change slots into the existing pluggable design: every backend-specific branch lives behind
if config.text_search_extension == ..., so this PR just adds a fourth arm wherever the existing three live. No new abstraction is introduced.Notes on the SQL shape
@@@predicate asparadedb.boolean(should => ARRAY[paradedb.match('text', \$4), paradedb.match('context', \$4), paradedb.match('text_signals', \$4)])rather than the simplerid @@@ \$4. The@@@operator on thekey_fieldrequires field-qualified terms (text:foo), and the boolean fan-out keeps multi-field coverage without query-string interpolation.ensure_text_search_extension) disambiguatespg_textsearchvspg_searchby inspectingpg_indexes.indexdeffor thekey_fieldreloption — both extensions register abm25access method on atextcolumn, so the access-method check alone is ambiguous.pg_search(same aspg_textsearch); the BM25 index reads base columns directly.What ships
pg_searchengine/sql/postgresql.pybuild_bm25_arm/prepare_bm25_textbranchesmemory_units,learnings,reflections, and thetext_signalsfollow-up migrationmigrations.py(incl. pg_textsearch vs pg_search disambiguation)paradedb/paradedb:latest-pg17configuration.mdandretrieval.mdTest plan
uv run pytest tests/test_db_abstraction.py tests/test_migration_shape.py tests/test_admin_backup_restore.py— 151 passed./scripts/hooks/lint.sh— passesuv run ty check hindsight_api/— passesUSING bm25 (id, text, context, text_signals) WITH (key_field=id)POST /memoriestext @@@ \$4form scoped to a single column if that's preferredpg_search, distributememory_unitsonbank_id, confirm recall still works