feat(bm25): configurable native language + opt-in pgroonga backend

[nicoloboschi]

Summary

Makes Hindsight's BM25 indexing language configurable, adds pgroonga as a fourth opt-in text-search backend, and adds a single knob to force the LLM output language across every LLM-driven pipeline. The goal is to serve non-English banks (especially CJK) out of the box.

Three new env vars

Variable	Default	Purpose
HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE	english	PG text search dictionary used by the native backend in to_tsvector / to_tsquery. Validated as a PG identifier so it can be embedded safely in raw SQL. Users with zhparser etc. can point to custom dictionaries.
HINDSIGHT_API_TEXT_SEARCH_EXTENSION	native	Now accepts pgroonga (fourth backend). pgroonga's TokenBigram polyglot tokenizer + NormalizerNFKC150 Unicode normalization handles English, French, Japanese, Chinese, etc. in a single index. Docker recipe at docker/docker-compose/pgroonga/.
HINDSIGHT_API_LLM_OUTPUT_LANGUAGE	unset	When set, every LLM-generated artifact is forced into this language regardless of source content. Applies uniformly to retain (fact extraction), consolidation (observations / mental models), and reflect (response synthesis).

The two language knobs are deliberately independent — search tokenization and LLM output language are separate concerns. Common patterns are documented on the Multilingual Support page.

Schema change

New migration p4q5r6s7t8u9_configurable_bm25_language drops the GENERATED ALWAYS expression on memory_units.search_vector (and reflections.search_vector), turning them into regular tsvector columns. The application now populates them at INSERT time via to_tsvector(:lang, ...) using the configured native language. Existing rows retain their English-derived lexemes — switching languages only affects newly-written rows. A backfill SQL snippet is documented for users who need to re-index existing data.

The initial migration's _detect_text_search_extension was extended to accept pgroonga (returning native so the migration still creates valid columns); ensure_text_search_extension() at startup converts the schema to pgroonga's structure on first boot, installing the extension if necessary.

LLM output language wiring

A single shared helper engine.prompt_utils.output_language_directive(language) is consumed by all three pipelines so the directive is identical everywhere:

• retain — appended to the fact-extraction system prompt in _build_extraction_prompt_and_schema.
• consolidation — appended to the batch observation prompt via a new llm_output_language param on build_batch_consolidation_prompt.
• reflect — appended to the final response prompt via a new llm_output_language param on build_final_system_prompt, threaded through run_reflect_agent and MemoryEngine.

Documentation

The multilingual story is consolidated on hindsight-docs/docs/developer/multilingual.md alongside the existing LLM / embedding / reranker guidance. configuration.md is a brief env-var table that links out. Backend-choice patterns ("CJK-out-of-the-box → pgroonga", "single-language bank → native + language", "cross-lingual unification → llm_output_language=English") are written up there.

Test plan

• tests/test_config_validation.py — defaults, lowercasing, PG-identifier rejection, pgroonga accepted, llm_output_language empty-string handling.
• tests/test_db_abstraction.py — native arm honours configured language, pgroonga branch uses &@~ + pgroonga_score, pgroonga ignores the native language knob.
• tests/test_multilingual_bm25.py — output_language_directive helper, per-pipeline directive injection (retain, consolidation, reflect), str.format safety for the consolidation prompt, migration shape regression.
• All existing tests pass (tests/test_migration_shape.py, tests/test_alembic_dag.py, tests/test_admin_backup_restore.py).
• ./scripts/hooks/lint.sh clean.
• End-to-end smoke against the new pgroonga docker-compose recipe (manual).

CI failures unrelated to this PR

The current CI run has 5 failures that all reproduce on a clean origin/main checkout:

• test-api → test_hierarchical_fields_categorization count assertion (35 vs 36) — pre-existing breakage from feat(api): add targeted consolidation by observation scopes #1746 which added consolidation_max_memories_per_round to _CONFIGURABLE_FIELDS without updating the count.
• test-rust-cli / test-embed-windows / test-doc-examples (cli) — all the same Rust compilation error (E0061: this method takes 3 arguments but 2 arguments were supplied) in hindsight-cli. I touched no Rust / API / OpenAPI files.
• LLM acceptance (vertexai) — test_llm_api_methods: known Vertex/Gemini flake (LLM occasionally doesn't emit the expected tool call).