feat(v0.4.23): caller-LLM-driven ingest retrieval + search_hint recall booster

[jinhongkuan]

Summary

Fixes the BM25 vocab-mismatch problem that became visible after v0.4.20
made grounding status honest. Ingest-time retrieval now defaults to
caller-LLM-resolved explicit code_regions via the existing MCP
retrieval tools; server-side BM25 becomes the fallback for abstract
decisions. For the fallback path, a new optional search_hint field
widens the BM25 query with caller-supplied synonyms.

Both within the existing deterministic-retrieval tech moat: the server
still does no LLM calls at runtime.

Motivation

Real failure from 2026-04-20: 12 dispatcher-subscription-status decisions
ingested against an Accountable branch. BM25 bound every "dispatch"
decision to use-toast.ts:dispatch (React toast reducer), "active"
decisions to AcquisitionFunnel.tsx:ActiveUser, "source" decisions to
PresentationSlideshow.tsx:SlideSource. Under v0.4.19 these would
have silently auto-promoted to REFLECTED; under v0.4.20 they stay at
PENDING forever (honest but unactionable).

Changes

Lever 1 — caller-LLM retrieval is the new default

• skills/bicameral-ingest/SKILL.md Step 2 now instructs the caller
  LLM to use validate_symbols + search_code + get_neighbors to
  resolve explicit code_regions BEFORE ingesting.
• Step 3 leads with internal format (explicit regions) as preferred;
  natural format is the fallback for abstract decisions.
• No server-side code change. The server already accepted internal-
  format payloads.

Lever 2 — search_hint recall booster

• IngestMapping.search_hint: str and IngestDecision.search_hint: str
  — optional caller-supplied synonym / identifier-name hints.
• ground_mappings concatenates description + search_hint as the
  BM25 query when the hint is non-empty. Strictly additive.
• Query-only metadata: never stored on intent.description, never
  surfaces in briefs / status / gap-judge responses.

Test plan

• 8 new tests in tests/test_v0423_search_hint.py (propagation,
  query construction, backward compat, span-text pollution guard)
• Regression: 34/34 pass across test_v0416_natural_format_fields,
  test_v0423_search_hint, test_phase1_l1_wiring,
  test_resolve_compliance

Upgrade notes

Pre-existing false-positive bindings from BM25 auto-grounding persist
in the graph. To clean up: bicameral.reset → re-ingest under the
new skill defaults. Targeted edge pruning tracked for a future
release.

Next up

This is incremental improvement within the current architecture.
Separate plan document coming for the structural refactor that
collapses intent → maps_to → symbol → implements → region into
intent → binds_to → region directly (symbol becomes a retrieval-tier-only
primitive). That's v0.5.0 territory — decided to ship this today so
you're unblocked on the dispatcher ingest while the refactor is scoped.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added optional search_hint parameter to enhance code search accuracy during ingestion, allowing callers to provide additional context for better symbol location resolution.

• Documentation

  • Updated ingest workflow guidance to emphasize explicit symbol resolution using validation tools before ingestion.

• Tests

  • Added comprehensive test coverage for search hint propagation and BM25 query construction.

• Chores

  • Bumped version to v0.4.23.

[coderabbitai]

📝 Walkthrough

Walkthrough

Added optional search_hint field to IngestMapping and IngestDecision contracts, with propagation through the ingest handler and integration into BM25 query construction in the grounding adapter. Updated skill documentation to reflect MCP-driven grounding workflow, incremented package version to 0.4.23, and added comprehensive regression tests.

Changes

Cohort / File(s)	Summary
Version Updates CHANGELOG.md, RECOMMENDED_VERSION, pyproject.toml	Bumped version from 0.4.22 to 0.4.23 with changelog entry describing search_hint feature and MCP-driven grounding workflow updates.
Contract Definitions contracts.py	Added optional search_hint: str field (default empty) to both IngestMapping and IngestDecision models to carry BM25 query-only metadata through ingest pipeline.
Handler Logic handlers/ingest.py	Updated _normalize_payload to propagate search_hint from natural-format decisions into generated mappings during normalization.
Adapter & Grounding adapters/code_locator.py	Modified ground_mappings() to conditionally concatenate trimmed search_hint to description for BM25 query construction; used in token counting and search operations only, not stored in output.
Skill Documentation skills/bicameral-ingest/SKILL.md	Restructured ingest workflow to emphasize MCP tool-driven grounding; added symbol hypothesis generation, validate_symbols, search_code, and optional get_neighbors steps; introduced search_hint as standardized BM25 booster; clarified internal vs. natural format selection criteria.
Test Suite tests/test_v0423_search_hint.py	Added 203 lines of regression tests validating search_hint propagation through normalization, BM25 query construction with conditional hint concatenation, backward compatibility (absent/empty hints), and skipping of grounding when code_regions present.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Possibly related PRs

• bicameral-mcp#9: Modifies the grounding pipeline with tiered coverage and reuse logic; overlaps with this PR's changes to ground_mappings() and BM25 query construction.
• bicameral-mcp#3: Introduced the original IngestMapping/IngestDecision models and _normalize_payload handler; this PR extends those same contracts and propagation paths with the new search_hint field.

Poem

🐰 A hint in the query, a whisper so fine,
To boost the retrieval when symbols align,
Version bumped up to point-two-three,
The MCP tools dance in harmony!
Through contracts and handlers it flows like a stream,
Grounding made better—a rabbit's sweet dream. ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 78.57% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main changes: caller-LLM-driven ingest retrieval and the search_hint recall booster feature added in v0.4.23.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jin/ingest-retrieval-search-hint

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (2)

skills/bicameral-ingest/SKILL.md (1)

161-279: Skill guidance reads well; grounds the Lever 1 / Lever 2 split clearly.

One micro-nit (non-blocking): example at line 232 puts search_hint alongside explicit code_regions, while the contract doc and this skill both note search_hint is only consulted when code_regions is empty. The example's own prose at lines 277-278 correctly says to treat it as a safety net for future re-grounding — but a casual reader may miss that and assume it's active. Consider adding a one-line parenthetical to the code block, e.g. // optional — consulted only if code_regions is ever cleared / re-grounded.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@skills/bicameral-ingest/SKILL.md` around lines 161 - 279, Add a clarifying
parenthetical to the internal-format example to indicate that search_hint is
only used when code_regions is empty or re-grounded; update the example block
that contains mappings[].code_regions and the search_hint field (the payload
example with mappings, symbols, code_regions, and search_hint) to include a
one-line note such as "(optional — consulted only if code_regions is later
cleared or during re-grounding)" beside the search_hint so readers don't assume
it is active when explicit code_regions are provided.

adapters/code_locator.py (1)

312-324: Stage-2 fused re-search drops search_hint.

Inside _ground_single, when Stage 1 fuzzy-matching produced symbols, line 319 re-invokes self.search_code(description, symbol_ids=...) with the bare description rather than the widened bm25_query used at the top of ground_mappings. The result is that the search_hint recall booster is applied only to the initial BM25 pass (whose hits are then ignored down this branch because fused is replaced) and not to the symbol-seeded RRF fusion re-search — which is exactly the path search_hint would most plausibly help.

Consider threading the same widened query down:

♻️ Proposed fix

     def _ground_single(
         self,
         description: str,
         db,
         max_files: int,
         fuzzy_threshold: int,
         max_symbols: int,
         hits: list[dict] | None = None,
         mapping_symbol_names: list[str] | None = None,
+        bm25_query: str | None = None,
     ) -> list[dict]:
@@
+        search_query = bm25_query or description
@@
                 if matched_ids:
-                    fused = self.search_code(description, symbol_ids=sorted(matched_ids))
+                    fused = self.search_code(search_query, symbol_ids=sorted(matched_ids))
                 else:
                     fused = hits

And pass bm25_query=bm25_query from the _ground_single call site in ground_mappings.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@adapters/code_locator.py` around lines 312 - 324, _stage-2 fused re-search in
_ground_single calls self.search_code(description, symbol_ids=...) and thus
drops the widened bm25_query (and associated search_hint) used higher in
ground_mappings; modify the code so the same bm25_query is threaded through:
update the call sites so ground_mappings passes bm25_query into _ground_single,
and inside _ground_single call self.search_code(...,
symbol_ids=sorted(matched_ids), bm25_query=bm25_query) (or pass along whatever
parameter name search_code expects) so the search_hint/boosting is applied to
the symbol-seeded RRF re-search as well.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@adapters/code_locator.py`:
- Around line 312-324: _stage-2 fused re-search in _ground_single calls
self.search_code(description, symbol_ids=...) and thus drops the widened
bm25_query (and associated search_hint) used higher in ground_mappings; modify
the code so the same bm25_query is threaded through: update the call sites so
ground_mappings passes bm25_query into _ground_single, and inside _ground_single
call self.search_code(..., symbol_ids=sorted(matched_ids),
bm25_query=bm25_query) (or pass along whatever parameter name search_code
expects) so the search_hint/boosting is applied to the symbol-seeded RRF
re-search as well.

In `@skills/bicameral-ingest/SKILL.md`:
- Around line 161-279: Add a clarifying parenthetical to the internal-format
example to indicate that search_hint is only used when code_regions is empty or
re-grounded; update the example block that contains mappings[].code_regions and
the search_hint field (the payload example with mappings, symbols, code_regions,
and search_hint) to include a one-line note such as "(optional — consulted only
if code_regions is later cleared or during re-grounding)" beside the search_hint
so readers don't assume it is active when explicit code_regions are provided.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a7887c10-9417-4dca-87e2-81388ac45d3a

📥 Commits

Reviewing files that changed from the base of the PR and between fe2ba95 and c7a8141.

📒 Files selected for processing (8)

• CHANGELOG.md
• RECOMMENDED_VERSION
• adapters/code_locator.py
• contracts.py
• handlers/ingest.py
• pyproject.toml
• skills/bicameral-ingest/SKILL.md
• tests/test_v0423_search_hint.py