fix: compound token extraction for grounding recall (9%→14%)

[silongtan]

Summary

• The tokenizer in _ground_single used re.findall(r"[a-zA-Z]{4,}") which stripped underscores and dots, destroying compound identifiers (decrease_stock →
  ["decrease", "stock"]) before they reached validate_symbols
• Added _COMPOUND_RE regex to extract snake_case and dotted.name identifiers intact, prepended to the token list before word-level tokens
• 6 lines changed in one file (adapters/code_locator.py)

Results

Metric	Before	After
Recall	9.3%	13.9% (+50%)
MRR@3	0.577	0.592 (held)
Saleor recall	13.6%	27.3% (2x)

Key wins: decrease_stock, update_order_status, transaction.atomic now match their exact symbols instead of generic fragments like Stock or Order.

Test plan

• 29/29 unit tests pass
• Eval harness: recall 9.3% → 13.9%, MRR@3 held at 0.59
• No regressions in hit rate or grounding rate

PR 2: Parent Repo (bicameral)

Title: mcp: bump submodule — compound token recall fix

Body:

Summary

• Bumps pilot/mcp submodule to include compound token extraction fix
• See BicameralAI/bicameral-mcp#<PR_NUMBER> for details

Summary by CodeRabbit

• Improvements
  • Enhanced code location detection to better recognize compound identifiers (such as names with underscores and dots), improving search and navigation accuracy.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f3e3623b-e82a-40e5-9205-040fab719518

📥 Commits

Reviewing files that changed from the base of the PR and between 8c73e68 and ba35e4b.

📒 Files selected for processing (1)

• adapters/code_locator.py

---

📝 Walkthrough

Walkthrough

Modified tokenization in _ground_single() to extract compound tokens (containing underscores or dots) using a new regex pattern, concatenating them before word tokens for fuzzy validation and ranking.

Changes

Cohort / File(s)	Summary
Tokenization Enhancement adapters/code_locator.py	Added _COMPOUND_RE regex and enhanced _ground_single() tokenization to extract compound tokens matching [A-Za-z]\w*(?:[_.][A-Za-z]\w*)+ (≥4 chars) before word tokens for ranking.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

• fix: FC-2 follow-ups — symbol seeding, team-mode test unwrap, step-1 parser  #24: Also modifies _ground_single() tokenization in adapters/code_locator.py; adds symbol-name seeding/lookup logic alongside this PR's compound-token extraction.
• fix: Stage 1 symbol ranking by fuzzy score — MRR@3 0.39→0.58  #25: Also modifies RealCodeLocatorAdapter._ground_single() in adapters/code_locator.py; alters matched_scores tracking to complement this PR's tokenization changes.

Poem

🐰 Hop, skip, through underscores we go,
Compound tokens dancing in a row,
With dots and dashes, fuzzy and fine,
Four chars or more—such code divine!
A rabbit's extraction, swift and neat,
Makes symbol grounding oh-so sweet! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: improving compound token extraction to increase grounding recall from 9% to 14%.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch silong/p0-compound-token-recall

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}