Skip to content

Consolidate FTS5/BM25 search into flare-search-kit crate (prerequisite for #51) - #189

Merged
getappz merged 3 commits into
masterfrom
task/63
Jul 15, 2026
Merged

Consolidate FTS5/BM25 search into flare-search-kit crate (prerequisite for #51)#189
getappz merged 3 commits into
masterfrom
task/63

Conversation

@getappz

@getappz getappz commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Auto-opened on item done for 019f62aa-6020-72f3-b346-5ec86af244e7.

Summary by CodeRabbit

  • New Features
    • Added shared full-text search utilities (token sanitization, phrase queries, AND/OR match modes, and BM25 weighting) for consistent query construction.
    • Reused these utilities across the gateway and skill registries to standardize search behavior.
    • Added optional serialization support for search-related configuration.
  • Bug Fixes
    • Improved protection against extremely large result limits by clamping values to prevent failures.
    • Updated/extended tests to cover large-limit behavior and query building outcomes.

Create shared crate with fts_query, MatchMode, Bm25Weights, clamped_limit.
Refactor gateway-registry and skill-registry to import from it instead of
duplicating the sanitization logic.
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 333de2f8-ea0a-4860-9199-f9f9e4135087

📥 Commits

Reviewing files that changed from the base of the PR and between 858bad2 and 9989add.

📒 Files selected for processing (1)
  • crates/skill-registry/src/search.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/skill-registry/src/search.rs

📝 Walkthrough

Walkthrough

Changes

Adds the flare-search-kit workspace crate with shared FTS5 query construction, phrase handling, BM25 weight rendering, match modes, and limit clamping. Gateway and skill registry search modules now consume these shared primitives instead of maintaining local implementations.

Shared search primitives

Layer / File(s) Summary
Search kit foundation
Cargo.toml, crates/flare-search-kit/*
Registers and defines the new crate, its public search helpers, optional Serde support, lint settings, and unit tests.
Gateway search integration
crates/gateway-registry/Cargo.toml, crates/gateway-registry/src/search.rs
Routes gateway search through shared query and limit helpers and adds coverage for very large limits.
Skill search integration
crates/skill-registry/Cargo.toml, crates/skill-registry/src/search.rs
Routes skill search through shared query and limit helpers and adds coverage for very large limits.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description is missing the required Summary, Test plan, and Notes for reviewers sections and only contains an auto-opened notice. Add the template sections with a summary of changes and why, the test plan checklist, and reviewer notes covering risks and compatibility.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: consolidating FTS5/BM25 search into a shared crate, though the prerequisite note is extra.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch task/63

Comment @coderabbitai help to get the list of available commands.

… impl

Satisfies clippy::derivable_impls under -D warnings, which CI's clippy job enforces.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
crates/flare-search-kit/src/lib.rs (1)

76-87: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Consider bounding token count/length for defense-in-depth.

fts_query sanitizes arbitrary-length input with no cap on token count or overall size. Since callers (gateway/skill registries) pass user-supplied query strings straight through, a pathological huge input could still build an unbounded MATCH expression even though clamped_limit protects the result-set size on the other end.

♻️ Example hardening
 pub fn fts_query(query: &str, mode: MatchMode) -> Option<String> {
     let tokens: Vec<String> = query
         .split_whitespace()
         .map(|t| t.replace('"', ""))
         .filter(|t| !t.is_empty())
         .map(|t| format!("\"{t}\""))
+        .take(MAX_TOKENS)
         .collect();
     if tokens.is_empty() {
         return None;
     }
     Some(tokens.join(mode.joiner()))
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/flare-search-kit/src/lib.rs` around lines 76 - 87, Update fts_query to
enforce bounds on both the number of tokens and the total generated query length
before constructing the final MATCH expression. Use defined limits, stop or
reject input that exceeds them, and preserve the existing None result for empty
input and join behavior for valid queries.
crates/skill-registry/src/search.rs (1)

60-167: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

No huge-limit test for skill-registry, unlike gateway-registry.

Once clamped_limit is wired in (see the comment on lines 4-5/37), add a test mirroring gateway-registry's search_with_a_huge_limit_does_not_panic_and_still_returns_results to lock in the fix.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/skill-registry/src/search.rs` around lines 60 - 167, The search tests
lack coverage for oversized limits. After `clamped_limit` is wired into
`search`, add a test near the existing search tests that calls `search` with a
very large limit and verifies it does not panic and still returns expected
results, mirroring gateway-registry’s
`search_with_a_huge_limit_does_not_panic_and_still_returns_results` behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/skill-registry/src/search.rs`:
- Around line 4-5: Update the search query setup in search.rs to pass
clamped_limit(limit) rather than casting the raw limit to i64 before binding it.
Reuse the existing clamped_limit helper and preserve the current query behavior
while ensuring oversized inputs remain capped.

---

Nitpick comments:
In `@crates/flare-search-kit/src/lib.rs`:
- Around line 76-87: Update fts_query to enforce bounds on both the number of
tokens and the total generated query length before constructing the final MATCH
expression. Use defined limits, stop or reject input that exceeds them, and
preserve the existing None result for empty input and join behavior for valid
queries.

In `@crates/skill-registry/src/search.rs`:
- Around line 60-167: The search tests lack coverage for oversized limits. After
`clamped_limit` is wired into `search`, add a test near the existing search
tests that calls `search` with a very large limit and verifies it does not panic
and still returns expected results, mirroring gateway-registry’s
`search_with_a_huge_limit_does_not_panic_and_still_returns_results` behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0b5a957a-5bc6-444f-b6f7-0731fbddb866

📥 Commits

Reviewing files that changed from the base of the PR and between 1ca98ef and e9c8c4e.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (7)
  • Cargo.toml
  • crates/flare-search-kit/Cargo.toml
  • crates/flare-search-kit/src/lib.rs
  • crates/gateway-registry/Cargo.toml
  • crates/gateway-registry/src/search.rs
  • crates/skill-registry/Cargo.toml
  • crates/skill-registry/src/search.rs

Comment thread crates/skill-registry/src/search.rs Outdated
…egression test

CodeRabbit caught it on #189: search() still cast the raw usize limit to
i64 directly instead of using flare-search-kit's clamped_limit, leaving
the negative-LIMIT-on-usize::MAX bug (already fixed in gateway-registry)
unguarded here.
@getappz
getappz merged commit 5584110 into master Jul 15, 2026
15 checks passed
@getappz
getappz deleted the task/63 branch July 15, 2026 05:20
getappz added a commit that referenced this pull request Jul 15, 2026
- Don't hold any lock (skills_registry mutex or gateway Registry's conn
  mutex) across the fallback HTTP call to registry.modelcontextprotocol.io:
  local search now runs and releases its lock before the network fallback
  fires, in both the skill and tool MCP actions.
- Add a 3s timeout to the registry HTTP request so a slow/unreachable
  registry can't hang a search indefinitely.
- Wire the fallback into tool_search (the "tool"/"search" MCP action) too,
  matching the item's original scope -- gateway_registry::Registry's
  fallback method existed but was dead code, never called anywhere.
- Fix score semantics: local bm25() scores are ascending (lower = better,
  no DESC), but registry hits got a flat 0.5 with a comment claiming that
  ranked them below local results -- backwards, since 0.5 > every negative
  local score. Registry hits now score f64::MAX, so they always sort last
  under the same ascending convention if the merged list is ever re-sorted.
- Resolve merge conflicts from master's flare-search-kit consolidation
  (#189) by rebuilding gateway-registry/search.rs and skill-registry's
  merge logic against the new shared MatchMode/fts_query/clamped_limit.
getappz added a commit that referenced this pull request Aug 25, 2026
…tch (#601)

execute_work chdir'd into an item's worktree via std::env::set_current_dir,
which mutates the whole process's cwd, not per-thread. The daemon dispatches
multiple in-process work-item jobs concurrently by design
(work_max_concurrency), so two jobs racing here could have one item's
pipeline run against a different item's checked-out worktree -- observed
live, twice, with two different item pairs (items #189/#70 and #189/#188).

Add run_in_worktree (src/cli/work_cwd_lock.rs), which serializes the
chdir -> run -> restore critical section behind a mutex, same mitigation
shape as flare_git_core::worktree::WORKTREE_ADD_LOCK. Only this section is
serialized -- claiming, agent resolution, and DB reads above it still run
concurrently. Split into a satellite file (and the new regression test into
another) to keep work.rs under the repo's LOC gate.

Regression test dispatches two items concurrently on separate threads and
asserts neither thread's cwd drifts into the other's worktree mid-run;
confirmed it reliably reproduces the race (3/3 runs) with the lock removed
and passes reliably (3/3) with it restored, both before and after factoring
the lock+chdir logic into run_in_worktree.

Agentflare-Agent: claude-code
Agentflare-Branch: fix/execute-work-cwd-race

Co-authored-by: shiva <shiva@gosysinfo.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant