Skip to content

feat(recall): structured per-stage scores and two-level min_scores filtering - #2422

Merged
nicoloboschi merged 1 commit into
mainfrom
feat/recall-scores-min-scores
Jun 26, 2026
Merged

feat(recall): structured per-stage scores and two-level min_scores filtering#2422
nicoloboschi merged 1 commit into
mainfrom
feat/recall-scores-min-scores

Conversation

@nicoloboschi

@nicoloboschi nicoloboschi commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

What

Replaces recall's single score field with a structured scores object (one score per pipeline stage) and replaces the min_score request param with min_scores, a per-stage filter that operates at two levels of the pipeline.

Response — scores on every result

field meaning null when
final the score results are ranked by
reranker cross-encoder normalized relevance (0–1) passthrough reranker (rrf/interleave)
semantic raw vector cosine similarity (0–1) not surfaced semantically
keyword raw keyword/BM25 score (≥0) not surfaced by keyword search

Per-arm semantic/text scores are aggregated across retrieval arms during RRF / interleave fusion (ArmScores on MergedCandidate) — fusion otherwise keeps only the first-seen arm's score per doc.

Request — min_scores (opt-in, inclusive, AND-ed; default = no filtering)

field level effect
semantic retrieval min vector similarity, pushed into the SQL arm (overrides global similarity min for the request) — prunes before fusion
keyword retrieval min keyword/BM25 score, pushed into the SQL arm
reranker post-query min normalized cross-encoder score on ranked results
final post-query min final ranking score on ranked results
{ "query": "...", "min_scores": { "reranker": 0.5 } }

Why

The recall pipeline computes several scores but only exposed the final one, and only via trace. Exposing them lets callers inspect ranking and threshold on the stage they care about.

There is deliberately no default threshold. The cross-encoder's absolute scores are reliable for ordering but not calibrated across queries — e.g. for query software, the correctly-#1-ranked "Alice prefers Python over Java" scores ~0.001, while a natural-question phrasing of the same intent scores ~1.0. A fixed default cutoff would silently drop good results, so min_scores is fully opt-in.

Also in this PR

  • Surfaces proof_norm in the search trace so the scoring breakdown reconciles.
  • Control-plane trace view: renders scores at full precision (no rounding — rounded 0.001 hid real differences) and shows the per-stage scores breakdown; relabels the trace's "CE" → "reranker score".

Surfaces touched

Engine (response_models, fusion, retrieval, reranking, tracer, types, memory_engine), HTTP (http.py), MCP (both recall tools), control-plane proxy (recall/route.ts, lib/api.ts, search-debug-view.tsx). OpenAPI + Python/TS/Go/Rust clients + docs-skill mirror regenerated; recall.mdx + mcp-server.md updated.

Tests

tests/test_recall_min_score.py (9 tests): scores present, post-query final/reranker floors, retrieval-level semantic floor (SQL pruning), default None no-op. Fusion/scoring/trace suites pass (53). Lint + ty clean.

Notes for review

  • Breaking response/request shape: scorescores.final, min_scoremin_scores.final; retrieval-arm score fields are semantic + keyword. The old fields were unreleased, so no compatibility shim.
  • Branch is currently a few commits behind main — will rebase before marking ready.

🤖 Draft generated with Claude Code.

@nicoloboschi
nicoloboschi force-pushed the feat/recall-scores-min-scores branch from 82547ef to 093ae42 Compare June 26, 2026 13:48
@nicoloboschi
nicoloboschi marked this pull request as ready for review June 26, 2026 13:52
@nicoloboschi
nicoloboschi force-pushed the feat/recall-scores-min-scores branch from 093ae42 to 4ee9de6 Compare June 26, 2026 14:29
…ltering

Replace the recall result's single `score` with a `scores` object exposing the
scores from each pipeline stage, and replace the `min_score` request param with
`min_scores`, a per-stage filter that operates at two levels.

Response — each result carries `scores`:
- final     : the value results are ranked by
- reranker  : cross-encoder normalized relevance (null for passthrough rerankers)
- semantic  : raw vector cosine similarity (null if not surfaced semantically)
- text      : raw keyword/BM25 score (null if not surfaced by keyword search)

Per-arm semantic/text scores are aggregated across retrieval arms during RRF /
interleave fusion (ArmScores on MergedCandidate), since fusion otherwise keeps
only the first-seen arm's score per doc.

Request — `min_scores` floors (inclusive, AND-ed, opt-in; default no filtering):
- semantic / text : retrieval-level cutoffs pushed into the SQL arms, overriding
  the global similarity / BM25 minimums for the request (prune before fusion)
- reranker / final: post-query filters on the scored results

There is deliberately no default threshold: the cross-encoder's absolute scores
are reliable for ordering but not calibrated across queries (a clearly-relevant
match can score ~0.001 on one query and ~1.0 on another), so a fixed cutoff would
silently drop good results.

Also surfaces proof_norm in the search trace and reworks the control-plane trace
view to render scores at full precision (no rounding) and show the per-stage
`scores` breakdown; relabels the trace's "CE" column to "reranker score".

Threaded through engine, HTTP, MCP (both recall tools), and the control-plane
proxy; OpenAPI spec, Python/TS/Go/Rust clients, and the docs-skill mirror
regenerated; docs updated.
@nicoloboschi
nicoloboschi force-pushed the feat/recall-scores-min-scores branch from 4ee9de6 to 6655561 Compare June 26, 2026 14:47
@nicoloboschi
nicoloboschi merged commit 758f346 into main Jun 26, 2026
200 of 202 checks passed
@nicoloboschi
nicoloboschi deleted the feat/recall-scores-min-scores branch June 26, 2026 15:12
nicoloboschi pushed a commit that referenced this pull request Jun 30, 2026
…SDK wrapper (#2446)

#2422 added the public RecallRequest.min_scores (per-stage score floors) to
the HTTP/MCP API and the generated clients, but the hand-maintained
high-level Python wrapper (hindsight_client.recall/arecall) never got it, so
high-level SDK users can't use the feature without dropping to the raw
generated client.

Thread an optional min_scores dict through recall()/arecall() into
RecallRequest, mirroring the existing tag_groups dict->from_dict pattern.
Unknown keys raise ValueError so a misspelled floor fails loud instead of
silently applying no filter. Parity test mirrors
tests/test_recall_prefer_observations.py.

Follow-up to #2422.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant