chore(eval): FlashRank rerank quality-lift probe (#46) - #64
Conversation
Before/after eval harness comparing baseline (vector-distance) vs FlashRank-reranked ordering on an identical candidate pool, plus a findings report with the metrics and a recommendation. Result on 11 usable queries (live, pool=20): MRR +15.3% (0.761 -> 0.877), R@10 unchanged, one R@5 regression (3 -> 7) from cross-encoder score compression. Rerank latency ~47 ms mean. Verdict: KEEP nano now, follow up with an A/B against ms-marco-MiniLM-L-12-v2 on the frozen candidate pools (L-6 from the issue is not shipped in this FlashRank build). - scripts/evals/rerank_eval.py: harness, live + in-process candidates modes - scripts/evals/collect_candidates.py: freeze candidate pools for offline A/B - scripts/evals/rerank_eval_queries.json: 12 hand-verified labeled queries - tests/test_rerank_eval.py: 9 cases (full suite 252 passed / 1 skipped) - docs/evals/: report + raw metrics + frozen candidate pools Read-only against the production palace; no writes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a comprehensive evaluation framework to measure the effectiveness of the FlashRank reranking pass in the palace-daemon. By comparing baseline retrieval orderings against reranked outputs on frozen candidate pools, the changes provide a clear, reproducible signal on the reranker's impact. The evaluation confirms a positive lift in retrieval quality and provides a roadmap for future model optimizations. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive evaluation suite for the FlashRank reranking feature, including a markdown report, a frozen candidate JSON, candidate collection and evaluation scripts, a labeled query set, and corresponding unit tests. The reviewer's feedback identifies a critical issue in both collect_candidates.py and rerank_eval.py where r.raise_for_status() can raise an uncaught HTTPError (such as transient 502/503/504 errors), bypassing the retry mechanism and crashing the scripts. The reviewer suggests catching requests.RequestException to handle all request-related exceptions gracefully.
| out[q["id"]] = hits | ||
| print(f"ok {q['id']:<28} n={len(hits)}") | ||
| break | ||
| except (requests.Timeout, requests.ConnectionError) as e: |
There was a problem hiding this comment.
The except block currently only catches requests.Timeout and requests.ConnectionError. However, r.raise_for_status() on line 81 can raise requests.exceptions.HTTPError (e.g., on transient 502/503/504 errors). Because HTTPError is not caught here, any HTTP error will immediately crash the script instead of being retried or logged in the errors dictionary.
Catching requests.RequestException ensures that all request-related errors, including HTTP errors, are gracefully caught, retried, and logged.
| except (requests.Timeout, requests.ConnectionError) as e: | |
| except requests.RequestException as e: |
| ) | ||
| r.raise_for_status() | ||
| return r.json() | ||
| except (requests.Timeout, requests.ConnectionError) as e: |
There was a problem hiding this comment.
Similar to the candidate collection script, r.raise_for_status() on line 164 can raise requests.exceptions.HTTPError on transient server errors (e.g., 502/503/504). Since only Timeout and ConnectionError are caught, any HTTP error will bypass the retry mechanism and immediately propagate, failing the query evaluation without retrying.
Catching requests.RequestException allows the harness to retry on transient HTTP errors as well.
| except (requests.Timeout, requests.ConnectionError) as e: | |
| except requests.RequestException as e: |
) (#69) Addresses Gemini review on #64 + live eval. Both eval scripts only caught Timeout/ConnectionError around raise_for_status(), so a transient 502/503/504 from the contended daemon crashed instead of retrying. Now retry on 5xx (transient) and let 4xx (bad auth / bad request) propagate immediately rather than burning retries masking a permanent error. Live re-run against production (env URL fixed to http://familiar:8085) confirms the round-1 finding and is stronger: MRR 0.748 -> 0.921 (+23.1%), R@10 unchanged, R@5 a wash, one persistent regression (daemon-deploy-arch, rank 3->8) from cross-encoder score compression. Verdict stands: KEEP nano, follow up with an A/B vs ms-marco-MiniLM-L-12-v2 on the frozen pools. - scripts/evals/rerank_eval.py, collect_candidates.py: _is_retryable_http helper + 5xx retry / 4xx propagate - tests/test_rerank_eval.py: 5 new cases (5xx retryable, 4xx not, fetch_live retries 503 then succeeds, 4xx fails fast) — full suite 307 passed / 1 skipped - docs/evals/rerank-eval-2026-05-27.md: two-run results table + reconciliation - docs/evals/rerank-eval-live-2026-05-27.json: raw live run #2 output Read-only against the palace. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 participant
Summary
Before/after eval probe quantifying the retrieval-quality lift of the live FlashRank rerank pass (
rerank.py,ms-marco-TinyBERT-L-2-v2), per issue #46. Adds a re-runnable harness, a hand-verified labeled query set, frozen candidate pools, unit tests, and a findings report with a recommendation.Method: for each labeled query, pull one candidate pool from the production palace (read-only
GET /search) and score two orderings of the identical pool — baseline (vector distance asc, = rerank disabled) vs reranked (FlashRank score desc, = today). Holding retrieval constant isolates the reranker's contribution. Relevance is matched by hand-verified structural predicates (source file + content substring), not frozen drawer IDs, so the set survives palace re-mining.Results (live, 11/12 usable, pool=20)
Rerank latency: mean 47 ms (min 20, max 157), CPU-only host. 3 queries improved (incl. a rank 5→1 rescue), 7 unchanged (retrieval already optimal — reranker correctly left them alone), 1 regression (rank 3→7). The regression is a near-tie shuffle: 7 genuinely-relevant passages scored within 0.002 of each other (TinyBERT score compression at ~0.999).
Cross-validated: replaying the frozen pools in-process via
rerank.py(--mode candidates) reproduces the metrics exactly.Recommendation: KEEP nano now; follow up with an A/B vs MiniLM L-12
ms-marco-MiniLM-L-12-v2against nano on the same frozen pools (zero retrieval cost, no daemon restart) and decide from that head-to-head. Note: the issue mentioned L-6, but this FlashRank build only ships L-12.Full methodology, limitations, and per-query analysis:
docs/evals/rerank-eval-2026-05-27.md.What's included
scripts/evals/rerank_eval.py— harness (live + in-process candidates modes)scripts/evals/collect_candidates.py— freeze candidate pools for offline A/Bscripts/evals/rerank_eval_queries.json— 12 labeled queriestests/test_rerank_eval.py— 9 casesdocs/evals/— report, raw metrics JSON, frozen candidate poolsTest plan
venv/bin/python -m pytest tests/ -q→ 252 passed, 1 skipped--mode candidatesreplay reproduces live metrics exactly🤖 Generated with Claude Code