fix(eval): retry transient 5xx in eval HTTP calls + live re-run (#46 #64)

[jphein]

Summary

Round-2 follow-up to #64 (rerank quality-lift eval). Addresses the Gemini review and adds a confirming live production run.

Gemini fixes (both real boundary bugs):

• scripts/evals/rerank_eval.py and scripts/evals/collect_candidates.py only caught Timeout/ConnectionError around raise_for_status(), so a transient 502/503/504 from the contended daemon crashed instead of retrying.
• Fix: retry on 5xx (transient — daemon restart settling), but let 4xx (bad auth / bad request) propagate immediately rather than burning retries on a permanent error. New _is_retryable_http() helper in both scripts.

Live re-run (env URL fixed to http://familiar:8085): re-confirms #64's finding and is stronger.

Live results — two runs agree

run	metric	baseline	reranked	delta
#2 (this PR, 12:15)	R@5	0.909	0.909	0.000
	R@10	1.000	1.000	0.000
	MRR	0.748	0.921	+0.173 (+23.1%)
#1 (in #64, 11:39)	R@5	1.000	0.909	−0.091
	R@10	1.000	1.000	0.000
	MRR	0.761	0.877	+0.116 (+15.3%)

MRR lift is robust (+15–23%) — rerank reliably surfaces the best answer higher (rank 7→1 rescue in run #2). R@10 untouched; R@5 a wash (rerank's win and its one regression cancel at the top-5 cutoff, which is why MRR is the load-bearing metric). One persistent regression — daemon-deploy-arch rank 3→8 — from cross-encoder score compression (~0.999 ties on a saturated pool).

Final call: KEEP nano; follow up with an A/B vs MiniLM L-12

Lift is real and repeatable, latency acceptable (mean 47–126 ms depending on host load) → rules out REVERT. The persistent regression + flat scores are the ESCALATE candidate signal. Keep nano live; A/B ms-marco-MiniLM-L-12-v2 on the frozen pools (zero retrieval cost). Note: this FlashRank build ships only L-12, no L-6.

Test plan

• venv/bin/python -m pytest tests/ -q → 307 passed, 1 skipped
• 5 new retry tests: 5xx retryable, 4xx not, fetch_live retries 503→succeeds, 4xx fails fast without consuming retries
• Live eval completes clean (0 errors) against production at the fixed URL
• Read-only against the palace (no writes)
• (Follow-up) A/B MiniLM L-12 vs nano on the frozen pools

🤖 Generated with Claude Code

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request hardens the evaluation harness by implementing intelligent retry logic for transient HTTP 5xx errors, ensuring that temporary daemon instability does not crash evaluation runs. It also includes a second, confirmed live production run that validates the rerank quality-lift metrics previously reported, providing more robust evidence for the model's performance.

Highlights

• HTTP Retry Logic: Added an _is_retryable_http helper to distinguish between transient 5xx server errors and permanent 4xx client errors.
• Evaluation Hardening: Updated scripts/evals/rerank_eval.py and scripts/evals/collect_candidates.py to implement intelligent retry logic for transient HTTP failures.
• Documentation Update: Updated docs/evals/rerank-eval-2026-05-27.md with results from a second, confirmed live production run.
• Test Coverage: Added comprehensive unit tests in tests/test_rerank_eval.py to verify retry behavior for 5xx and fast-fail behavior for 4xx errors.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the rerank evaluation documentation with results from a second live run, adds a new raw metrics JSON file, and implements HTTP retry hardening in the evaluation scripts to retry transient 5xx server errors while failing fast on permanent 4xx errors. It also adds comprehensive unit tests for this new retry logic. No review comments were provided, so there is no feedback to address.