benchmarks: v3.3.0 reproduction results + Ollama rerank backend

[igorls]

Summary

Reproduces every headline claim in benchmarks/BENCHMARKS.md against the tagged v3.3.0 release on a clean Linux x86_64 environment, and commits the per-question result JSONLs that the doc already references but were not previously in the repo. Also adds an --llm-backend ollama flag so the rerank pipeline can be exercised with any OpenAI-compatible endpoint — important for #875, because it lets the "100% with Haiku rerank" claim be verified with a different LLM family and zero Anthropic dependency.

Paired with #875's public-surface rewrite (separate PR).

Scorecard — every BENCHMARKS.md claim reproduces

Run on v3.3.0, seeded ChromaDB embeddings, seed=42 for the 50/450 LongMemEval split.

Benchmark	Mode	Claimed	Observed	Notes
LongMemEval R@5	raw (no LLM)	96.6%	96.6% (500/500)	✅ exact
LongMemEval R@5	hybrid_v4 held-out 450q	98.4%	98.4% (442/450)	✅ exact
LongMemEval R@5	hybrid_v4 + rerank	100% Haiku	99.2% minimax, 100% R@10	* model-swap drops 4q at R@5; pipeline-level identical at R@10
LoCoMo R@10	session, top-10, no rerank	60.3%	60.3% (1986q)	✅ exact
LoCoMo R@10	hybrid v5 top-10, no rerank	88.9%	88.9% (1986q)	✅ exact
ConvoMem	avg recall, all categories	92.9%	92.9% (250 items)	✅ exact
MemBench R@5	all categories, hybrid top-5	80.3%	80.3% (6828/8500)	✅ exact

All 10 per-category MemBench numbers and all 5 per-category ConvoMem numbers match to the decimal; the same for LoCoMo per-category and LongMemEval per-category breakdowns.

What the minimax rerank result tells us

The hybrid_v4 + rerank pipeline was run on the full 500 LongMemEval questions with minimax-m2.7:cloud via Ollama Cloud's OpenAI-compatible endpoint — no Anthropic anywhere in the pipeline. Result: R@5 99.2% (496/500), R@10 100.0% (500/500), per-category R@10 all 6 at 100%.

Three things fall out of this:

1. The pipeline's retrieval layer already has the target session in top-10 for all 500 questions. The rerank step is a reading-comprehension task, not a retrieval task.
2. "100% with Haiku rerank" is a pipeline-level result, not a Haiku-specific one. Any capable LLM reader gets there.
3. The 99.4% → 100% step that landed the perfect Haiku score is the three hand-tuned fixes in hybrid_v4 (quoted-phrase boost, person-name boost, nostalgia patterns) — which benchmarks/BENCHMARKS.md has itself called "teaching to the test" since February. The generalisable number when an LLM rerank is in the loop is the clean held-out 98.4% R@5 on 450 questions the pipeline was never tuned on.

Commits

• 8df7b9b — benchmarks: add --llm-backend ollama for non-Anthropic rerank (both scripts, +169/−66). Adds --llm-backend {anthropic,ollama} and --llm-base-url to longmemeval_bench.py and locomo_bench.py. Response parsing handles reasoning models (minimax emits its final digit in message.content when max_tokens is large enough, or in message.reasoning when it isn't — we parse the last integer in either field). API key required only for the Anthropic backend; diary/palace modes still need it.
• ca0682a — benchmarks: apply ruff-format to llm_rerank (trivial line wrap, 3 lines).
• 61d02e1 — benchmarks: add v3.3.0 reproduction results + 50/450 split. Adds 8 result JSONLs (one per headline figure, including the smoke test and the full 500 minimax rerank) and the committed seed-42 train/held-out split.

Reproducing

# no API key needed
python benchmarks/longmemeval_bench.py /tmp/longmemeval_s_cleaned.json
python benchmarks/longmemeval_bench.py /tmp/longmemeval_s_cleaned.json \
  --mode hybrid_v4 --held-out --split-file benchmarks/lme_split_50_450.json
python benchmarks/locomo_bench.py /tmp/locomo/data/locomo10.json \
  --granularity session --top-k 10
python benchmarks/locomo_bench.py /tmp/locomo/data/locomo10.json \
  --granularity session --top-k 10 --mode hybrid
python benchmarks/convomem_bench.py --category all --limit 50
python benchmarks/membench_bench.py /tmp/membench/MemData/FirstAgent --mode hybrid --top-k 5

# any OpenAI-compatible endpoint — no Anthropic key required
python benchmarks/longmemeval_bench.py /tmp/longmemeval_s_cleaned.json \
  --mode hybrid_v4 --llm-rerank \
  --llm-backend ollama --llm-model minimax-m2.7:cloud

Test plan

• ruff check + ruff format --check (benchmarks/ is in ruff's extend-exclude, but pre-commit hook ran clean on both commits).
• Smoke-test the Ollama backend end-to-end on 20 questions before the 2.4-hour full run (results_mempal_hybrid_v4_llmrerank_session_20260414_1654.jsonl is that smoke run).
• Every committed JSONL was produced by the exact command in its result_file_path header; running R@5 per-batch lines match the final aggregate in each log.
• CI green on this branch.