docs: retract subset-200 Single-hop pick after full-1540 validation - #67
Merged
Conversation
…ailure
The May 7-9 12 GB hardware-tier table promoted llama3.1:8b + RRF + temp 0.2 as
the best Single-hop recipe based on a 200-QA subset result of 0.65 SH under
gemma4:e2b judge. Validating that pick at the full 1540 QAs measured Single-hop
at 0.49 — a -0.16 regression that puts it below the qwen3.5:9b + mem0_additive
leader's 0.55 SH on the same dataset.
Subset 200 over-represented questions where llama+RRF won at our generator/recipe
pairing. The leader recipe (qwen+mem0+temp 0.2) generalises within noise (-0.01
SH from subset 200 to full 1540), so the production default holds and gets a
small Overall number refresh (0.71 -> 0.68).
README changes:
- Headline number now cites full-1540 g4e2b (0.68 / 0.55 SH) instead of the
subset-200 figure
- Recipe in the headline updated to mem0_additive + temp 0.2 (the actual
leader; previous text named the older RRF-default recipe)
- 'Best factual recall: llama+RRF' row removed
- Surviving subset-200 rows (llama+mem0+temp 0, mistral+rrf+temp 0.2) carry
explicit subset-200 footnotes pending full-1540 validation
- New callout above the table explaining the subset->full discipline going
forward
docs/benchmarks.md adds a 'Subset 200 -> full 1540 validation' subsection
under the temp-sweep section with the side-by-side comparison and methodology
takeaways. Cell 4 qwen3:4b rescore is still running at time of this commit
(~6h remaining); only the gemma4:e2b half is reflected. q34b half lands as a
follow-up commit.
📝 WalkthroughWalkthroughDocumentation updates to the LoCoMo benchmark section add full 1540-QA validation results for temperature-sweep recipes, reporting subset-to-full score deltas and revising the README generator recommendations table based on qwen3.5:9b generalization and llama3.1:8b regression. ChangesLoCoMo Benchmark Rejudge Results
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (2 files)
Reviewed by grok-code-fast-1:optimized:free · 82,435 tokens |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Around line 184-185: The README claims “same predictions, same recipe” but
attributes the strict-score 0.557 to the qwen3.5:9b + mem0_additive recipe while
the actual strict value for that recipe under the strict judge (qwen3:4b) is
0.54; replace the incorrect 0.557 string with 0.54 (and update the repeated
mentions of 0.557 at the other occurrences that refer to the same-recipe claim)
or alternatively clarify that 0.557 is from the RRF leaderboard recipe (not the
same-recipe run) so the sentence correctly contrasts 0.54 (strict, same
predictions) vs 0.68 (lenient judge), ensuring the recipe string
(--retrieval-top-k 20 --adjacent-turns 2 --llm-query-expansion --fusion
mem0_additive --gen-temp 0.2) and judge names (qwen3:4b, gemma4:e2b) remain
aligned with the corrected numbers.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 41eaaf99-649e-4782-a2e2-ce91cb7ded81
📒 Files selected for processing (2)
README.mddocs/benchmarks.md
Comment on lines
+184
to
185
| **0.557 ext rejudge** on the full 1540-QA test set under our strict default judge (`qwen3:4b`) — and **0.68 overall / 0.55 Single-hop** on the **full 1540 QAs** under the lenient matched-with-paper-SOTA judge (`gemma4:e2b`). Same predictions, same recipe (qwen3.5:9b + `--retrieval-top-k 20 --adjacent-turns 2 --llm-query-expansion --fusion mem0_additive --gen-temp 0.2`), only the judge differs. Both numbers are honest measurements of the same system; published numbers from Mem0/EMem/Zep use a lenient frontier judge (gpt-4o-mini), so 0.68 is the more apples-to-apples comparison number with their headlines. See [docs/benchmarks.md](docs/benchmarks.md#judge-sensitivity--what-we-are-really-measuring) for the full multi-judge analysis. | ||
|
|
There was a problem hiding this comment.
Fix strict-score attribution mismatch for the “same recipe” claim.
Line 184 says the strict score is 0.557 for qwen3.5:9b + mem0_additive + temp 0.2, but Line 192 (and the new full-1540 validation section in docs) reports 0.54 for that recipe under qwen3:4b.
If the recipe is truly “same predictions, only judge differs,” the strict value should be 0.54; 0.557 appears to be from the RRF leaderboard recipe.
Suggested README fix
-**0.557 ext rejudge** on the full 1540-QA test set under our strict default judge (`qwen3:4b`) — and **0.68 overall / 0.55 Single-hop** on the **full 1540 QAs** under the lenient matched-with-paper-SOTA judge (`gemma4:e2b`). Same predictions, same recipe (qwen3.5:9b + `--retrieval-top-k 20 --adjacent-turns 2 --llm-query-expansion --fusion mem0_additive --gen-temp 0.2`), only the judge differs.
+**0.54 ext rejudge** on the full 1540-QA test set under our strict default judge (`qwen3:4b`) — and **0.68 overall / 0.55 Single-hop** on the **full 1540 QAs** under the lenient matched-with-paper-SOTA judge (`gemma4:e2b`). Same predictions, same recipe (qwen3.5:9b + `--retrieval-top-k 20 --adjacent-turns 2 --llm-query-expansion --fusion mem0_additive --gen-temp 0.2`), only the judge differs.Also applies to: 192-193
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 184 - 185, The README claims “same predictions, same
recipe” but attributes the strict-score 0.557 to the qwen3.5:9b + mem0_additive
recipe while the actual strict value for that recipe under the strict judge
(qwen3:4b) is 0.54; replace the incorrect 0.557 string with 0.54 (and update the
repeated mentions of 0.557 at the other occurrences that refer to the
same-recipe claim) or alternatively clarify that 0.557 is from the RRF
leaderboard recipe (not the same-recipe run) so the sentence correctly contrasts
0.54 (strict, same predictions) vs 0.68 (lenient judge), ensuring the recipe
string (--retrieval-top-k 20 --adjacent-turns 2 --llm-query-expansion --fusion
mem0_additive --gen-temp 0.2) and judge names (qwen3:4b, gemma4:e2b) remain
aligned with the corrected numbers.
2 tasks
jaylfc
added a commit
that referenced
this pull request
May 12, 2026
…68) * docs: cell-4 qwen3:4b rescore — both full-1540 cells now dual-judge complete Follow-up to PR #67. Cell 4 (llama3.1:8b + RRF + temp 0.2 full 1540) qwen3:4b rescore landed at 0.53 Overall / 0.27 SH, vs cell 3's 0.54 / 0.28. Both judges now agree the qwen+mem0 leader wins Overall and Single-hop at full scale — subset 200 was over-representing the slice where llama+RRF won. Removes the 'qwen3:4b rescore still running' note and adds a small both-judges-both-cells summary table so readers don't have to cross-reference two paragraphs. * ci: retrigger Kilo (sandbox provisioning error on first run)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Status of the underlying bench
The gemma4:e2b half alone is enough to flip the Single-hop ranking, so shipping the retraction now rather than waiting another ~6h.
Test plan
Summary by CodeRabbit