docs(benchmarks): judge-ceiling finding + dual-judge methodology

[jaylfc]

Summary

The most important methodology finding to date. After hitting a stubborn ~0.28 Single-hop ceiling across four independent architectural levers, the multi-judge experiment shows the ceiling was the judge's strictness, not the system's quality.

Multi-judge sweep results (same predictions, 5 judges)

`qwen3.5:9b` leader recipe, 200-QA subset, identical predictions:

Judge	Overall	Single-hop
gemma4:e2b	0.71	0.53
qwen3.5:9b (self)	0.64	0.35
gemma4:e4b	0.56	0.26
qwen3:4b (our published baseline)	0.54	0.28
llama3.1:8b ⚠	0.35	0.21

⚠ `llama3.1:8b` rescore returned in 26 s (vs ~30 min for others) — parsing failure, treated as outlier.

Same pattern reproduces under `qwen3.6:35b-a3b` (Phase 1 MoE) — gemma4:e2b judge gives 0.51 Single-hop on the same predictions that scored 0.21 under qwen3:4b.

What this means

• Same predictions, different judges, 2× variation in Single-hop (0.26 → 0.53 across working judges).
• The "unmovable Single-hop" framing across 4 prior negative results was largely measurement, not architecture.
• Our pipeline at gemma4:e2b judge is 0.53 Single-hop / 0.71 overall — close to the published-SOTA band (EMem 0.83 with gpt-4o-mini).
• Real architectural gap to EMem is ~0.30, not the ~0.55 our qwen3:4b numbers suggested.

Methodology change going forward

1. `qwen3:4b` stays as headline judge (strictest, locally-runnable, honest under-reporting).
2. `gemma4:e2b` becomes secondary judge, reported alongside qwen3:4b for matched-with-published-SOTA comparisons.
3. New cells get dual-rescored under both judges. Every published number carries both attributions.
4. Hardware-tier guidance: 4-6 GB VRAM uses `qwen3:4b` judge (low-VRAM, never refuses); 8 GB+ can run `gemma4:e2b` (7.2 GB) for matched-with-paper benchmarks.
5. Existing leaderboard rows are NOT retroactively rescored — that would look like number-massaging. The 0.557 leader stands; future rows include both judge scores explicitly.

Test plan

• Read end-to-end alongside the prompt + embedder + ENGRAM negative-result sections — confirms the four-null narrative is properly contextualised by judge-ceiling
• Verify all numbers match `/tmp/multijudge_overnight_summary.tsv` and `/tmp/multijudge_phase2_summary.tsv` on the bench host
• Confirm the methodology-change paragraph reads as honest disclosure rather than retroactive number-massaging

Summary by CodeRabbit

• Documentation
  • Refreshed LoCoMo benchmark results with dual-judge evaluation methodology for improved scoring consistency and reliability
  • Updated GPU tier recommendations with specific generator configurations, temperature settings, and comprehensive performance metrics across multiple evaluation judges
  • Added judge sensitivity analysis demonstrating how evaluator choice significantly affects benchmark scores
  • Expanded benchmark documentation with detailed architectural comparison tables, temperature optimization insights, and reproducibility information

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR expands LoCoMo benchmarking documentation with a dual-judge methodology for improved metric reliability. A new "Judge sensitivity" section in docs/benchmarks.md explains that Single-hop scores vary ~2× across different LLM judges and establishes qwen3:4b as the strict headline judge and gemma4:e2b as a secondary lenient judge. Dual-judge results are presented across architectural levers, generator candidates, and temperature sweeps, with updated 12 GB GPU tier recommendations. README.md is concurrently updated to reflect these findings.

Changes

LoCoMo Dual-Judge Methodology

Layer / File(s)	Summary
Judge Sensitivity Analysis & Methodology docs/benchmarks.md	Introduces "Judge sensitivity" section showing Single-hop score spread across judges (~2× range) and establishing dual-judge forward methodology: qwen3:4b (strict, headline) and gemma4:e2b (lenient, secondary) with mandatory dual attribution for new cells.
Benchmark Results Across Scenarios docs/benchmarks.md	Presents dual-judge comparative results for May 7 architectural levers, May 5 generator candidates, and generator-temperature sweep. Reports Overall, Single-hop/Factual, and Multi-hop/Temporal scores under both judges; updates 12 GB GPU production guidance per metric priorities.
Updated Generator Recommendations README.md	Replaces prior LoCoMo description with strict-judge (qwen3:4b) results on 1540-QA set and lenient-judge (gemma4:e2b) metrics on 200-QA subset. Adds recommended generators table for 12 GB GPU tier with fusion strategy, temperature, and both-judge scores; includes temperature sweet spots and dual-judge explanation.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• jaylfc/taosmd#63: Introduces May 5 generator candidate sweep and initial 12 GB GPU recommendations; this PR extends that with dual-judge rescoring and judge-sensitivity analysis on top.
• jaylfc/taosmd#34: Introduces qwen3 external rescoring and gemma4 secondary judge methodology; this PR applies and documents that dual-judge approach comprehensively across LoCoMo benchmarks.
• jaylfc/taosmd#56: Modifies LoCoMo benchmark documentation and leaderboard results; this PR adds judge-sensitivity analysis and dual-judge rescoring to the same benchmark evaluation framework.

Poem

🐰 Two judges now reign, one strict and one kind,
Their verdicts diverge—the truth, intertwined.
A rabbity sweep of temperatures true,
Refined GPU guidance from this balanced view.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main findings and methodology changes: the discovery that the Single-hop ceiling is due to judge strictness rather than system quality, and the shift to dual-judge evaluation methodology for future results.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docs/judge-ceiling-may7

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (2 files)

• README.md
• docs/benchmarks.md

---

_{Reviewed by grok-code-fast-1:optimized:free · 161,754 tokens}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

docs/benchmarks.md (1)

406-409: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Methodology disclosure is now internally inconsistent with the new dual-judge section.

This block still states a single external judge (qwen3:4b) and says a thinking-mode control run is “queued,” but the new sections above already report dual-judge methodology and measured thinking-mode outcomes. Please update this section so readers don’t get conflicting methodology statements.

Suggested doc fix

-- **External judge**: `qwen3:4b`, temperature 0.0, via `locomo_rescore_streaming.py`.
+- **External judges**:
+  - `qwen3:4b` (strict, locally-runnable) is the headline judge for the full leaderboard.
+  - `gemma4:e2b` (lenient, matched-with-paper-SOTA comparisons) is reported alongside `qwen3:4b` for new cells from May 7, 2026 onward.
+  - Dual-judge rescoring uses `locomo_rescore_streaming.py`.
...
-- **`think=false` on generator** for Qwen3/3.5/3.6 (PR `#42`). A `thinking_mode_on` control run is queued to measure whether chain-of-thought changes the result.
+- **`think=false` on generator** for Qwen3/3.5/3.6 (PR `#42`). A `thinking_mode_on` control run was measured and regressed on LoCoMo memory-recall settings.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/benchmarks.md` around lines 406 - 409, Update the inconsistent
methodology text in docs/benchmarks.md: replace the single-external-judge
statement and the “queued” thinking-mode phrasing with wording that reflects the
dual-judge setup and the measured thinking-mode results; specifically edit the
lines mentioning `qwen3:4b`, the “No cherry-picking” sentence, and the
`think=false` / `thinking_mode_on` references so they state the dual-judge
methodology and the observed thinking-mode outcomes (rather than saying a
control run is queued).

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@README.md`:
- Around line 197-200: The blockquote containing the paragraphs starting with
"Per-generator temp sweet spots" and "About the dual judge." has an empty quoted
line between paragraphs which violates MD028; remove that blank quoted line (the
solitary "> " line) so the two quoted paragraphs are directly adjacent with no
empty blockquote line between them.

---

Outside diff comments:
In `@docs/benchmarks.md`:
- Around line 406-409: Update the inconsistent methodology text in
docs/benchmarks.md: replace the single-external-judge statement and the “queued”
thinking-mode phrasing with wording that reflects the dual-judge setup and the
measured thinking-mode results; specifically edit the lines mentioning
`qwen3:4b`, the “No cherry-picking” sentence, and the `think=false` /
`thinking_mode_on` references so they state the dual-judge methodology and the
observed thinking-mode outcomes (rather than saying a control run is queued).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 26b61da3-f7e1-40c4-8fab-97025661a7f7

📥 Commits

Reviewing files that changed from the base of the PR and between 54e8983 and b93a65a.

📒 Files selected for processing (2)

• README.md
• docs/benchmarks.md

[coderabbitai]

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Remove the blank line inside the blockquote (MD028).

There’s a blank quoted line between blockquote paragraphs here, which trips markdownlint rule MD028.

🧰 Tools

🪛 markdownlint-cli2 (0.22.1)

[warning] 198-198: Blank line inside blockquote

(MD028, no-blanks-blockquote)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` around lines 197 - 200, The blockquote containing the paragraphs
starting with "Per-generator temp sweet spots" and "About the dual judge." has
an empty quoted line between paragraphs which violates MD028; remove that blank
quoted line (the solitary "> " line) so the two quoted paragraphs are directly
adjacent with no empty blockquote line between them.