fix(bench): stop silent-failure patterns from biasing LoCoMo scores - #35
Closed
jaylfc wants to merge 1 commit into
Closed
fix(bench): stop silent-failure patterns from biasing LoCoMo scores#35jaylfc wants to merge 1 commit into
jaylfc wants to merge 1 commit into
Conversation
CodeRabbit flagged four benchmark-integrity issues on #30. All real — they let infra failures and adapter artefacts masquerade as real negative results, biasing the published numbers and hiding problems. 1. _judge() returned 0.0 on Ollama timeout / network error — identical to a genuine "NO" grade. Now returns None; _summary filters None out of the Judge average so transport flakiness doesn't depress the score. The row still records 0.0 vs None distinctly. 2. Generation errors stored a synthetic "[generation_error: ...]" prediction and then computed F1/BLEU/Judge on it. That folded infra failures into the benchmark averages, and failed_qa stayed zero so the run reported "complete" despite missing answers. Now: on gen failure, predicted=""; f1/bleu/judge=None; row carries an `error` field; _guarded increments failed_qa for those rows; _summary excludes None metrics. 3. mem0_locomo_runner.py hardcoded evidence_hits=0 in every row because mem0 2.x doesn't round-trip per-turn dia_ids. _summary then published retrieval_recall=0.0 for every mem0 run — a fake miss. Now sets evidence_hits and evidence_total to None (metric unavailable, not zero), and _summary skips None rows from recall. 4. mem0 runner inherited both the 0.0-on-failure judge and the error- folding pattern. Both paths fixed to the same None-on-failure convention. _summary now also emits judge_scored + recall_scored alongside count so the JSON shows the denominator honestly (e.g. "Judge 0.41 over 1487 scored of 1540 total"), making infra flakiness inspectable rather than invisible. No schema-breaking changes: existing .rescored.json outputs that have evidence_hits=0 or judge=0.0 remain readable — the new _summary treats them as real zeros, which is how they were when written. Only forward runs produce None for "metric unavailable".
📝 WalkthroughWalkthroughBoth benchmark runners updated to handle errors more explicitly: Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Code Review SummaryStatus: 3 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
Files Reviewed (2 files)
Fix these issues in Kilo Cloud Reviewed by seed-2-0-pro-260328 · 207,083 tokens |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
benchmarks/mem0_locomo_runner.py (1)
318-330:⚠️ Potential issue | 🟠 Major
failed_qais still undercounted for returned error rows.These paths now annotate the row with
error, but_guarded()only incrementsfailed_qawhen_process_qa_mem0()throws. Retrieval/generation failures that return a row will still show up as successes inmeta.failed_qa.Also applies to: 371-372
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@benchmarks/mem0_locomo_runner.py` around lines 318 - 330, The returned error rows (e.g., the dict built in _process_qa_mem0() that contains an "error" key) are not causing meta.failed_qa to increment because _guarded() only counts failures when _process_qa_mem0() raises; update _guarded() to treat a non-empty "error" field in the returned result as a failure: after calling result = _process_qa_mem0(...) check if isinstance(result, dict) and result.get("error") is truthy, and if so increment meta.failed_qa and append the result to meta.failed_examples (same behavior as the exception path); apply the same change to the other analogous call sites noted around lines 371-372 so returned-error rows are counted consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@benchmarks/mem0_locomo_runner.py`:
- Around line 348-352: The code treats judge outages as valid zeros because
_judge currently returns 0.0 on exceptions; update the implementation so
transport/remote failures return None (not 0.0), and keep this call site logic
(the variable judge and the rounding into judge_val) intact; specifically modify
the _judge(...) function to catch transport/HTTP errors and return None on those
failure paths (rather than 0.0) so judge_val becomes None and outages are
excluded from averages.
- Around line 303-309: The _summary aggregation in this file is failing because
rows now contain None sentinels (EVIDENCE_UNAVAILABLE) for evidence fields and
None for f1/bleu1/judge, so change the _summary implementation to mirror the
safe filtering/counting used in benchmarks/locomo_runner.py: skip rows where
evidence_total is None or where metric values are None before summing/averaging,
treat None as "unavailable" not zero, and compute counts using explicit filters
(e.g., only include rows with r.get("evidence_total") and r.get("f1") is not
None) so sums/averages use numeric values only; update all places referencing
evidence_total/evidence_hits and f1/bleu1/judge in _summary to use this guarded
logic and preserve EVIDENCE_UNAVAILABLE behavior.
---
Outside diff comments:
In `@benchmarks/mem0_locomo_runner.py`:
- Around line 318-330: The returned error rows (e.g., the dict built in
_process_qa_mem0() that contains an "error" key) are not causing meta.failed_qa
to increment because _guarded() only counts failures when _process_qa_mem0()
raises; update _guarded() to treat a non-empty "error" field in the returned
result as a failure: after calling result = _process_qa_mem0(...) check if
isinstance(result, dict) and result.get("error") is truthy, and if so increment
meta.failed_qa and append the result to meta.failed_examples (same behavior as
the exception path); apply the same change to the other analogous call sites
noted around lines 371-372 so returned-error rows are counted consistently.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 3096a247-78cf-4301-8f70-e2f20efe1a2a
📒 Files selected for processing (2)
benchmarks/locomo_runner.pybenchmarks/mem0_locomo_runner.py
Comment on lines
+303
to
+309
| # mem0 stored facts don't carry original LoCoMo dia_id metadata, so we | ||
| # cannot match against the gold evidence list. Report both hits and total | ||
| # as None — the metric is _unavailable_, not zero. _summary in the taosmd | ||
| # runner (and any downstream scorecard builder) skips None-valued rows | ||
| # from retrieval_recall so this artefact doesn't publish a fake 0.0. | ||
| # TODO: wire dia_id pass-through if mem0 adds metadata preservation. | ||
| EVIDENCE_UNAVAILABLE = {"evidence_hits": None, "evidence_total": None} |
There was a problem hiding this comment.
These None sentinels currently break _summary() in this file.
Every returned row now carries evidence_total=None, but the local _summary() still does r.get("evidence_total", 0) > 0, so aggregation will raise TypeError on the first mem0 result. The new f1/bleu1/judge=None paths also conflict with its raw sum(...) / n averages. Please port the benchmarks/locomo_runner.py::_summary filtering/counting logic here before merging.
Also applies to: 318-330, 348-369
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@benchmarks/mem0_locomo_runner.py` around lines 303 - 309, The _summary
aggregation in this file is failing because rows now contain None sentinels
(EVIDENCE_UNAVAILABLE) for evidence fields and None for f1/bleu1/judge, so
change the _summary implementation to mirror the safe filtering/counting used in
benchmarks/locomo_runner.py: skip rows where evidence_total is None or where
metric values are None before summing/averaging, treat None as "unavailable" not
zero, and compute counts using explicit filters (e.g., only include rows with
r.get("evidence_total") and r.get("f1") is not None) so sums/averages use
numeric values only; update all places referencing evidence_total/evidence_hits
and f1/bleu1/judge in _summary to use this guarded logic and preserve
EVIDENCE_UNAVAILABLE behavior.
Comment on lines
+348
to
+352
| if generation_error is None: | ||
| judge = await _judge(client, ollama_url, model, question, reference, predicted) | ||
| f1_val = round(_f1(predicted, reference), 4) | ||
| bleu_val = round(_bleu1(predicted, reference), 4) | ||
| judge_val = round(judge, 4) if judge is not None else None |
There was a problem hiding this comment.
Judge transport failures are still treated as wrong answers here.
This branch expects _judge() to return None, but the local implementation still returns 0.0 on exceptions. As written, mem0 judge outages will continue to depress the Judge average instead of being excluded.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@benchmarks/mem0_locomo_runner.py` around lines 348 - 352, The code treats
judge outages as valid zeros because _judge currently returns 0.0 on exceptions;
update the implementation so transport/remote failures return None (not 0.0),
and keep this call site logic (the variable judge and the rounding into
judge_val) intact; specifically modify the _judge(...) function to catch
transport/HTTP errors and return None on those failure paths (rather than 0.0)
so judge_val becomes None and outages are excluded from averages.
jaylfc
added a commit
that referenced
this pull request
Apr 19, 2026
Captures every model actually used during the benchmark (generator variants, external judge, embedders, cross-encoder, fact extractor) with params, quant, VRAM footprint, and backend. Adds the runtime/host row so anyone reproducing knows the Ollama parallel limit and rescore timeout. Derives hardware-tier recommendations from what we measured: - Orange Pi (RK3588 NPU, 16 GB): qwen3:4b gen on rkllama, external judge, MiniLM ONNX embed, taosmd arch - Fedora 3060 (12 GB VRAM): gemma4:e2b gen, qwen3:4b judge co-resident, prompt-opt on by default - Laptop / Mac Mini: qwen3:4b gen via Ollama, external judge - High-end (≥24 GB): qwen3.5:9b gen viable; e2b still competitive Documents the seven lessons that drive the defaults: bigger-gen-≠-better at small scale, qwen for structured output, NUM_PARALLEL is the real ceiling, nomic context forces batching, architecture dominates generator choice, self-judge inflates, R@K needs dia_id round-trip. Also corrects the Commits row: superseded SHAs (ca0ccb7 → 571d8af for mempalace) and references the right open PRs (#34, #35, #36).
Owner
Author
|
Closing — branch has rebased away from master after 2 weeks of The intent is still right and still needed. Verified just now: ```python benchmarks/locomo_runner.py:253-263 (current master)async def _judge(client, url, model, question, reference, predicted, backend="ollama") -> float: Follow-up work (not in this PR):
Smaller, focused successor PR rather than this one. |
jaylfc
added a commit
that referenced
this pull request
May 3, 2026
) * docs(specs): LoCoMo scorecard log — taosmd × 3 variants + mem0, rescored Single source of truth for every LoCoMo number we've produced so they don't live only in chat transcripts. Captures: - Self-judge scorecards for taosmd-e2b, taosmd-e4b, taosmd-e2b+prompt-opt, mem0-e2b (all runs 2026-04-17 to 2026-04-19) - External qwen3:4b rescore numbers for the three taosmd variants (100% coverage, 0 errors). mem0 rescore queued. - Per-category tables, not just headlines — Temporal 0.29 vs 0.02 (14.5x) is the most dramatic architecture signal - Known artefacts: mem0 R@K=0.0 is an adapter limitation (no dia_id pass-through), patched in PR #33 - Methodology disclosures: same generator (gemma4:e2b), same prompt, same dataset, same top-K=10, same judge (qwen3:4b), commit SHAs for every input - Follow-up: mem0 external rescore in flight, MemPalace adapter queued — will add scorecards to this doc as they complete * docs(specs): correct stale commit SHAs in scorecard methodology CodeRabbit CRITICAL on #34 caught that 40403cc / 86c4c19 / 3c5c6c2 are no longer reachable — rewritten out of history by PR #30's rebase to a single commit. Replaced with the reachable SHAs and noted that the old ones were intentionally rewritten so anyone reading git log won't be confused. * docs(specs): correct external-judge scorecards + record mem0 rescore Two corrections in one: 1. The external qwen3:4b scorecards table had wrong numbers (0.27 / 0.22 / 0.34 for taosmd variants). Those were the earlier qwen3.5:9b biased- sample numbers that got superseded but I left in the table by mistake. Now corrected to the actual qwen3:4b 100%-coverage numbers (0.40 / 0.38 / 0.41) directly from the streaming rescore log. Per-category rows also restated from source. 2. mem0 rescore completed in 116.9 min, 100% coverage, 0 errors: - Single-hop 0.04 / Temporal 0.02 / Multi-hop 0.10 / Open-dom 0.07 - Overall Judge 0.06 Added to the same table. Biggest architecture gap is Temporal (taosmd-e2b+prompt-opt 0.41 vs mem0 0.02 = 20.5x). Overall gap ~7x under identical external judge, same generator. Also refreshed the "In flight / queued" section: mem0 rescore done, MemPalace adapter already built as `ca0ccb7` (landed in PR #30, ready to run — just needs `pip install mempalace` on the Fedora host). The earlier stale numbers are kept in the caveat block so anyone comparing against chat history or the push notifications knows why they shifted. * docs(specs): add Configuration log + hardware tier recommendations Captures every model actually used during the benchmark (generator variants, external judge, embedders, cross-encoder, fact extractor) with params, quant, VRAM footprint, and backend. Adds the runtime/host row so anyone reproducing knows the Ollama parallel limit and rescore timeout. Derives hardware-tier recommendations from what we measured: - Orange Pi (RK3588 NPU, 16 GB): qwen3:4b gen on rkllama, external judge, MiniLM ONNX embed, taosmd arch - Fedora 3060 (12 GB VRAM): gemma4:e2b gen, qwen3:4b judge co-resident, prompt-opt on by default - Laptop / Mac Mini: qwen3:4b gen via Ollama, external judge - High-end (≥24 GB): qwen3.5:9b gen viable; e2b still competitive Documents the seven lessons that drive the defaults: bigger-gen-≠-better at small scale, qwen for structured output, NUM_PARALLEL is the real ceiling, nomic context forces batching, architecture dominates generator choice, self-judge inflates, R@K needs dia_id round-trip. Also corrects the Commits row: superseded SHAs (ca0ccb7 → 571d8af for mempalace) and references the right open PRs (#34, #35, #36). * docs(specs): MemPalace self-judge landed — surprise on the per-category split MemPalace-e2b full run completed. Self-judge Overall 0.42 — much closer to taosmd (0.48) than to mem0 (0.09). Per-category: - MemPalace beats baseline taosmd on Temporal (0.33 vs 0.29) + Multi-hop (0.24 vs 0.22) - taosmd pulls ahead on Open-dom (0.64 vs 0.51) + Single-hop (0.34 vs 0.29) - prompt-opt variant still the Overall leader at 0.51 - mem0 a distant fourth on every category Story shifts from "taosmd wins by 7x over competitors" to "taosmd and MemPalace are in the same tier, mem0 is much further behind — and raw verbatim-store + a sensible default embedder is a strong baseline on its own." Also added ingest-timing comparison: MemPalace fastest at ~100s for all 10 convs (simpler architecture = less processing per turn). External rescore for MemPalace is running now on Fedora, ETA ~01:55 BST. * docs(specs): MemPalace external rescore complete — final 5-row scorecard MemPalace-e2b external qwen3:4b rescore: Overall Judge 0.34 (180.5 min, 100% coverage, 0 errors). All three architectures now have the same treatment: same generator, same prompt, same external judge, same 1540 QAs. Only the memory layer varies. Final headline numbers (external Judge, gemma4:e2b generator): - taosmd-e2b+prompt-opt 0.41 - taosmd-e2b 0.40 - taosmd-e4b 0.38 - MemPalace-e2b 0.34 - mem0-e2b (infer=False) 0.06 Per-category reveals a more nuanced story than the Overall numbers: - Single-hop: three-way tie at ~0.16-0.17 — solved at this tier by any competent semantic-retrieval system - Temporal: taosmd (0.36) and MemPalace (0.35) nearly tied; only prompt-opt breaks away at 0.41 - Multi-hop: taosmd-opt leads at 0.24; KG + query expansion help on synthesis questions - Open-dom: taosmd's clearest architectural win (0.51 vs MemPalace 0.41, +24% relative) - mem0 distant fourth everywhere Reframes the positioning: taosmd's architecture edge concentrates on harder question types that benefit from rerank + synthesis (Open-dom, Multi-hop); on simpler retrieval (Single-hop, Temporal) MemPalace's verbatim-store + default embedder is nearly as good. Cleaner story than "we dominate" and more useful for positioning against the target audiences documented in project_taosmd_positioning.md. Next: README rewrite aligned with that positioning memory and these nuanced numbers — lead with target audiences (SBC, taOS clusters, offline/compliance, long-horizon agents), frame benchmark numbers as "at the compute tier we target," highlight architectural edge on the categories where it actually shows. * docs(specs): matrix C1-C6 complete — log results, lessons, c_stack in flight - Add Parametric retrieval matrix (C1-C6) scorecard: C3 adjacent_turns is the biggest single-lever win at 0.465; C6 multihop_decompose regresses to 0.317; C5 bge_reranker deferred pending refactor. - Add lessons #8 (multihop decomposition regresses at small-LLM scale) and #9 (context stitching beats retrieval width). - Reorganise 'In flight / queued' section into Complete / In flight / Queued sub-headings. Log the c_stack run currently mid-bench and the three queued follow-ups (qwen9b dense, Qwen3.6 HLWQ via vLLM, Qwen3.6 MoE via Ollama). * docs(specs): adj=2 is new leader at 0.499; stacking is additive (retract yesterday's claim) Five new results logged (2026-04-21 evening + 2026-04-22): - c_stack final 0.482 — stacking IS additive (+0.017 over adj=1). Yesterday's 'stacking didn't stack' read was from a 62% partial rescore. - adj_sweep_adj2 0.499 — new leader, +0.089 vs baseline-opt. - adj_sweep_adj3 0.487 — regresses from adj=2, sweet spot is 2. - adj1_k20 0.479 — k=20 adds +0.014 on adj=1. - adj1_llm partial 0.464 — llm-exp flat on adj=1. Clean stack decomposition: adj=1 alone = 0.465 adj=1 + k=20 = 0.479 (+0.014 from k=20) adj=1 + llm-exp = 0.464 (+0.00 from llm-exp) adj=1 + k=20 + llm = 0.482 (+0.003 from llm-exp on top of k=20) Next queued: adj2_k20 (predicted ~0.513), then qwen3.5:9b block, then Qwen3.6 MoE (HLWQ via vLLM + GGUF via Ollama). * docs(specs): 9B generator block — c_stack_plus_qwen9b new leader at 0.509 - Add qwen3.5:9b generator block section with three results and the stacking-at-9B insight: full stack gains +0.028 at 9B vs +0.017 at 5B. Bigger model can use the wider retrieval surface the 5B couldn't. - Tier crossover flagged: 0.509 matches the Letta/LangMem/OpenAI-memory band (0.50–0.52) on a local 12 GB GPU. Mem0 paper (0.66) and audited Zep (0.584) remain ahead cross-tier. - Retract the adj=2 + k=20 = 0.513 prediction. Actual measurement was 0.477. Context token budget saturates at adj=2 on 5B; adding k=20 floods it. - Update complete/in-flight/queued with today's timeline, adj1_llm final (0.458, not 0.464 partial), qwen9b numbers, qwen9b_k20_thinking_on queued as the post-POSTMATRIX control run. - Note PR #42 (think=false on generator, 20x speedup), PR #43 (revert think=false on judge after the 1452 silent-zero bug), PR #44 (--thinking-mode opt-in flag). * docs(specs): adj2_full_stack_qwen9b 0.545 — new leader, parity with audited Zep Today's key landings: - adj2_full_stack_qwen9b: 0.545 — overall leader, +0.029 over adj=2 alone at 9B, +0.046 over the previous adj=1+stack 9B leader. - adj2_qwen9b: 0.516 — adj=2 alone at 9B (logged earlier today). - c6_multihop_qwen9b: 0.306 — multihop regression worsened at 9B (was 0.317 at 5B). Footgun confirmed across all model sizes. - qwen35_9b_full_context: 0.090 — retrieval ablation. Full conversation in context collapses to slightly above mem0 floor. Empirical proof that retrieval is essential, not just a context-window workaround. Headline revision (3rd this week): stacking is adj-dependent AND model-size-dependent. 5B + adj=2 + stack regresses (-0.022); 9B + adj=2 + stack compounds (+0.029). Smaller model attention saturates earlier; bigger model can absorb wider retrieval surface even at adj=2. Tier crossover: 0.545 is within 0.04 of audited Zep (0.584) on gpt-4o-mini. Functional parity on a local 12 GB GPU + 9B quant. Mem0 paper (0.66) and Mem0^g (0.68) remain ahead — both reported by mem0's own harness, not independently audited.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-open of #33 after its original base branch (
feat/locomo-prompt-opt) was deleted by PR #30's merge. Same content, retargeted at master.Addresses four CodeRabbit MAJOR findings that were posted on #30 (now merged) — they flagged silent-failure patterns in the runner + mem0 adapter that let infra flakiness and adapter artefacts masquerade as legitimate negative results.
Findings addressed
locomo_runner.py:_judgeNone;_summaryexcludes from Judge averagelocomo_runner.pygen error[generation_error: ...]folded into F1/BLEU/Judge;failed_qastayed 0predicted="", metrics=None, row carrieserrorfield,failed_qaincrementedmem0_locomo_runner.pymem0_locomo_runner.pyR@K=0.0evidence_hits=0, evidence_total=len(evidence)hardcoded because mem0 doesn't round-trip dia_id → published fake 0.0None(metric unavailable);_summaryskips from recall aggregationBonus
_summarynow emitsjudge_scored+recall_scoredalongsidecountso the denominator is visible (e.g. "Judge 0.41 over 1535 scored of 1540 total").Test plan
pytest tests/ -q --ignore=tests/integration)_summary— no need to re-run end-to-endSummary by CodeRabbit
Release Notes
Bug Fixes
Refactor