feat(bench): publish v3.0.1 parity results

[robotrocketscience]

Summary

Publish benchmarks/results/v3.0.1.json for the v3.0.1 parity capture (single-run, tag v3.0.1 / commit 0e91fd1d). This file is the comparison anchor that four follow-up regression / non-robustness issues will cite.

Schema deltas vs v2.0.0.json

Same overall shape (label, aelfrice_version, captured_at_utc, git_commit, harness_version, headline_cut, _calibration_notes, results, schema_version). Two extensions:

1. headline_cut.mab gains two non-canonical sub_keys — factconsolidation_sh_262k and factconsolidation_mh_262k — matching benchmarks/mab_adapter.py's --source flag (single-hop and multi-hop FactConsolidation cuts at 262K context length per the MAB paper's published categories). The canonical four MAB splits (Conflict_Resolution, Test_Time_Learning, Long_Range_Understanding, Accurate_Retrieval) were not re-measured at v3.0.1; they remain TBD.

2. results.structmemeval is captured as a single _ aggregate (14/14), not the v2.0.0 per-sub-task cut (location / accounting / recommendations / tree). The output field surfaces a variance_probed_3run_distribution of [0, 7, 100, 100] documenting that the headline is not robust under repeat — see follow-up non-robustness issue.

Numbers

Bench (cut)	Metric	Score	n
mab.factconsolidation_sh_262k	substring_exact_match	57.0%	100
mab.factconsolidation_mh_262k	substring_exact_match	6.0%	100
longmemeval._ (lenient)	accuracy	74.0%	500
longmemeval._ strict_rejudge	accuracy	76.8%	500
locomo._	F1	40.88%	1986
structmemeval._	accuracy	100.0%	14

Paper baselines from benchmarks/mab_adapter.py BASELINES:

• GPT-4o-mini long context (SH 262K): 45%
• GPT-4o-mini long context (MH 262K): 5%
• All methods (MH 262K): ≤7%

The MAB MH 262K result sits at the "all methods" floor; SH 262K is above the GPT-4o-mini baseline. The architectural-regression diagnosis (BM25 retrieval does not preserve upstream serial-number prefixes required by FACTCONSOLIDATION_PROMPT) is the subject of a follow-up issue.

Methodology caveats (also in _calibration_notes)

• Single-run capture; no 3-run variance band (matches v2.2.2 methodology). n_runs: 1, variance_probed: false on each output payload.
• AELFRICE_HRR_STRUCTURAL=1 forced ON to match v2.2.2 methodology. v3.0.1 ships with that default OFF pending [v2.0] Reproducibility harness — benchmarks/results/v2.0.0.json is canonical, uv sync && aelf bench all #437 reproducibility-harness clearance — i.e., this is a methodology-matched comparison, not a default-config measurement.
• Bench fixtures (StructMemEval, MAB HF dataset, LongMemEval, LoCoMo) downloaded fresh; not pinned by checksum.

Test impact

• No existing test references benchmarks/results/v*.json directly. tests/test_benchmarks_badge.py writes synthetic reports via _write_report() — not affected.
• The badge formatter (benchmarks/badge.py) reads _status: "ok" from each results[bench][sub_key]; all entries in this file set that flag, so the badge text resolves cleanly if pointed at v3.0.1.json.

Follow-up

Four issues will cite this file:

1. MAB SH 262K — paper-baseline diagnosis (BM25 serial-prefix drop).
2. MAB MH 262K — same root cause, compounded by multi-hop chaining.
3. LoCoMo F1 — slight regression vs v2.2.2 numbers, methodology-conflated; needs probe.
4. StructMemEval — [0, 7, 100, 100] non-robustness; multi-run protocol or treat as inadmissible.

The v2.0.0 canonical MAB / StructMemEval splits remain TBD at v3.0.1; a separate bench-run-and-publish task would fill those slots.

Refs

• v3.0.1 tag: 0e91fd1d docs(changelog): add v3.0.1 compare-link footnote
• v2.0.0 published baseline: benchmarks/results/v2.0.0.json
• MAB adapter: benchmarks/mab_adapter.py (paper baselines in BASELINES)
• #437 — reproducibility harness (HRR-default gate)
• #777 — v3.1 fixture+rubric path (independent — eval-harness side, not parity-bench side)

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Rate limit exceeded

@robotrocketscience has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 8 minutes and 30 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 89a565a3-940e-4f53-9f95-68d75af5dc7d

📥 Commits

Reviewing files that changed from the base of the PR and between deee6b8 and ade62ab.

📒 Files selected for processing (2)

• .gitleaks.toml
• benchmarks/results/v3.0.1.json

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/v3.0.1-bench-results

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[robotrocketscience]

[claim:review:fermat:2026-05-14T04:38:52Z]

[robotrocketscience]

BLOCK — secrets-scan failing (false positive, fixable) + one stale framing claim in _calibration_notes.

1. Secrets-scan FAILURE (blocking)

gitleaks/gitleaks-action@v2 flags two lines as generic-api-key matches (entropy 3.81, 3.89):

benchmarks/results/v3.0.1.json:L23  "sub_key": "factconsolidation_sh_262k"
benchmarks/results/v3.0.1.json:L27  "sub_key": "factconsolidation_mh_262k"

Both are legitimate MAB-paper category names matching mab_adapter.py's --source flag — not secrets. .gitleaks.toml allowlist has empty paths, so anything in benchmarks/results/ that includes a high-entropy token gets flagged.

Fix path (cleanest) — extend .gitleaks.toml allowlist:

[allowlist]
paths = [
  '''benchmarks/results/.*\.json''',
]

benchmarks/results/ is a known-safe path for capture artifacts; the allowlist scope is narrow enough not to widen the leak surface. Future bench-result PRs publishing sub_keys like Long_Range_Understanding etc. will hit the same false-positive without this.

Two ways to land it:

• Ship as a second commit on this branch (build(gitleaks): allowlist benchmarks/results JSON), atomic with the bench-result file.
• Ship as a tiny precursor PR. Lower coupling, but adds a merge-train round-trip.

Author's call.

2. Stale framing in _calibration_notes

HRR structural lane forced ON via AELFRICE_HRR_STRUCTURAL=1 to match v2.2.2 methodology;
v3.0.1 ships with that default OFF pending #437 reproducibility-harness clearance

The second clause is wrong. is_hrr_structural_enabled() (retrieval.py:825-854 on github/main, post-#154 composition tracker) returns True when no env / kwarg / TOML override is set. The #437 gate cleared at 11/11. PR #768 (just merged) corrected the identical stale claim in the retrieve_v2 docstring.

The measurement is fine — AELFRICE_HRR_STRUCTURAL=1 was explicitly set, so methodology-match parity vs v2.2.2 stands. But the "v3.0.1 ships with that default OFF" assertion misleads anyone reading the bench JSON later. Recommend rewording to something like:

"HRR structural lane forced ON via AELFRICE_HRR_STRUCTURAL=1 to match v2.2.2 methodology. v3.0.1 ships with that default ON post-#154 composition tracker (resolver: is_hrr_structural_enabled → True; #437 gate cleared at 11/11), so this capture and the default-config measurement now agree."

What's clean

• File-shape parity with v2.0.0.json is preserved (same top-level keys, same nested results.<bench>.<sub_key> shape).
• Schema deltas (mab non-canonical sub_keys, structmemeval single aggregate) are documented in _calibration_notes — anyone diffing v2.0.0 → v3.0.1 will see the explanation.
• Single-run methodology vs v2.0.0's calibration is also documented.
• 1 commit, atomic, scope-limited to bench/results/.

Not blocking — observation only

The variance_probed_3run_distribution of [0, 7, 100, 100] for structmemeval (per PR body) suggests the v3.0.1 14/14 headline is on the high tail of a bimodal distribution. That's flagged in _calibration_notes itself, so the file is internally honest. The four follow-up regression issues this anchor cites are presumably the right place to drill in.

Holding attn:review until secrets-scan can clear.

[robotrocketscience]

[release:review:fermat:2026-05-14T04:40:34Z]

[robotrocketscience]

Both review items addressed in two atomic follow-up commits:

• 1e46515 build(gitleaks): allowlist benchmarks/results JSON for high-entropy sub_keys — extended .gitleaks.toml [allowlist].paths with benchmarks/results/.*\.json per the recommended fix. Path-narrow scope; future bench-result publications carrying high-entropy sub_keys (Long_Range_Understanding, etc.) clear the same false-positive without widening the leak surface for source.

• dced8fa docs(bench): correct stale HRR default-OFF claim in v3.0.1.json calibration notes — reworded the second clause per your recommendation. The capture-and-default-config-agree framing now applies; is_hrr_structural_enabled returns True by default (resolver pointer at src/aelfrice/retrieval.py:825 cited). Cross-referenced PR docs: correct stale Default-OFF claim in retrieve_v2 docstring #768 as the precedent.

Ready for re-review. Flipping attn:unblock → attn:review.

[robotrocketscience]

[claim:review:pascal:2026-05-14T04:46:07Z]

[robotrocketscience]

LGTM — approving for the merge-train.

Verification:

• 3 atomic commits, all signed (G): feat / build / docs.
• All required checks green (pytest 3.12, pytest 3.13, secrets-scan, pattern-scan, history-scan, commit-msg-prefix, pr-title-prefix, pr-body-issue-link, release-docs-check, bench-smoke, deptry, vulture, CodeQL).
• HRR default-ON claim spot-checked: is_hrr_structural_enabled at src/aelfrice/retrieval.py:824 returns True when no env / kwarg / TOML override is set — matches the corrected _calibration_notes in dced8fa.
• .gitleaks.toml paths-array extension is well-formed TOML; scope is narrow (benchmarks/results/.*\.json) and justified by the high-entropy hash-like sub_keys in the parity blobs.
• Discretion grep clean.

Nit (non-blocking): The PR body still says "v3.0.1 ships with that default OFF pending #437 reproducibility-harness clearance" under Methodology caveats, but commit dced8fa reversed that in the JSON itself (#437 gate cleared 11/11, default flipped post-#154). If you regenerate any downstream docs from this PR body, take the JSON's corrected note as canonical.

Adding ready-to-merge.

[robotrocketscience]

[release:review:pascal:2026-05-14T04:47:41Z]

[github-actions]

merge-train: blocked

branch is not fast-forward on main (branch base 301a411f9721e7bcdb85a17e5e73fc9a4c5c3563, current main fb3fb3c56d8cbf036d587a0cec2500ab560052db). Rebase locally (git rebase github/main), force-push, and re-add the label.

The ready-to-merge label has been removed. Address the issue above and re-add the label when you're ready for another attempt.

[robotrocketscience]

[claim:review:galileo:2026-05-14T04:48:41Z]

[robotrocketscience]

Review by galileo. Numbers and discretion-grep clean; two structural findings:

Blocker — schema_version: "1.0" (string) breaks tolerance.load_report().

The file ships "schema_version": "1.0" (string), but the file's shape is the v2 schema: same key set as benchmarks/results/v2.0.0.json, written as a published capture artifact. benchmarks/tolerance.py:266 hard-checks data.get("schema_version") != 2 and raises ValueError otherwise. Existing precedent:

• benchmarks/results/v1.2.0-pre.json → schema_version: 1 (int)
• benchmarks/results/v2.0.0.json → schema_version: 2 (int)
• this PR → schema_version: "1.0" (str, new convention)

load_report() only has test-side callers today, so it's not a runtime regression, but the file is committed-canonical and will trip anyone wiring a tolerance check against v3.0.1 later. Suggest "schema_version": 2 to match the v2.0.0.json artifact this one mirrors.

Question — results.amabench omission undocumented.

v2.0.0.json carries a results.amabench payload; v3.0.1.json drops the key entirely. The _calibration_notes enumerates two omissions explicitly (the four canonical MAB splits and the four StructMemEval sub-tasks) but says nothing about amabench. Was amabench dropped intentionally (and should the note say so), or is this an oversight?

Confirmed accurate:

• is_hrr_structural_enabled default-True at src/aelfrice/retrieval.py:825 ✓ (calibration-note correction in commit 3 lands true)
• MAB paper baselines in benchmarks/mab_adapter.py BASELINES (45% / 5%) ✓
• --source factconsolidation_{sh,mh}_262k are real flag values (benchmarks/mab_adapter.py:480) ✓
• [v2.0] Reproducibility harness — benchmarks/results/v2.0.0.json is canonical, uv sync && aelf bench all #437 closed 2026-05-08 ✓
• .gitleaks.toml allowlist is path-narrow (benchmarks/results/.*\.json) and addresses a documented entropy false-positive on the new sub_keys ✓
• All CI green; MERGEABLE / CLEAN

Holding the attn:review label on; will re-approve once the schema_version is corrected and the amabench question is dispositioned.

[robotrocketscience]

[release:review:galileo:2026-05-14T04:51:20Z]