docs: hardware-tier defaults + May 5 LoCoMo sweep - #63
Merged
Conversation
…B tier refresh May 5 LoCoMo subset-200 sweep on Fedora 12 GB 3060: - 6 generators × 200 QAs at leader recipe (k=20 + adj=2 + llm-exp + RRF): qwen3.5:9b 0.56, mistral-small3.2 0.56 (2.8× slower), llama3.1:8b 0.54 (2.4× faster), gemma4:e4b 0.51, granite4:tiny-h 0.41, phi4-reasoning timeout. qwen stays as production; llama3.1:8b is the new fast-tier rec. - 8 9B quants × 200 QAs same recipe: cliff is shallow above Q3 (Q4_K_M / Q5_K_M / IQ4_XS within ±0.01). IQ4_XS at 4.81 GB / 0.55 is the 8 GB tier candidate. Q3_K_S at 0.49 (-0.07) is the floor. - Methodology note: custom quants need full Modelfile metadata cloned from ollama show <native> --modelfile (TEMPLATE / RENDERER / PARSER / PARAMETER). Bare FROM <gguf> Modelfiles silently apply the wrong template and look ~130× slower per QA — the IQ-kernel-is-slow theory was incorrect. Hardware-tiers section: 12 GB GPU now leads with 0.557 leader (was 0.516 adj=2 alone) + adds llama3.1:8b as fast-tier; new 8 GB GPU section extrapolated from quant data (IQ4_XS at 0.55 predicted ext rejudge).
…LoCoMo subsection benchmarks.md: - New "Answer-prompt variants" subsection: 4 prompt templates (concise, refusal, citation, memobase) × 2 generators (qwen3.5:9b, llama3.1:8b) at the leader recipe. Single-hop hypothesis refuted — every variant regresses Single-hop on both models. qwen3.5:9b regresses -0.15 to -0.23; llama3.1:8b regresses -0.15/-0.16/-0.16 except citation which holds at -0.01 — a prompt × model interaction worth flagging. - Pi NPU section: removed the residual "LoCoMo not yet measured (planned)" paragraph that contradicted the LoCoMo measurements above it; folded into a single sentence noting the same stack measured both LongMemEval-S and LoCoMo at this tier. README.md: - New "LoCoMo — same-tier leader on a 12 GB GPU" subsection: 0.557 ext rejudge headline, plus the two preferred generators (qwen3.5:9b default, llama3.1:8b fast-tier alt) with quick rationale and pointer to the full doc. - Updated tier list in the existing benchmarks-doc cross-link to include the 8 GB tier.
📝 WalkthroughWalkthroughThis PR updates benchmark documentation in README.md and docs/benchmarks.md: adds a 12 GB GPU tier, expands 12/8/4 GB hardware-tier guidance, inserts a LoCoMo — same-tier leader on a 12 GB GPU section (LoCoMo-10, 0.557 ext rejudge, generator recommendations), and reorganizes benchmark details and caveats. ChangesBenchmark Documentation Updates
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (2 files)
Reviewed by grok-code-fast-1:optimized:free · 186,326 tokens |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Line 191: The README sentence currently presents the "8 GB-tier IQ4_XS at
0.55" as a measured result; update that phrase to clarify it is extrapolated
(for example: "8 GB-tier IQ4_XS (extrapolated from the 12 GB quant sweep) at
0.55") so readers understand it was not measured on native 8 GB hardware and is
an extrapolation from the 12 GB quant sweep.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: e1146c2f-0e75-4bda-9384-1aa82d10d898
📒 Files selected for processing (2)
README.mddocs/benchmarks.md
| - **`qwen3.5:9b`** Q4_K_M (5.3 GB on disk) — **production default**. Best measured quality. | ||
| - **`llama3.1:8b`** (4.9 GB on disk) — **fast-tier alternative**. -0.02 ext rejudge from the qwen leader, **2.4× faster per QA**. Right pick for realtime turn latency or running multiple agents on one card. | ||
|
|
||
| May 5 2026 generator-candidate sweep (200 QAs, leader recipe): qwen3.5:9b 0.56, mistral-small3.2 0.56 (2.8× slower per QA — not promoted), llama3.1:8b 0.54, gemma4:e4b 0.51, granite4:tiny-h 0.41. Full table, the 9B quant cliff (8 quants from Q2 through Q6, including the 8 GB-tier IQ4_XS at 0.55), the answer-prompt-variants negative result, and per-hardware-tier configurations in [docs/benchmarks.md](docs/benchmarks.md). |
There was a problem hiding this comment.
Clarify that the 8 GB tier figure is extrapolated, not measured on native 8 GB hardware.
This line currently reads like a direct 8 GB benchmark result. Add a short qualifier (e.g., “8 GB-tier candidate extrapolated from the 12 GB quant sweep”) to prevent misinterpretation.
Suggested wording tweak
-May 5 2026 generator-candidate sweep (200 QAs, leader recipe): qwen3.5:9b 0.56, mistral-small3.2 0.56 (2.8× slower per QA — not promoted), llama3.1:8b 0.54, gemma4:e4b 0.51, granite4:tiny-h 0.41. Full table, the 9B quant cliff (8 quants from Q2 through Q6, including the 8 GB-tier IQ4_XS at 0.55), the answer-prompt-variants negative result, and per-hardware-tier configurations in [docs/benchmarks.md](docs/benchmarks.md).
+May 5 2026 generator-candidate sweep (200 QAs, leader recipe): qwen3.5:9b 0.56, mistral-small3.2 0.56 (2.8× slower per QA — not promoted), llama3.1:8b 0.54, gemma4:e4b 0.51, granite4:tiny-h 0.41. Full table, the 9B quant cliff (8 quants from Q2 through Q6, including the IQ4_XS 0.55 row used as the **extrapolated** 8 GB-tier candidate), the answer-prompt-variants negative result, and per-hardware-tier configurations in [docs/benchmarks.md](docs/benchmarks.md).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| May 5 2026 generator-candidate sweep (200 QAs, leader recipe): qwen3.5:9b 0.56, mistral-small3.2 0.56 (2.8× slower per QA — not promoted), llama3.1:8b 0.54, gemma4:e4b 0.51, granite4:tiny-h 0.41. Full table, the 9B quant cliff (8 quants from Q2 through Q6, including the 8 GB-tier IQ4_XS at 0.55), the answer-prompt-variants negative result, and per-hardware-tier configurations in [docs/benchmarks.md](docs/benchmarks.md). | |
| May 5 2026 generator-candidate sweep (200 QAs, leader recipe): qwen3.5:9b 0.56, mistral-small3.2 0.56 (2.8× slower per QA — not promoted), llama3.1:8b 0.54, gemma4:e4b 0.51, granite4:tiny-h 0.41. Full table, the 9B quant cliff (8 quants from Q2 through Q6, including the IQ4_XS 0.55 row used as the **extrapolated** 8 GB-tier candidate), the answer-prompt-variants negative result, and per-hardware-tier configurations in [docs/benchmarks.md](docs/benchmarks.md). |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 191, The README sentence currently presents the "8 GB-tier
IQ4_XS at 0.55" as a measured result; update that phrase to clarify it is
extrapolated (for example: "8 GB-tier IQ4_XS (extrapolated from the 12 GB quant
sweep) at 0.55") so readers understand it was not measured on native 8 GB
hardware and is an extrapolation from the 12 GB quant sweep.
…no Single-hop lift May 5/6 sweep at the leader recipe (k=20 + adj=2 + llm-exp + RRF) on qwen3.5:9b, swapping the embedder layer: - Arctic Embed L v2.0 (Snowflake, 568M): 0.55 (-0.01) - EmbeddingGemma-300m (Google): 0.54 (-0.02) - Nomic Embed v1.5 (Nomic): 0.54 (-0.02) Single-hop drops to 0.26 across all three, vs ~0.34 baseline. Combined with the prompt-variant sweep (yesterday), this is the second leverage point that doesn't move Single-hop — the ceiling on that category sits in the architecture, not in prompt or embedder choice. Architectural levers (typed memory routing per ENGRAM, multi-vector retrieval, hybrid lexical+vector) are the next slot. EmbeddingGemma's Multi-hop collapse (0.31 vs Arctic/Nomic both 0.62) is worth a follow-up — likely an interaction between the asymmetric 'title: none | text: …' document prefix and conversational turns. Pending: Qwen3-Embedding-0.6B and -4B via --embed-mode local (sentence-transformers + PyTorch CPU), since the onnx-community export keeps causal-LM KV-cache inputs.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/benchmarks.md`:
- Line 264: The docs reference an ephemeral host-local path
`/tmp/embedder_sweep_summary.tsv`; replace it with a checked-in artifact path
(or add the TSV into the repo) and update the sentence in docs/benchmarks.md
that mentions branch `feat/embedder-asymmetric-prefixes-may5` to point to the
committed location (e.g., docs/artifacts/embedder_sweep_summary.tsv) so readers
can reproduce; ensure the file is added to the repository and the markdown
link/path is updated accordingly.
- Around line 333-335: Update the conflicting claim that the LongMemEval-S 97.0%
run used the “same exact stack” as the LoCoMo measurements: either (A) confirm
and document that the LoCoMo-adjacent-turns setting has been validated and that
LongMemEval-S used the same explicit adjacent-turn flag, or (B) clarify that
LongMemEval-S used the Qwen3-4B via rkllama stack without the adjacent-turn flag
while LoCoMo used a distinct LoCoMo-adjacent-turns configuration (not yet
validated). Specifically edit the phrasing around the “LongMemEval-S 97.0%
reference stack” and the “LoCoMo measurements” lines to state which
configuration (including the presence/absence of the LoCoMo-adjacent-turns flag)
was actually used so the benchmark provenance is unambiguous.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
|
|
||
| Pending: Qwen3-Embedding-0.6B and Qwen3-Embedding-4B — the `onnx-community` ONNX exports keep causal-LM KV-cache inputs that don't fit our embedder loader, so they're queued via `--embed-mode local` (sentence-transformers + PyTorch CPU) on the same recipe. Results will be appended to this section once the chain finishes. | ||
|
|
||
| Measured on Fedora 12 GB 3060 host, May 6 2026. The asymmetric query/document prefix detection lives on branch `feat/embedder-asymmetric-prefixes-may5` for reproduction; full sweep summary at `/tmp/embedder_sweep_summary.tsv` on the bench host. |
There was a problem hiding this comment.
Avoid host-local /tmp paths in reproducibility docs.
Line 264 points to /tmp/embedder_sweep_summary.tsv, which is ephemeral and not accessible to readers. Please reference a checked-in artifact path (or attach the TSV location in-repo) to keep reproduction practical.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/benchmarks.md` at line 264, The docs reference an ephemeral host-local
path `/tmp/embedder_sweep_summary.tsv`; replace it with a checked-in artifact
path (or add the TSV into the repo) and update the sentence in
docs/benchmarks.md that mentions branch `feat/embedder-asymmetric-prefixes-may5`
to point to the committed location (e.g.,
docs/artifacts/embedder_sweep_summary.tsv) so readers can reproduce; ensure the
file is added to the repository and the markdown link/path is updated
accordingly.
Comment on lines
+333
to
+335
| This is also the LongMemEval-S 97.0% reference stack — same configuration as the LoCoMo measurements above. | ||
|
|
||
|
|
||
| This is the LongMemEval-S 97.0% reference stack. LoCoMo on this hardware is **not yet measured** (planned). | ||
|
|
||
| - **Generator**: `Qwen3-4B` via `rkllama` on the RK3588 NPU (~17 s/turn). Exact stack used for the 97% claim. | ||
| - **Generator**: `Qwen3-4B` via `rkllama` on the RK3588 NPU (~17 s/turn). Exact stack used for both the 97 % LongMemEval-S claim and the LoCoMo measurements at this tier. |
There was a problem hiding this comment.
Conflicting methodology claim for Pi-tier “same configuration.”
Line 333 and Line 335 claim the LongMemEval-S 97.0% run used the same exact stack as LoCoMo on this tier, but the methodology section still says the LoCoMo-adjacent-turns setting is “not yet validated” and LongMemEval used no explicit adj flag. Please align these statements so the benchmark provenance is unambiguous.
Proposed wording fix
-This is also the LongMemEval-S 97.0% reference stack — same configuration as the LoCoMo measurements above.
+This section shares the same hardware tier and core components as the LoCoMo runs, but not every retrieval flag is identical.
-- **Generator**: `Qwen3-4B` via `rkllama` on the RK3588 NPU (~17 s/turn). Exact stack used for both the 97 % LongMemEval-S claim and the LoCoMo measurements at this tier.
+- **Generator**: `Qwen3-4B` via `rkllama` on the RK3588 NPU (~17 s/turn). This is the measured stack for the 97 % LongMemEval-S claim on this tier; LoCoMo rows above list their measured retrieval flags explicitly.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| This is also the LongMemEval-S 97.0% reference stack — same configuration as the LoCoMo measurements above. | |
| This is the LongMemEval-S 97.0% reference stack. LoCoMo on this hardware is **not yet measured** (planned). | |
| - **Generator**: `Qwen3-4B` via `rkllama` on the RK3588 NPU (~17 s/turn). Exact stack used for the 97% claim. | |
| - **Generator**: `Qwen3-4B` via `rkllama` on the RK3588 NPU (~17 s/turn). Exact stack used for both the 97 % LongMemEval-S claim and the LoCoMo measurements at this tier. | |
| This section shares the same hardware tier and core components as the LoCoMo runs, but not every retrieval flag is identical. | |
| - **Generator**: `Qwen3-4B` via `rkllama` on the RK3588 NPU (~17 s/turn). This is the measured stack for the 97 % LongMemEval-S claim on this tier; LoCoMo rows above list their measured retrieval flags explicitly. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/benchmarks.md` around lines 333 - 335, Update the conflicting claim that
the LongMemEval-S 97.0% run used the “same exact stack” as the LoCoMo
measurements: either (A) confirm and document that the LoCoMo-adjacent-turns
setting has been validated and that LongMemEval-S used the same explicit
adjacent-turn flag, or (B) clarify that LongMemEval-S used the Qwen3-4B via
rkllama stack without the adjacent-turn flag while LoCoMo used a distinct
LoCoMo-adjacent-turns configuration (not yet validated). Specifically edit the
phrasing around the “LongMemEval-S 97.0% reference stack” and the “LoCoMo
measurements” lines to state which configuration (including the presence/absence
of the LoCoMo-adjacent-turns flag) was actually used so the benchmark provenance
is unambiguous.
This was referenced May 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the May 5 2026 LoCoMo subset-200 sweep into the docs and refreshes the hardware-tier configuration recommendations now that we have data for 4 / 8 / 12 GB GPU and Pi NPU tiers.
docs/benchmarks.mdANSWER_PROMPTis locally optimal at this tier. One worth-noting interaction:citationcosts qwen -0.17 but only -0.01 on llama3.1:8b — a prompt × model effect, not a universal lever.README.mdTest plan
Summary by CodeRabbit