fix(bench): disable Ollama reasoning mode — 20x+ bench/rescore speedup - #42
Merged
Conversation
Qwen3/3.5/3.6 models emit 200+ hidden reasoning tokens per call by default. Ollama strips them from the visible response but bills full generation time. A probe on qwen3.5:9b measured 173s for a trivial 50-token prompt (225 eval tokens, 10-char visible answer). Setting `think: false` top-level in the /api/generate payload brings the same probe to 0.5s (9 eval tokens, correct answer). qwen9b_k20 was running 24% done after 16h at this rate (projected ~64h total). With think=false it should finish in ~3-4h. Rescore tool similarly — qwen3:4b judge was limited to 0.12 items/s by the same mechanism. Also adds `benchmarks/locomo_rescore_streaming.py` to the repo — this tool was being used on Fedora but had not been checked in.
jaylfc
added a commit
that referenced
this pull request
Apr 24, 2026
….509 - Add qwen3.5:9b generator block section with three results and the stacking-at-9B insight: full stack gains +0.028 at 9B vs +0.017 at 5B. Bigger model can use the wider retrieval surface the 5B couldn't. - Tier crossover flagged: 0.509 matches the Letta/LangMem/OpenAI-memory band (0.50–0.52) on a local 12 GB GPU. Mem0 paper (0.66) and audited Zep (0.584) remain ahead cross-tier. - Retract the adj=2 + k=20 = 0.513 prediction. Actual measurement was 0.477. Context token budget saturates at adj=2 on 5B; adding k=20 floods it. - Update complete/in-flight/queued with today's timeline, adj1_llm final (0.458, not 0.464 partial), qwen9b numbers, qwen9b_k20_thinking_on queued as the post-POSTMATRIX control run. - Note PR #42 (think=false on generator, 20x speedup), PR #43 (revert think=false on judge after the 1452 silent-zero bug), PR #44 (--thinking-mode opt-in flag).
jaylfc
added a commit
that referenced
this pull request
May 3, 2026
) * docs(specs): LoCoMo scorecard log — taosmd × 3 variants + mem0, rescored Single source of truth for every LoCoMo number we've produced so they don't live only in chat transcripts. Captures: - Self-judge scorecards for taosmd-e2b, taosmd-e4b, taosmd-e2b+prompt-opt, mem0-e2b (all runs 2026-04-17 to 2026-04-19) - External qwen3:4b rescore numbers for the three taosmd variants (100% coverage, 0 errors). mem0 rescore queued. - Per-category tables, not just headlines — Temporal 0.29 vs 0.02 (14.5x) is the most dramatic architecture signal - Known artefacts: mem0 R@K=0.0 is an adapter limitation (no dia_id pass-through), patched in PR #33 - Methodology disclosures: same generator (gemma4:e2b), same prompt, same dataset, same top-K=10, same judge (qwen3:4b), commit SHAs for every input - Follow-up: mem0 external rescore in flight, MemPalace adapter queued — will add scorecards to this doc as they complete * docs(specs): correct stale commit SHAs in scorecard methodology CodeRabbit CRITICAL on #34 caught that 40403cc / 86c4c19 / 3c5c6c2 are no longer reachable — rewritten out of history by PR #30's rebase to a single commit. Replaced with the reachable SHAs and noted that the old ones were intentionally rewritten so anyone reading git log won't be confused. * docs(specs): correct external-judge scorecards + record mem0 rescore Two corrections in one: 1. The external qwen3:4b scorecards table had wrong numbers (0.27 / 0.22 / 0.34 for taosmd variants). Those were the earlier qwen3.5:9b biased- sample numbers that got superseded but I left in the table by mistake. Now corrected to the actual qwen3:4b 100%-coverage numbers (0.40 / 0.38 / 0.41) directly from the streaming rescore log. Per-category rows also restated from source. 2. mem0 rescore completed in 116.9 min, 100% coverage, 0 errors: - Single-hop 0.04 / Temporal 0.02 / Multi-hop 0.10 / Open-dom 0.07 - Overall Judge 0.06 Added to the same table. Biggest architecture gap is Temporal (taosmd-e2b+prompt-opt 0.41 vs mem0 0.02 = 20.5x). Overall gap ~7x under identical external judge, same generator. Also refreshed the "In flight / queued" section: mem0 rescore done, MemPalace adapter already built as `ca0ccb7` (landed in PR #30, ready to run — just needs `pip install mempalace` on the Fedora host). The earlier stale numbers are kept in the caveat block so anyone comparing against chat history or the push notifications knows why they shifted. * docs(specs): add Configuration log + hardware tier recommendations Captures every model actually used during the benchmark (generator variants, external judge, embedders, cross-encoder, fact extractor) with params, quant, VRAM footprint, and backend. Adds the runtime/host row so anyone reproducing knows the Ollama parallel limit and rescore timeout. Derives hardware-tier recommendations from what we measured: - Orange Pi (RK3588 NPU, 16 GB): qwen3:4b gen on rkllama, external judge, MiniLM ONNX embed, taosmd arch - Fedora 3060 (12 GB VRAM): gemma4:e2b gen, qwen3:4b judge co-resident, prompt-opt on by default - Laptop / Mac Mini: qwen3:4b gen via Ollama, external judge - High-end (≥24 GB): qwen3.5:9b gen viable; e2b still competitive Documents the seven lessons that drive the defaults: bigger-gen-≠-better at small scale, qwen for structured output, NUM_PARALLEL is the real ceiling, nomic context forces batching, architecture dominates generator choice, self-judge inflates, R@K needs dia_id round-trip. Also corrects the Commits row: superseded SHAs (ca0ccb7 → 571d8af for mempalace) and references the right open PRs (#34, #35, #36). * docs(specs): MemPalace self-judge landed — surprise on the per-category split MemPalace-e2b full run completed. Self-judge Overall 0.42 — much closer to taosmd (0.48) than to mem0 (0.09). Per-category: - MemPalace beats baseline taosmd on Temporal (0.33 vs 0.29) + Multi-hop (0.24 vs 0.22) - taosmd pulls ahead on Open-dom (0.64 vs 0.51) + Single-hop (0.34 vs 0.29) - prompt-opt variant still the Overall leader at 0.51 - mem0 a distant fourth on every category Story shifts from "taosmd wins by 7x over competitors" to "taosmd and MemPalace are in the same tier, mem0 is much further behind — and raw verbatim-store + a sensible default embedder is a strong baseline on its own." Also added ingest-timing comparison: MemPalace fastest at ~100s for all 10 convs (simpler architecture = less processing per turn). External rescore for MemPalace is running now on Fedora, ETA ~01:55 BST. * docs(specs): MemPalace external rescore complete — final 5-row scorecard MemPalace-e2b external qwen3:4b rescore: Overall Judge 0.34 (180.5 min, 100% coverage, 0 errors). All three architectures now have the same treatment: same generator, same prompt, same external judge, same 1540 QAs. Only the memory layer varies. Final headline numbers (external Judge, gemma4:e2b generator): - taosmd-e2b+prompt-opt 0.41 - taosmd-e2b 0.40 - taosmd-e4b 0.38 - MemPalace-e2b 0.34 - mem0-e2b (infer=False) 0.06 Per-category reveals a more nuanced story than the Overall numbers: - Single-hop: three-way tie at ~0.16-0.17 — solved at this tier by any competent semantic-retrieval system - Temporal: taosmd (0.36) and MemPalace (0.35) nearly tied; only prompt-opt breaks away at 0.41 - Multi-hop: taosmd-opt leads at 0.24; KG + query expansion help on synthesis questions - Open-dom: taosmd's clearest architectural win (0.51 vs MemPalace 0.41, +24% relative) - mem0 distant fourth everywhere Reframes the positioning: taosmd's architecture edge concentrates on harder question types that benefit from rerank + synthesis (Open-dom, Multi-hop); on simpler retrieval (Single-hop, Temporal) MemPalace's verbatim-store + default embedder is nearly as good. Cleaner story than "we dominate" and more useful for positioning against the target audiences documented in project_taosmd_positioning.md. Next: README rewrite aligned with that positioning memory and these nuanced numbers — lead with target audiences (SBC, taOS clusters, offline/compliance, long-horizon agents), frame benchmark numbers as "at the compute tier we target," highlight architectural edge on the categories where it actually shows. * docs(specs): matrix C1-C6 complete — log results, lessons, c_stack in flight - Add Parametric retrieval matrix (C1-C6) scorecard: C3 adjacent_turns is the biggest single-lever win at 0.465; C6 multihop_decompose regresses to 0.317; C5 bge_reranker deferred pending refactor. - Add lessons #8 (multihop decomposition regresses at small-LLM scale) and #9 (context stitching beats retrieval width). - Reorganise 'In flight / queued' section into Complete / In flight / Queued sub-headings. Log the c_stack run currently mid-bench and the three queued follow-ups (qwen9b dense, Qwen3.6 HLWQ via vLLM, Qwen3.6 MoE via Ollama). * docs(specs): adj=2 is new leader at 0.499; stacking is additive (retract yesterday's claim) Five new results logged (2026-04-21 evening + 2026-04-22): - c_stack final 0.482 — stacking IS additive (+0.017 over adj=1). Yesterday's 'stacking didn't stack' read was from a 62% partial rescore. - adj_sweep_adj2 0.499 — new leader, +0.089 vs baseline-opt. - adj_sweep_adj3 0.487 — regresses from adj=2, sweet spot is 2. - adj1_k20 0.479 — k=20 adds +0.014 on adj=1. - adj1_llm partial 0.464 — llm-exp flat on adj=1. Clean stack decomposition: adj=1 alone = 0.465 adj=1 + k=20 = 0.479 (+0.014 from k=20) adj=1 + llm-exp = 0.464 (+0.00 from llm-exp) adj=1 + k=20 + llm = 0.482 (+0.003 from llm-exp on top of k=20) Next queued: adj2_k20 (predicted ~0.513), then qwen3.5:9b block, then Qwen3.6 MoE (HLWQ via vLLM + GGUF via Ollama). * docs(specs): 9B generator block — c_stack_plus_qwen9b new leader at 0.509 - Add qwen3.5:9b generator block section with three results and the stacking-at-9B insight: full stack gains +0.028 at 9B vs +0.017 at 5B. Bigger model can use the wider retrieval surface the 5B couldn't. - Tier crossover flagged: 0.509 matches the Letta/LangMem/OpenAI-memory band (0.50–0.52) on a local 12 GB GPU. Mem0 paper (0.66) and audited Zep (0.584) remain ahead cross-tier. - Retract the adj=2 + k=20 = 0.513 prediction. Actual measurement was 0.477. Context token budget saturates at adj=2 on 5B; adding k=20 floods it. - Update complete/in-flight/queued with today's timeline, adj1_llm final (0.458, not 0.464 partial), qwen9b numbers, qwen9b_k20_thinking_on queued as the post-POSTMATRIX control run. - Note PR #42 (think=false on generator, 20x speedup), PR #43 (revert think=false on judge after the 1452 silent-zero bug), PR #44 (--thinking-mode opt-in flag). * docs(specs): adj2_full_stack_qwen9b 0.545 — new leader, parity with audited Zep Today's key landings: - adj2_full_stack_qwen9b: 0.545 — overall leader, +0.029 over adj=2 alone at 9B, +0.046 over the previous adj=1+stack 9B leader. - adj2_qwen9b: 0.516 — adj=2 alone at 9B (logged earlier today). - c6_multihop_qwen9b: 0.306 — multihop regression worsened at 9B (was 0.317 at 5B). Footgun confirmed across all model sizes. - qwen35_9b_full_context: 0.090 — retrieval ablation. Full conversation in context collapses to slightly above mem0 floor. Empirical proof that retrieval is essential, not just a context-window workaround. Headline revision (3rd this week): stacking is adj-dependent AND model-size-dependent. 5B + adj=2 + stack regresses (-0.022); 9B + adj=2 + stack compounds (+0.029). Smaller model attention saturates earlier; bigger model can absorb wider retrieval surface even at adj=2. Tier crossover: 0.545 is within 0.04 of audited Zep (0.584) on gpt-4o-mini. Functional parity on a local 12 GB GPU + 9B quant. Mem0 paper (0.66) and Mem0^g (0.68) remain ahead — both reported by mem0's own harness, not independently audited.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Measured impact
Probe on `qwen3.5:9b` with a trivial 50-token prompt ("What is 2+2?"):
`/no_think` prefix doesn't work on this Ollama — it's stripped from output but the model still thinks.
Projected impact on the running chain
qwen9b_k20 was at 24% after 16h (rate ~1.3 tok/s due to thinking). With `think: false` the rate should be ~20 tok/s → ~3-4h total for 1540 QAs. Same multiplier applies to every remaining qwen3.5:9b run plus every qwen3:4b rescore in the queue.
Test plan