From b34123b3d825492a0a350fcb591ab8899b62869a Mon Sep 17 00:00:00 2001 From: jaylfc Date: Tue, 28 Apr 2026 22:16:04 +0100 Subject: [PATCH] docs: multilevel_full_stack 0.552 + capture Kinthai retention suggestions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Update LoCoMo same-tier leaderboard with multilevel_full_stack at 0.552. Adding multi-level retrieval on top of the rrf_full_stack leader (0.557) regresses by -0.005, confirming the architectural plateau at the 9B + adj=2 + retrieval-augmented tier. - Capture Kinthai retention-scoring suggestions (issue #182 comment 4337347476) as a future-work spec rather than half-implementing them. The suggestions target taosmd/retention.py's consolidation pipeline (Light/REM/Deep), which is NOT exercised by the LoCoMo benchmark — retention weights have zero effect on retrieval at query time. Running LoCoMo with modified weights would burn ~5h to measure no change. The right validation path is the supersede + routing benchmarks already designed (2026-04-16-*), which exercise long-horizon memory management. Spec note documents what would need to ship to test these ideas honestly. --- docs/benchmarks.md | 3 +- ...026-04-28-kinthai-retention-suggestions.md | 45 +++++++++++++++++++ 2 files changed, 47 insertions(+), 1 deletion(-) create mode 100644 docs/specs/2026-04-28-kinthai-retention-suggestions.md diff --git a/docs/benchmarks.md b/docs/benchmarks.md index a67ad5a9..4e34bcfb 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -56,7 +56,8 @@ Harness: [`benchmarks/locomo_runner.py`](../benchmarks/locomo_runner.py). Rescor | System | Generator | Retrieval config | Ext Judge | Notes | |---|---|---|---|---| | **taosmd** | qwen3.5:9b | k=20 + adj=2 + llm-exp + RRF | **0.557** | **leader — full stack with RRF fusion** | -| **taosmd** | qwen3.5:9b | k=20 + adj=2 + llm-exp | 0.545 | previous leader (no RRF) | +| **taosmd** | qwen3.5:9b | k=20 + adj=2 + llm-exp + RRF + multi-level | 0.552 | adding multi-level on top of leader regresses by -0.005 | +| **taosmd** | qwen3.5:9b | k=20 + adj=2 + llm-exp | 0.545 | full stack without RRF | | **taosmd** | qwen3.5:9b | adj=3 | 0.532 | broader context window | | **taosmd** | qwen3.5:9b | adj=2 + multi-level retrieval | 0.524 | turns + summaries + events | | **taosmd** | qwen3.5:9b | adj=2 + RRF only | 0.500 | RRF without scaffolding regresses at 9B | diff --git a/docs/specs/2026-04-28-kinthai-retention-suggestions.md b/docs/specs/2026-04-28-kinthai-retention-suggestions.md new file mode 100644 index 00000000..137be9db --- /dev/null +++ b/docs/specs/2026-04-28-kinthai-retention-suggestions.md @@ -0,0 +1,45 @@ +# Kinthai retention-scoring suggestions — captured for future-work evaluation + +External feedback from `@kinthaiofficial` on tinyagentos issue #182 (memory consolidation), 2026-04-28. Three suggestions for `taosmd/retention.py`'s consolidation/decay scoring weights. + +## The suggestions + +1. **Bump `relevance` weight from 0.30 → 0.35–0.40.** Rationale per Kinthai: relevance is the single best predictor of whether a memory will be useful in future sessions. + +2. **Drop `frequency` weight from 0.24 → 0.15–0.20.** Rationale: high-frequency memories are often trivial (greetings, common phrases) — over-weighting frequency promotes noise. + +3. **Add a negative signal for retrieved-but-unused memories.** Rationale: catches the "seemed relevant in retrieval, agent didn't actually reference it in output" failure mode. + +## Why we haven't tested these in LoCoMo (and shouldn't) + +The LoCoMo benchmark exercises **retrieval-augmented QA over freshly-ingested conversations**. Retrieval at query time uses vector similarity + keyword fusion + cross-encoder rerank. The retention/consolidation scoring weights are **not consulted** during retrieval — they only affect the nightly consolidation pipeline (Light/REM/Deep) which decides what stays "hot" in long-term memory. + +LoCoMo's structure: each conversation's turns are ingested fresh, queries run immediately. Nothing has decayed; nothing has been promoted; nothing has been evicted. Changing weights in `retention.py` has zero effect on the LoCoMo numbers. + +Running a LoCoMo bench with modified weights would burn ~5h to measure no change — exactly the kind of "blindly add" we want to avoid. + +## What would actually validate these suggestions + +A benchmark that exercises long-horizon memory management: + +- **Supersede chains** (already designed in `docs/specs/2026-04-16-supersede-benchmark-design.md`) — tests temporal-reasoning over contradictory facts across time. Retention's recency + relevance weighting is directly relevant. +- **Multi-store routing** (already designed in `docs/specs/2026-04-16-routing-benchmark-design.md`) — tests promote/demote decisions across vmem, KG, archive. Retention scoring weights determine what gets promoted. +- **Live taOS user data** (when a long-running deployment accumulates real-world session history). Closest to Kinthai's "future sessions" evaluation framing. + +For the *negative-signal-on-unused* idea specifically: requires usage-tracking instrumentation we don't have today. A retrieved memory needs to be flagged in the response as "actually referenced" vs "ignored". Implementation paths: + +- Per-question post-processing: ask the judge "did the answer rely on this passage?" — expensive (extra judge call per retrieved chunk). +- Token-overlap heuristic: count overlap between retrieved-passage tokens and predicted-answer tokens; high overlap = referenced. Cheap, noisy. +- Attention-extraction from a generator that exposes attention weights — accurate but limited to local generators with that hook. + +## Decision + +- **Capture the suggestions here**, don't ship them blindly. +- **Revisit when** either the supersede or routing benchmarks ship, OR when we have multi-day taOS user data to ablate against. Kinthai's framing ("predicts future-session usefulness") needs a benchmark that actually has multiple sessions per memory. +- The `negative-signal-on-unused` idea is the most architecturally interesting; if we ever build a usage-tracking layer, that benchmark would need to come with it. + +## Provenance + +- Comment ID: 4337347476 +- URL: https://github.com/jaylfc/tinyagentos/issues/182#issuecomment-4337347476 +- Linked external write-up: https://blog.kinthai.ai/why-character-ai-forgets-you-persistent-memory-architecture