docs: multilevel_full_stack 0.552 + Kinthai retention suggestions captured - #56
Merged
Merged
Conversation
…ions - Update LoCoMo same-tier leaderboard with multilevel_full_stack at 0.552. Adding multi-level retrieval on top of the rrf_full_stack leader (0.557) regresses by -0.005, confirming the architectural plateau at the 9B + adj=2 + retrieval-augmented tier. - Capture Kinthai retention-scoring suggestions (issue #182 comment 4337347476) as a future-work spec rather than half-implementing them. The suggestions target taosmd/retention.py's consolidation pipeline (Light/REM/Deep), which is NOT exercised by the LoCoMo benchmark — retention weights have zero effect on retrieval at query time. Running LoCoMo with modified weights would burn ~5h to measure no change. The right validation path is the supersede + routing benchmarks already designed (2026-04-16-*), which exercise long-horizon memory management. Spec note documents what would need to ship to test these ideas honestly.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR updates the LoCoMo leaderboard documentation with explicit comparison rows for RRF variants and adds a new specification document recording retention-scoring weight suggestions from external feedback on an open issue. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~4 minutes Possibly related PRs
Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This was referenced May 5, 2026
jaylfc
added a commit
that referenced
this pull request
May 30, 2026
…80) Closes task #56. Lifts cognee's LoaderInterface ABC (66 lines at cognee/infrastructure/loaders/LoaderInterface.py) with memobase's discriminated-union envelope (BlobType: chat | transcript | email | doc) on top. taosmd/loaders/ __init__.py — package exports interface.py — LoaderInterface ABC (loader_name, supported_extensions, supported_mime_types, can_handle, async load -> Blob) blob.py — typed envelopes: Blob (base), ChatBlob, TranscriptBlob, EmailBlob, DocBlob. Stdlib dataclasses only — no pydantic dependency to keep Pi-tier dep matrix tight. chat_loader.py — JSON list / {messages: [...]} -> ChatBlob. Multi-suffix '.chat.json' / '.messages.json' extension matching. transcript_loader.py — Whisper segments / canonical TranscriptStamp list / plain list -> TranscriptBlob. JSON only (VTT/SRT deferred). email_loader.py — RFC 5322 .eml via stdlib email -> EmailBlob. Threading headers (Message-Id, In-Reply-To) preserved for downstream thread-tree work. doc_loader.py — text / markdown -> DocBlob. Markdown title extracted from first '# ' heading. registry.py — pick_loader(path) -> LoaderInterface. First registered loader whose can_handle() returns True wins; DocLoader catches all. Scope-honest note: this PR ships the abstraction. process_conversation_turn and the existing string-based ingest paths keep working unchanged — nothing migrated to the new path. Migration is a separate refactor (swap call sites once we have a real consumer driving the choice). Test plan - 20 new tests in tests/test_loaders.py exercising each loader against its expected file format, the registry's path-extension extraction (multi-suffix support), pick_loader for each blob type + the unknown- extension fallback to DocLoader, register_loader's insert-before-doc ordering, and an end-to-end pick+load. - All 192 total tests pass (172 before + 20 new).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two updates:
Add multilevel_full_stack_qwen9b_adj2 at 0.552 to the leaderboard. Adding multi-level on top of the 0.557 leader regresses by -0.005 — confirms the 9B + adj=2 ceiling.
New spec doc capturing Kinthai's retention-scoring suggestions from tinyagentos#182. The suggestions don't transfer to the LoCoMo benchmark (retention weights aren't consulted at retrieval time); proper validation needs the supersede/routing benchmarks already designed but not yet shipped. Documented honestly instead of speculatively implementing.
Summary by CodeRabbit