diff --git a/blueprints/memory-layer-s3/PREREGISTRATION.md b/blueprints/memory-layer-s3/PREREGISTRATION.md new file mode 100644 index 000000000..6968b7d87 --- /dev/null +++ b/blueprints/memory-layer-s3/PREREGISTRATION.md @@ -0,0 +1,1268 @@ +# Memory layer S3: head-to-head preregistration (DRAFT r7, 2026-09-30) + +**Status: DRAFT r7, not frozen.** +- r7 applies the coordinator's r7 brief of 2026-09-30, which relays the user's standing requirements of that date (quoted in "r6 → r7 changes"). It removes r6's reference arm and replacement routes in favour of a symmetric merit rule (sections 6–7), adds a maintenance gate (section 1), a token ledger (5.2) and a RAG section (8), and re-pins every system at its current release (3.1). +- **r7 repair round 1 (2026-09-30)** applies the GPT-6 cross-family review of this draft (verdict needs_changes, seven findings; [`inputs/r7/gpt6-review-20260930.md`](inputs/r7/gpt6-review-20260930.md)). The resolutions are listed under "r7 repair round 1" below. The text stays DRAFT r7. +- **r7 repair round 2 (2026-09-30)** applies the independent Opus review of `a266a383`, GPT-6's subsequent N1 finding, and the user's two accepted decisions: LongMemEval-V2 is the main usefulness test, and a host without a qualified winner selects the best-scoring eligible system (5.5, 7.6). Local pinned-source observations and separately labelled synthetic checks are in [`inputs/r7/repair2-verification-20260930.json`](inputs/r7/repair2-verification-20260930.json). +- **r7 repair round 3 (2026-09-30)** applies the pre-freeze review of `0d027e06` and the accepted dependency-maintenance scope. Local source verification and synthetic positive/negative controls are in [`inputs/r7/repair3-verification-20260930.json`](inputs/r7/repair3-verification-20260930.json). Landscape completeness was deferred by that brief; the pre-freeze candidate additions below supersede that deferral for the named candidates. This round makes no network request or model run. +- **r7 pre-freeze candidate additions (2026-09-30)** add four memory primaries, HippoRAG 2 for document RAG and the named reserves, with pinned admission screens and pending deployment requirements. The additions remain DRAFT r7, subject to section 11's freeze acceptance. +- This is a protocol revision. It made no deployment, provider call or scored run. Its live checks are metadata and source reads (GitHub, PyPI, Hugging Face, container registries, `nvidia-smi`, client `--version`), captured in [`inputs/r7/live-verification-20260930.json`](inputs/r7/live-verification-20260930.json) and, for the repair round, [`inputs/r7/repair1-verification-20260930.json`](inputs/r7/repair1-verification-20260930.json). The pre-freeze discovery and gate captures are [`inputs/r7/landscape-memory-20260930.json`](inputs/r7/landscape-memory-20260930.json) and [`inputs/r7/landscape-foldin-gate-20260930.json`](inputs/r7/landscape-foldin-gate-20260930.json). +- The freeze needs **both the user's acceptance and a GPT-6 cross-family review** of the complete bundle (section 11). Until then nothing runs under this protocol. +- Once frozen, it is never edited; later changes are dated amendments. A result that misses a bar is a documented FAIL, never a reason to loosen the bar or rerun silently. +- The r1–r6 review history is kept unchanged in Appendix A, with r6's section numbers. Decision record: [`docs/decisions/2026-09-30-memory-s3-r7.md`](../../docs/decisions/2026-09-30-memory-s3-r7.md). + +**Decision this experiment makes:** which durable-memory system each host deploys for Claude Code and Codex, and, per role, which document-RAG system and local embedder and reranker it uses (section 8). Client versions are recorded at freeze; on 2026-09-30 `claude --version` returned 2.1.285 and `codex --version` returned 0.157.1. The two hosts decide separately: +- the workstation `nativestack-5975wx-20260925`; +- the Mac `mac-coordinator-64gb-20260925`. + +**Merit rules:** +- no system has protected status; the deployed ai-memory is one contender among the others, and r6's reference role is gone; +- license, stars, popularity, installed or incumbent status are not evidence (the user's 2026-09-26 verdict-wave direction, retained from r6); +- every repository selected to run passes the maintenance gate; transitive dependencies require an SBOM and a clean OSV scan, with stale dependencies flagged (section 1); +- a missing result is `pending`, never `refuted`, the rule [`2026-09-23-verdict-integrity.md`](../../docs/decisions/2026-09-23-verdict-integrity.md) applies to missing lanes. + +## r6 → r7 changes + +**The user's requirements, 2026-09-30,** as relayed verbatim by the coordinator's r7 brief (this revision did not hear them directly): +1. The memory layer uses “the best sota repos, with highest quality performance, rag and token efficiency for full memory foundation”. +2. Repositories must be “the best candidates only with evidence, always check the repos quality, active maintenance, NEVER STALED REPOS”. +3. “No bias but evaluate on the sotaness and the quality of repos itself”; the current incumbent is not the default. + +| # | r7 change | Sections | Sources | +|---|---|---|---| +| 1 | Maintenance gate: stale means archived or no default-branch commit in 90 days; selected repositories are checked at freeze and before every run; transitive dependencies receive OSV scans, SBOMs and stale flags; LongMemEval `9e0b455f` is a citation only | 1, 4.2 | OpenSSF Scorecard v5.5.0 Maintained check (section 1); accepted user decision 3; repository CI OSV recipe (1) | +| 2 | Every system at its current release, with tag and full commit; ai-memory v2.4.2 is a contender with no protected status; reserves engram, Graphiti and supermemory-local; Mem0, Letta and beads are out on fit, with locators | 3 | 3.1–3.3 | +| 3 | Harness: AMB `03c1d0f1` runs byte-identical; our LongMemEval adapters (retrieval and open-QA variants), scorer and thin provider clients live in this repository; AMB's locked mem0, cognee and hindsight libraries serve no arm | 4 | AMB locators in 4.3–4.4 | +| 4 | A symmetric merit rule replaces the reference, both replacement routes and the 14-hypothesis family: all-pairs Holm tests, the leader wins, a 0.02 practical-equivalence band, then tie-breaks on cost and on an exact repo-quality formula | 6, 7 | SciPy and statsmodels pins (6); Scorecard aggregate (7.4) | +| 5 | Token ledger: OmniRoute ingest usage per system and role, o200k recall payloads (gpt-tokenizer 4.0.0), native per-session usage from `claude -p` and `codex exec --json` in scratch config homes, latency p95 and resources | 5.2 | client docs and source in 5.2 | +| 6 | RAG section: a document-RAG head-to-head with each system wrapped as an MTEB `SearchProtocol` model, and embedder and reranker selection on MTEB 2.21.10 (LoCoMo, CoIR, English retrieval) over task sets disjoint from the evaluation set | 8 | MTEB 2.21.10 locators in 8 | +| 7 | Header DRAFT r7; the freeze needs the user's acceptance and a GPT-6 cross-family review | Status, 11 | the brief's item 7 | + +**Retained from r6, with cross-references and pins updated:** the GPT-6 role contract and gateway invariants (2.1–2.2), the upstream deployment and isolation rules (3.5, 4.6), the dataset, tracks and dependence clusters (4.1), the dispatch wrapper (4.3), the 96-hour feasibility rule with its prospective stratified subset (5.2, 6), the statistics seed, resample count and quantile rule (6), latency (5.3), the coding-agent acceptance and early lifecycle screens (5.4), the research-memory usefulness tasks (5.5) and the host rules (9). Source keys I, R, M, W, V, U and B in retained text resolve through r6 §14 (Appendix A). + +**Removed:** the C3′/C4′ reference and its preflight, the superiority and non-inferiority replacement routes, the 14-hypothesis family, the "no replacement qualified" label, any execution of LongMemEval's scorer or judge prompt, MTEB 2.21.8 (now 2.21.10) and gpt-tokenizer 3.4.0 (now 4.0.0, this repository's pin since #493). + +**Brief items adjusted on primary evidence**, each also in the decision record: +- (a) The retrieval variant has to run AMB's LLM-free `retrieval` mode; "rag mode only" governs the answer-generating variant (4.3). +- (b) AMB's `supermemory` provider cannot score a local supermemory server, so supermemory-local gets a thin client in this repository (4.3). +- (c) Mem0's plugin defaults to, rather than hard-codes, `https://api.mem0.ai`; the exclusion stands on the Platform routes it calls (3.3). +- (d) Free VRAM measured 12,726 and later 12,005 MiB, not about 13.7 GB (8.2). +- (e) `CoIR-team/coir` is also stale; CoIR enters only as MTEB task data (1, 8.2). +- (f) AMB's LongMemEval judge prompts have no abstention branch; the 30 abstention questions form their own reported stratum (5.5). + +**ACCEPTED by the user, 2026-09-30:** the best-scoring eligible-system fallback (7.6), LongMemEval-V2 as the main usefulness test with MemoryAgentBench secondary only where applicable and S8 as a do-no-harm check (5.5, 8.4), and the selected-repository versus transitive-dependency maintenance scope (1). These replace the minus-one floor, S8 usefulness coupling and round-2 executing-closure maintenance gate. Other coordinator choices still require bundle acceptance at freeze: the optional-stage rule and observable fallback preflight (2, 7.5), agentmemory's embedder (3.1), the priced-cost boundary (5.2, 7.3), repo-quality heuristics (7.4), RAG partition (8.1), common depth and failure accounting (4.5), and S8's route and future margin (8.4). The decision record distinguishes those choices from the three accepted decisions. + +### r7 repair round 1 (2026-09-30) + +A GPT-6 cross-family review of the initial draft (gateway `cx/gpt-6-astra`, effort max) returned **needs_changes** with seven findings. The coordinator reports that all 12 quoted source excerpts matched. The findings are captured verbatim in [`inputs/r7/gpt6-review-20260930.md`](inputs/r7/gpt6-review-20260930.md). This subsection records the original bounded repair round, before the separately issued round-2 brief. Its upstream reads went through `gh api graphql` and are captured in [`inputs/r7/repair1-verification-20260930.json`](inputs/r7/repair1-verification-20260930.json). + +| Finding | Resolution | Sections | +|---|---|---| +| **F1 (high)**: stale harbor-datasets task bundles could execute code | "Static data" excludes any executed or interpreted file, and harbor-datasets supplies nothing. A task-level provenance manifest, enforced before every trial, governs any lane that executes tasks. The S8 code-task stage regenerates SWE-bench Verified tasks through Harbor's maintained adapter, with every executable part from a gate-passing source, and stays `pending` until its dated amendment. | 1, 8.4 | +| **F2 (high)**: AMB's Hindsight provider turns native observations off; "equal candidate depth" undefined | Hindsight gets a thin local provider like every other system, and AMB stays byte-identical. Banks keep the vendor's documented defaults; the effective bank configuration is read back and matched to the frozen settings before any ingest, and ingest ends only after consolidation. One operational depth policy covers every provider. The parity audit exercises every provider's actual ingest and recall paths, failures included. | 2, 3.5, 4.3, 4.5 | +| **F3 (high)**: tie-break membership undefined when several systems share the maximum | The maximizer set M is defined. A non-maximizer joins B only within 0.02 of the maximum and only if Holm finds it significantly worse than no leader. Comparisons use exact rationals, with frozen executable examples. | 7.2, 7.3 | +| **F4 (medium)**: gross versus control-subtracted cost | C is the gross, nonnegative priced cost of 5.2's frozen workload; the difference from the no-memory control is reported separately. Several equal minima, zero included, go to 7.4. C_RAG is defined for 8.3. | 5.2, 7.3, 8.3 | +| **F5 (medium)**: the fallback preflight mixed concurrency K with concurrency 1 | Two measurements: the fallback rate under the confirmatory dispatch at K, with a stated denominator, which alone controls stage disabling; and warm latency at concurrency 1. | 7.5 | +| **F6 (medium)**: the CI term could score unfinished runs | K = 10 needs the complete, paginated check and status set, with every run completed and passing. Missing, incomplete or truncated sets give K = 0. R and K are labelled coordinator heuristics. | 7.4 | +| **F7 (medium)**: no minimum usefulness for deployment | Round 1 coupled selection to S8 non-inferiority. **Superseded in round 2:** the accepted main LongMemEval-V2 rule supplies usefulness, and S8 is only the separate short-code do-no-harm check. | 5.5, 7.1, 8.4 | + +**Corrections and clarifications made with these fixes:** +- r7 said AMB's locked `hindsight-all` 0.4.17 serves no arm, but AMB's `hindsight-http` provider builds its synchronous client from it (`from hindsight import HindsightClient`, `memory/hindsight.py:1252-1255`). With the thin provider, the statement now holds (4.3). +- C counts only 5.2's frozen workload. LongMemEval ingest usage is reported in ledger (a) but never enters C; r7 had left the workload implicit. +- At the 7.3 boundary, a member whose C is exactly c_min / 0.80 is dominated, as r6's selection condition c_min ≤ 0.80 × C implies. r7's wording had also kept it in the tied set. + +**Round-1 residuals, historical:** the old 0/25 usefulness gap and coupling to pending S8 are superseded by repair round 2. S8's design, margin, test and relayed 36-task receipt, and execution of the depth and effective-bank checks, remain outstanding before qualification in their respective lanes. + +### r7 repair round 2 (2026-09-30) + +The repair brief reports Opus's **needs_changes** review of `a266a383` and GPT-6's **accept_with_minor** re-check with new finding N1. This revision resolves their protocol defects using the local upstream checkouts at the pins; it makes no network request, deployment or model run. The source receipt records the clone identities and corrections. Section 7.5's invalid-score example is a synthetic provider-boundary check, not a completed real gateway experiment. Native acceptance remains required before scoring. + +| Finding | Protocol resolution | Sections | +|---|---|---| +| H1 | Exhausted question-seeds score zero against the frozen manifest; failure rate above 5% makes the arm pending. | 4.5 | +| H2 / F7 | Paired, multi-seed, Holm-adjusted LongMemEval-V2 usefulness replaces the minus-one floor and S8 coupling. | 5.5, 6, 7.1, 8.4 | +| M1 | Best-scoring operationally eligible system is the no-winner fallback; no incumbent privilege. | 7.6 | +| M2 | Round 2 gated the executing closure. **Superseded by accepted user decision 3:** selected repositories pass Maintained; transitive dependencies require OSV scans/SBOMs, with stale flags. Runtime egress remains denied except owned local services, gateway and embedder. | 1, 4.3, 4.6 | +| M3 | Both thin-provider recall methods default to k=20; every provider returns raw_response=None, giving the same answer-prompt context shape. | 4.3, 4.5 | +| M4 | Unique output directory per host, lane, track, question, seed, arm and attempt; analysis reads the exact native split file. | 4.3 | +| M5 | All providers expand and deduplicate to five source sessions, escalating to a frozen cap by one common rule. | 4.5 | +| M6 | Header-limited clients need a source-backed, measured exemption or frozen route; agentmemory and MemPalace currently lack that receipt and remain pending. | 2.2, 4.4 | +| M7 | Selection must fit the embedder, reranker and local generative models together under serving load. | 8.2, 9 | +| L1 | Local inference is unpriced; the gross-C cost boundary and its sensitivity are explicit at acceptance. | 5.2, 7.3 | +| L2 | cx/GPT-6 aliases are unpriced until a defensible price mapping exists; no zero or partial-total C is invented. | 5.2, 7.1 | +| L3 | Empty-context QA scores are unjudged zeros, separately flagged; provider-facing question and user IDs are opaque. | 4.1, 5.5 | +| L4 | Prospective cost/power preflight sets usefulness and QA subsets; identical retrieval ingest may be reused only from a verified pre-query snapshot. | 5.2, 5.5 | +| N1 | Every stage request needs an accepted/fallback outcome, including invalid scores after gateway success; unobservable outcomes leave the arm pending. | 7.5 | + +Deployment corrections for all eleven systems are summarized in 3.5. The source review establishes recipes and measurement requirements, not a host's passed status. + +### r7 repair round 3 (2026-09-30) + +This round verifies the final review's source claims in the supplied pinned local clones. It corrects the protocol and executable decision examples; it does not execute a benchmark, refresh online maintenance metadata, download haystacks or perform a vulnerability scan. The corrections and their verification paths also appear in the decision record's anti-pattern log. + +| Item | Protocol resolution | Sections | +|---|---|---| +| 1 | Every LongMemEval-V2 arm, including no retrieval, uses the same direct native harness invocation per domain, with identical reader/judge arguments and request-body wire checks. | 5.5 | +| 2 | Outcome-independent shared-trajectory haystack components are the resampling unit; the downloaded-file preflight must find at least 20 clusters, or usefulness stays pending. | 5.5, 6, 7.1 | +| 3 | Open QA receives the shortest prefix of the final ranking covering five sessions, capped at 20 items; emitted context tokens are recorded per arm. | 4.3, 4.5 | +| 4 | MemoryAgentBench is secondary only for applicable native modes; the per-system table excludes vendored candidate copies. | 5.5 | +| 5 | The executable usefulness rule computes the best set only over eligible, completely observed memory systems. | 7.1 | +| 6 | Fallback remains unresolved while a pending contender could change qualification or the selected primary maximizer. | 7.6 | +| 7 | A no-retrieval final-failure rate above 5% makes the entire usefulness family pending. | 4.5, 7.1 | +| 8 | Judge-effort, privacy and question-ID locators are corrected in this protocol, the decision record and the source receipt. | 5.5; round-2 receipt | +| User decision 3 | **ACCEPTED:** Maintained applies to selected repositories; dependencies pass OSV and appear in each arm's SBOM; stale transitive dependencies are report flags. | 1, 4.3 | + +### r7 pre-freeze candidate additions (2026-09-30) + +The coordinator makes these changes to the unfrozen draft under the user's 2026-09-30 instruction that every layer's candidates be the current best maintained repositories, evaluated on merit with no incumbent default. These are pre-freeze additions, not admissions through 3.2's late-candidate rule, which "never edits this text". Discovery and maintenance metadata are captured in section 1; deployment sources were read, not executed. The complete bundle still requires section 11's freeze acceptance. + +| Item | Change | Sections | +|---|---|---| +| 1 | Add claude-mem v13.28.0, MCP Memory Service v11.14.0, MemMachine v0.3.9 and ReMe v0.4.1.12 as primaries at their latest GitHub releases resolved to commits; each passed the official local-install and both-client admission screen. Add their source-reviewed deployment recipes and required corrections/pending items. | 1, 3.1, 3.5 | +| 2 | SimpleMem v0.3.0 is out on fit: no official Codex setup at the release or screened main head; re-screen when a release supplies it. | 3.3 | +| 3 | Retain Mem0's exclusion despite the landscape's "Mem0 OSS core" primary recommendation: the official Codex plugin calls Platform routes, and the pinned tree supplies no self-hosted OpenMemory directory. | 3.3 | +| 4 | Add HippoRAG 2 to document RAG, with a dated default-branch pin exception because the GitHub release is HippoRAG 1 and PyPI predates maintained fixes; freeze a retrieval-only SDK wrapper. Seven document-RAG candidates give all 21 Holm pairs. | 8.3 | +| 5 | Append memory reserves 4–9: SuperLocalMemory, memU, AgentMem, EverOS, MemOS and MemoryOS, with the ordering basis and release identities; AgentMem needs a release or dated head-pin decision before screening. | 3.2 | +| 6 | Add RAGFlow as a document-RAG reserve. GitNexus and Serena stay outside S3, which has no code-retrieval tool arm. Serena's measurement in the separate Harbor token-tool E2E and GitNexus's absence from that run are relayed; the Harbor receipt is required before freeze. GitNexus is a candidate for the next round. | 8.3, 8.4 | +| 7 | MemMachine and MCP Memory Service leave r6's lower-priority list for the primaries; SimpleMem moves to exclusions; Honcho and the pending names retain the late-candidate route. | 3.2, 3.3 | +| 8 | Record the coordinator's pre-freeze authority and distinction from late-candidate admissions. | change log; decision record | +| Counts and power | Twelve memory primaries give m = 66 and m_U = 78. The all-system ingest scenario is 3.243B tokens per host, 6.486B across both hosts. Strictest-step power for a 0.05 difference is 0.54/0.18/0.08 at discordance 0.10/0.20/0.30; for 0.02 it is at most 0.02. Last-step figures remain 0.93/0.68/0.51 and at most 0.28. Route (a)'s full-scenario process starts become 470 × 5 × 12 = 28,200 per host. | 2.2, 5.2, 6 | + +## 1. Maintenance gate (r7) + +**Rule.** A repository is **stale** if GitHub reports it archived, or if its default branch has no commit in the 90 days before the check. This is the floor of OpenSSF Scorecard's **Maintained** check: Scorecard gives an archived project the minimum score and otherwise scores commit and collaborator issue activity over a 90-day look-back, with one activity per week earning the maximum ([`checks/evaluation/maintained.go:31-32,88-100`](https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/checks/evaluation/maintained.go#L31-L100) and [`docs/checks.md:400-422`](https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/docs/checks.md#L400-L422) at v5.5.0). Scorecard itself suggests checking the `archived` probe instead of relying on an aggregate or Maintained score alone ([`README.md:86-92`](https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/README.md#L86-L92)). The gate is necessary, not sufficient: a project created less than 90 days ago passes it but gets Maintained 0 (`maintained.go:92-94`), and the full Maintained score enters the repo-quality score (7.4). + +**Commands**, with `SINCE` the check time minus 90 days in UTC ISO-8601; `true` from the first or `0` from the second marks the repository stale: + +```sh +gh api repos/OWNER/REPO --jq .archived +gh api "repos/OWNER/REPO/commits?since=SINCE&per_page=1" --jq length +``` + +When the account's REST quota is exhausted, `gh api graphql` gives the same verdict from the repository's `isArchived` field and its default branch's `history(since: SINCE) { totalCount }`; repair round 1 used that route ([`inputs/r7/repair1-verification-20260930.json`](inputs/r7/repair1-verification-20260930.json)). + +**When.** At freeze, and before each development and confirmatory run on each host. A repository that becomes stale stops executing from that point; results it already produced stay recorded as they are, and the change needs a dated amendment. + +**Consequences.** +- **Selected repositories — ACCEPTED 2026-09-30, user decision 3.** The 90-day Maintained rule excludes any stale repository we choose to run: harnesses, memory systems, RAG systems and tools, including selected runtimes, model servers, sidecars and task-code sources. Selection is by role, not by whether it is launched directly or through an adapter: a candidate implementation cannot be relabelled a dependency to evade the gate. No selected stale scripts, evaluators, prompt code or packages run. A pinned release still requires its source repository's current gate receipt. +- **Transitive dependencies — ACCEPTED 2026-09-30; replaces round 2's executing-closure/no-shim maintenance rule.** Imported libraries, native extensions and other dependencies of the selected implementations pass **osv-scanner with no known vulnerability** and are listed in an **SBOM per arm**, including harness import-time dependencies and installed but unused packages. Use the repository CI's checksum-pinned **google/osv-scanner v2.6.0**, `scan source --no-resolve` on resolved lockfiles (`.github/workflows/security-scan.yml:46-86`); freeze scanner identity, lock hashes, advisory snapshot/time, complete coverage and returned findings/exit status. S3 requires no known vulnerability, including any finding a CI-specific suppression would hide. Missing locks, coverage, SBOM or scan observation leaves the arm/harness pending; a known vulnerability blocks it until resolved. Staleness alone is a report flag and does **not** block the harness. Each SBOM entry records version, artifact hash, source repository/revision when known, execution class and vulnerability status; unknown provenance/activity remains unknown in the report. Neither a patched import shim nor a vendored candidate substitutes for the selected implementation. Fetch/build networking belongs to separately recorded provisioning; scored runtime egress remains governed by 4.6. +- **Stale transitive flags found so far.** AMB imports `rank_bm25` **0.2.2** at package load (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/__init__.py:2` → `bm25.py:1`). The supplied `dorianbrown/rank_bm25@47aa3ddf8dc1ebeb7ef4e65f2b4536af44594099` checkout's last commit is **2024-10-08**, beyond 90 days; flag it as stale in the AMB arm SBOM/report. This local observation does not refresh archival/default-branch metadata, and no clean OSV scan is claimed. The stale flag itself does not block AMB. The other lock entries are not labelled stale from version age alone; their vulnerability/provenance inventory remains pending. +- A stale source may supply static data only: a file pinned by the sha256 of its downloaded bytes and read by code from active repositories or from this one. **Static data is inert input that is only read (r7 repair).** A file that is executed, sourced, built or interpreted, such as a Dockerfile, a setup or build script, a test, a judge or grading script or a solution script, is code wherever it is hosted, and a bundle that contains one is not static data. Registry metadata, such as a Hugging Face LFS oid, is recorded but never replaces the downloaded-file hash. +- **LongMemEval `9e0b455f` appears only as a citation**: for the paper's definitions, the dataset format and the reference implementation that 4.2 reimplements. The dataset file is static data pinned by sha256 (4.1). +- LoCoMo enters only as MTEB data (`mteb/LoCoMo`, revision `02e2c3dea15d9fdfd1cd7a0f65f5f8ae2ed4c1ac`, read by MTEB); CoIR only as MTEB's CoIR tasks (8.2). +- **harbor-datasets supplies nothing (r7 repair).** Its task bundles carry executable tests and judges: [`datasets/aa-lcr/aa-lcr-1/tests/test.sh:4`](https://github.com/harbor-framework/harbor-datasets/blob/37db108843a49bb31a592e37a75e2c40dc3f9749/datasets/aa-lcr/aa-lcr-1/tests/test.sh#L4) at `37db1088` runs `python /tests/llm_judge.py`, and Harbor's verifier uploads a task's tests and executes its test script ([`src/harbor/verifier/verifier.py:175-232`](https://github.com/harbor-framework/harbor/blob/1e5c5c6db929a10a140d05e606882c671ae20729/src/harbor/verifier/verifier.py#L175-L232) at v0.23.0). No bundle, file or data from it enters any S3 process; the S8 stage regenerates its tasks from maintained sources (8.4). +- **Task-level provenance manifest (r7 repair).** A lane that executes task bundles, today only 8.4's S8 stage, freezes one manifest per task. It lists every file of the generated task and every artifact the task fetches when built or run, such as the base image, installers and verifier dependencies. Each entry carries its sha256 or image digest, its class (inert input; environment or Dockerfile; setup or build script; test; judge or grader; solution), its source (repository and commit, or dataset and revision) and its selected-repository/dependency role under the accepted scope above. Selected executable sources need a current ACTIVE gate; transitive dependencies need SBOM/OSV coverage, with stale flags retained. Before every trial the runner re-hashes the task directory and checks the built image's digest; it refuses the trial if any file is unlisted or differs, or if its applicable maintenance/vulnerability requirement is not met. A refused task is recorded, and excluding a task after any result needs a dated amendment. + +**Results at 2026-09-30T02:52:42Z** (window start 2026-07-02T02:52:42Z). The full capture, with every head, release date and the 300-commit counting cap, is [`inputs/r7/live-verification-20260930.json`](inputs/r7/live-verification-20260930.json) (sha256 `c02d1c506d31164891d3f1782a55993d1d5ae61f9a5c4b9c7e285aa839e24f69`). "300+" means the count reached the cap. No repository below is archived. + +| Repository | Role in r7 | Last default-branch commit | Commits in window | Verdict | +|---|---|---|---|---| +| snap-research/locomo | LoCoMo origin; data only through MTEB | 2024-08-13 (`3eb6f2c5`) | 0 | **STALE** | +| xiaowu0162/LongMemEval | citation only | 2026-05-11 (`9e0b455f`) | 0 | **STALE** | +| harbor-framework/harbor-datasets (formerly laude-institute/harbor-datasets) | none; supplies nothing (1, 8.4) | 2026-05-16 (`37db1088`) | 0 | **STALE** | +| CoIR-team/coir | CoIR origin; data only through MTEB | 2025-06-30 (`89d0e769`) | 0 | **STALE** | +| vectorize-io/agent-memory-benchmark (AMB) | harness; the pin `03c1d0f1` is its head | 2026-09-22 | 12 | ACTIVE | +| embeddings-benchmark/mteb | harness | 2026-09-29 | 300+ | ACTIVE | +| ossf/scorecard | repo-quality score | 2026-09-29 | 22 | ACTIVE | +| niieani/gpt-tokenizer | payload counts | 2026-08-16 | 11 | ACTIVE | +| scipy/scipy; statsmodels/statsmodels; numpy/numpy | statistics | 2026-09-29; 2026-09-29; 2026-09-30 | 300+ each | ACTIVE | +| diegosouzapw/OmniRoute | GPT-6 gateway | 2026-09-29 (default branch `release/v3.8.52`) | 300+ | ACTIVE | +| harbor-framework/harbor | Harbor E2E harness (8.4) | 2026-09-30 | 300+ | ACTIVE | +| xiaowu0162/LongMemEval-V2 | main usefulness harness (5.5) | 2026-08-09 | 3 | ACTIVE | +| rohitg00/agentmemory; vectorize-io/hindsight; volcengine/OpenViking; topoteretes/cognee; MemPalace/mempalace; basicmachines-co/basic-memory; zilliztech/memsearch; akitaonrails/ai-memory | primary candidates (3.1) | 2026-09-24 to 2026-09-30 | 39; 300+; 300+; 300+; 300+; 300+; 79; 300+ | ACTIVE | +| Gentleman-Programming/engram; getzep/graphiti; supermemoryai/supermemory | reserves (3.2) | 2026-09-29 to 2026-09-30 | 300+; 116; 183 | ACTIVE | +| mem0ai/mem0; letta-ai/letta; gastownhall/beads (formerly steveyegge/beads) | out on fit (3.3) | 2026-09-25; 2026-09-10; 2026-09-30 | 236; 8; 300+ | ACTIVE | +| garrytan/gbrain | late-candidate screen (3.2) | 2026-09-29 | 300+ | ACTIVE | +| tobi/qmd; HKUDS/LightRAG; run-llama/llama_index | document-RAG candidates (8.3) | 2026-09-09; 2026-09-26; 2026-09-29 | 110; 300+; 102 | ACTIVE | +| DeusData/codebase-memory-mcp | code-retrieval fit record (8.4) | 2026-09-28 | 300+ | ACTIVE | + +**Repair-round rerun at 2026-09-30T04:23:48Z** (window start 2026-07-02T04:23:48Z), for the sources of 8.4's S8 stage, through `gh api graphql` ([`inputs/r7/repair1-verification-20260930.json`](inputs/r7/repair1-verification-20260930.json), sha256 `e17484588948dc7150afba464d4806924bc08ae7858a726bda5ef1b70c9a2b35`). None is archived. +- ACTIVE: harbor-framework/harbor (526 commits in the window), SWE-bench/SWE-bench (113), huggingface/datasets (62), AnswerDotAI/fastcore (262) and astral-sh/uv (1,218). +- STALE: harbor-framework/harbor-datasets, still with no commit in the window (head `37db1088`). + +**Round-2 local source observations, not a fresh online gate.** The supplied LongMemEval-V2 pin `2cc8c540` has commit date 2026-08-09; MemoryAgentBench `538026089` has commit date 2026-09-25. The brief designates them active under the 90-day rule; their archival/default-branch gate receipts are still refreshed at freeze. MemoryArena `6cd9de14` has last commit 2026-05-31 as relayed by the brief and confirmed by the supplied checkout's commit date: it is **STALE**, cited for ICML 2026 context only, and none of its code runs. Source receipt: `inputs/r7/repair2-verification-20260930.json`. + +**Pre-freeze additions gate at 2026-09-30T16:30:31Z** (window start 2026-07-02T16:30:31Z). The capture and admission screens are [`inputs/r7/landscape-foldin-gate-20260930.json`](inputs/r7/landscape-foldin-gate-20260930.json) (sha256 `bed16fa91e45d116d0ee968bb86b5e9c7fcd503eff820b1c0a92bc33c5a4ac65`). All 16 repositories are ACTIVE; none is archived. Counts retain the capture's `300+` notation. The discovery input is the GPT-6 landscape check at 2026-09-30T15:09:53Z, [`inputs/r7/landscape-memory-20260930.json`](inputs/r7/landscape-memory-20260930.json) (sha256 `8f877d85ea57421f1c50704b6a2f9e88b0f8e03e8f281e9f165e03b78cfd75b8`); its recommendations are discovery leads, with admission determined by the screen and the coordinator's decisions below. + +| Repository | Role in r7 | Last default-branch commit | Commits in window | Verdict | +|---|---|---|---|---| +| thedotmack/claude-mem; doobidoo/mcp-memory-service; MemMachine/MemMachine; agentscope-ai/ReMe | pre-freeze primary additions (3.1) | 2026-09-30 (`a077975e`); 2026-09-30 (`9079737b`); 2026-09-25 (`d57f5cb3`); 2026-09-30 (`81c16d3e`) | 300+; 300+; 28; 174 | ACTIVE | +| qualixar/superlocalmemory; NevaMind-AI/memU; JaceHo/AgentMem; EverMind-AI/EverOS; MemTensor/MemOS; BAI-LAB/MemoryOS | memory reserves 4–9 (3.2) | 2026-09-12 (`5bfa47c9`); 2026-09-21 (`2c050bc9`); 2026-08-31 (`0770f5d8`); 2026-09-30 (`6aff7351`); 2026-09-22 (`a7367d07`); 2026-07-07 (`587ed775`) | 300+; 133; 3; 83; 264; 2 | ACTIVE | +| aiming-lab/SimpleMem; mem0ai/mem0 | out on fit (3.3) | 2026-07-24 (`db80b6a7`); 2026-09-25 (`94c3fe9f`) | 14; 234 | ACTIVE | +| OSU-NLP-Group/HippoRAG | document-RAG candidate (8.3) | 2026-09-29 (`398bfdc3`) | 21 | ACTIVE | +| infiniflow/ragflow | document-RAG reserve (8.3) | 2026-09-30 (`9766c8c5`) | 300+ | ACTIVE | +| abhigyanpatwari/GitNexus | outside S3; Harbor token-tool candidate (8.4) | 2026-09-29 (`acb65b95`) | 300+ | ACTIVE | +| oraios/serena | outside S3; relayed Harbor token-tool E2E measurement, receipt required before freeze (8.4) | 2026-09-29 (`8a3ce35c`) | 300+ | ACTIVE | + +## 2. Systems, arms and the GPT-6 backbone + +There is **no reference arm**. Every system in the family, ai-memory included, has exactly **one primary arm**, defined by the same rule: +- its upstream-recommended self-hosted deployment for coding-agent memory at the release pinned in 3.1, installed as 3.5 requires; +- every active LLM role on GPT-6 through the host's own OmniRoute (2.1–2.2); +- a current local embedder and dedicated reranker selected by 8.2 wherever upstream supports configuring them; otherwise the shipped model, labelled **shipped default**; +- **optional-stage rule (r7):** where upstream documents an optional recall stage, such as an LLM or cross-encoder reranker, without recommending it for this use case, the stage setting is chosen per system on the development split by the frozen comparison of 2.1 (LoCoMo supporting-evidence recall@5 with a one-sided cluster bootstrap at α = 0.05) before any confirmatory run. An inconclusive comparison keeps the upstream default. The unselected setting is a diagnostic outside the family; +- **native store configuration (r7 repair):** each arm keeps the system's documented defaults for per-store settings, such as Hindsight's bank configuration. Neither the harness nor a provider client overrides them, and the effective settings are frozen and read back before ingest (4.3, 4.5). A departure from them is a frozen, labelled configuration choice made before freeze, never a harness side effect. + +**ai-memory v2.4.2 (r7).** ai-memory competes under exactly this rule. Its LLM reranker is "optional and off by default" and makes at most one LLM call per query to reorder candidates ([`docs/llm-providers.md:331-341`](https://github.com/akitaonrails/ai-memory/blob/a0ca8d1a5fbd5920799411fa891fe6d49c90efc1/docs/llm-providers.md#L331-L341); `AI_MEMORY_RERANKER=llm` is validated at [`crates/ai-memory-cli/src/config.rs:427-433,1762-1774`](https://github.com/akitaonrails/ai-memory/blob/a0ca8d1a5fbd5920799411fa891fe6d49c90efc1/crates/ai-memory-cli/src/config.rs#L427-L433)), so the optional-stage rule decides it. If selected, it runs through the openai-compatible provider on GPT-6 and is subject to the fallback preflight in 7.5. Its embedder follows 8.2 like any configurable system, through its `openai-compat` embedding provider (`docs/llm-providers.md:343-349`). r6's C3′/C4′ definitions and v2.4.0 feasibility notes are historical (r6 §1 in the r6 text at `f7e5e228`; review history in Appendix A). Of the ai-memory files r6 cited, `crates/ai-memory-llm/src/embedding.rs` and `reranker.rs` are byte-identical from v2.4.0 to v2.4.2; `config.rs` and `docs/llm-providers.md` changed (live capture, `changed_between_releases`), so r6's line references into those two files do not carry over. + +**Deployed configurations (descriptive only).** The workstation runs ai-memory 2.4.1 (this checkout's `manifests/stack.json`); the Mac runs its production build `19b6429` in a C4-shaped configuration (r6 §1, in the r6 text at `f7e5e228`). Any measurement of them uses fresh diagnostic copies with the GPT-6 overlay, labelled D-workstation-GPT6 and D-mac-prod-GPT6, outside the family. The v2.4.2 re-pin is the user's 2026-09-30 decision as relayed by the brief; the base's `manifests/stack.json` still lists 2.4.1. + +**Eligibility applies to every configuration** (7.1). + +### 2.1 GPT-6 role contract (r6 1.1, retained) + +Every active LLM role inside every scored system and diagnostic copy uses GPT-6 through **that host's own local OmniRoute**: extraction, retain/reflect, graph building, consolidation, summarization, LLM reranking, query rewriting/expansion, and any auxiliary or fallback LLM call. Deterministic operations and local embedding/cross-encoder inference remain local; an absent LLM role is recorded `not applicable`, not invented. A shipped local generative model that upstream offers no way to replace is recorded as a fixed-provider shipped default, not patched (for example qmd's query expansion, 8.3). The workstation endpoint is `http://127.0.0.1:20128/v1`. The Mac uses its own loopback gateway and LaunchAgent, on the same OmniRoute tree and patch IDs as the workstation (section 9). This supersedes “local-only providers” for **LLMs**; the memory service, stores, embeddings and non-LLM rerankers stay on the host. Sources: coordinator brief B:A/I; I:4–14,18–58; R:31–53; pinned gateway sources in 2.2 and host-receipt requirements in 9. + +- Freeze a **per-host, per-arm, per-role matrix** containing the GPT-6 model ID, effort, provider/endpoint, prompt and schema hashes, output/context caps, timeouts, retries and fallback routing before confirmatory execution. Use the same selected model/effort for a matched role across systems where supported. If upstream shares one provider across roles, freeze that coupling and test the combined configuration; do not claim independent overrides. No fallback may silently invoke a shipped older model. +- **R02 development selection:** compare `cx/gpt-6-astra` at medium with `cx/gpt-6-astra-max`, using paired, interleaved calls with identical prompts/schema on S3's separate development split. **Before the first development call**, freeze, per role or coupled role set and on **each host**: LoCoMo full-system question-weighted supporting-evidence recall@5 plus schema validity, question count **n**, conversation/cluster manifest, task/source hashes, and a **one-sided cluster bootstrap at α=0.05 using section 6's SciPy binding with `alternative="greater"`** (development cluster arrays replace the confirmatory arrays). Coupled roles are selected jointly. Freeze the schema-validity acceptance rule, then select max only when the paired quality lower bound is >0; otherwise select medium when its lower bound is ≥−0.02 and it is cheaper. Record complete usage, latency, timeouts and fresh 429s. A `cx/gpt-6-sol` medium alternative, or a newer GPT-6-family model current at freeze, needs the same prespecified comparison. The optional-stage choices in section 2 use the same split, metric and test. An arm with any unresolved role is **pending**, and needs a **dated amendment before its first confirmatory run**; there is no post-result role choice. The development split shares no question or canonical conversation content with either confirmatory LongMemEval track or the 25 usefulness tasks. The li26 wire check is not memory-quality evidence. **Answerer and judge are fixed separately at Astra max** (5.5), not selected on these results. Sources: R:25–29,31–53; W:83–97. +- Record requested and returned model identity and the gateway build at freeze and on execution. Recheck freshness before freezing; any later change to model/effort, gateway tree/patches or role policy requires a dated amendment, not an in-run substitution (r6 user directions 1–4; section 11). + +### 2.2 Gateway invariants for every scored GPT-6 call (r6 1.2, retained) + +These also apply to ingestion, retries, development comparisons and QA/judging, not just the final query. + +1. A role needing max sends the **`-max` model suffix**, e.g. `cx/gpt-6-astra-max` (cognee's LiteLLM spelling is `openai/cx/gpt-6-astra-max`). A chat/completions request without `reasoning_effort` runs at **medium**; the suffix wins over the body. Never use `auto`. Verify effective effort from the `call_logs` effort columns, including `reasoning_effort_upstream`; missing values are unverified, not inferred from the alias. These columns are populated only on rows with encrypted reasoning, so development wire checks and a value-free per-call audit must account for missing coverage. Sources: I:5–8; R:11,42,50; [OmniRoute executor at a58000c7](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/executors/codex.ts#L1424-L1465). +2. A **conversation is one question × seed × arm run**, including that run's ingest and recall; non-retrieval lanes substitute their frozen task/session ID. Give distinct runs distinct affinity identities and tag the arm separately. Allocate one stable `x-omniroute-session` key per conversation, never one per arm, where supported. `LLM_ARGS` and `HINDSIGHT_API_LLM_DEFAULT_HEADERS` are **process-static**, not per-call callbacks. **Route (a):** start one fresh arm process per run with that run's session key (470 × 5 × 12 = **28,200 starts per host** in the full, all-stochastic scenario, before deterministic reuse; official-track/QA/development extras are separate). **Route (b), Hindsight only:** omit the static session header and set `HINDSIGHT_API_{OP}_LLM_CACHE_AFFINITY=openai_prompt_cache_key` for the supported operations; this remains pending until wire checks show the emitted value gives the required stable within-run, distinct between-run gateway affinity. Neither route establishes dynamic idempotency headers. This corrects I:13/31 using R:7–11; an arm-wide constant pins its conversations to one account. Sources: [Hindsight config:176–182,403](https://github.com/vectorize-io/hindsight/blob/f8950b0c07d9e34c76493dba802bb309f0ce60fd/hindsight-api-slim/hindsight_api/config.py#L176-L182), [affinity implementation](https://github.com/vectorize-io/hindsight/blob/f8950b0c07d9e34c76493dba802bb309f0ce60fd/hindsight-api-slim/hindsight_api/engine/cache_affinity.py) (the same names sit at the same lines in 0.10.2, `5fc4ce20`; `cache_affinity.py` changed and is re-read before freeze); section 3.5. Native prompt-cache affinity remains separate from response-cache reuse. +3. Every call sends **`X-OmniRoute-No-Cache: true`** and is **dedup-ineligible**: `stream: true`, or a resolved temperature that is **absent or >0.1**, verified at the wire and against the frozen `maxTemperatureForDedup=0.1`. Explicit **1.0 is eligible for this exclusion**, just as omission resolves to 1.0 in the dedup gate; this corrects the original brief's stricter rule. Use a fresh `Idempotency-Key` per attempt where per-call headers exist; a static key is not fresh. The pinned [key composition](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/handlers/chatCore/idempotency.ts#L74-L110) combines raw key, provider/model and a digest of semantic fields, including messages: different bodies separate, but identical retries can replay. Therefore static-only adapters **omit both `Idempotency-Key` and the `x-request-id` fallback**, rather than sending a constant key; verify SDK-added headers and prove no replay in `call_logs` before scoring. [Header lookup and absent-key behavior](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/src/lib/idempotencyLayer.ts#L42-L55) and [chatCore:759–766](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/handlers/chatCore.ts#L759-L766) establish this choice. Sources: I:9–14,30–32; R:7–11; [cache gate](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/handlers/chatCore.ts#L1247-L1249), [dedup gate](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/services/requestDedup.ts#L205-L215). Header-limited clients, including AMB's answerer/judge, use 4.4's measured source-backed exemption or a frozen native per-arm route; otherwise they remain pending. +4. Keep OmniRoute compression **off**, with **`codex/*` excluded**; freeze and verify the byte-preserving request path. Sources: I:14; [compression dispatch](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/handlers/chatCore.ts#L1416-L1478). +5. Prove the actual request envelope on each pinned adapter before scoring, including nested/background calls and retries. Use upstream header/SDK controls where available; a header-limited client needs the source-backed, measured exemption or separate native route in 4.4, otherwise it remains `pending`. A body `EXTRA_BODY` field is not an HTTP-header control. A gateway HTTP 200 or a model alias alone is not this proof. Keep only sanitized request metadata and effort/usage observations; never read credential stores or retain authorization headers. + +**Gateway identity (r7).** r6 recorded "OmniRoute 3.8.51, tree `a58000c7`". The released `v3.8.51` tag resolves to a different commit, `c1e30b7676975feb298b49eff6ff58923c04b89e` (published 2026-09-30T01:41:50Z). Each host's own sanitized build receipt at freeze decides the gateway identity (section 9); the `a58000c7` links above remain source citations. + +## 3. Candidates, reserves, exclusions and deployments + +### 3.1 Primary candidates (the family) + +Each release below is the repository's latest GitHub release on 2026-09-30 (`releases/latest`), resolved to its commit through `repos/OWNER/REPO/commits/TAG`. The run manifest pins each to an immutable artifact and sha256 (3.5). + +| System | Release | Commit | Published (UTC) | r6 pin | +|---|---|---|---|---| +| [rohitg00/agentmemory](https://github.com/rohitg00/agentmemory) | [v0.9.29](https://github.com/rohitg00/agentmemory/releases/tag/v0.9.29) | `2d38dafede67d0d4ed920cde94d2106e98825b8a` | 2026-08-16 | same | +| [vectorize-io/hindsight](https://github.com/vectorize-io/hindsight) | [v0.10.2](https://github.com/vectorize-io/hindsight/releases/tag/v0.10.2) | `5fc4ce20917b916240cef27c212c387a177f115b` | 2026-09-29 | v0.10.1 (`f8950b0c`) | +| [volcengine/OpenViking](https://github.com/volcengine/OpenViking) | [v0.4.22](https://github.com/volcengine/OpenViking/releases/tag/v0.4.22) | `e8716760934d95ef73414484a45bfe475b9296e2` | 2026-09-28 | v0.4.21 | +| [topoteretes/cognee](https://github.com/topoteretes/cognee) | [v1.6.2](https://github.com/topoteretes/cognee/releases/tag/v1.6.2) | `ba3631f2ed363a6ea50d649c34c56885af6b36fe` | 2026-09-29 | v1.6.1 (`eb90d037`) | +| [MemPalace/mempalace](https://github.com/MemPalace/mempalace) | [v3.10.0](https://github.com/MemPalace/mempalace/releases/tag/v3.10.0) | `22fd87f09c19d5ffb2d6966486483353937931c0` | 2026-09-16 | same | +| [basicmachines-co/basic-memory](https://github.com/basicmachines-co/basic-memory) | [v0.23.2](https://github.com/basicmachines-co/basic-memory/releases/tag/v0.23.2) | `c0bd87c6d5a4a58034b1d6c8c5018e443b0bd048` | 2026-08-25 | same | +| [zilliztech/memsearch](https://github.com/zilliztech/memsearch) | [v0.4.21](https://github.com/zilliztech/memsearch/releases/tag/v0.4.21) | `2a4652fa086fbd45e92bfd8da7781ebe1642baa7` | 2026-09-24 | same | +| [akitaonrails/ai-memory](https://github.com/akitaonrails/ai-memory) | [v2.4.2](https://github.com/akitaonrails/ai-memory/releases/tag/v2.4.2) | `a0ca8d1a5fbd5920799411fa891fe6d49c90efc1` | 2026-09-29 | r6 reference v2.4.0 (`b1b25219`) | +| [thedotmack/claude-mem](https://github.com/thedotmack/claude-mem) | [v13.28.0](https://github.com/thedotmack/claude-mem/releases/tag/v13.28.0) | [`57c38303734482a596b3e2a052b969b30584be0d`](https://github.com/thedotmack/claude-mem/commit/57c38303734482a596b3e2a052b969b30584be0d) | 2026-09-26 | none (pre-freeze addition, 2026-09-30) | +| [doobidoo/mcp-memory-service](https://github.com/doobidoo/mcp-memory-service) | [v11.14.0](https://github.com/doobidoo/mcp-memory-service/releases/tag/v11.14.0) | [`bed24ef7214192bfe59ad46f6852165678735a77`](https://github.com/doobidoo/mcp-memory-service/commit/bed24ef7214192bfe59ad46f6852165678735a77) | 2026-09-25 | none (pre-freeze addition, 2026-09-30) | +| [MemMachine/MemMachine](https://github.com/MemMachine/MemMachine) | [v0.3.9](https://github.com/MemMachine/MemMachine/releases/tag/v0.3.9) | [`1d3e0ce2782b83571e1974198c4d317e601cb263`](https://github.com/MemMachine/MemMachine/commit/1d3e0ce2782b83571e1974198c4d317e601cb263) | 2026-05-18 | none (pre-freeze addition, 2026-09-30) | +| [agentscope-ai/ReMe](https://github.com/agentscope-ai/ReMe) | [v0.4.1.12](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.1.12) | [`4f7c8786e3975b6ff1b20685ed56c3ccbaa4ed9f`](https://github.com/agentscope-ai/ReMe/commit/4f7c8786e3975b6ff1b20685ed56c3ccbaa4ed9f) | 2026-09-14 | none (pre-freeze addition, 2026-09-30) | + +**Pre-freeze admission screen (2026-09-30).** Each addition passed the same official local-install and both-client integration rule as the existing primaries; this is source admission, with deployment acceptance still required (3.5). The screens are in [`inputs/r7/landscape-foldin-gate-20260930.json`](inputs/r7/landscape-foldin-gate-20260930.json): + +| Addition | Pinned admission-screen locators | +|---|---| +| thedotmack/claude-mem | `thedotmack/claude-mem@57c38303:docs/public/installation.mdx:27`; `thedotmack/claude-mem@57c38303:docs/public/installation.mdx:55`; `thedotmack/claude-mem@57c38303:src/services/integrations/CodexCliInstaller.ts:221`; `thedotmack/claude-mem@57c38303:src/services/integrations/CodexCliInstaller.ts:507` | +| doobidoo/mcp-memory-service | `doobidoo/mcp-memory-service@bed24ef7:README.md:111`; `doobidoo/mcp-memory-service@bed24ef7:docs/integration/multi-client.md:450-477` | +| MemMachine/MemMachine | `MemMachine/MemMachine@1d3e0ce2:docs/install_guide/integrate/claude_code_mcp.mdx:62`; `MemMachine/MemMachine@1d3e0ce2:packages/skills/README.md:13-24`; `MemMachine/MemMachine@1d3e0ce2:packages/client/pyproject.toml:38` | +| agentscope-ai/ReMe | `agentscope-ai/ReMe@4f7c8786:integrations/claude_code/README.md:23`; `agentscope-ai/ReMe@4f7c8786:README.md:188` | + +**agentmemory under the symmetric rule (r7).** r6 kept agentmemory's shipped local MiniLM as its primary by coordinator requirement B:C, an explicit exception to the current-model rule B:E. r7 applies one rule to every system. agentmemory supports an OpenAI-compatible embedder, so its primary arm uses the 8.2-selected embedder through that provider, and the shipped MiniLM configuration becomes a labelled diagnostic. If the coordinator or the user reaffirms B:C, it becomes a dated exception before freeze, recorded as one. The pinned-source facts r6 verified still hold: +- the local provider hard-codes Xenova MiniLM q8 ([v0.9.29 local provider](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/providers/embedding/local.ts#L8-L46)); +- with `EMBEDDING_PROVIDER=openai`, `OPENAI_EMBEDDING_BASE_URL`, `OPENAI_EMBEDDING_MODEL` and `OPENAI_EMBEDDING_DIMENSIONS` select the endpoint, model and dimensions, but a non-empty **`OPENAI_API_KEY` takes precedence over `OPENAI_EMBEDDING_API_KEY`** ([embedding/index.ts:37–38](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/providers/embedding/index.ts#L37-L38), [embedding/openai.ts:64–68](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/providers/embedding/openai.ts#L64-L68)). The primary therefore needs a deliberately safe local-endpoint key policy and verified dimensions, without reading or copying a credential value. + +The rest of r6's agentmemory contract is retained: +- Set `OPENAI_BASE_URL` to the host's gateway and `OPENAI_MODEL` and `OPENAI_REASONING_EFFORT` to the frozen GPT-6 selection. The pinned worker shares one LLM provider across roles, so its role matrix records a shared model/effort rather than fictitious per-role environment variables ([config.ts](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/config.ts#L70-L97), [provider wiring](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/index.ts#L250-L341)). +- The native chat path is non-streaming and omits temperature; its request/header support still has to meet 2.2 before scoring ([openai.ts:78–113](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/providers/openai.ts#L78-L113)). +- **Hook equivalence is required.** The primary adapter must execute the pinned shipped hooks, or demonstrate equivalent behaviour on frozen client-shaped fixtures, including SessionEnd transcript handling (upstream `src/hooks/session-end.ts:20`). The existing D2h replay alone does not establish that equivalence: v4's `am_hook_session` emulates HTTP events and omits transcript replay (`am-minilm-hooks`: v4 `lme_harness.py:759-784,1618-1619`; v4 `PREREGISTRATION.md:456-459,497-498`). +- Pins: agentmemory 0.9.29 at `2d38dafe`; iii 0.11.2, Linux asset `9c83c477…`; Node 24.21.0; and, for the MiniLM diagnostic, Xenova MiniLM q8 at `751bff37`, ONNX `afdb6f1a…`, staged by v4 `lane_tools.py` `install_minilm` (all in v4 `pins.json:36-39,70-77,89-93,1072-1100`). On macOS the Mac's own sanitized host receipt records its platform assets. + +**Discovery inputs (r4, retained).** The Mac session's five model and repository sweeps (embedders, rerankers, memory LLMs, generation models and memory systems), each Opus-verified and dated 2026-09-27, are committed verbatim under `inputs/sweep-20260927-mac/` (commit ff12bf13), with their sha256 in `manifests/evidence.json`. Vendor-reported scores are motivation for the candidate list only (section 10), never S3 evidence. + +### 3.2 Reserves and late candidates + +**Reserves**, in priority order: +1. [Gentleman-Programming/engram](https://github.com/Gentleman-Programming/engram) v2.2.1 (`818be842f57a95063f62f1b745297a96c407c143`): local SQLite/FTS5, official Claude and Codex setup; +2. [getzep/graphiti](https://github.com/getzep/graphiti) v0.30.2 (`eaa4128681bc53487138a4bbc22d58336ebe70d2`); +3. **supermemory-local**: `supermemory-server` release [`server-v0.0.8`](https://github.com/supermemoryai/supermemory/releases/tag/server-v0.0.8) (`5d2b5855fe492a3682a1cde4a255e2db0c4db595`, published 2026-08-17). +4. [qualixar/superlocalmemory](https://github.com/qualixar/superlocalmemory) [v4.1.17](https://github.com/qualixar/superlocalmemory/releases/tag/v4.1.17) (`5bfa47c941b6329b23f93b17f3dce7f6d70a2b9a`, published 2026-09-12). +5. [NevaMind-AI/memU](https://github.com/NevaMind-AI/memU) [v1.5.1](https://github.com/NevaMind-AI/memU/releases/tag/v1.5.1) (`357aefc8012705bfde723f141d52f675fe712bed`, published 2026-03-23). +6. [JaceHo/AgentMem](https://github.com/JaceHo/AgentMem): no GitHub release; head `0770f5d861142486a6e8d57599ae36db7ddd93be` (2026-08-31); needs a pinnable release or a dated head-pin decision before screening. +7. [EverMind-AI/EverOS](https://github.com/EverMind-AI/EverOS) [v1.4.1](https://github.com/EverMind-AI/EverOS/releases/tag/v1.4.1) (`462ebf9fd59b55c03fefb8eec855c62500f2a3cf`, published 2026-09-24). +8. [MemTensor/MemOS](https://github.com/MemTensor/MemOS) [v2.0.34](https://github.com/MemTensor/MemOS/releases/tag/v2.0.34) (`41bf5c7fa89ee08ebedf3c662638b06fc29aca8d`, published 2026-09-23). +9. [BAI-LAB/MemoryOS](https://github.com/BAI-LAB/MemoryOS) [V1.2](https://github.com/BAI-LAB/MemoryOS/releases/tag/V1.2) (`cbb4c74c7b8005f29d9bb363f93d966ed7f82452`, published 2025-07-18). + +**Ordering basis for additions 4–9:** first a landscape-reported official integration with both clients (verification at a pinned source is the admission screen itself), then a published release that can be pinned under 3.1's rule, then the newer release. Release commits and dates come from [`inputs/r7/landscape-foldin-gate-20260930.json`](inputs/r7/landscape-foldin-gate-20260930.json); a reserve listing does not establish admission. + +**supermemory-local, verified from the self-hosting docs at that commit** (read, not executed): +- the server listens on `PORT` or `SUPERMEMORY_PORT`, default **6767** ([`apps/docs/self-hosting/configuration.mdx:16`](https://github.com/supermemoryai/supermemory/blob/5d2b5855fe492a3682a1cde4a255e2db0c4db595/apps/docs/self-hosting/configuration.mdx#L16)); +- **any OpenAI-compatible LLM endpoint** works through `OPENAI_API_KEY` plus `OPENAI_BASE_URL` (`configuration.mdx:44`; `quickstart.mdx:72`), so the GPT-6 gateway can serve its LLM roles; +- embeddings default to local `Xenova/bge-base-en-v1.5` (`overview.mdx:27`); +- the Claude Code and Codex plugins target the local server with `SUPERMEMORY_API_URL=http://localhost:6767` (`overview.mdx:57`); +- `SUPERMEMORY_DISABLE_TELEMETRY` exists (`configuration.mdx:119-125`); +- self-hosting runs the pipeline on the model supplied, while the hosted service uses proprietary models (`quickstart.mdx:72`), so hosted results do not transfer. AMB's provider cannot drive it (4.3). + +**Admission rule (r6, retained):** a reserve is screened for eligibility, meaning an official local install and official integration with both clients, in priority order. It is run whenever a primary candidate is ineligible, or when compute remains after the primary arms. An eligible system that is not run stays `pending`. Every winner claim is bounded to the evaluated set. A reserve admitted before confirmatory execution joins the family (section 6); one admitted afterwards is exploratory until a separately frozen comparison. + +**Late candidates (r4, retained).** Any later survivor of the 32-layer sweep, or any other dated discovery, is handled like a reserve admitted after confirmatory execution. It must also pass the maintenance gate: +- it is exploratory until a separately frozen, dated amendment arm names it before its first run; +- it uses the same lanes, thresholds and decision rule; +- it can change a host's selection only through that amendment's own frozen comparison; +- it never edits this text. + +r6's lower-priority reserves MemMachine and MCP Memory Service are now primaries through the pre-freeze addition of 2026-09-30 (3.1); SimpleMem is out on fit (3.3). Honcho and r6's pending names (byterover-cli, deja-vu, total-agent-memory) are not r7 reserves and still enter only through the late-candidate rule. + +**GBrain (r6 late-candidate screen, now at the current release).** `garrytan/gbrain` v0.60.10.0 (`608a174dcfa1d39d5ea2d8fb5b296122b1cc78c5`, 2026-09-29) is ACTIVE. Its published **449/470 `recall_all@5`** was verified by r6 at `e78f1c38` ([CHANGELOG:2039–2061](https://github.com/garrytan/gbrain/blob/e78f1c38b947b053f3a46881340f74f316be855a/CHANGELOG.md#L2039-L2061)). That run used hosted OpenAI embeddings and a [Voyage reranker](https://github.com/garrytan/gbrain/blob/e78f1c38b947b053f3a46881340f74f316be855a/CHANGELOG.md#L1960), and its local path is unmeasured, so it decides nothing. The screen covers official local installation, both clients, configurable local retrieval, GPT-6 routing and lifecycle at the current release. GBrain stays outside the family, and even a passing screen changes selection only through the late-candidate rule. + +### 3.3 Exclusions, with evidence + +- **Mem0** (OSS v2.2.1, 2026-09-25; the repository passes the gate). Its official integrations send memory off the host. + - The Claude Code plugin sets `DEFAULT_API_URL = "https://api.mem0.ai"` ([`integrations/claude-code-plugin/core/memory_core.py:31`](https://github.com/mem0ai/mem0/blob/94c3fe9f238f3dbf29c9ce98643bd71eb13077cd/integrations/claude-code-plugin/core/memory_core.py#L31) at `94c3fe9f`, the main head on 2026-09-30). + - A `MEM0_API_URL` variable can override it (`:1995,2334,2584,2622`), so the brief's "hard-codes" is corrected here. The override does not bring memory home, though: the plugin calls Platform routes, `/v3/memories/add/` (`:1996`), `/v1/event/{id}/` (`:1916`) and `/v2/memories/` (`:2532`). The self-hosted server serves `/memories`, `/search` and related routes instead ([`server/main.py:367-547`](https://github.com/mem0ai/mem0/blob/94c3fe9f238f3dbf29c9ce98643bd71eb13077cd/server/main.py#L367-L547)). + - The official Codex path is `codex mcp add mem0 --url https://mcp.mem0.ai/mcp/` with a Platform account and key ([`docs/integrations/codex.mdx:16-18,71,78`](https://github.com/mem0ai/mem0/blob/94c3fe9f238f3dbf29c9ce98643bd71eb13077cd/docs/integrations/codex.mdx#L71-L79)). + - The 2026-09-30 landscape check recommended "Mem0 OSS core" as a primary ([discovery input](inputs/r7/landscape-memory-20260930.json)); this recommendation is not adopted. The official Codex plugin core also posts to Platform routes, including `/v3/memories/add/` (`mem0ai/mem0@94c3fe9f:integrations/codex-plugin/core/memory_core.py:1996`). The pinned tree has no self-hosted OpenMemory directory; OpenMemory survives only in changelog entries and documentation redirects (`mem0ai/mem0@94c3fe9f:docs/docs.json:1182`), as the [gate screen](inputs/r7/landscape-foldin-gate-20260930.json) records. + - Mem0 is at most a diagnostic arm outside the family: its self-hosted OSS server at the current release, through a thin client, never the Platform. +- **Letta**. The repository passes the gate, with 8 commits in the window, so the exclusion is on fit, not staleness. Its README says the `archive` branch "contains the retired Letta V1 API server" and sends active projects to `letta-ai/letta-code` ([`README.md:5,39`](https://github.com/letta-ai/letta/blob/5bcdd177d70fa2b31a754cfcd801e77b2e1ab16a/README.md#L39) at `5bcdd177`). `letta-code` v0.33.8 (`21daa38a`, ACTIVE) is "a stateful agent harness", an agent client of its own, not a memory provider for Claude Code or Codex. +- **beads** (`gastownhall/beads` v1.3.0, `f45b249ce6b40ba62aecc03949e6371e8f7c79d8`, ACTIVE). Its README calls it a "Distributed graph issue tracker for AI agents" ([`README.md:3`](https://github.com/gastownhall/beads/blob/3f9561db2bd1b1903ca8cbb39e9fc0cb37b0bccb/README.md#L3) at `3f9561db`) that provides "persistent, structured memory for coding agents" by replacing "messy markdown plans with a dependency-aware graph" (`README.md:15`). That is work tracking (`bd create`, `ready`, `claim`, `close`), not query-time recall of earlier sessions, so it is out on fit. +- **Attemory v0.1.3** (r6, retained): ineligible on fit. It has not established the required official local integration with both clients; its pinned README still lists MCP as planned ([603c03af:318–322](https://github.com/AttemorySystem/attemory/blob/603c03afa9a04e48b778922eae156a0024752f37/README.md#L318-L322); M:100–112). +- **SimpleMem v0.3.0** (`aiming-lab/SimpleMem@c830efa3fd2614bc4eb0b4a80bc5836ece9ded96`, ACTIVE): out on fit, like Attemory. The 2026-09-30 [admission screen](inputs/r7/landscape-foldin-gate-20260930.json) found no Codex setup in any document at release `c830efa3` or main `db80b6a7`; the Claude path is a skill copy (`aiming-lab/SimpleMem@c830efa3:SKILL/README.md:13`). Re-screen when a release ships an official Codex path. + +### 3.4 Controls and diagnostics (outside the family) + +- **Native no-external-memory controls, both clients (r6, retained):** Claude Code auto memory at the recorded client version, isolated with `--settings autoMemoryDirectory`, where `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` gives the no-memory control; and Codex native memories at 0.157.1 (`memories`, stable, off by default: [features/src/lib.rs](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/features/src/lib.rs)), enabled in an isolated `CODEX_HOME`. Controls use the same lanes. Unsupported cross-client behaviour is recorded as such, and a control is never eligible for deployment. +- Plain BM25 and dense retrieval with the 8.2-selected memory embedder, on the same eligible corpus, session mapping, deduplication and top-5 scorer (r6, M:61–67,117). +- Unselected optional-stage settings (section 2), shipped-default replays such as agentmemory's MiniLM, the deployed-configuration copies (section 2) and, if run, the Mem0 diagnostic (3.3). + +### 3.5 Upstream deployment and isolated arm configurations (r6 2.1, updated) + +Every system uses its upstream-recommended deployment for the selected host/use case, with the executable image **pinned by digest**, or a build/install from the selected tag's **lockfile**. A floating `latest`, a package version with newly resolved dependencies, or a setup-only smoke is insufficient. Freeze the upstream install/start/readiness/test/stop/backup/restore commands, source/lock hashes, platform image digest, native client integration pin and effective configuration per arm. If upstream documents several supported modes without a single recommendation, name the selected mode and its source rather than inventing an endorsement. All twelve release identities in 3.1 remain fixed; their deployment recipes must be resolved before the bundle freeze (section 11). + +- Each arm receives a **fresh container, volume and home**, loopback-only endpoints, its own database/index/model cache and isolated client profiles. The worker and every sidecar are inside that arm's isolation boundary; upstream compose that isolates only an engine does not also isolate its host worker. Use fresh empty stores for independent question/seed runs, not additive import/restore as a reset. Never score against a live deployment, including the persistent cognee service. Freeze how the container reaches its host-local gateway: container loopback is not host loopback. Verify the supported network/forwarding path and endpoint identity without exposing a public listener. These are experiment requirements, not claims that the upstream default deployment already enforces them. +- Keep upstream installs intact. Use supported configuration, not edits to upstream databases, libraries or model implementations, to satisfy isolation. Record upstream tests separately from S3 integration checks and preserve failures (section 4 and `docs/acceptance-evidence-policy.md`). +- **Recorded live hosting, never scored or touched (r6):** I:37–54 records `cognee-live` from the pinned digest on rootless Docker **29.8.1**, `127.0.0.1:3800`, auth on; and `hindsight-live` as a `systemd --user` unit on **3710/5433** with pg0. Archive sanitized copies of **both deploy receipts** in the freeze bundle. Ports **3800, 3710 and 5433** are on the protected fail-closed list (4.6). **`10.0.2.2:20128`** is the recorded candidate container-to-host transport; re-verify endpoint identity per isolated arm. Live envelopes use static session headers and are **not scored envelopes**. Hindsight's stale-`postmaster.pid` guard is a **local lifecycle modification**; a scored arm must not inherit it silently, and the retained unguarded first failure counts as Hindsight lifecycle evidence. + +| System | Pinned native deployment and verified configuration to carry into the isolated arm | Evidence and boundary | +|---|---|---| +| **cognee 1.6.2** | Release commit `ba3631f2ed363a6ea50d649c34c56885af6b36fe`; image `cognee/cognee:1.6.2@sha256:41c06180608a053f07fffc4da56dd1efba62967dd3d4a1023b33939abee5d45d`, or that tag's `uv.lock` with `uv sync --python 3.12 --frozen --no-dev --no-editable` in a fresh container. `LLM_PROVIDER=custom`, `LLM_ENDPOINT` = local gateway, and LiteLLM model spelling `openai/cx/gpt-6-…`. Set base plus `LLM_EXTRACTION_MODEL`, `LLM_SUMMARIZATION_MODEL`, `LLM_QUERY_MODEL` from the role matrix; all other roles inherit the GPT-6 base. Set `STRUCTURED_OUTPUT_FRAMEWORK=instructor`, `LLM_INSTRUCTOR_MODE=tool_call`, `GRAPH_EXTRACTOR=llm`, `ENABLE_BACKEND_ACCESS_CONTROL=true`, `TELEMETRY_DISABLED=1`, `LITELLM_LOCAL_MODEL_COST_MAP=True`. Give each arm its own `/cognee-storage` and home, with no inherited `.env`. Use shipped SQLite/LanceDB/Ladybug stores. Pin `DataItem(data_id=uuid5(...))` per opaque source session, with frozen namespace/mapping. | I:16–35; [Dockerfile](https://github.com/topoteretes/cognee/blob/ba3631f2ed363a6ea50d649c34c56885af6b36fe/Dockerfile#L59-L140) and [DataItem](https://github.com/topoteretes/cognee/blob/ba3631f2ed363a6ea50d649c34c56885af6b36fe/cognee/tasks/ingestion/data_item.py#L14-L22), both byte-identical to 1.6.1; [LLM config at 1.6.2](https://github.com/topoteretes/cognee/blob/ba3631f2ed363a6ea50d649c34c56885af6b36fe/cognee/infrastructure/llm/config.py#L107-L128): `structured_output_framework` defaults to `litellm_native` (L107), so `instructor` is a frozen departure; per-role models L116–128; `llm_args` L193. The supplied 1.6.1 bare-metal smoke is setup evidence only; it is not the scored container. | +| **Hindsight 0.10.2** | Release commit `5fc4ce20917b916240cef27c212c387a177f115b`; upstream Docker path, e.g. standalone `ghcr.io/vectorize-io/hindsight:0.10.2@sha256:d1840062a5b79940ab7a9f4809ceb90fc776d4ad737cd9329e9b5836cc64ab70`, with the selected database mode pinned separately. Set `HINDSIGHT_API_LLM_PROVIDER=openai`, `_BASE_URL` = local gateway, `_MODEL` = frozen GPT-6; explicitly freeze the `HINDSIGHT_API_{RETAIN,REFLECT,CONSOLIDATION,MENTAL_MODEL_REFRESH}_LLM_{MODEL,REASONING_EFFORT}` model/effort/provider/base overrides. Set `HINDSIGHT_API_LLM_TEMPERATURE=none` and leave `HINDSIGHT_API_LLM_TEMPERATURE_{RETAIN,REFLECT,CONSOLIDATION,VERIFICATION}` unset: explicit operation values override the global value, and their built-in defaults are respectively 0.1/0.9/0.0/0.0. Verify omission at the wire. Freeze local embedding/reranker choices under 8.2. Use owned API/database/worker identities and loopback publishing, and external PostgreSQL, since upstream calls embedded pg0 "not recommended for production". Leave `HINDSIGHT_API_ENABLE_OBSERVATIONS` and `HINDSIGHT_API_ENABLE_AUTO_CONSOLIDATION` unset, so observations and automatic consolidation keep their documented default `true` ([`configuration.mdx:2257-2258`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/configuration.mdx#L2257-L2258)), and create banks without configuration fields (4.3, r7 repair). | I:56–58; at 0.10.2: [temperature names and defaults](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-api-slim/hindsight_api/config.py#L216-L245), [operation config](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-api-slim/hindsight_api/config.py#L383-L462), [installation](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/installation.md#L37-L39); [loopback compose](https://github.com/vectorize-io/hindsight/blob/f8950b0c07d9e34c76493dba802bb309f0ce60fd/docker/docker-compose/claude-code/docker-compose.yaml#L14-L17) (unchanged at 0.10.2). Four of the five Hindsight files r6 cited at `f8950b0c` changed at 0.10.2, so these links supersede r6's line references. The recorded gateway request-shape probes (**9/9** across sol/luna/astra, `max_tokens` up to 64000, JSON-object and required-tool shapes) are wire compatibility for the 0.10.1 service, not a ≥65000-output or complete lifecycle acceptance. | +| **agentmemory 0.9.29** | Release commit `2d38dafede67d0d4ed920cde94d2106e98825b8a`, its lockfile, iii **0.11.2**, and the platform-specific iii digest. Use the upstream CLI/source-build path inside a fresh arm container, including the local Transformers optional dependency for the MiniLM diagnostic, plus an isolated engine volume/home. Pin the worker and MCP package independently; keep Node/model pins above. Embedder per 3.1 and GPT-6 LLM configuration as specified above. | [upstream start](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/README.md#L76-L115), [source build](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/README.md#L748-L765), [engine-only compose](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/docker-compose.yml#L1-L47). The MCP package's `~0.9.0` runtime dependency is not a runtime pin. | + +**Deployment-source supplement (repair round 2).** The supplied deployment-spec review found six source-supported recipes deployable as specified (agentmemory, Hindsight, memsearch, basic-memory, OpenViking and engram) and five requiring the corrections below. These are source-review verdicts, not executed deployment acceptance. Every row still needs the frozen installation, test, lifecycle, configuration and artifact receipts above. Version/tag or health-only checks cannot replace a native ingest→ready→recall→delete→restart→restore check through both clients. + +| System at the pin in 3.1–3.2 | Native recipe and required correction/configuration | Pinned upstream locators | +|---|---|---| +| agentmemory | Documented npm CLI install plus iii engine; root compose isolates only iii, so the CLI/worker and MCP must also be in the owned arm. Pin the package, iii binary and complete dependency closure; freeze the source-build/lock path if the npm install cannot reproduce it. | `rohitg00/agentmemory@2d38dafe:INSTALL_FOR_AGENTS.md:24`; `rohitg00/agentmemory@2d38dafe:deploy/README.md:4`; `rohitg00/agentmemory@2d38dafe:docker-compose.yml:26` | +| Hindsight | Versioned upstream Docker deployment with external PostgreSQL as selected above; pin both artifacts. Read back bank defaults, wait for retain and consolidation operations, and run the native durability checks. A full-image health response does not pre-pass these. | `vectorize-io/hindsight@5fc4ce20:hindsight-docs/docs/developer/installation.md:37`; `vectorize-io/hindsight@5fc4ce20:hindsight-docs/docs/developer/installation.md:152`; `vectorize-io/hindsight@5fc4ce20:hindsight-docs/docs/developer/api/operations.mdx:109` | +| memsearch | CLI/library with Milvus Lite by default; `uv tool install memsearch` is the documented form, while the version pin is an S3 adaptation and does not pin transitive dependencies. Use the source checkout and CI's `uv sync --all-extras` with a verified lock for reproducibility. Configure embedding and each plugin's named LLM provider separately in the isolated global config; run `uv run python -m pytest`, then native index/stats/search and both shipped hook paths. | `zilliztech/memsearch@2a4652fa:README.md:430`; `zilliztech/memsearch@2a4652fa:.github/workflows/test.yml:34`; `zilliztech/memsearch@2a4652fa:docs/cli.md:303`; `zilliztech/memsearch@2a4652fa:docs/cli.md:465` | +| basic-memory | Build the pinned Docker source (`docker build -t basic-memory .` upstream form), freeze its lock and image identity, and publish only the owned loopback port. Mount fresh data/config using the image's effective paths and UID, not a guessed host path. `basic-memory status` and `reindex` are native readiness steps; connect clients to this same store through upstream MCP transport. | `basicmachines-co/basic-memory@c0bd87c6:docs/Docker.md:64`; `basicmachines-co/basic-memory@c0bd87c6:docs/Docker.md:111`; `basicmachines-co/basic-memory@c0bd87c6:docs/Docker.md:292` | +| OpenViking | Upstream Docker run with owned config/data mounts; derive and verify the version tag/digest, or source-build with `UV_LOCK_STRATEGY=locked`. Persist `plugin.recallQueryExpansion=off` and `plugin.recallCompress=off` in `ovcli.conf` for both clients, then restart them; setting variables only during plugin installation does not configure later hooks. Verify `/health` and `/ready` plus actual ingest/find/client capture. | `volcengine/OpenViking@e8716760:docs/en/guides/03-deployment.md:214`; `volcengine/OpenViking@e8716760:Dockerfile:30`; `volcengine/OpenViking@e8716760:docs/en/agent-integrations/01-overview.md:54`; `volcengine/OpenViking@e8716760:openviking/server/routers/system.py:100` | +| engram (reserve) | Native binary, owned SQLite store and official MCP integration; prefer bare MCP without spawning plugin hooks, or freeze an explicit `ENGRAM_URL` pointing to the supervised owned server. Keeping just the same data-dir/port does not prevent a hook from starting an unsupervised server during restart. Cloud autosync stays off. Verify the version/store-aware `/health` and full lifecycle. | `Gentleman-Programming/engram@818be842:docs/AGENT-SETUP.md:328`; `Gentleman-Programming/engram@818be842:plugin/claude-code/scripts/_helpers.sh:86`; `Gentleman-Programming/engram@818be842:DOCS.md:293` | +| cognee | Keep auth on as selected above: set a fresh `DEFAULT_USER_PASSWORD`, provision the owned default user and API key, and set MCP `COGNEE_API_AUTH_SCHEME=x-api-key`; otherwise the first API calls 401. Apply `TELEMETRY_DISABLED=1` to the API **and** MCP processes. A source-built image may report `1.6.2-local`, not exactly `1.6.2`; compare this with the frozen image/source identity. Set the embedding provider/endpoint/model/dimensions explicitly to the selected local service. | `topoteretes/cognee@ba3631f2:cognee/api/client.py:132`; `topoteretes/cognee@ba3631f2:cognee/api/client.py:297`; `topoteretes/cognee@ba3631f2:cognee-mcp/src/cognee_client.py:84`; `topoteretes/cognee@ba3631f2:cognee/shared/utils.py:400`; `topoteretes/cognee@ba3631f2:cognee/version.py:22`; `topoteretes/cognee@ba3631f2:cognee/infrastructure/databases/vector/embeddings/config.py:97` | +| supermemory-local (reserve) | Pinned `supermemory-server` binary and documented env configuration, not AMB's cloud provider. Bounded native status polling must stop on `failed`, HTTP 404 (including auto-deletion), or deadline; `done` alone is insufficient without searchable expected evidence. Test auth with a wrong-key control and verify actual bind/isolation. The supplied review leaves the v0.0.8 loopback/container boundary unresolved; this reserve stays pending until it is source-backed and observed. | `supermemoryai/supermemory@5d2b5855:apps/docs/self-hosting/quickstart.mdx:55`; `supermemoryai/supermemory@5d2b5855:apps/docs/concepts/how-it-works.mdx:109`; `supermemoryai/supermemory@5d2b5855:apps/docs/ingestion/add-memories.mdx:73`; `supermemoryai/supermemory@5d2b5855:apps/docs/self-hosting/configuration.mdx:16` | +| MemPalace | Use the pinned source/image and the native remote-team compose with Qdrant, pinning both images. Run init/mine in that **same compose project/store/backend**, with a writable corpus mount; an unrelated `docker run` volume creates a separate Chroma palace. Init writes origin/config, entities, `mempalace.yaml` and `.gitignore`; freeze supported LLM-init settings. Retain compose's bearer-token guard and explicitly set Qdrant backend for helper processes. Confirm HTTP MCP initialize/list/call, `/healthz` and authenticated `/statusz`; telemetry suppression is not an egress barrier. | `MemPalace/mempalace@22fd87f0:deploy/docker-compose.server.yml:7`; `MemPalace/mempalace@22fd87f0:deploy/docker-compose.server.yml:47`; `MemPalace/mempalace@22fd87f0:mempalace/palace/backend.py:41`; `MemPalace/mempalace@22fd87f0:mempalace/cli/cmd_init.py:110`; `MemPalace/mempalace@22fd87f0:mempalace/cli/cmd_init.py:178`; `MemPalace/mempalace@22fd87f0:mempalace/cli/cmd_serve.py:145` | +| ai-memory | Build the pinned local image through `docker compose build`, then use the upstream `docker run --env-file` shape with the private 0600 env file, a new owned name/volume and a proven free loopback port. Compose's optional `docker/.env` does not load the external provider file and its fixed name/volume/49374 allocation is unsuitable for isolated scoring. Never allocate the live 49474 service or the known occupied 49374 default. Run value-free effective-provider checks and native lifecycle; liveness alone cannot prove the LLM overlay loaded. | `akitaonrails/ai-memory@a0ca8d1a:docker/docker-compose.yml:13`; `akitaonrails/ai-memory@a0ca8d1a:docker/docker-compose.yml:25`; `akitaonrails/ai-memory@a0ca8d1a:docker/docker-compose.yml:26`; `akitaonrails/ai-memory@a0ca8d1a:README.md:288`; `akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-cli/src/commands/serve.rs:2497` | +| Graphiti (reserve) | Build the standalone MCP image from the pinned source with core version 0.30.2, not its older default; record the resolved dependency lock and built digest. Use the documented Neo4j compose with `NEO4J_URI=bolt://neo4j:7687` inside the network, not the example env's localhost. Pull Neo4j 5.26.0 and pin its digest before using `--pull never`; mount the effective config at the command's actual path, and prevent startup re-resolution with the frozen native `uv run --no-sync` command. The endpoint/key/embedding split and lifecycle need functional acceptance. | `getzep/graphiti@eaa41286:mcp_server/docker/Dockerfile.standalone:46`; `getzep/graphiti@eaa41286:mcp_server/docker/README.md:92`; `getzep/graphiti@eaa41286:mcp_server/docker/docker-compose-neo4j.yml:3`; `getzep/graphiti@eaa41286:mcp_server/docker/docker-compose-neo4j.yml:38`; `getzep/graphiti@eaa41286:mcp_server/src/graphiti_mcp_server.py:1188` | + +**Deployment sources for the pre-freeze additions (2026-09-30)** + +These are source-review extractions, not executed deployment acceptance. Every model below comes from 2.1's frozen GPT-6 role matrix; active, background and fallback calls still need 2.2/4.4 wire acceptance. Local embedders use 8.2's selection wherever the pinned implementation supports it. Fresh stores, isolated client profiles, owned loopback ports, immutable artifacts, dependency SBOM/OSV coverage and full native lifecycle acceptance remain required for every row. MemMachine was re-read at release `1d3e0ce2`; the supplied extraction's `899ace1a` was 31 commits later, and unsupported transfers remain pending. + +| System at the pin | Native recipe and required correction/configuration | Pinned upstream locators | +|---|---|---| +| claude-mem v13.28.0 | Build the pinned plugin with the documented npm install/build, then use the Postgres/Valkey/server/worker Compose recipe with fresh database, queue and data volumes. Supply the required database secrets; publish only `127.0.0.1:37877:37877`, with sidecars unpublished. Explicitly forward `CLAUDE_MEM_SERVER_PROVIDER=openrouter`, `CLAUDE_MEM_SERVER_MODEL`, `CLAUDE_MEM_OPENROUTER_BASE_URL` and a nonempty `OPENROUTER_API_KEY` into the worker; the shipped environment omits the model/base settings. Observation extraction and summaries share that GPT-6 gateway provider. Client capture uses `CLAUDE_MEM_RUNTIME=server` and `CLAUDE_MEM_SERVER_{URL,API_KEY,PROJECT_ID}`; any local-worker fallback also needs `CLAUDE_MEM_PROVIDER=openrouter`, `CLAUDE_MEM_OPENROUTER_{BASE_URL,MODEL,API_KEY}`, `CLAUDE_MEM_MODEL` and `CLAUDE_MEM_TIER_ROUTING_ENABLED=false`. Set `DO_NOT_TRACK=1` in every process and leave cloud-sync URL/token/user fields empty. **Required corrections/pending:** Compose disables Chroma and uses full-text recall, so its embedder is not applicable; the local Chroma route's 8.2 override and offline model-artifact recipe remain unverified, with MiniLM only a labelled shipped default. Optional KnowledgeAgent priming/repriming/Q&A needs a supported Anthropic-protocol GPT-6 route before activation. Manual Compose avoids the installer's account-gated OpenRouter path despite the README's explicit-provider exemption. Freeze floating images, dependencies and the installed client; verify gateway networking and rootless/WSL2 operation. Probe port 37877; the Docker guide's 37777 is wrong for this recipe. Experiment headers and full lifecycle remain pending. Published memory-quality reproduction is also pending: the changelog names an absent SWE-bench directory, while Smart Explore measures code exploration. | `thedotmack/claude-mem@57c38303:docs/public/installation.mdx:85`; `thedotmack/claude-mem@57c38303:docs/server.md:79,144`; `thedotmack/claude-mem@57c38303:docker-compose.yml:56,72,121,126,135,168`; `thedotmack/claude-mem@57c38303:src/server/runtime/create-server-service.ts:294`; `thedotmack/claude-mem@57c38303:src/services/hooks/runtime-selector.ts:39`; `thedotmack/claude-mem@57c38303:src/services/worker/model-aliases.ts:36`; `thedotmack/claude-mem@57c38303:src/shared/openrouter-base-url.ts:31`; `thedotmack/claude-mem@57c38303:src/server/routes/v1/ServerV1PostgresRoutes.ts:925`; `thedotmack/claude-mem@57c38303:docs/public/telemetry.mdx:232`; `thedotmack/claude-mem@57c38303:src/shared/SettingsDefaultsManager.ts:261`; `thedotmack/claude-mem@57c38303:src/services/sync/ChromaSync.ts:169`; `thedotmack/claude-mem@57c38303:src/services/sync/ChromaMcpManager.ts:45`; `thedotmack/claude-mem@57c38303:src/services/worker/knowledge/KnowledgeAgent.ts:14`; `thedotmack/claude-mem@57c38303:docs/public/configuration/litellm-gateway.mdx:77`; `thedotmack/claude-mem@57c38303:src/npx-cli/commands/install.ts:1961`; `thedotmack/claude-mem@57c38303:README.md:155`; `thedotmack/claude-mem@57c38303:docker/claude-mem/Dockerfile:2,36,55`; `thedotmack/claude-mem@57c38303:docs/docker.md:7`; `thedotmack/claude-mem@57c38303:src/server/generation/providers/OpenRouterObservationProvider.ts:144`; `thedotmack/claude-mem@57c38303:CHANGELOG.md:2135`; `thedotmack/claude-mem@57c38303:docs/public/smart-explore-benchmark.mdx:3` | +| MCP Memory Service v11.14.0 | Build the pinned HTTP Compose recipe; set `HTTP_PORT=127.0.0.1:8000`, a fresh API key and `MCP_MEMORY_STORAGE_BACKEND=sqlite_vec`, with fresh data, backup/archive and model-cache stores. For active quality scoring use `MCP_QUALITY_AI_PROVIDER=openai-compatible` and `MCP_QUALITY_AI_{BASE_URL,MODEL,API_KEY}`. Harvest classification/rewriting and enabled NLI use the sole `HARVEST_LLM_PROVIDERS=gateway` entry, `HARVEST_LLM_GATEWAY_{BASE_URL,MODEL,API_KEY}` and `MCP_NLI_BACKEND=cascade`; all LLM models/routes come from the GPT-6 matrix. Set `MCP_EXTERNAL_EMBEDDING_URL` to 8.2's selected local service's full `/v1/embeddings` endpoint and set `MCP_EXTERNAL_EMBEDDING_MODEL`/`MCP_EXTERNAL_EMBEDDING_API_KEY`; this adapter requires SQLite-vec. Set `MCP_MEMORY_ALLOW_HASH_EMBEDDINGS=0`. Correct the Claude plugin's hybrid default to this backend and set `ENSURE_SERVER_NO_SPAWN=1` so hooks cannot start another server. **Codex route pending:** choose the recipe's third-party `mcp-proxy` sidecar, pinned and passing section 1's selected-repository maintenance gate, or a labelled adaptation using Codex 0.159.2's native streamable-HTTP MCP client (`codex mcp add … --url`, with `--bearer-token-env-var MCP_API_KEY`) pointed at the owned loopback `/mcp` endpoint with its `Authorization: Bearer` header. The recipe's claim that Codex does not natively support HTTP is contradicted by that Codex source; its referenced `examples/codex-mcp-config.json` is absent at `bed24ef7`. Either Codex path must be pinned and gated before freeze. **Required corrections/pending:** the embedder must be reachable; the Dockerfile's purported preload only prints a runtime-download message, so any ONNX path needs a prepared cache and download controls. Verify GPT-6 compatibility: quality scoring sends `max_tokens=50` and temperature 0.1 for `cx/` aliases; harvest sends 200 and 0.3. The request headers and quality scorer's dedup exclusion remain pending under 2.2/4.4. Optional native LoCoMo QA and pattern-discovery gateway routes are unverified and stay unused until resolved. Use `/api/search` with `n_results`, correcting the HTTP guide's path/field. Freeze base-image/dependency artifacts and any benchmark dataset bytes; rootless/WSL2 operation, gateway networking, writable mounts and full native lifecycle remain pending. | `doobidoo/mcp-memory-service@bed24ef7:tools/docker/docker-compose.http.yml:4,12,27`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/quality/config.py:70`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/harvest/rewriter.py:27,401`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/reasoning/nli.py:100,131`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/embeddings/external_api.py:14`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/storage/mixins/embeddings.py:138,199,417`; `doobidoo/mcp-memory-service@bed24ef7:claude-hooks/.mcp.json:7`; `doobidoo/mcp-memory-service@bed24ef7:claude-hooks/PLUGIN.md:48`; `doobidoo/mcp-memory-service@bed24ef7:tools/docker/Dockerfile:3,77,90`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/embeddings/onnx_embeddings.py:150`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/quality/ai_evaluator.py:408,425`; `doobidoo/mcp-memory-service@bed24ef7:scripts/benchmarks/benchmark_locomo.py:395`; `doobidoo/mcp-memory-service@bed24ef7:scripts/maintenance/discover_harvest_patterns.py:233`; `doobidoo/mcp-memory-service@bed24ef7:docs/agents/http-generic.md:161`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/web/api/search.py:56,150`; `doobidoo/mcp-memory-service@bed24ef7:src/mcp_memory_service/web/app.py:325`; `doobidoo/mcp-memory-service@bed24ef7:scripts/benchmarks/locomo_dataset.py:18`; `doobidoo/mcp-memory-service@bed24ef7:scripts/benchmarks/longmemeval_dataset.py:128`; `doobidoo/mcp-memory-service@bed24ef7:tools/docker/README.md:51`; `doobidoo/mcp-memory-service@bed24ef7:docs/integration/multi-client.md:447-477`; `openai/codex@ff6aec96:codex-rs/cli/src/mcp_cmd.rs:145-157,343-359`; `openai/codex@ff6aec96:codex-rs/config/src/mcp_types.rs:640-644` | +| MemMachine v0.3.9 | Use the documented PostgreSQL/Neo4j Compose deployment, building the release source through `memmachine-compose.sh build` and freezing the resulting image and dependency artifacts. Start from the CPU sample; make `.env` and YAML database settings agree, with fresh database/log volumes and owned container/network identities. Replace retained publications with loopback mappings and omit unused database/docs publications; do not put a bind address into variables also used as database connection ports. Configure `resources.language_models.` with `provider=openai-chat-completions` and gateway `config.base_url`/`api_key` plus the frozen GPT-6 `config.model`. Bind `episodic_memory.short_term_memory.llm_model`, `semantic_memory.llm_model` and `retrieval_agent.{llm_model,answer_llm_model,judge_llm_model}` to those resources, including persisted semantic-set overrides. Bind `episodic_memory.long_term_memory.embedder` and `semantic_memory.embedding_model` to a `resources.embedders.` OpenAI-compatible resource with 8.2's local endpoint, model and dimensions; remove unused cloud resources. **Codex route pending:** use the skill from the pinned `packages/skills/memmachine-memory` tree or through a pinned, section-1-gated installer (`npx skills add` / `pipx run shskills install --agent codex`). It drives `mem-cli` from `memmachine-client`, which must be pinned in the arm's SBOM. The Codex path must be pinned and gated before freeze. **Required corrections/pending:** gateway/embedder reachability and rootless/WSL2 topology; floating application/builder images and OS/dependency artifacts; telemetry/offline provisioning for Neo4j plugins, NLTK and model loaders; GPT-6 structured-output/tool acceptance; the guide's unregistered `memmachine-sync-profile-schema`; the native evaluation's `LongTermMemoryParams` constructor/type-alias mismatch; and legacy evaluation models/embeddings/AWS reranking that do not follow this YAML. Their unchanged reproduction stays pending. Verify MCP identity variables inside the executed container and use fresh projects/stores, since recall has no automatic user filter. Health is liveness only. The later extraction's episodic-only HTTP defaults, optimized-CoQ/spaCy settings and lazy-initialization assertion are not transferred: release handlers default to all memory types, and release-specific acceptance of the other claims remains pending. | `MemMachine/MemMachine@1d3e0ce2:DOCKER_COMPOSE_README.md:12,149`; `MemMachine/MemMachine@1d3e0ce2:memmachine-compose.sh:1116`; `MemMachine/MemMachine@1d3e0ce2:docker-compose.yml:8,31,44,61,66,108,133`; `MemMachine/MemMachine@1d3e0ce2:sample_configs/episodic_memory_config.cpu.sample:8,109,163`; `MemMachine/MemMachine@1d3e0ce2:packages/server/src/memmachine_server/common/configuration/retrieval_config.py:11`; `MemMachine/MemMachine@1d3e0ce2:packages/server/src/memmachine_server/semantic_memory/semantic_memory.py:519`; `MemMachine/MemMachine@1d3e0ce2:Dockerfile:9,17,71`; `MemMachine/MemMachine@1d3e0ce2:docs/install_guide/install_guide.mdx:343`; `MemMachine/MemMachine@1d3e0ce2:packages/server/pyproject.toml:71`; `MemMachine/MemMachine@1d3e0ce2:evaluation/utils/agent_utils.py:453`; `MemMachine/MemMachine@1d3e0ce2:packages/server/src/memmachine_server/episodic_memory/long_term_memory/long_term_memory.py:162`; `MemMachine/MemMachine@1d3e0ce2:evaluation/episodic_memory/llm_judge.py:46`; `MemMachine/MemMachine@1d3e0ce2:evaluation/episodic_memory/longmemeval_search.py:119`; `MemMachine/MemMachine@1d3e0ce2:evaluation/episodic_memory/README.md:98`; `MemMachine/MemMachine@1d3e0ce2:packages/server/src/memmachine_server/common/language_model/openai_chat_completions_language_model.py:143,216`; `MemMachine/MemMachine@1d3e0ce2:docs/install_guide/integrate/claude_code_mcp.mdx:68`; `MemMachine/MemMachine@1d3e0ce2:packages/server/src/memmachine_server/server/api_v2/mcp.py:158,233`; `MemMachine/MemMachine@1d3e0ce2:packages/server/src/memmachine_server/server/api_v2/router.py:292,310,1080`; `MemMachine/MemMachine@1d3e0ce2:packages/server/src/memmachine_server/retrieval_agent/service_locator.py:17`; `MemMachine/MemMachine@1d3e0ce2:packages/skills/README.md:13-24`; `MemMachine/MemMachine@1d3e0ce2:packages/skills/memmachine-memory/SKILL.md:31-33`; `MemMachine/MemMachine@1d3e0ce2:packages/client/pyproject.toml:6,38` | +| ReMe v0.4.1.12 | Use the documented Python 3.11+ source/package installation and `reme start` HTTP/MCP service; freeze the dependency closure and source-backed isolation adaptation. Bind an owned loopback port (default 2333), set `service.web_enabled=false`, and use a fresh complete `workspace_dir` per independent run. Select an explicit full config with only the default AgentScope wrapper bound to `as_llm.default`; all declared components are instantiated, so remove the native Claude/Codex wrappers whose Anthropic/OAuth paths do not establish the required gateway route. Set every active `as_llm` component to `backend=openai`, the frozen GPT-6 model and gateway `credential.base_url`/`api_key`: default extraction/update, resource interpretation, dreaming and proactive work share the wrapper; any enabled vision, benchmark, compressor and judge components need explicit routing too. Clear inherited provider-specific `parameters` through supported config and verify the wire envelope. Enable `as_embedding`, the local `embedding_store` and its `file_store` binding explicitly, with 8.2's selected local model, dimensions, endpoint and credentials; local storage does not supply an embedder. **Required corrections/pending:** local embedding provisioning; GPT-6/tool/image acceptance; container/rootless/WSL2/gateway topology; dependency telemetry/offline provisioning; isolated Claude transcript access and both-client lifecycle capture. Keep the plugin's `.mcp.json` URL synchronized because it takes precedence over `REME_HOST`/`REME_PORT`. Supply `session_id` to `auto_memory`; shared-workspace recall still requires separate workspaces for isolation. Check `metadata.health.healthy` and full lifecycle. Published-score reproduction remains pending because its reference environment, reused memory and dataset revision are not frozen for this arm. | `agentscope-ai/ReMe@4f7c8786:README.md:68,81,188`; `agentscope-ai/ReMe@4f7c8786:docs/en/services.md:8,84,109`; `agentscope-ai/ReMe@4f7c8786:reme/config/config_parser.py:277`; `agentscope-ai/ReMe@4f7c8786:reme/config/default.yaml:840`; `agentscope-ai/ReMe@4f7c8786:reme/application.py:80`; `agentscope-ai/ReMe@4f7c8786:reme/components/agent_wrapper/cc_agent_wrapper.py:176`; `agentscope-ai/ReMe@4f7c8786:reme/components/agent_wrapper/codex_agent_wrapper.py:199`; `agentscope-ai/ReMe@4f7c8786:reme/components/as_llm/__init__.py:41`; `agentscope-ai/ReMe@4f7c8786:reme/config/benchmark.yaml:394`; `agentscope-ai/ReMe@4f7c8786:plugins/lme-judge/src/judge_lme/plugin.yaml:22`; `agentscope-ai/ReMe@4f7c8786:plugins/beam-judge/src/judge_beam/plugin.yaml:24`; `agentscope-ai/ReMe@4f7c8786:docs/en/configuration.md:94`; `agentscope-ai/ReMe@4f7c8786:reme/components/embedding_store/local_embedding_store.py:22`; `agentscope-ai/ReMe@4f7c8786:skills/reme_memory/SKILL.md:224`; `agentscope-ai/ReMe@4f7c8786:reme/steps/evolve/auto_memory.py:298,358`; `agentscope-ai/ReMe@4f7c8786:reme/steps/evolve/auto_memory_cc.py:116`; `agentscope-ai/ReMe@4f7c8786:integrations/claude_code/reme/hooks/auto_memory.py:34`; `agentscope-ai/ReMe@4f7c8786:docs/en/operations.md:30`; `agentscope-ai/ReMe@4f7c8786:pyproject.toml:45`; `agentscope-ai/ReMe@4f7c8786:benchmark/longmemeval/README.md:99`; `agentscope-ai/ReMe@4f7c8786:benchmark/longmemeval/download.py:18`; `agentscope-ai/ReMe@4f7c8786:reme/steps/common/health_check.py:200` | + +These corrections do not permit editing upstream source. Supported config and isolated launch adaptations must be frozen, with any unresolved image, lock, selected-repository gate, dependency SBOM/OSV scan, route or transport left pending. The private deployment-spec file is an input lead; the public evidence is the upstream locators above and the round-2 source receipt. + +**cognee request envelope (r6, retained).** For route (a), carry `LLM_ARGS={"temperature":1.0,"extra_headers":{"x-omniroute-session":"","X-OmniRoute-No-Cache":"true"}}` in a fresh process for each run. `` is resolved **once at process start**, not dynamically per call; this static-only configuration omits idempotency and request-ID headers, subject to 2.2's wire/replay check. Temperature 1.0 satisfies the dedup exclusion without requiring instructor streaming. r6 verified that `LLM_ARGS` takes precedence over folded sampling defaults at 1.6.1 ([config.py:271–298](https://github.com/topoteretes/cognee/blob/eb90d03740755f5252b8b12cce91fd09970f2d81/cognee/infrastructure/llm/config.py#L271-L298)); `llm_args` is still present at 1.6.2 (`config.py:193,201`), and the wire check confirms the precedence before scoring. The gateway removes temperature downstream for Codex models, so 1.0 concerns gateway eligibility, not a claimed GPT-6 sampling temperature. The successful instructor `tool_call` smoke supersedes the research inference that LiteLLM's JSON-object fallback would work: that fallback and the strict KnowledgeGraph schema returned 400 in the supplied smoke (I:25–35). BGE-small from that smoke is only a shipped-default diagnostic; configurable scored retrieval uses 8.2's selected local model. + +## 4. Dataset, metric and harness + +### 4.1 Dataset and tracks + +- **Dataset:** `longmemeval_s_cleaned.json`, sha256 `d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442`, 277,383,467 bytes, from [xiaowu0162/longmemeval-cleaned @98d7416c](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/98d7416c24c778c2fee6e6f3006e7a073259d48f/longmemeval_s_cleaned.json). It is static data under section 1. On 2026-09-30 the Hugging Face LFS oid of that file equalled the pin; the freeze hashes the downloaded file. + - 500 questions, 30 of them abstention questions, whose `question_id` ends in `_abs` (LongMemEval `README.md:81` at `9e0b455f`, cited). + - Full-track population: **n=470**. The upstream README states that retrieval evaluation skips the 30 abstention instances because they have no answer location (`README.md:206`, cited). r7 plans all 470 unless the **prospective 96 h feasibility rule** in 5.2 and section 6 selects a stratified subset before any confirmatory result. The full population and any selected subset each have a frozen manifest. + - Official track: n=419, which also excludes 51 of the 56 `single-session-assistant` questions; 5 remain (frozen `eligible-manifest.json`, checked against the pinned dataset). +- **Tracks (r5), reimplemented in r7:** + - **Official track:** upstream's user-text construction and relabelled gold IDs, reimplemented in the S3 adapter from the cited definition ([LongMemEval @9e0b455f `run_retrieval.py:202`](https://github.com/xiaowu0162/LongMemEval/blob/9e0b455f4ef0e2ab8f2e582289761153549043fc/src/retrieval/run_retrieval.py#L202), `process_item_flat_index`; citation only). + - **Full-session track (primary):** presents complete sessions to the native adapters and scores raw session IDs against `answer_session_ids`, including assistant-side evidence, following A2. This construction is a local extension. + - Both are scored by the S3 scorer (4.2) and frozen as question manifests with sha256. +- **What candidates see (r6, extended in repair round 2):** answers, `has_answer`, gold-session labels and scoring feedback are never visible to candidates. Session, question, `Query.id`, `Query.user_id`, `Document.user_id` and provider namespace IDs are opaque, independently assigned without raw-ID substrings, category names or `_abs` suffixes. AMB's stock adapter passes the raw question ID as `user_id` (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/dataset/longmemeval.py:296`); the S3 adapters replace that field too. The evaluator alone holds the opaque-to-original map and judge-only metadata; no provider request receives that map or category/gold annotations. +- **Development separation (r6, extended):** freeze separate development question/history manifests for role-effort, optional-stage, local-model and volume/power checks, with source hashes and overlap checks against all 500 LongMemEval QA questions, both retrieval tracks and every confirmatory LongMemEval-V2 trajectory/question. No confirmatory question, answer, gold label, score or per-model leaderboard number may choose an embedder, reranker, effort, optional stage or prompt. Published numbers motivate the shortlist only. The retained 25 coding/research-memory diagnostics are also excluded from tuning (R:41–53; coordinator brief B:E). A cost/power preflight may inspect outcome-free corpus size/strata but never tune on held-out answers or model outcomes. +- **Ranking (r6, retained):** artifact-to-session ranking and expansion (graph nodes, summaries and pages that reference several sessions) are frozen per adapter before any scoring. Ranked source-session IDs are deduplicated before top-5 selection. The scorer receives the complete eligible corpus IDs. A system whose output cannot be mapped to source sessions gets retrieval `N/A`, never a constructed score. +- **Dependence (r5, retained):** questions share evidence histories, so resampling is by cluster (section 6). Construction: union questions whose gold sessions share identical canonical sequences of `(role, content)`, excluding session IDs and gold annotations from the content hash (this repository's v4 `lme_summarize.py:234`, `clusters`). Counts: 452 full-track clusters (434 singletons and 18 two-question clusters) and 401 official-track clusters (383 and 18), reproduced with the frozen function on the pinned dataset. Session-ID matching would give 465 and 414, understating dependence. The manifest is frozen and hashed before scoring. + +### 4.2 Metric definitions (the S3 scorer, in this repository) + +- **Source.** The LongMemEval paper (Wu et al., "LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory", ICLR 2025, [arXiv:2410.10813](https://arxiv.org/abs/2410.10813)), §3.3 "Evaluation Metric", paragraph "Memory Recall" (p. 6): because the benchmark has human-annotated answer-location labels, it reports "Recall@k and NDCG@k, where k is the number of top items retrieved by the system". At session level the labels are `answer_session_ids` (`README.md:88` at `9e0b455f`, cited). +- **Definitions**, for question q with non-empty gold set G_q and the system's ranking R_q of distinct source sessions (4.1): + - `recall_all@k(q)` = 1 if every session in G_q is among the first k sessions of R_q, else 0; + - `recall_any@k(q)` = 1 if at least one is; + - `ndcg_any@k(q)` = DCG_k / IDCG_k with binary relevance, DCG_k = rel_1 + Σ_{i=2..k} rel_i / log2(i), and IDCG_k the same sum over min(|G_q|, k) relevant items; + - each is averaged over questions and then seed-averaged as in section 6. +- These are the "all" and "any" session forms of the paper's Recall@k, and its NDCG@k, as upstream's reference code writes them ([`src/retrieval/eval_utils.py:4-29`](https://github.com/xiaowu0162/LongMemEval/blob/9e0b455f4ef0e2ab8f2e582289761153549043fc/src/retrieval/eval_utils.py#L4-L29) at `9e0b455f`, cited, never executed). Some questions have up to 6 gold sessions, so a perfect `recall_all@5` is unattainable for them; this is reported, not corrected. +- **Why reimplemented:** the stale-repository rule (section 1). Separately, `eval_utils.py:6` calls `np.asfarray`, which NumPy 2.0 removed ([`doc/source/release/2.0.0-notes.rst:197`](https://github.com/numpy/numpy/blob/v2.5.3/doc/source/release/2.0.0-notes.rst#L197)), so it cannot run unchanged under the pinned NumPy 2.x. +- **Conformance:** the scorer ships with unit tests written from these definitions (duplicates in R_q, |G_q| > k, gold sessions absent from R_q, rankings shorter than k), reviewed against the cited text before freeze. No upstream code runs. + +### 4.3 Harness + +AMB at [`03c1d0f1d27da63034f0931121c858faba512383`](https://github.com/vectorize-io/agent-memory-benchmark/tree/03c1d0f1d27da63034f0931121c858faba512383) runs **byte-identical**; the freeze bundle verifies the checkout's tree hash. It is also AMB's default-branch head on 2026-09-30. All S3 code lives in this repository: + +1. **LongMemEval dataset adapters** on the pinned file (4.1). Each is passed as the `dataset` instance to unmodified `EvalRunner.run` ([`runner.py:51-71`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/runner.py#L51-L71)). Each builds documents with opaque session IDs and an ID-free context (r6 N2, Appendix A) in place of AMB's `"{question_id}_{session_id}"` IDs and ID-bearing context ([`dataset/longmemeval.py:285-289,329-342`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/dataset/longmemeval.py#L285-L342)): + - **retrieval variant**: `task_type = "retrieval"`, run in AMB's LLM-free `retrieval` mode. For a retrieval-typed dataset the runner scores only `raw_response["documents"]` ([`runner.py:222-227`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/runner.py#L222-L227)), which only [`modes/retrieval.py:56-61`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/modes/retrieval.py#L56-L61) supplies. In `rag` mode that key is absent, so the brief's "rag mode only" governs the next variant, not this one. `retrieval_limit` is the common depth (4.5), `score_retrieval` only exports each ranked opaque-ID list, and the S3 scorer (4.2) computes the metrics; + - **open-QA variant**: `task_type = "open"`, run in AMB's `rag` mode only, because `agentic-rag` constructs `GeminiLLM()` whenever no LLM is passed in ([`modes/agentic_rag.py:8,26-27`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/modes/agentic_rag.py#L26-L27)). It inherits AMB's LongMemEval answer prompt (`build_rag_prompt`, `dataset/longmemeval.py:129-168`) and per-category judge prompts (`get_judge_prompt_fn`, `:177-258`) unchanged (5.5). +2. **Thin `MemoryProvider` clients** (HTTP, MCP or CLI) to each system's current upstream deployment (3.5). They implement AMB's interface ([`memory/base.py:8-79`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/base.py#L8-L79)) without changing ingest or recall semantics, including each system's native store configuration (section 2). Since repair round 1 every system uses one, Hindsight included. +3. **No AMB provider is reused (r7 repair).** + - r7 had reused AMB's `hindsight-http` provider for Hindsight ([`memory/hindsight.py:1242-1255`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/hindsight.py#L1242-L1255)). That provider cannot keep Hindsight's native pipeline: + - It creates every bank with `enable_observations=False` ([`_bank_kwargs`, `:218-219`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/hindsight.py#L218-L219)), on the synchronous path (`:236-243`) and on the asynchronous path that AMB's per-unit ingest takes (`:425-433`). The 0.10.2 server stores that field as a bank override ([`api/http.py:1951-1954,1989-2030`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-api-slim/hindsight_api/api/http.py#L1951-L2030); `PUT /v1/default/banks/{bank_id}`, `:8364-8384`), and a bank with it `false` runs no consolidation at all, automatic or manual ([`memory-banks.mdx:242`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/api/memory-banks.mdx#L242)). Hindsight's documented defaults turn observations and automatic consolidation on ([`config.py:1711-1712`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-api-slim/hindsight_api/config.py#L1711-L1712); `memory-banks.mdx:240-246`; `configuration.mdx:2257-2258`). AMB has no setting that restores them: its only bank-level overrides are `AMB_RETAIN_MISSION` and `AMB_RETAIN_EXTRACTION_MODE` (`:224-233`). + - Its asynchronous recall ignores the `k` that AMB's retrieval mode passes and sends `budget: "high"` with `max_tokens` 32768 (`:708-747`). + - It builds its synchronous client from AMB's locked `hindsight-all` 0.4.17 (`from hindsight import HindsightClient`, `:1252-1255`; that wheel's `hindsight/__init__.py:60`). + - A timed-out or five-times-failed asynchronous recall returns an empty list (`:1155-1180`). + - Hindsight therefore gets a **thin local provider** in this repository. It speaks the 0.10.2 service's documented REST API over `httpx`, as AMB's own provider already does for recall (`:838-861`). It: + - creates each bank with no configuration field, so the bank keeps the server's defaults; configuration fields belong to the separate config API, not to bank creation ([`memory-banks.mdx:508`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/api/memory-banks.mdx#L508)); + - before any ingest, reads `GET /v1/default/banks/{bank_id}/config` ([`api/http.py:9380-9382`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-api-slim/hindsight_api/api/http.py#L9380-L9382)), whose response separates the resolved `config` from the bank's `overrides` (`memory-banks.mdx:546-548`). It proceeds only if `overrides` is empty and the resolved `config` matches the frozen effective settings, whose canonical-JSON sha256 is frozen at freeze from a development bank created the same way. A mismatch stops the run before ingest; + - treats ingest as complete only when `GET /v1/default/banks/{bank_id}/operations` ([`operations.mdx:109-117`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/api/operations.mdx#L109-L117)) lists no pending or processing operation of any type for the bank, including the `consolidation` operations that follow each retain (`operations.mdx:67-73`). A failed operation is a logged ingest failure; + - maps facts and observations to source sessions under the common expansion rule (4.1, 4.5); an observation reaches its sessions through the facts it cites (`source_fact_ids`, [`recall.mdx:163`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/api/recall.mdx#L163)); + - uses the common depth policy and retry policy of 4.5. + - AMB's `supermemory` provider is **not** reusable against the local server. Its indexing-status poll hard-codes `https://api.supermemory.ai/v3/memories/{id}` and sends the API key there ([`memory/supermemory.py:77-85`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/supermemory.py#L77-L85)). It ignores `k` in favour of 30 results (`:7,113-119`), and it returns one merged context document (`:127-128`), so no session ranking exists. The provider's SDK honours `SUPERMEMORY_BASE_URL` (supermemory 3.28.0 `_client.py:113-115`), but the poll does not. + - supermemory-local therefore uses a thin client in this repository. This departs from the brief on the evidence above. +4. **AMB's locked libraries are transitive dependencies, not candidate implementations (accepted user decision 3).** AMB imports every provider when importing the memory package (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/__init__.py:1-13`). This pulls in `rank_bm25` 0.2.2, `mem0ai` 1.0.5, `qdrant_client` and `sentence_transformers` even with thin providers (`src/memory_bench/memory/bm25.py:1`, `mem0.py:5`, `hybrid_search.py:5` at that pin). `cognee` 0.5.4 and Hindsight libraries are also inventoried from the lock; they implement no scored candidate. Include all these dependencies in each AMB arm's SBOM and clean OSV scan. Flag `rank_bm25`'s 2024-10-08 last commit as stale (1); it does not block the harness. **AMB execution remains pending for missing SBOM/vulnerability acceptance**, not for a transitive Maintained verdict. Candidate services still use the selected releases in 3.1, never the harness's older bundled providers. +5. **The frozen dispatch wrapper (r6, retained)** holds no ingest, retrieval, scoring or statistics logic. It launches per-unit `EvalRunner.run(..., unit=)` processes ([unit filter 119–121](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/runner.py#L119-L121)) in the frozen block-randomized order at the frozen process concurrency, and records each process identity and exit status. It exists because AMB runs questions one at a time in one process, and its provider `concurrency` bounds only queries within a unit (runner.py:282–385, 338–363). The parity audit and the bundle review cover it. The v4 driver (#386) remains a cited adapter/provenance reference, never an executable S3 runner. + +**Output ownership (repair round 2, M4).** Allocate a fresh `EvalRunner(output_dir=)` for **each host × lane × track × opaque question × seed × arm × attempt**. `` is an owned location outside the checkout, with a different directory for every tuple. AMB's native result file is exactly `////.json` (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:474`). The run manifest names that file and the ranked-session export sidecar; analysis reads those exact paths, never a glob, shared arm directory or inferred last file. Only after process completion does the collector verify the expected opaque query IDs, final file/hash, attempt status and sidecar, then reconcile them against the frozen manifest (4.5). Concurrent processes never write one native result file: final saves filter to the current query set and `_save` performs non-atomic read/merge/write (`runner.py:459`, `runner.py:492` at the same pin). Repeated attempts retain separate files and usage; only the frozen retry policy chooses the final score. + +**Open-QA parity (repair rounds 2–3, M3/M5).** Both `retrieve` and `async_retrieve` on **every** thin provider explicitly default to `k=20`; do not inherit the base class's async default of 10. RAG mode omits k (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/modes/rag.py:49`; `src/memory_bench/memory/base.py:54`). After 4.5's common escalation, the open-QA provider returns **the shortest prefix of the final ranking that covers five distinct source sessions, capped at k=20 items**, with `raw_response=None`. At exhaustion/cap, return the available prefix up to 20 and flag fewer than five sessions; never reorder, replace items or add gold evidence to fill it. Retrieval-only scoring keeps the full final ranking and its first-five-session expansion. The lane is frozen in the provider configuration, not inferred from question/category labels or the value of k. AMB formats every returned document into the answer context (`modes/rag.py:52-54`), so limiting the request's initial depth alone was insufficient. The unchanged LongMemEval answer prompt uses the same formatted text contract for all arms (`src/memory_bench/dataset/longmemeval.py:131` at that pin). Provenance/session mappings remain evaluator-only. Record the exact emitted memory-context **o200k tokens per question×seed×arm**, plus per-arm totals/distributions, prefix items/sessions and native reader input tokens separately (5.2). Development parity checks exercise omitted-k RAG calls, explicit-k retrieval calls, cap/exhaustion cases and emitted prompt bytes for every provider. + +The adapters are offered upstream to AMB after the S3 decision; the offer is not a precondition. + +**LongMemEval-V2 is the main usefulness harness, not a finalists-only diagnostic (user decision, 2026-09-30).** Its native backend contract, commands, baseline and independent usefulness family are frozen in 5.5 and section 6. AMB retrieval remains the primary ranking lane. + +### 4.4 Runtime settings for every AMB process + +- `OMB_ANSWER_LLM=openai`, `OMB_ANSWER_MODEL=cx/gpt-6-astra-max`, `OMB_JUDGE_LLM=openai`, `OMB_JUDGE_MODEL=cx/gpt-6-astra-max` ([`llm/__init__.py:21-40`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/llm/__init__.py#L21-L40)). The class named `GeminiJudge` takes `get_judge_llm()` (`judge.py:36-39`), and the runner builds it at `runner.py:42`. +- `OPENAI_BASE_URL=http://127.0.0.1:20128/v1` on the workstation, and the Mac's own loopback gateway on the Mac (section 9). The locked `openai` 2.26.0 SDK reads it (`_client.py:164`). `OPENAI_API_KEY` is the gateway key, handled per `docs/secret-storage.md` and never recorded. AMB's client requests strict `json_schema` output (`llm/openai.py:32-39`). The brief relays that the gateway supports this; the wire check must confirm it before scoring. +- `GEMINI_API_KEY` is a non-secret placeholder for an unused provider. AMB's CLI only checks that a Gemini or Google key is present ([`cli.py:25-30`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/cli.py#L25-L30)). The dispatch wrapper does not import the CLI. With both `OMB_*_LLM=openai`, in `rag` or `retrieval` mode, no Gemini or Groq client is constructed (`llm/__init__.py:21-40`; `modes/rag.py:33-35`). Egress to Google and Groq endpoints is denied at the arm boundary, so any accidental call fails loudly, and the wire proof records zero such connections. +- There is no `.env` in the checkout, because `cli.py:12` would load one with `override=True`. `EvalRunner(output_dir=…)` points outside the checkout (`runner.py:40,474-475`). `LONGMEMEVAL_DATA_PATH` names the pinned file, never the mutable download (`dataset/longmemeval.py:73-90`). +- **Envelope.** AMB's OpenAI client sends none of the 2.2 headers (`llm/openai.py:14,32-39`), but it omits `temperature`: + - the gateway's semantic cache cannot serve these calls, because `isCacheableForRead` requires `temperature` to be exactly 0 ([`src/lib/semanticCache.ts:483-493`](https://github.com/diegosouzapw/OmniRoute/blob/c1e30b7676975feb298b49eff6ff58923c04b89e/src/lib/semanticCache.ts#L483-L493), identical at `a58000c7` and at `v3.8.51`); + - an absent temperature resolves to 1.0, which makes the calls dedup-ineligible (2.2 item 3); + - the missing session header affects only account affinity; + - before scoring, the wire check confirms in `call_logs` that no answer or judge call was served from cache or replayed. + +**Header-limited clients (repair round 2, M6; extends the exception in 2.2 item 3).** A client without arbitrary gateway headers can use the same exemption only when pinned-source request review and a complete per-attempt `call_logs`/wire receipt prove endpoint, model/effective effort, within-run affinity, zero response-cache hits, zero dedup/replay and complete usage attribution for every nested/retried call. Missing headers are not themselves proof of these properties. Where the request shape cannot exclude cache/dedup, freeze a separate per-arm gateway route using **supported native controls**, with its source locators, configuration hash and measured negative controls before scoring; do not patch the client or silently alter the shared route. + +agentmemory's auth-header builder provides no S3 custom-header control (`rohitg00/agentmemory@2d38dafe:src/providers/_openai-shared.ts:148`); its omitted-temperature path may qualify for the measured exemption, but no such receipt exists here. MemPalace's OpenAI-compatible client sends only auth and temperature **0.1** (`MemPalace/mempalace@22fd87f0:mempalace/llm_client.py:347`), which is **dedup-eligible** under 2.2, so the AMB reasoning cannot exempt it. **Both scored configurations are pending now** until the appropriate verified exemption or per-arm route is frozen. AMB's own exemption likewise remains conditional on its wire receipt. A successful response or an absent cache marker alone does not establish eligibility. + +### 4.5 Vendor-bias controls and parity + +AMB is maintained by vectorize-io, Hindsight's vendor. The controls: +1. **A primary metric with no LLM judge:** `recall_all@5` (4.2, 5.1). +2. **Equal depth, defined operationally (r7 repair).** + - *Scoring depth:* the first 5 distinct source sessions of each ranking (4.2), for every provider. + - *Initial request depth:* k = 20 for every provider. That is AMB's retrieval-mode default ([`modes/retrieval.py:34,45-47`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/modes/retrieval.py#L34-L47)); both thin-provider methods also default to 20 for open QA (4.3). + - *One escalation rule (repair round 2, M5):* expand each native ranked artifact through the frozen source-session mapping, deduplicate in first-occurrence order, and stop as soon as it represents **five distinct source sessions**, the store is exhausted, or the frozen native cap is reached. Otherwise rerun the same native recall with depth multipliers **1, 2, 4, 8, 16**, clamped to its documented maximum. Count APIs start at 20 items (global cap 320); token-sized APIs start at their upstream-default token limit (global cap 16× that limit). Hindsight starts at `max_tokens=4096` and `budget=mid` (`vectorize-io/hindsight@5fc4ce20:hindsight-api-slim/hindsight_api/api/http.py:448`). The adapter manifest freezes each supported knob/cap and item→session expansion before scoring; a smaller immutable native limit is labelled and retained, never bypassed. Every provider uses this stopping rule, including count-based systems. No score/gold label chooses a depth, and the last returned ranking, not a best-of-attempts ranking, is scored. Record each request's depth, native items, expanded distinct sessions, stopping reason, usage and total latency. Unsupported escalation is recorded as a fixed native cap; unmappable output remains N/A (4.1). + - *Recall effort:* upstream defaults; for Hindsight, `budget` `mid` (`api/http.py:448`; [`configuration.mdx:1630`](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/configuration.mdx#L1630)). A documented non-default level, such as the `budget: "high"` that AMB's replaced provider sent (`memory/hindsight.py:743`), is an optional stage under section 2's rule. + - A ranking shorter than 5 distinct sessions at exhaustion/cap is scored as returned and flagged, with item/session counts and cap reason; it is never topped up from another arm or from gold evidence. Open QA uses the same retrieval escalation, then independently applies 4.3's shortest-prefix **20-item prompt cap**; a 320-item final ranking never enters the answer prompt in full. Report context tokens per arm, including the formatted headings/separators. If five sessions appear only beyond item 20, QA gets the first 20 and the insufficient-session flag. +3. **One parity checklist for every system, Hindsight included** (r6's pre-freeze audit, extended in repair round 1): + - native ingest and recall at the common depth above, with identical session mapping and deduplication, and one expansion rule for artifacts that cite several sessions, such as graph nodes, summaries, pages and observations (4.1); + - no oracle input, the frozen GPT-6 role matrix, and no vendor-specific shortcut; + - each system's effective store configuration, read back before ingest and matched to its frozen settings (for Hindsight, 4.3); + - wire proof that no provider receives a raw session ID; + - the actual ingest and recall paths as AMB calls them under the per-unit dispatch (`async_ingest` and `async_retrieve`), exercised on development fixtures. Ingest ends only when the system's own status interface reports that every background operation it started has finished. Injected failures at ingest and at recall (refused connection, timeout, server error) must each end in a logged failure, never in an empty ranking or a silently dropped document; + - one frozen retry policy (attempts, backoff and per-call timeout) in every thin provider; + - provider logs that separate a failed recall from an empty ranking. AMB's replaced Hindsight provider did not: it returned an empty list after a timeout or five failed attempts ([`memory/hindsight.py:1155-1180`](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/hindsight.py#L1155-L1180)). + + Registry coverage and source inspection are not parity acceptance. +4. **A GPT-6 adversarial audit** of the adapters, providers, scorer and checklist before freeze, in addition to the section 11 bundle review. Its findings are resolved before freeze. +5. **The vendor leaderboard and AMB's published results are ignored.** + +**Failure accounting (repair round 2, H1; baseline clarification in round 3).** Before any result, freeze the retry attempts, backoff, timeouts and a **5% final-failure threshold** for every system/lane. A scheduled question×seed that still fails ingest, recall, answering, judging or result publication after that policy scores **0**, with a failure reason, every attempt and all known usage retained. It remains in the frozen score denominator and paired arrays; a failed retrieval has zero on all 4.2 metrics. Valid empty retrieval is a scored outcome, not an execution failure. This is an S3 collector rule: AMB may abort at its asynchronous gather and publish no row (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:363`; `:385`), so native summary accuracy is not the analysis denominator. Reconcile the exact result files against the expected manifest, including zero-score failure records stored in the collector ledger, without editing native results. If final failures exceed 5% of scheduled unique question-seeds in **any required lane**, that arm is `pending`; its all-pairs p-values are 1, never removed from the planned family. **If the no-retrieval baseline exceeds 5% final failures, the entire main-usefulness family is pending**, with p=1 placeholders for all m_U pairs; zero-scored baseline failures cannot manufacture superiority. Exactly 5% does not exceed the threshold. Source-reviewed deterministic executions count once for this rate, though their result is reused in seed averaging. Unstarted work, missing observer coverage or an interrupted run awaiting its retry policy is pending, not an invented zero or a complete result. Shared baseline/judge/infrastructure failures are retained and reported; no post-result deletion repairs the denominator. + +**Executable open-QA prefix examples (synthetic integration rule, not native prompt acceptance).** The mapping is evaluator-side provenance from the final native ranking, as in 4.1. The answerer receives item text only. + +```python +def open_qa_prefix(items): + prefix, sessions = [], set() + for item in items[:20]: + prefix.append(item) + sessions.update(item["sessions"]) + if len(sessions) >= 5: + break + return prefix, len(sessions) < 5 + +items = [{"id": i, "sessions": {str(i // 3)}} for i in range(320)] +prefix, short = open_qa_prefix(items) +assert [item["id"] for item in prefix] == list(range(13)) and not short +late = [{"id": i, "sessions": {"same"}} for i in range(20)] + [ + {"id": 20, "sessions": {"b", "c", "d", "e"}}] +prefix, short = open_qa_prefix(late) +assert len(prefix) == 20 and short +assert open_qa_prefix([{ "sessions": {"a", "b", "c", "d", "e"}}] * 20)[0] == [ + {"sessions": {"a", "b", "c", "d", "e"}}] +assert open_qa_prefix([]) == ([], True) +print("4.5 open-QA prefix examples: all assertions hold") +``` + +### 4.6 Isolation, cache proxy and private inputs (r6, retained) + +- **Isolation**, which AMB and every provider-driven process must satisfy before any scored run (requirements from the GPT-6 reviews on #386): + - owned ports, stores, homes and caches inside the fresh per-arm container/volume/home boundary in 3.5, with lane-owned `HOME` and `HF_HOME`; + - server readiness that checks identity; + - process and process-group identity (pid, start ticks, boot id, executable) checked before any signal; + - fail closed on an occupied port or unreadable identity; protected workstation ports include **49474, 3800, 3710 and 5433**, with no scored allocation, signal or store access to those services; + - `env -i` for vendor harnesses; + - production services and stores are never touched. On the workstation that includes ai-memory on 127.0.0.1:49474, cognee-live on 3800 and hindsight-live on 3710/5433 (I:37–54). +- **Deny-by-default runtime egress (repair round 2, M2).** Permit only owned loopback/arm-local service addresses (including its database/sidecars), the exact host-local gateway endpoint and the selected local embedder endpoint; block **all other destinations**, DNS/external fallbacks and telemetry included, for every process/sidecar and IPv4/IPv6 path. Container-to-host transport is a narrowly verified endpoint exception, not permission to reach the whole host/subnet. External dependency/model downloads happen only in separate provisioning, with frozen artifacts available offline for scoring. Verify both allowed calls and refused external connection attempts, and retain value-free network observations. Do not rely on endpoint env variables or telemetry log suppression: cognee defaults embeddings to OpenAI (`topoteretes/cognee@ba3631f2:cognee/infrastructure/databases/vector/embeddings/config.py:19`), memsearch defaults its embedding and compact providers to OpenAI (`zilliztech/memsearch@2a4652fa:src/memsearch/core.py:62`, `:274`), agentmemory's fallback URL is `api.openai.com` (`rohitg00/agentmemory@2d38dafe:src/providers/_openai-shared.ts:25`), and MemPalace only suppresses a Chroma telemetry logger (`MemPalace/mempalace@22fd87f0:mempalace/__init__.py:59`). Any unverified network boundary leaves the arm pending. +- **Embedding-cache proxy: omitted, preregistered (r4).** v4's A16.3 says the cache gives no reuse between arms, because ai-memory writes wall-clock timestamps into session pages, and stays "only for byte-identical repeats, such as infra retries", with no arm ever cache-only (v4 `PREREGISTRATION.md:566-573`). So neither the v4 proxy (sha256 `e08e11f8…`) nor #380's is used. Omitting it means only that a byte-identical repeat is recomputed rather than served from the cache. Each run records its embedding server's runtime and device, so a recomputation difference stays attributable. +- **Private inputs (r6).** S3 freezes its own model/effort matrix, sanitized gateway identity and wire-check receipts. v4's Modelfiles, `embed_gates.json` and v3-check IDs gate v4's own driver, which S3 does not run. No credential store, authorization value or production memory store is an experiment input (2.1–2.2). + +## 5. Lanes + +### 5.1 Retrieval (primary lane) + +- **Metric:** `recall_all@5` (every supporting session in the top 5) on the full track (4.2). +- **Secondary:** the official track, and `ndcg_any@5`. +- **Arms:** one primary arm per system (section 2). Every departure from defaults is frozen. +- **Stale defaults:** MiniLM and BGE-small appear only in controls, diagnostics or genuinely fixed-provider configurations labelled **shipped default**. A provider/model switch changes the arm manifest before freeze, never after scores. Native pipelines that cannot host an alternative are recorded explicitly, not patched to manufacture a SOTA claim (I:21). +- **Diagnostics outside the family:** 3.4. + +### 5.2 Token ledger and workload cost + +**Frozen workload (r6, retained):** identical recorded coding sessions from this repository's own work, frozen with sha256; a frozen recall schedule (N recalls per session at fixed points); frozen queries, context boundaries and aggregation. + +**Ledgers**, per host and per system, never added to one another: +- **(a) Ingest and internal LLM tokens per system and role**, from OmniRoute `call_logs` native usage (uncached input, cached input, output and reasoning kept apart). This covers every call the system makes (capture, extraction, consolidation, summarization, reranking, query rewriting and retries), attributed by the frozen per-run correlation key. +- **(b) Recall payload tokens:** `o200k_base` counts of each recall's returned and injected payload with [gpt-tokenizer 4.0.0](https://github.com/niieani/gpt-tokenizer/releases/tag/4.0.0) (`fb04ebca53f662200e737caefe9a5ef372a5e41a`; this repository's token-report pin since #493, `docs/token-practice.md:273`). +- **(c) Per-session injected context in each client.** + - Claude Code: native usage from `claude -p --output-format json`, whose payload carries usage and `total_cost_usd` with a per-model breakdown ([headless docs](https://code.claude.com/docs/en/headless)). + - Codex: `codex exec --json`, whose `turn.completed` events carry `input_tokens`, `cached_input_tokens`, `cache_write_input_tokens`, `output_tokens` and `reasoning_output_tokens` ([`codex-rs/exec/src/exec_events.rs:20-21,50-72`](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/exec/src/exec_events.rs#L50-L72)). + - Each system is installed upstream-clean in a scratch `CLAUDE_CONFIG_DIR` or `CODEX_HOME` and runs the frozen workload. `CLAUDE_CONFIG_DIR` overrides `~/.claude` and holds settings, session history and plugins ([env-vars docs](https://code.claude.com/docs/en/env-vars)). `CODEX_HOME` must already exist as a directory ([`codex-rs/utils/home-dir/src/lib.rs:6-11`](https://github.com/openai/codex/blob/rust-v0.157.1/codex-rs/utils/home-dir/src/lib.rs#L6-L11)). + - The same client's no-memory control runs the same workload. Its usage is reported beside each system's as a signed difference, never subtracted inside C (r7 repair). + - The user's real `~/.claude` and `~/.codex` are never read or written; they stay frozen until Gate A window W closes. + - Scratch homes sign in through a supported non-interactive route the user sets up; no credential is copied from the real homes (section 11). +- **(d) Latency p95 and resources** (5.3). + +Count once: a payload count is never added to a usage total that already contains it. Unknown usage stays unknown and leaves that system's cost `pending`, never zero. MCP-only paths are measured at the client tool-call boundary. Payload-only counts support only a payload-size claim; being under a budget is not a token-saving claim. + +**Ingest reuse and volume preflight (repair round 2, L4).** Open QA on a selected non-abstention full-session question may reuse retrieval's ingest only if corpus bytes, track construction, arm settings, opaque-ID map and seed are identical, and an upstream-supported pre-query snapshot/restore provides an independently verified clean copy of the fully ingested store. The query/output/affinity identities remain separately scoped by lane; QA cannot inherit retrieval-induced state changes. If native snapshot/reset cannot establish this, ingest separately and price/project that extra work. Official-track construction and the 30 abstention histories require their own ingest unless an identical eligible snapshot is already recorded. LongMemEval-V2 has different trajectories and never reuses LongMemEval-S ingestion. Charge actual reused ingest once, with the shared artifact hash and reuse edges recorded. Before freeze, project retrieval, optional QA, main usefulness, secondary TTL/CR, development, reader/judge and retries separately; choose and hash their prospective subsets using measured cost and power, then publish the combined host volume without double counting. + +**Decision cost scalar C (tie-break 1 in 7.3; repaired in round 1).** +- C is the **gross, nonnegative** priced cost of the frozen workload above, per host and system. It sums, over every model invocation that workload causes in ledger (a) (the system's own calls while serving it) and ledger (c) (both clients' own model calls in its sessions), the sum over the invocation's token classes (uncached input, cached input, cache write, output including reasoning) of tokens × the frozen list price of that model and class. Usage from LongMemEval and the other benchmark lanes is reported in ledger (a) but never enters C. +- The control difference ΔC = C − C₀, where C₀ is the same clients' no-memory control on the same workload, is reported beside C and may be zero or negative. It never enters 7.3, because a ratio of control-subtracted costs moves with the baseline: gross costs of 120 and 130 stay tied under 7.3's 0.80 ratio, while 20 and 30 left after subtracting 100 would not. +- C is computed exactly, as a rational number from integer token counts and the frozen decimal prices, so that 7.3's equalities are exact. +- The price table is frozen at freeze for every model ID involved, with the authoritative model/provider price source, exact route-to-model mapping, token classes, currency/unit, retrieval date and saved-page sha256. A gateway alias or embedded cost-map estimate is not a list-price source. **The `cx/` GPT-6 routes are currently `unpriced`**: this offline repair has no authoritative price/mapping receipt and invents none. +- **One missing-price policy (repair round 2, L2):** omit unpriced invocations from the labelled priced subtotal, list their measured token classes separately, and leave **total C unavailable** if any workload invocation has unknown usage or an unpriced class. Never call the subtotal C, substitute zero, use subscription spend as a token price or compare a partial total to a complete one. Unknown prices do not exclude a system on merit (7.1); when 7.3 could change the selection, its cost decision is unresolved. A model-price receipt or a dated pre-result amendment is required to resolve it. +- **Unpriced local inference (repair round 2, L1):** local embedder/reranker/generative inference has no list-token charge inside C. Report its serving time, hardware/energy assumptions and resources separately (5.3); the resulting C comparison is a **priced-model-call cost** comparison, not total operating cost. At acceptance state the exact boundary: a member j with C_j > c_min leaves the cost-tied set iff **c_min ≤ 0.80 C_j**, equivalently **C_j ≥ c_min / 0.80**; all exact minima, including zero, stay. Report sensitivity with an explicitly labelled omitted local cost L_i: the total-operating-cost boundary would use **min_i(C_i + L_i) ≤ 0.80(C_j + L_j)** for non-minima. Do not insert guessed L_i into the frozen decision after results. +- 8.3's RAG decision uses C_RAG, defined there. + +**r6 ingest budget, throughput and wall-clock feasibility (retained).** Use **about 115K extraction-input tokens per LongMemEval-S question per GPT-6-powered system** as coordinator brief B:G's planning assumption, not a measured total for every architecture (coordinator brief B:G; the approximately 115K-token history scale is also documented in [Attemory's pinned benchmark overview:125](https://github.com/AttemorySystem/attemory/blob/603c03afa9a04e48b778922eae156a0024752f37/README.md#L125)). Native calls may revisit text, and some systems have no extraction role; measure each complete workload rather than treating this estimate as usage. + +- Full 470 implies **54.05M** extraction-input tokens per system per ingest pass. Five independent ingest/seed passes imply **270.25M** per such system per host; if all twelve systems incurred that rate, the planning envelope would be **3.243B per host**, **6.486B across both hosts**. These are arithmetic scenarios, not invoices or accepted evidence. QA's 30 abstentions, the official-track construction when separately ingested, development, diagnostics, replies/reasoning, reflect/consolidation and retries add work. Record actual overlap once; do not multiply reused artifacts or sum cached subsets twice. +- **Concurrency is a frozen throughput parameter, not a quota gate.** The user's recorded 2026-09-27 direction for GPT-6 is **“full parallel utilization”** (relayed at V:32 from memory `provider-limits-and-logins.md:17–25`, historical evidence, not authority); coordinator repair decision 2 applies it as the K rule below. Use **K in-flight GPT-6 calls per host**, with K chosen on the development split from **measured throughput and fresh 429s**. Set it through each system's supported knobs, including nested/background calls, record the departure from defaults, and verify effective concurrency from `call_logs`; unsupported enforcement leaves the configuration pending. Use a **block-randomized dispatch order, identical across arms**, with its randomization seed frozen. Preserve conversation affinity and each host's own gateway; record any shared-account contention. Seeds replicate **only stochastic components**: an arm shown deterministic by pinned source review is **ingested once and counted once**, with the same value used in paired seed comparisons without inventing independent runs or duplicate usage. +- **Wall-clock decision:** the confirmatory retrieval run has **96 hours of wall-clock per host**, covering its selected arms, ingestion and retrieval, plus separately ingested track work, startup and retry overhead. This is the **coordinator's decision under delegated authority**, and the **user may overturn it before the freeze**. Before freeze, record development **tokens/s**, measured ingest/query durations, effective K and startup/queue overhead, and the projected wall-clock for **each host** under the complete planned work. If either projection exceeds **96 h**, adopt B:G's **prospective stratified subset by question type and dependence cluster**, with section 6's power calculation for the pairwise Holm family, before **any confirmatory result**. Freeze the selected manifest and recomputed cluster counts; do not truncate retrospectively to fit the window. If no justified subset fits, the retrieval plan remains pending before freeze. +- Preserve native pool cooldown/backoff and current `Retry-After` handling. Fresh 429s, timeouts or provider exhaustion are recorded with usage and handled by supported backoff. **Never gate on a recorded quota**, infer a current refusal from an old receipt, or replace GPT-6 with a stale model. Unexpected window overruns preserve incomplete work as `pending` and require a dated amendment, never an unplanned subset or early success stop. The separate latency lane in 5.3 excludes other arms' in-flight calls. Sources: coordinator repair decision 2 and B:G; I:35 is one small canary, **not** measured full-run throughput; R:31–53 supplies the development role comparison, **not** a pacing limit. + +### 5.3 Latency and resources (r6 4.3, retained) + +- **Measured:** client-observed time until usable evidence is delivered, including embedding, expansion, reranking and IPC or network time. +- **Frozen per host:** corpus, request schedule, 200 requests, concurrency 1, a 20-request warm-up discarded, and the nearest-rank quantile estimator. +- **Distributions:** + - cold (the first request after a server start, 10 starts) and warm, reported separately; + - the **warm p95 is the gated statistic**; + - timeouts (10 s) and failures count at the timeout value and are reported. Latency is never computed from successful requests only. +- **Query-path GPT-6 calls are inside the gated warm p95**. The **1.0 s bar and 10 s timeout are unchanged**; a native query path requiring GPT-6 may be **latency-ineligible**, declared before execution. Run the latency lane with **no other arm's call in flight on that host**, keeping its frozen concurrency 1. Report dispatch queue wait separately; queueing/internal work after query submission remains in client-observed latency. Throughput-lane scheduling cannot hide or contaminate this gate. +- Also measured: peak RSS, VRAM on the workstation, disk growth, ingest-to-query-ready time, and cold-start time. + +### 5.4 Coding-agent acceptance and early lifecycle screen (r6 4.4, retained) + +**Coding-agent acceptance (both clients, official integrations only).** Order: Claude→Codex, then Codex→Claude. Isolated client profiles, with the enabled memory owners recorded explicitly. MCP-only systems must capture through real client tool calls. Required: +- capture of a unique decision, correction, failed approach and fix; +- scoped cross-client recall, with canaries in an adjacent project that must stay out; +- supersession of an updated fact; +- stored untrusted text treated as data; +- deletion, verified after restart and consolidation; +- restore into an empty instance; +- restart and interrupted-ingestion durability. + +**Early development lifecycle screen (r6), before expensive confirmatory ingest.** Run these on disposable development fixtures on **each host**. They triage reported risks; issue reports are not new reproductions or automatic disqualifications. Reproduce the exact selected pin/path, retain commands/output and independent observations, and distinguish a reproduced failure (`FAIL`) from an unexecuted check (`pending`). A failed required lifecycle item makes the arm ineligible under section 7; missing arms do not shrink the planned family. Passing these checks does not replace the complete acceptance above after retrieval ranking. + +| Report to probe | Frozen development check and required observation | Source | +|---|---|---| +| agentmemory **#1273**, governance deletion reports a match but deletes nothing | Capture a unique observation through each native client; delete through the supported intended path; prove absence in storage and BM25/vector/graph recall after restart and consolidation. A success response or `deleted:0` alone cannot pass. | [upstream report](https://github.com/rohitg00/agentmemory/issues/1273); M:45,128 | +| agentmemory **#1190**, additive restore / omitted stores | Snapshot records spanning the stores used by the arm; restore into a **fresh empty instance**, compare the full expected inventory and search/graph state, then verify a post-snapshot sentinel is absent. Separately probe restore into a dirty development instance to expose additive behavior; do not use it to reset scored questions. | [upstream report](https://github.com/rohitg00/agentmemory/issues/1190); [export/import at the pin](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/functions/export-import.ts#L306-L399) | +| agentmemory **#1389**, iii 0.11.2 memory growth/OOM | A bounded four-hour development hook-write soak, including provider-backoff windows, with frozen write schedule and host resource ceiling. Observe RSS/queue growth, completion and restart recovery; stop at the ceiling, preserving the failure. No claim that a shorter smoke rules out the reported sustained-load failure. | [upstream report](https://github.com/rohitg00/agentmemory/issues/1389); M:76,128 | +| MemPalace **P0 #1581**, HNSW corruption | Exercise the selected release's own overlapping MCP, hook and auto-ingest writers in one disposable arm; interrupt/restart, then verify index readability and every expected canary. Do not suppress native concurrency merely to avoid the report's trigger. The report names 3.3.0; behavior on the S3 3.10.0 pin must be measured. | [upstream report](https://github.com/MemPalace/mempalace/issues/1581); M:77,128 | +| Hindsight, reported **pg0 fsync-off** | Treat M's pg0 0.15.2 statement as an unverified lead. Inspect effective `SHOW fsync`, `SHOW synchronous_commit`, `SHOW full_page_writes`, database version and startup arguments in the isolated deployment; retain acknowledged-write inventory across interrupted ingestion and database/container crash/restart. Use upstream-supported durable external PostgreSQL if the embedded setup fails; freeze that choice before scoring. Process restart alone does not prove power-loss durability. | M:139,172; [upstream embedded-DB caveat at 0.10.2](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/installation.md#L37-L39) | +| Hindsight, reported **65K output-token requirement** vs GPT-6 limits | Check the frozen GPT-6 route's actual context/output limits and native retain behavior before ingest. At 0.10.2 the docs require at least 65,000 output tokens for listed-model parity (models.mdx:156) while advising the retain cap stay at its **64000** default (models.mdx:466; `config.py:1628`); 64000 does **not** prove 65000 output capacity or full extraction. Freeze any supported `HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS` adjustment, keep it greater than `HINDSIGHT_API_RETAIN_CHUNK_SIZE`, and probe truncation, schema completion, retries and retained-evidence completeness. An unknown GPT-6 limit remains a gate to resolve, not an invented number or a switch to an older model. | [models.mdx:156,466 at 0.10.2](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-docs/docs/developer/models.mdx#L156); I:58 | + +### 5.5 Usefulness + +**Main test: LongMemEval-V2 (ACCEPTED 2026-09-30; replaces the minus-one floor and S8 usefulness coupling).** Use `xiaowu0162/LongMemEval-V2@2cc8c540bdb87fe6761629b585e727e1c4704520`, active under section 1. It has 451 curated questions over long web/enterprise agent trajectories and five abilities: static state recall, dynamic state tracking, workflow knowledge, environment gotchas and premise awareness (`xiaowu0162/LongMemEval-V2@2cc8c540:README.md:32`, `:41`, `:50`). This measures experience accumulated over long histories; short single-session coding outcomes cannot veto its usefulness conclusion. + +- **Native integration.** Implement each thin backend against upstream `memory_modules.memory.Memory`, with `@register_memory`, a unique `memory_type`, native `insert(trajectory)` and `query(query, query_image=None)` returning typed text/image context; preserve the pinned service's ingest/recall semantics and shared-memory lifecycle. A memory-config JSON contains `memory_type` and `memory_params`. The harness exposes `--memory-config-path`, `--save-memory` and `--load-memory-dir`; it keeps question IDs/categories/gold/evaluator settings private and gives the backend only a random `query_invocation_id` (`xiaowu0162/LongMemEval-V2@2cc8c540:memory_modules/memory.py:25`, `:146`; **`README.md:219-223`**; `evaluation/harness.py:128`). Config JSON does not import custom classes: the S3 registration bootstrap must import its adapter before calling the unchanged harness entry point, following the published registration pattern; it contains no evaluator logic. Review and freeze that integration before native acceptance. +- **One native invocation per domain, including the control (round-3 item 1).** Use upstream data preparation/validation (`README.md:114-117`), then run **every arm through the same direct `evaluation/harness.py` entry point**, separately for `web` and `enterprise`. The no-memory control uses upstream **`evaluation/memory_configs/no_retrieval.json`**, byte-identical; `run_eval.py:188-189,297-354` constructs and calls that same native no-retrieval backend internally. Do not score the control through `evaluation/scripts/run_no_retrieval.sh`: its two-domain wrapper forwards temperature 0.6, top-p 0.95, top-k 20 and reader concurrency 16, omits reader reasoning effort and evaluator base URL (`run_eval.py:75-78,301-343`; `harness.py:853-854,875-880`). All custom backends and the control use the same frozen registration bootstrap, which only imports backend classes and calls unchanged `evaluation.harness.main()` with the direct-harness argv below. The bootstrap module in this template is a required, separately hashed S3 integration artifact; it is not built or run by this revision. Keep upstream source byte-identical. +- **Backbone and judge; frozen common arguments.** For every memory arm and no-retrieval control, freeze the following same direct-harness argv. Only the domain, its preselected runtime files, memory config and owned output directory vary by arm/domain/replication; the schedule seed varies by paired replication, identically across arms. Reader and evaluator models are **`cx/gpt-6-astra-max`**, both endpoints are the host gateway, and both CLI reasoning-effort arguments are **`high`**: reader choices are at `evaluation/harness.py:169`, judge choices at **`:193-196`**. There is no native `max` argument; the route suffix and 2.2 wire check must establish effective max. Reader/evaluator completion limits are 20,000/4,096, both timeouts 43,200 s, memory context cap 200,000, reader concurrency 16, prompt-build workers 1 and native reader thinking enabled. **Omit temperature, top-p, top-k, presence/repetition penalties for every arm**; the evaluator also omits sampling overrides (`evaluation/qa_eval_metrics.py:517-528`). Freeze the effective defaults and retry configuration as well. No Qwen-specific `top_k` or thinking-template request field is allowed on the GPT-6 route. + +```sh +# Template, one domain/arm/paired replication; variables name frozen public artifacts. +rtk python -m s3_lme_registration \ + --domain "$S3_LME_DOMAIN" \ + --questions-path "$S3_LME_QUESTIONS" --haystack-path "$S3_LME_HAYSTACK" \ + --trajectories-path "$S3_LME_TRAJECTORIES" \ + --memory-config-path "$S3_LME_MEMORY_CONFIG" --output-dir "$S3_LME_OUTPUT" \ + --model cx/gpt-6-astra-max --base-url "$S3_LME_GATEWAY" \ + --api-key-env OPENAI_API_KEY --reasoning-effort high \ + --max-completion-tokens 20000 --memory-context-max-tokens 200000 \ + --timeout-seconds 43200 --reader-max-concurrent-requests 16 \ + --prompt-build-max-workers 1 --reader-enable-thinking \ + --shuffle-questions-seed "$S3_LME_PAIRED_SCHEDULE_SEED" \ + --evaluator-model cx/gpt-6-astra-max --evaluator-base-url "$S3_LME_GATEWAY" \ + --evaluator-api-key-env OPENAI_API_KEY --evaluator-reasoning-effort high \ + --evaluator-max-completion-tokens 4096 --evaluator-timeout-seconds 43200 +``` + +- **Request-body wire acceptance is mandatory.** Capture sanitized reader and judge request bodies for every arm/control, domain and nested/retried call path. Compare all non-content parameters, including present/absent sampling and extra-body fields, completion limits, model and reasoning effort; confirm `--evaluator-base-url` actually routes the GPT-6 judge to the same host gateway. Record native concurrency, effective gateway/provider effort and 2.2's zero cache/dedup/replay and complete usage attribution. Deliberately add wrapper temperature/top-k in a development negative control: parity must fail. A matching argv, shell environment or route suffix does not prove matching emitted requests; any gap leaves that arm/control pending. +- **Prospective small-tier selection.** Start with tier **small**, both domains and all five abilities. Before any confirmatory result on either host, a held-out development preflight measures native ingest/reader/judge tokens, retries, startup, serving time and seed dependence. It first verifies section 6's outcome-independent haystack-component manifest and **minimum 20 clusters against the downloaded haystack/trajectory files**, then its cost/power estimate chooses full-small or a stratified subset by domain/ability and those fixed clusters, checking the minimum again for the selected sample. Freeze the manifests, selected IDs, downloaded-byte hashes, five replication IDs, run order, aggregation, context limits, budget/window and power at the actual sample/cluster count. Materialize the runtime files once **per domain**, using upstream `data.public_data.materialize_runtime_questions` and `materialize_runtime_haystack` (`data/public_data.py:79-124`), then use identical bytes for every arm and control in that domain. Each question-ID list contains only that domain's IDs; the native selector rejects a cross-domain list (`data/public_data.py:88-93`). `run_eval.py`'s question-ID flag is at **`:70`**, not `:69` (`--limit`); it is not a preparation-only command. No retrospective truncation, component splitting or best-of-seed selection. A shuffle seed is a schedule seed, not deterministic provider sampling. Deterministic ingestion may be reused once with provenance; stochastic reader/judge calls still get five paired replications. +- **Outcome and usefulness rule.** Aggregate native per-question correctness into question-weighted accuracy after averaging five paired replications. Final failed question-replications score zero under 4.5; missing execution and a baseline failure rate above 5% remain pending. Bootstrap section 6's **fixed shared-trajectory haystack components**, retaining each component's questions and all replications together, and apply Holm to the frozen usefulness family. **Fewer than 20 independent components leaves usefulness pending; no question-level substitute test is authorized.** A system passes if significantly better than no retrieval and not significantly worse than any point-estimate maximizer among operationally eligible, completely observed memory systems. A significantly worse system or lack of superiority evidence is unqualified, not evidence of no benefit. This is usefulness qualification; 7.6 separately defines the accepted fallback only when the qualification/selection is resolved. Report ability/domain strata, failures, power, cluster count and uncertainty. +- **Status.** No main-usefulness model run is claimed by this revision. Missing backend/native wire acceptance, dataset manifest, failure accounting, preflight or paired outcomes leaves this lane pending. + +**Secondary: MemoryAgentBench TTL and CR, applicable systems only.** Use `HUST-AI-HYZ/MemoryAgentBench@538026089d1a8a8eff05121d0db89b388f360eba`, subject to the selected-repository gate, dependency SBOM/OSV policy and isolation. Native TTL ICL uses `exact_match`; CR fact-consolidation uses `substring_exact_match` (`README.md:178-180`). The real script is `bash_files/sh/run_memagent_rag_agents.sh:18`, invoking `python main.py --agent_config --dataset_config `; the README's `bash_files/eniac/...` path is absent at this pin. Freeze supported config/commands on GPT-6 and the local embedder, with native wire/lifecycle acceptance, rather than the author's shell/GPU/model choices. **Secondary results are reported only for applicable systems/modes below; never reach a selected candidate through a vendored copy.** They do not replace the main test or exclude systems lacking a native route. + +The pinned dispatch recognizes `letta`, `mem0`, `cognee`, `zep`, `knowl`, `agentmemory`, `total_agent_memory` and `rag_*`, plus long-context controls (`agent.py:64-89`). Those names do not make the S3 memory deployments or the document-RAG candidates interchangeable. Applicability is source-reviewed at this pin, not a run or a universal capability claim: + +| Selected system | Pinned MemoryAgentBench path | Applicability / secondary status | +|---|---|---| +| agentmemory v0.9.29 | Native REST mode to the external service (`methods/agentmemory.py:3-17,111-128,142-190`) | **Applicable to normalized-fact CR only**, pending native acceptance. It parses and ingests prepared facts, not raw trajectory/session extraction. TTL is not established for this CR-specific parser (`:57-82`); no TTL result is claimed. | +| Hindsight v0.10.2 | No Hindsight dispatch at `agent.py:64-89` | Not applicable in the pinned native runner. | +| OpenViking v0.4.22 | No OpenViking dispatch at `agent.py:64-89` | Not applicable. | +| cognee v1.6.2 | `cognee` mode imports the benchmark's vendored tree (`agent.py:236-243,518-531`) | **Not applicable to the selected 1.6.2 release; vendored execution prohibited.** | +| MemPalace v3.10.0 | No MemPalace dispatch at `agent.py:64-89` | Not applicable. | +| basic-memory v0.23.2 | No basic-memory dispatch at `agent.py:64-89` | Not applicable. | +| memsearch v0.4.21 | No memsearch dispatch at `agent.py:64-89` | Not applicable. | +| ai-memory v2.4.2 | No ai-memory dispatch at `agent.py:64-89` | Not applicable. | +| claude-mem v13.28.0 | No claude-mem dispatch at `HUST-AI-HYZ/MemoryAgentBench@53802608:agent.py:64-89` | Not applicable. | +| MCP Memory Service v11.14.0 | No MCP Memory Service dispatch at `HUST-AI-HYZ/MemoryAgentBench@53802608:agent.py:64-89` | Not applicable. | +| MemMachine v0.3.9 | No MemMachine dispatch at `HUST-AI-HYZ/MemoryAgentBench@53802608:agent.py:64-89` | Not applicable. | +| ReMe v0.4.1.12 | No ReMe dispatch at `HUST-AI-HYZ/MemoryAgentBench@53802608:agent.py:64-89` | Not applicable. | +| engram (reserve) | No engram dispatch at `agent.py:64-89` | Not applicable. | +| Graphiti (reserve) | `zep` constructs `zep_cloud.Zep` (`agent.py:245-255`), not the selected Graphiti service | Not applicable; Zep cloud is no substitute. | +| supermemory-local (reserve) | No supermemory dispatch at `agent.py:64-89` | Not applicable. | +| HippoRAG 2 | `hippo_rag_v2_*` mode imports the benchmark's bundled tree (`HUST-AI-HYZ/MemoryAgentBench@53802608:agent.py:744-745,826-848`; `HUST-AI-HYZ/MemoryAgentBench@53802608:README.md:149` names 2.0.0a3) | **Not applicable to the selected 398bfdc3 pin; bundled execution prohibited.** | +| qmd, LightRAG, llama_index document-RAG arms | Generic `rag_*` modes at `agent.py:86-87,258-265` do not dispatch these selected pipelines | Not applicable to those arms; native RAG controls may be labelled separate diagnostics. | + +Letta/Mem0 remain excluded on fit (3.3), and Knowl/total_agent_memory are not S3 selected arms. Their dispatch presence is not admission to the family; a vendored implementation never supplies secondary acceptance for a selected release. Adding support requires a separate upstream-backed amendment before results. No MemoryAgentBench run is claimed here. + +**MemoryArena is citation-only.** `ZexueHe/MemoryArena@6cd9de14b71915e39ac742a20dc33785e14b6aab`, ICML 2026, last commit 2026-05-31, is stale under section 1. Never run its scripts, evaluator or packages. + +- **(a) Coding/research-memory diagnostics (r6 tasks retained):** retain **20 frozen queries plus 5 bounded continuation tasks**. **At least 8 of the 20** are US-equities research-memory queries, anchored below in recorded decisions and their evidence. Each of all 25 tasks gets frozen expected evidence, source hashes, semantic-success criteria, answerer/context policy and a binary success result. These are memory tasks for research agents: the north-star diagram sends scoped memory into research, whose output is evidence and proposals (`blueprints/us-equities/north-star.md:33–35,57–60,74–79`; `catalogs/us-equities/README.md:52–58`). This adds no strategy, data-availability or broker qualification claim. They are reported diagnostics, not the new usefulness qualification. +- **(b) LongMemEval QA (r7):** + - Plan all **500** questions through the open-QA adapter variant (4.3) in AMB's `rag` mode; the cost/power preflight may choose a prospective, frozen stratified subset, with its actual counts reported. All 30 abstentions remain a separately identified population/audit stratum; subset reports never claim full-500 coverage. QA is secondary and consumes the separately projected workload in 5.2. + - **GPT-6 Astra at max** (`cx/gpt-6-astra-max`) is both the fixed answerer and the judge, through each host's gateway (4.4). + - AMB's LongMemEval answer prompt and per-category judge prompts are used unchanged (`dataset/longmemeval.py:129-168,177-258`). This replaces r6's plan to send upstream LongMemEval's `get_anscheck_prompt`, which is now citation-only (section 1). + - **Abstention and empty context (repair round 2, L3).** AMB scores **any open-QA empty context wrong without judging**, even if the answer appropriately abstains (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:228`). Retain that native zero, mark it `empty_context_unjudged`, and report its count/rate by arm and abstention stratum; never claim it was a judge verdict. Non-empty-context abstentions use the unchanged base-category prompt, which has **no abstention branch**. Report native QA accuracy with that validity limit and separately audited abstention correctness; the audit cannot silently replace native scores. Every included abstention is in the manual-audit sampling frame. Provider-facing IDs have no `_abs` suffix or other category hint (4.1). + - Keep the **stratified manual audit of abstention and knowledge-update questions**, its sampling rule and adjudication record. + - The answerer sees only the question and the allowed retrieved context. The judge sees the question, the gold answer and the response, without arm or model labels, cost or latency results, or rankings. Shared model family and blinding do not establish judge validity. + - 5.5(b) stays `pending` until the answerer and judge calls pass the 4.4 wire check. + - Results are reported as **S3 QA scores with a gateway-hosted answerer and judge**, never as official leaderboard or local-inference scores. + +**Research-memory task anchors for 5.5(a) (r6, retained).** The following eight query intents and expected evidence are mandatory in the 20-query manifest; freeze exact wording, eligible memory packet and source hashes before scoring. Their gold evidence is evaluator-only. The other 12 queries and all 5 continuations must be fully specified in the freeze bundle as in r5. Each task must retrieve the recorded decision **and its boundary**, not merely mention a repository. Decision-index paths below identify the retained decision lineage, not new installations; their line numbers refer to the input checkout at `75e30ea4` (r6 §14). + +| ID / query intent | Expected evidence and semantic success | Frozen source locations | +|---|---|---| +| RM01 — What changed the SEC access decision? | Distinguish the initial native 403/refusal from the later tested-archive access resolution; retain the failed receipt rather than rewriting it as a success. | `catalogs/us-equities/decision-index.json:6352–6355` (EdgarTools bounded adoption); `blueprints/us-equities/catalyst-provenance/README.md:3–14,67–72` | +| RM02 — May the parsed real 8-K be used at a historical trading cutoff? | Recover the pinned real fixture and its missing acceptance metadata; explain its quarantine and distinguish native parsing/byte reduction from historical availability. | `catalogs/us-equities/decision-index.json:6330–6333`; `blueprints/us-equities/catalyst-provenance/README.md:16–23,108–114` | +| RM03 — What did catalyst dataset acceptance actually cover? | Recover 371 memberships / 360 accessions, five acquired headers, 366 unfetched rows and nine header item links; distinguish observed metadata from verified catalysts and a historically available universe. | `catalogs/us-equities/decision-index.json:6330–6333`; `blueprints/us-equities/data-readiness/README.md:12–15,38–46` | +| RM04 — Is the AAPL split-basis change an investment loss or engine acceptance? | Recover 499.23 → 124.8075 as a basis change, the 25-session factor/map scope, and the failed native test-host attempt with zero NUnit cases. Do not turn normalization into return/P&L or a passing upstream suite. | `catalogs/us-equities/decision-index.json:17233–17236`; `blueprints/us-equities/data-readiness/README.md:14–20,43–46` | +| RM05 — Can an FB/META `asof` query and today's asset UUID establish historical membership? | Recall the selected bounded Alpaca identity probe; distinguish request symbol mapping, current asset identity and observation availability from historical universe membership and original publication/revision times. | `catalogs/us-equities/decision-index.json:1592–1595`; `blueprints/us-equities/identity-readiness/README.md:17–41` | +| RM06 — How was the zero-volume META case continued? | Recall the original rejection and the decision to fetch only the saved continuation; keep source zeros plus a separate quarantine. All four chains completed; that is neither ten independent sessions nor an atomic revision snapshot. The task recalls the recorded action, not a new authenticated request. | `catalogs/us-equities/decision-index.json:1592–1595`; `blueprints/us-equities/identity-readiness/README.md:43–56,78–98` | +| RM07 — Can the 2021 segment be called an untouched holdout again? | Recall that the chronological control's reserved segment has been inspected, with 20 development folds and 21 frozen selections; keep unselected alternatives visible and prohibit retuning on their better outcomes. | `catalogs/us-equities/decision-index.json:19176` (skfolio record); `blueprints/us-equities/simulation-research/README.md:20–24,38–51` | +| RM08 — Do the SPY/QQQ/IWM results justify historical +200% mover research or engine P&L claims? | Retrieve the curated-control scope, lack of all-equities/delisted/mover/catalyst-arrival reconstruction and distinction between raw-price labels and engine accounting; identify the still-open historical-universe/publication/revision gate. | `catalogs/us-equities/decision-index.json:17248,19176`; `blueprints/us-equities/simulation-research/README.md:20–30,47–51`; `blueprints/us-equities/identity-readiness/README.md:114–120` | + +Use GPT-6 Astra max with the same frozen answerer/context policy for all 25 diagnostic tasks; publish their total and eight-task research-memory success counts. **There is no minus-one floor on these diagnostics.** Missing diagnostic evidence is labelled pending for that diagnostic and cannot substitute for main usefulness. Extraction and other system roles use the separately frozen 2.1 matrix. Without the main 5.5 test, the result is a retrieval/lifecycle comparison or an explicitly unqualified 7.6 fallback, not accepted positive long-workflow usefulness. + +## 6. Statistics (frozen before any confirmatory run) + +- **Sample commitment (r6, retained):** plan all 470 full-track questions, all six types and all **452** canonical clusters; the full official population is **419 questions / 401 clusters**, secondary. If the development projection exceeds the **96 h per-host window**, the only permitted reduction is B:G's **prospective stratified subset**, sized and frozen with the power calculation below under 5.2 **before any confirmatory result on either host**. Freeze its question and cluster counts rather than continuing to quote full-population counts for a subset; its official-track intersection stays secondary. No adaptive subsampling, retrospective cost-driven truncation or early success stopping is authorized. Five seeds improve within-question averaging for stochastic components; they do not create 2,350 independent questions. Source-reviewed deterministic arms are ingested once and counted once. +- **Frozen before the first confirmatory result:** + - the question-to-cluster manifest (sha256), built by 4.1's canonical `(role, content)` rule (452 full-track and 401 official-track clusters for full coverage; prospectively recomputed counts if 5.2 selects a subset); + - seed aggregation: the five paired seeds (20260927–20260931) are averaged within each question for stochastic components; a deterministic arm supplies one source-reviewed result without multiplying ingestion or usage; + - question-weighted differences; + - the RNG (NumPy `PCG64`, bootstrap seed 20260927); + - 10,000 cluster resamples. Each resample keeps every question of each drawn cluster. +- **Upstream statistics binding:** pin **scipy==1.18.1** (`e4e854eaa8f18d807cd3496028e257e36caa93cc`) and **statsmodels==0.15.0** (`278ff9950636cdd4939b4055e339a8e681d79cab`), plus the exact NumPy version in the analysis lock (2.x; v2.5.3 was the current release on 2026-09-30). +- **Pairwise family (r7).** For every unordered pair (i, j) of planned family systems on a host, the per-cluster arrays are `s`, the sum of seed-averaged question differences q_i − q_j within the cluster, and `n`, the cluster's question count: + +```python +res = scipy.stats.bootstrap( + (s, n), + lambda s, n, axis=-1: s.sum(axis) / n.sum(axis), + paired=True, + vectorized=True, + n_resamples=10000, + method="percentile", + confidence_level=0.95, + alternative="two-sided", + rng=numpy.random.Generator(numpy.random.PCG64(20260927)), +) +d = res.bootstrap_distribution +p = min(1.0, 2 * min(numpy.mean(d <= 0), numpy.mean(d >= 0))) +``` + + - A fresh generator with the same seed for each pair draws identical cluster indices for every pair: a joint bootstrap over the shared clusters. + - Holm uses **`statsmodels.stats.multitest.multipletests(p, alpha=0.05, method='holm')`** over the frozen vector of all **m = k(k−1)/2** planned pairs, with **p = 1.0 for every pair involving a missing or ineligible system**, so m never shrinks; the placeholders are not imputed outcomes ([statsmodels v0.15.0](https://github.com/statsmodels/statsmodels/blob/v0.15.0/statsmodels/stats/multitest.py#L233-L247)). + - With the twelve primaries, **m = 66**. A reserve admitted before confirmatory execution raises k, which is frozen before execution. + - A rejected pair is decided in the direction of the observed difference. + - Two-sided 95% percentile intervals per pair are reported and decide nothing. Explicit `n_resamples=10000` overrides SciPy's 9999 default ([SciPy signature](https://github.com/scipy/scipy/blob/v1.18.1/scipy/stats/_resampling.py#L300-L303), [interval calculation](https://github.com/scipy/scipy/blob/v1.18.1/scipy/stats/_resampling.py#L661-L683)). + - The r6 cluster sign-flip `scipy.stats.permutation_test(permutation_type="samples")` stays as a diagnostic that decides nothing, with its RNG and resamples frozen and whole cluster sums sign-flipped at fixed question counts. +- **Power (r6's report, retargeted in r7).** Report, from the frozen analysis code, the achieved power to Holm-separate a true difference of 0.05 and of 0.02 at the frozen m and cluster structure, over discordance 0.10–0.30, with Monte Carlo uncertainty and the seed-dependence assumption stated. + - An illustrative normal approximation (470 paired questions, two-sided) for a difference of 0.05 gives power 0.54, 0.18 and 0.08 at discordance 0.10, 0.20 and 0.30 at the strictest Holm step (α = 0.05/66), and 0.93, 0.68 and 0.51 at the last step (α = 0.05). + - For a difference of 0.02 it is at most 0.02 at the strictest step and 0.28 at the last. + - Differences inside the 0.02 band will therefore rarely be separated, and even 0.05 differences may remain not separated. 7.2 therefore decides on the point estimate and uses the tests to label the result and to bar significantly worse systems from the tie-break. + - Power is information, not a gate or a margin change. A prospective subset (5.2) must report this calculation for its frozen size before any result. +- **LongMemEval-V2 resampling unit and minimum (round-3 item 2).** Build an outcome-independent graph on **all questions in the downloaded tier's haystack mapping**, before subset selection. Connect two questions when their haystacks share a trajectory ID or a canonical trajectory-content hash (canonical JSON with sorted keys, excluding only the trajectory `id`), or when they use the same native shared memory instance. The **connected components** are the resampling clusters; shared trajectories/memory never cross independent clusters, even through an unselected bridging question. Freeze trajectory/file hashes, the full question→component manifest and the component IDs represented in the selected subset; all five paired replications and all arms/control for a question stay in its component. The preflight verifies the mapping, referenced trajectory existence/content hashes and counts against the actual downloaded `haystacks/lme_v2_small.json` and `trajectories.jsonl`, then recomputes represented counts for any selected subset. The native loader supplies question→trajectory lists (`data/public_data.py:51-64,110-124`); the harness builds one shared memory for a common haystack (`evaluation/harness.py:1169-1170,1180-1213`), and the leaderboard requires the same web/enterprise haystacks across operating points (`leaderboard/README.md:149-150`). **Minimum: 20 independent components in the selected sample.** This is a preregistered design safeguard, not an upstream requirement or a guarantee of power/coverage. With one shared haystack per domain the design may yield only about two components (or fewer if connected): **the main-usefulness lane stays pending**, and cannot establish that no system qualifies or authorize a 7.6 fallback. Question count, repeated seeds or subset selection do not increase the cluster count; no question-level alternative test or outcome-based regrouping is allowed. No downloaded-file count is claimed by this source-only repair. +- **Separate main-usefulness family (repair rounds 2–3).** LongMemEval-V2 uses the same paired seed aggregation, cluster-bootstrap test and Holm α=0.05 **only after the minimum above passes**, with its own fixed haystack-component/sample manifest. Each draw retains every selected question and all replications of the drawn component; fresh same-seed generators give joint indices across pairs, with the `s`/`n` question weighting above. Include every unordered pair of k planned memory systems **and** each system versus no retrieval: **m_U = k(k+1)/2** (78 with twelve primaries). Missing/ineligible-arm pairs retain p=1; baseline comparisons are included even though baseline is outside the retrieval family. If the minimum is not met, the baseline exceeds 5% final failures or required comparison coverage is missing, the family is **pending**, using p=1 placeholders, not a negative usefulness conclusion. The best set is all exact point-estimate maximizers among operationally eligible memory systems with complete usefulness outcomes, never the no-retrieval control or an ineligible higher scorer. A pass requires an observed positive difference and Holm rejection versus no retrieval, and no Holm rejection in any best member's favour. Freeze the family before results; publish power at the verified cluster count for superiority and best comparisons. Low power with otherwise complete valid evidence may leave every system unqualified and invoke 7.6; insufficient clusters/coverage cannot. Neither case authorizes relaxing α or choosing a post-result subset. +- **Outside the retrieval and usefulness families:** other controls/diagnostics (3.4), development checks, the official track, `ndcg_any@5`, 5.5(a)/(b), secondary MemoryAgentBench TTL/CR and S8's separate non-inferiority check (8.4). +- **Missing results:** a missing system or lane result is `pending` and is not imputed. +- **Governing statistics (r4, retained):** these statistics govern S3 on both hosts. The A-protocol's family α values (0.0333, 0.0167), its +5 pp/−2 pp rule and its one-look-per-family rule do not transfer (r6 §12, Appendix A). + +## 7. Symmetric merit rule (per host) + +### 7.1 Eligibility, required of every system alike + +1. passes the current maintenance gate for every selected repository and the dependency SBOM/OSV requirements (section 1), source-backed deployment, network and GPT-6 wire/effort acceptance (2.2, 3.5, 4.4, 4.6); +2. deploys on that host and passes every item of the 5.4 lifecycle/coding-client acceptance; +3. warm p95 latency (5.3) at most 1.0 s, with the complete serving configuration co-resident (8.2); +4. has complete primary retrieval results and failure accounting (4.5); an exceeded failure threshold or missing required observer/result leaves it pending; +5. **main usefulness:** passes 5.5's LongMemEval-V2 superiority-to-no-retrieval and not-worse-than-eligible-best rule, with complete paired outcomes, at least 20 verified haystack components, baseline final failures at most 5% and the separate Holm family (section 6); +6. **cost readiness:** total C is available wherever the cost tie-break can change the decision; otherwise 7.3 is unresolved. Unpriced routes are handled only by 5.2's one policy and do not falsely disqualify a lone primary leader; +7. **short-code do-no-harm, separate from usefulness:** if S8 has been frozen/run, each client's resolve rate is non-inferior within its prospectively frozen margin, with token overhead reported (8.4). S8 cannot exclude a system **on usefulness** and its current pending status does not block a LongMemEval-V2-based host decision. Until S8 is qualified, report coding non-inferiority as pending, never passed; a later observed hard do-no-harm failure blocks cutover/retention in that coding scope. + +Items 1–4 are operational hard gates for both qualified selection and the 7.6 fallback; item 7 is a hard do-no-harm requirement in its frozen coding scope. Item 5 is statistical usefulness qualification. The no-winner fallback can select an operationally eligible system with complete evidence that fails to establish usefulness; it must say so. All-zero main-usefulness results cannot pass item 5. The 25 diagnostics and short-code tasks supply no replacement floor. + +**Executable usefulness examples (synthetic decisions, not bootstrap/evaluator acceptance).** `losses` contains Holm-rejected pairs `(loser, winner)` from the frozen family; scores are exact native-correctness aggregates. `eligible` is the independently established operationally eligible memory set with complete usefulness observations; it excludes the control. The analysis uses this rule after native statistics. Missing expected comparison/family observations use `complete=False`; a synthetic cluster count here does not verify downloaded data. + +```python +from fractions import Fraction as F + +def usefulness_status(system, scores, losses, *, eligible, cluster_count, + complete=True, failure_rate=F(0), baseline_failure_rate=F(0)): + if (not complete or cluster_count < 20 or failure_rate > F(5, 100) + or baseline_failure_rate > F(5, 100)): + return "pending" + if system not in eligible or system == "no_retrieval": + return "ineligible" + if "no_retrieval" not in scores or any(i not in scores for i in eligible): + return "pending" + memory = {i: q for i, q in scores.items() if i in eligible and i != "no_retrieval"} + top = max(memory.values()) + best = {i for i, q in memory.items() if q == top} + superior = (scores[system] > scores["no_retrieval"] + and ("no_retrieval", system) in losses) + not_worse = all((system, leader) not in losses for leader in best) + return "qualified" if superior and not_worse else "unqualified" + +zero = {"A": F(0), "B": F(0), "no_retrieval": F(0)} +ready = {"eligible": {"A", "B"}, "cluster_count": 20} +assert usefulness_status("A", zero, set(), **ready) == "unqualified" +scores = {"A": F(8, 10), "B": F(6, 10), "no_retrieval": F(5, 10)} +losses = {("no_retrieval", "A"), ("no_retrieval", "B"), ("B", "A")} +assert usefulness_status("A", scores, losses, **ready) == "qualified" +assert usefulness_status("B", scores, losses, **ready) == "unqualified" +assert usefulness_status("A", scores, set(), **ready) == "unqualified" # superiority unproved +tied = {"A": F(8, 10), "B": F(8, 10), "C": F(7, 10), "no_retrieval": F(5, 10)} +tied_ready = {"eligible": {"A", "B", "C"}, "cluster_count": 20} +assert usefulness_status("C", tied, {("no_retrieval", "C"), ("C", "B")}, **tied_ready) == "unqualified" +assert usefulness_status("C", tied, {("no_retrieval", "C")}, **tied_ready) == "qualified" +ineligible_high = dict(scores, X=F(9, 10)) +assert usefulness_status("A", ineligible_high, losses | {("A", "X")}, **ready) == "qualified" +assert usefulness_status("X", ineligible_high, losses, **ready) == "ineligible" +assert usefulness_status("A", scores, losses, failure_rate=F(5, 100), **ready) == "qualified" +assert usefulness_status("A", scores, losses, failure_rate=F(51, 1000), **ready) == "pending" +assert usefulness_status("A", scores, losses, baseline_failure_rate=F(5, 100), **ready) == "qualified" +for system in ready["eligible"]: + assert usefulness_status(system, scores, losses, baseline_failure_rate=F(51, 1000), **ready) == "pending" +assert usefulness_status("A", scores, losses, eligible={"A", "B"}, cluster_count=19) == "pending" +assert usefulness_status("A", scores, losses, eligible={"A", "B"}, cluster_count=2) == "pending" +assert usefulness_status("A", scores, losses, complete=False, **ready) == "pending" +print("7.1 usefulness examples: all assertions hold") +``` + +### 7.2 Winner and tie-break set + +Repaired in round 1 (finding F3): the maximizer set and the quantifier are explicit. +- Let q_i be each eligible system's full-track `recall_all@5` (4.2). Each q_i is computed exactly, as a rational number; a seed-averaged question score is a multiple of 1/5, so every comparison below is exact. +- Let q* be the largest q_i. The **maximizer set** is M = {i : q_i = q*}, and every member of M is a leader. +- The **tie-break set** B contains every member of M and each other eligible system j that meets both conditions: + - q_j ≥ q* − **0.02**, the preregistered practical-equivalence band; + - for **every** leader m in M, Holm does not reject the pair (j, m) in m's favour (section 6). A significant loss to any one leader excludes j. +- A system significantly worse than a leader is therefore never in B, however small the difference. A system that no leader is separated from, but that lies more than 0.02 below q*, is outside B: the leaders win over it on the point estimate, and the report states "not separated from" for each such pair. +- If B has one member, it is selected. It is labelled **best** if Holm separates it from every other eligible system, and otherwise **leader, not separated from** the listed systems. +- If B has several members, whether several leaders or leaders plus admitted systems, 7.3 applies, then 7.4. +- The frozen code at the end of 7.3 states this rule exactly, with examples that include tied leaders whose pairwise decisions against a third system conflict. + +### 7.3 Tie-break 1: token and cost efficiency + +Repaired in round 1 (finding F4): the rule handles several equal minima, zero included. +- Within B, let c_min be the lowest decision cost C (5.2), which is gross, nonnegative and exact. +- The **cost-tied set** A contains each member j with C_j = c_min or 0.80 × C_j < c_min, where 0.80 is the cost ratio r6 froze for its cost route. A member whose C exceeds c_min leaves A when c_min ≤ **0.80 ×** its C; every member at c_min stays. +- If A has one member, it is selected. Otherwise every member of A goes to 7.4; this covers several equal minimum costs, zero included. +- A member whose C is exactly c_min / 0.80 leaves A, matching r6's selection condition c_min ≤ 0.80 × C. r7's wording had also kept such a member tied (repair clarification). +- If an unknown C, or an unavailable price, could change the outcome, selection is unresolved. + +**Frozen rule and examples for 7.2 and 7.3 (repair round 1).** This code is part of the preregistration. The analysis applies exactly these two functions, and the examples must pass under the frozen Python before freeze. `worse` holds the pairs that section 6's Holm step rejects, oriented by the observed difference. + +```python +# S3 r7 sections 7.2-7.3: frozen rule and examples (r7 repair round 1, F3 and F4) +from fractions import Fraction as F + +BAND = F(2, 100) # 7.2 practical-equivalence band +RATIO = F(80, 100) # 7.3 cost ratio (r6) + + +def tie_break_set(q, worse): + """q: {system: exact full-track recall_all@5}. worse: the pairs (i, m) that + Holm rejects with m ahead of i (section 6); a pair with a zero observed + difference never enters it. Returns the maximizer set M and the set B.""" + top = max(q.values()) + M = {i for i in q if q[i] == top} + B = M | {j for j in q if j not in M and q[j] >= top - BAND + and all((j, m) not in worse for m in M)} + return M, B + + +def cost_tied(C): + """C: {member of B: exact gross decision cost, >= 0}. A single member is + selected by 7.3; several members go to 7.4.""" + c_min = min(C.values()) + return {j for j in C if C[j] == c_min or RATIO * C[j] < c_min} + + +n = F(470) # full-track questions; seed averages are multiples of 1/5 + +# (1) One maximizer; the runner-up is 20/470 below it, outside the band. +assert tie_break_set({"A": 300 / n, "B": 280 / n}, set()) == ({"A"}, {"A"}) +# (2) Tied leaders with conflicting pairwise decisions. C is inside the band +# but Holm separates it from co-leader A (not from B), so C is out. D is +# separated from neither co-leader, so D is in. +q = {"A": 300 / n, "B": 300 / n, "C": 295 / n, "D": F(1453, 5) / n} +assert tie_break_set(q, {("C", "A")}) == ({"A", "B"}, {"A", "B", "D"}) +assert tie_break_set(q, set()) == ({"A", "B"}, {"A", "B", "C", "D"}) +# (3) The band is compared exactly: 271.2 of 470 is exactly 0.02 below 280.6 +# of 470 and is in (binary floating point would put it out); one fifth of a +# question less is out. +assert tie_break_set({"A": F(1403, 5) / n, "E": F(1356, 5) / n}, set())[1] == {"A", "E"} +assert tie_break_set({"A": F(1403, 5) / n, "E": F(1355, 5) / n}, set())[1] == {"A"} +# (4) Several equal minimum costs, zero included: every minimum stays tied. +assert cost_tied({"A": F(10), "B": F(10), "D": F(13)}) == {"A", "B"} +assert cost_tied({"A": F(0), "B": F(0), "D": F(5)}) == {"A", "B"} +assert cost_tied({"A": F(0), "B": F(3)}) == {"A"} +# (5) A unique minimum is selected when it costs at most 0.80 x every other +# member, r6's boundary included; otherwise the members above stay tied. +assert cost_tied({"A": F(8), "B": F(10)}) == {"A"} +assert cost_tied({"A": F(8), "B": F("9.9")}) == {"A", "B"} +# (6) Gross costs, never control-subtracted ones (5.2): 120 and 130 stay tied, +# although 20 and 30 left after subtracting a common 100 would not. +assert cost_tied({"A": F(120), "B": F(130)}) == {"A", "B"} +assert cost_tied({"A": F(20), "B": F(30)}) == {"A"} +print("7.2-7.3 frozen examples: all assertions hold") +``` + +### 7.4 Tie-break 2: repo-quality score + +For each remaining member's repository, scored at the freeze gate run: + +RQ = (Σ_{c ∈ S} w_c · s_c + 2.5 · R + 2.5 · K) / (Σ_{c ∈ S} w_c + 5) + +- **s_c** ∈ [0, 10] is the OpenSSF Scorecard score of check c, over the seven checks Maintained, CI-Tests, Code-Review, Branch-Protection, Signed-Releases, Vulnerabilities and Dangerous-Workflow. **S** is the subset with a conclusive score, because Scorecard's own aggregate skips inconclusive (−1) checks ([`pkg/scorecard/scorecard_result.go:83-120`](https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/pkg/scorecard/scorecard_result.go#L83-L120) at v5.5.0). +- **w_c** is Scorecard's risk weight ([`README.md:605-613`](https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/README.md#L605-L613); `scorecard_result.go:86`), with each check's risk from `docs/checks/internal/checks.yaml:19,141,227,287,655,747,772`: + - Dangerous-Workflow is Critical, weight 10; + - Maintained, Code-Review, Branch-Protection, Signed-Releases and Vulnerabilities are High, weight 7.5; + - CI-Tests is Low, weight 2.5. + + The first term is therefore Scorecard's own aggregate over these seven checks. +- **R**, release cadence: R = 10 × min(1, n90 / 3). n90 counts GitHub releases that are neither drafts nor prereleases, published in the 90 days before the gate run; for a project that publishes no releases, it counts version tags. One release a month on average over Scorecard's 90-day window earns full marks. +- **K**, CI on the pinned release commit (repaired in round 1, finding F6). At the freeze gate run, capture the commit's complete check and status set: + + ```sh + gh api --paginate "repos/OWNER/REPO/commits/SHA/check-runs?filter=latest&per_page=100" + gh api --paginate "repos/OWNER/REPO/commits/SHA/status?per_page=100" + gh api "repos/OWNER/REPO/commits/SHA/check-suites?per_page=1" --jq .total_count + ``` + + GitHub's REST description ([`rest-api-description@af2c1025`](https://github.com/github/rest-api-description/blob/af2c102501766f14fe4edc688e449427a971e66a/descriptions/api.github.com/api.github.com.json), `api.github.com.json`) caps `per_page` at 100 and limits the check-run listing to a ref's 1000 most recent check suites (`:55478-55482`); the check-suite listing reports a `total_count` (`:55584`). The capture is complete when the retrieved check runs and status contexts each equal their reported `total_count` and the ref has at most 1000 check suites. The complete set is frozen, with its sha256, as the relevant set. + - K = 10 only if the set is complete and non-empty, every check run has status `completed` (none `queued`, `in_progress`, `waiting`, `requested` or `pending`; `:145900-145911`) and conclusion `success`, `neutral` or `skipped` (`:145913-145925`), and every status context's latest state is `success`. A commit with check runs but no status context meets the last condition; the combined `state` is not used alone, because GitHub reports it as `pending` when there are no statuses (`:55667`). + - Otherwise K = 0, with the reason recorded: missing (no check run and no status), incomplete (an unfinished run or a `pending` context), truncated (a retrieved count below `total_count`, or more than 1000 check suites) or failed (any other conclusion or state, such as `failure`, `error`, `cancelled`, `timed_out` or `action_required`). An API error or rate-limit refusal is retried after the reported reset, and the final response is the one recorded. +- R and K take the Low-risk weight 2.5, the same as CI-Tests. Their definitions and weights are coordinator heuristics, not Scorecard's measures, and the user may change them before freeze. +- **Scorecard runs.** + - The v5.5.0 CLI, `scorecard --repo=github.com/OWNER/REPO --checks=Maintained,CI-Tests,Code-Review,Branch-Protection,Signed-Releases,Vulnerabilities,Dangerous-Workflow --format=json`, runs against the default branch. + - It runs again with `--commit=` ([`options/flags.go:35-36,124-127`](https://github.com/ossf/scorecard/blob/c395761df6afe1a69e476bc60a013a94bcbc153f/options/flags.go#L124-L127)) for Vulnerabilities and Dangerous-Workflow. A check that cannot evaluate a non-HEAD commit keeps its default-branch result, recorded as such. + - The REST API at `api.scorecard.dev` is only a cross-check. + - The GitHub token is injected per `docs/secret-storage.md` and never recorded. +- The highest RQ, rounded to one decimal, is selected. An exact tie gives joint winners, and the host decision goes to the user. +- Stars, forks, downloads and popularity never count. Scorecard describes aggregate scores as telling "nothing about what individual behaviors a repository is or is not doing" (`README.md:109-112`), which is why RQ only breaks ties. + +### 7.5 Silent-fallback LLM stages (r6's C4′ preflight, generalized) + +- **Scope.** This applies to every system whose selected configuration has an LLM stage that can silently fall back. ai-memory's reranker is an example: "A timeout, provider error, or incomplete/invalid score set preserves the normal order", and queries over its four-call cap "keep their local ranking without waiting" (`docs/llm-providers.md:338-341` at `a0ca8d1a`). +- **Preflight: two separate development measurements under the frozen envelope** (repaired in round 1, finding F5): + 1. *Fallback rate under confirmatory load.* The system's development retrieval queries (4.1's development split) are dispatched as the confirmatory run will dispatch them: through the frozen dispatch wrapper (4.3), with K in-flight GPT-6 calls per host (5.2) and the same query concurrency per instance. The denominator is every recall request for which the selected stage is configured to run. The numerator is the requests on which the stage fell back, whether through timeout, provider error, invalid output or saturation such as ai-memory's four-call cap ([`docs/llm-providers.md:338-341`](https://github.com/akitaonrails/ai-memory/blob/a0ca8d1a5fbd5920799411fa891fe6d49c90efc1/docs/llm-providers.md#L338-L341)). **An observed stage outcome, not the presence/absence of a completed gateway call, decides that numerator** (N1). Require a per-request `accepted` or `fallback:` outcome from a supported native signal or a source-backed provider-boundary observer of the actual consumed stage output and validation result. A source-defined `not_applicable` precondition, such as fewer than two candidates, is separately counted and excluded from the denominator; saturation remains included. Unknown/unmatched outcomes are `unobserved`, never accepted or zero fallbacks, and leave the arm pending. Record gateway transport status separately: an HTTP 200 and completed `call_logs` row can still precede invalid-output rejection and preservation of the original ranking. + 2. *Gated warm latency at concurrency 1.* 5.3's frozen warm schedule (200 requests, concurrency 1, a 20-request warm-up discarded, no other arm's call in flight), with the stage on. Its p95 is the development estimate for 7.1 item 3. Fallbacks seen here are reported but do not enter the 5% rate. +- **Stage disabling** follows measurement 1 alone: development fallbacks above 5% disable the stage before any confirmatory run by dated amendment, or the arm is `pending`. Measurement 2 disables nothing; the confirmatory 5.3 run decides latency eligibility. +- **Confirmatory fallbacks above 5%**, counted as in measurement 1 over the system's confirmatory recall requests: that system's comparisons are inconclusive, and host selection is unresolved if they could change the winner. +- No configuration may switch after confirmatory scores. + +**Observable validation and fault control (repair round 2, N1).** ai-memory supplies native signals for saturation, provider error, timeout and incomplete/invalid scores (`akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-mcp/src/server.rs:2114`, `:2133`, `:2176`). Its acceptance predicate requires exactly one finite score in [0,1] for each candidate ID, with no extra/missing/duplicate ID (`server.rs:2157`). Freeze the native logging level and query correlation needed to attribute these events; a boundary observer must capture the actual candidate IDs, canonical scores consumed by that stage and resulting acceptance/ranking, validating the same predicate at the pin. Unchanged order alone is not a fallback signal: valid scores can preserve it. A stage without a trustworthy native accepted-output signal or this complete boundary observation stays pending; absence of a warning is not an accepted-output signal. + +Before either development preflight can pass, inject incomplete, duplicate, unknown-ID, nonfinite and out-of-range scores, plus timeout, provider error and saturation, through the actual selected query path. **The invalid-score response must complete successfully at the gateway**; retain the completed transport observation and then observe a native invalid-score fallback, preserved original ranking, a fallback numerator increment and complete outcome coverage. Run a valid-score control too, including one that preserves the original order. A proxy that sees only transport success must fail this control. Upstream's `reranker_degrades_on_partial_malformed_error_and_timeout` (`server.rs:6004`) is a source/test reference, not a run made here; the native gateway fault control remains required. The following source-backed synthetic example verifies the repaired discrete observation rule without claiming a service/gateway run: + +```python +import math + +def observed_stage_outcome(candidate_ids, consumed_scores, *, gateway_completed): + # gateway_completed is transport evidence only; the stage predicate decides. + # None means the consumed output/outcome could not be observed. + if consumed_scores is None: + return "pending" + expected = set(candidate_ids) + seen = set() + valid = len(consumed_scores) == len(candidate_ids) + for candidate_id, relevance in consumed_scores: + valid &= (candidate_id in expected and candidate_id not in seen + and math.isfinite(relevance) and 0 <= relevance <= 1) + seen.add(candidate_id) + return "accepted" if valid and seen == expected else "fallback:invalid_output" + +ids = ["opaque-a", "opaque-b"] +invalid = [("opaque-a", 0.9)] # successful transport, incomplete stage output +assert observed_stage_outcome(ids, invalid, gateway_completed=True) == "fallback:invalid_output" +assert observed_stage_outcome(ids, [(ids[0], 0.9), (ids[1], 0.8)], gateway_completed=True) == "accepted" +for scores in [[(ids[0], 0.9), (ids[0], 0.8)], [(ids[0], 0.9), ("unknown", 0.8)], + [(ids[0], float("nan")), (ids[1], 0.8)], [(ids[0], 1.1), (ids[1], 0.8)]]: + assert observed_stage_outcome(ids, scores, gateway_completed=True) == "fallback:invalid_output" +assert observed_stage_outcome(ids, None, gateway_completed=True) == "pending" +assert observed_stage_outcome(ids, None, gateway_completed=False) == "pending" +outcomes = [observed_stage_outcome(ids, invalid, gateway_completed=True), "accepted"] +assert sum(o.startswith("fallback:") for o in outcomes) / len(outcomes) == 0.5 +print("7.5 successful-transport invalid-score example: fallback observed; unknown is pending") +``` + +### 7.6 No qualified winner: best-scoring eligible-system fallback + +**ACCEPTED 2026-09-30: "Best-scoring eligible system."** If no system qualifies under 7.1, the host deploys the operationally eligible system with the highest observed primary `recall_all@5`, labelled **fallback; usefulness not qualified**, with the failed/inconclusive statistical qualification reported. The incumbent is one contender and is never kept by default. + +**Operationally eligible** means the system passes the current selected-repository maintenance gate and dependency SBOM/OSV requirements, has a source-backed deployment that actually starts and works on that host, passes all required lifecycle, wire/effort, network, latency and observer hard gates, and has **no failed hard gate** (7.1 items 1–4 and any already-frozen coding do-no-harm requirement). A missing required deployment/hard-gate observation leaves it pending and excludes it from the currently selectable set, **but does not permit ignoring it if its eventual eligibility could change the result**. Lack of significant main-usefulness superiority is not an operational hard-gate failure. Before declaring that no system qualifies, main-usefulness outcomes/comparisons must be complete, the 20-cluster minimum must pass and baseline final failures must be at most 5%; missing or invalid evidence is pending, not failed qualification. + +Apply this definition symmetrically to every system, ai-memory included. Require complete comparable primary results for all operationally eligible contenders that could change the choice; select the exact primary maximizer(s), with **no lower-score equivalence-band promotion** in this fallback. Exact primary ties use 7.3, then 7.4, on those maximizers; an unavailable cost capable of changing that tie remains unresolved under 5.2. A cold-copy cutover and rollback receipt is still required. If there are **zero operationally eligible systems**, deployment is unresolved with each hard-gate reason listed; do not deploy a failed/pending arm or call retention of the current installation a qualified/fallback decision. This empty-set limit does not reinstate an incumbent default. + +**Pending-contender guard (round-3 item 6).** The fallback stays **unresolved** while a planned pending contender could qualify, exceed the best currently eligible primary score, or tie it and change 7.3/7.4. This includes missing 7.5 stage-outcome coverage and section 9 host evidence. An unknown/incomplete primary score is potentially higher; only a complete comparable score strictly below the eligible maximum, with no remaining qualification uncertainty, cannot change this fallback. A demonstrated failed hard gate excludes a contender; an unobserved gate does not. Do not shrink the choice to whichever arms finished first. + +```python +from fractions import Fraction as F + +def fallback_status(scores, eligible, pending, *, usefulness_complete=True, + any_qualified=False, pending_qualification=False): + # scores contains complete primary results only, never partial estimates. + if not usefulness_complete or pending_qualification: + return "pending", set() + if any_qualified: + return "qualified_selection", set() + if not eligible: + return "unresolved", set() + if any(i not in scores for i in eligible): + return "pending", set() + top = max(scores[i] for i in eligible) + if any(i not in scores or scores[i] >= top for i in pending): + return "pending", set() + return "fallback", {i for i in eligible if scores[i] == top} + +scores = {"A": F(8, 10), "B": F(6, 10), "P": F(9, 10)} +assert fallback_status(scores, {"A", "B"}, {"P"}) == ("pending", set()) +assert fallback_status({**scores, "P": F(8, 10)}, {"A", "B"}, {"P"}) == ("pending", set()) +assert fallback_status(scores, {"A", "B"}, {"unknown"}) == ("pending", set()) +assert fallback_status({**scores, "P": F(7, 10)}, {"A", "B"}, {"P"}) == ("fallback", {"A"}) +assert fallback_status({**scores, "P": F(7, 10)}, {"A", "B"}, {"P"}, + pending_qualification=True) == ("pending", set()) +assert fallback_status(scores, {"A", "B"}, set()) == ("fallback", {"A"}) # P failed a hard gate +assert fallback_status(scores, {"A", "B"}, set(), usefulness_complete=False) == ("pending", set()) +assert fallback_status(scores, set(), set()) == ("unresolved", set()) +print("7.6 pending-contender examples: all assertions hold") +``` + +## 8. RAG and retrieval-model selection (r7) + +There is no incumbent default here either: qmd, the host's deployed embedders and rerankers, and ai-memory's MiniLM compete on the same terms as every challenger. + +### 8.1 Task sets, frozen before any result + +- **Source.** MTEB 2.21.10 (`7fac921e8672cfb6799d7c6a9a542832bc10cbb6`) defines ten retrieval tasks in `MTEB(eng, v2)` ([`mteb/benchmarks/benchmarks/benchmarks.py:21-84`](https://github.com/embeddings-benchmark/mteb/blob/7fac921e8672cfb6799d7c6a9a542832bc10cbb6/mteb/benchmarks/benchmarks/benchmarks.py#L21-L84)). Their corpora are disjoint from the memory data. +- **Partition rule.** To keep embedder selection and RAG evaluation apart, the tasks are split by corpus size alone. Corpus size is outcome-blind, and it also bounds ingest cost for systems that extract with an LLM. +- **Sizes.** They come from `mteb/descriptive_stats/Retrieval/.json` at that commit. The three HardNegatives tasks report only `num_samples`, which counts documents plus queries. +- **E_doc, document-RAG evaluation, the five smallest corpora:** ArguAna (8,674 documents), SCIDOCS (25,657), CQADupstackGamingRetrieval (45,301), CQADupstackUnixRetrieval (47,382) and ClimateFEVERHardNegatives (48,416 samples). +- **S_doc, document embedder and reranker selection:** FiQA2018 (57,638 documents), FEVERHardNegatives (164,698 samples), TRECCOVID (171,332 documents), HotpotQAHardNegatives (226,621 samples) and Touche2020Retrieval.v3 (303,732 documents). + +### 8.2 Embedder and reranker selection + +MTEB 2.21.10, run one model at a time on the workstation's RTX 4090 without stopping live services: +- **Memory role:** MTEB's LoCoMo task ([`mteb/tasks/retrieval/eng/lmeb_retrieval.py:996-1043`](https://github.com/embeddings-benchmark/mteb/blob/7fac921e8672cfb6799d7c6a9a542832bc10cbb6/mteb/tasks/retrieval/eng/lmeb_retrieval.py#L996-L1043); data `mteb/LoCoMo` at `02e2c3dea15d9fdfd1cd7a0f65f5f8ae2ed4c1ac`, cc-by-nc-4.0). + - The selection metric stays r6's question-weighted supporting-evidence recall@5, frozen before the first check, not the task default `ndcg_at_10` (`:119-124`). + - Rerankers use MTEB's two-stage procedure ([`docs/get_started/advanced_usage/two_stage_reranking.md`](https://github.com/embeddings-benchmark/mteb/blob/7fac921e8672cfb6799d7c6a9a542832bc10cbb6/docs/get_started/advanced_usage/two_stage_reranking.md)): stage-1 predictions, then `convert_to_reranking(top_k=100)`, then stage 2. + - Freeze the corpus, native inference, pooling and prompt, dimensions, truncation, artifact revision and quantization, candidate depth and scoring before the check (r6). +- **Code role:** the `CoIR` benchmark's tasks in MTEB (`benchmarks.py:528-558`: AppsRetrieval, CodeFeedbackMT, CodeFeedbackST, CodeSearchNetCCRetrieval, CodeTransOceanContest, CodeTransOceanDL, CosQA, COIRCodeSearchNetRetrieval, StackOverflowQA and SyntheticText2SQL), scored by nDCG@10. These run as MTEB task data; no code from the stale `CoIR-team/coir` runs. +- **Document role:** S_doc (8.1), scored by nDCG@10. + +**Candidates are current models only.** The challengers are r6's verified 2026-09-27 shortlist (r6 §14 E row; `inputs/sweep-20260927-mac/embed.md`, `rerank.md`): Nemotron-3-Embed-8B-BF16 and -1B-BF16, F2LLM-v2-4B and -0.6B, harrier-oss-v1-0.6b, MemReranker-4B, Qwen3-Reranker-0.6B and ettin-reranker-400m. Full revisions are expanded at freeze. The incumbents compete on the same tasks: + +| Incumbent | Artifact and revision | Where it runs today | +|---|---|---| +| Nemotron-3-Embed-1B | `nvidia/Nemotron-3-Embed-1B-BF16` @ `c0c9fea93ea424587517f2c59e20db9f1d6bf615` | the workstation's vLLM embedder (`manifests/stack.json`) | +| embeddinggemma-300M | `ggml-org/embeddinggemma-300M-GGUF` @ `0f741b5a6585bd53aeb15cd1372c56f2a0f65e12` (base `google/embeddinggemma-300m` @ `57c266a740f537b4dc058e1b0cda161fd15afa75`, gated) | qmd's default embedder ([qmd v2.8.3 `README.md:560,729,1272`](https://github.com/tobi/qmd/blob/facd35e01359e59d938bc9418e93fb9318addee3/README.md#L560)) | +| qwen3-reranker-0.6b | `ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF` @ `a02f48bb4f057028298c21fa033da2b30d7742d5` (base `Qwen/Qwen3-Reranker-0.6B` @ `e61197ed45024b0ed8a2d74b80b4d909f1255473`) | qmd's default reranker (`README.md:561,730,1273`) | +| MiniLM | `sentence-transformers/all-MiniLM-L6-v2` @ `1110a243fdf4706b3f48f1d95db1a4f5529b4d41` | ai-memory's pinned baseline (`manifests/stack.json`) | + +**VRAM budget.** +- On 2026-09-30, `nvidia-smi` showed 12,726 MiB free (time not recorded), then 12,005 of 24,564 MiB free at 03:13:24Z. Neither reproduces the brief's "about 13.7 GB". +- Each check runs within the free memory measured at its own start, less 1 GiB. +- **Co-residency (repair round 2, M7):** individual model checks do not establish that their selected combination serves the arms. Before selection/freeze, enumerate and measure feasible embedder+reranker combinations **together with every required local generative/shipped model**, native runtimes, buffers and the frozen concurrent serving schedule for each arm. The workstation constraint is **peak additional serving VRAM ≤ measured free VRAM − 1 GiB** (the recorded roughly 12 GB is historical, re-measured at qualification); Mac shared-memory/RSS is measured separately. Only a jointly feasible combination may serve every arm that needs it. If independent role leaders cannot co-reside, freeze a feasible jointly tested combination from the shortlist before confirmatory scores or leave the serving plan pending until a pre-approved GPU stop/restore window. Never qualify models one at a time and silently unload one during another arm's measured work. Record cold/warm transitions and all local serving costs; no live service is stopped by this repair. +- Nemotron-3-Embed-8B-BF16 needs about 16 GB for its BF16 weights alone (8 billion parameters × 2 bytes), so it does not fit. It stays `pending`, not refuted, until the user approves a GPU window (section 9) or an official quantized artifact is qualified. + +**Selection rule, per role.** +- The model with the best selection metric wins. +- Tests use the paired bootstrap of section 6. LoCoMo resamples conversation clusters; MTEB tasks resample queries stratified by task, with the unweighted mean over tasks as the statistic. Holm runs over the role's frozen candidate list. +- Within the 0.02 band, 7.2's rule applies, followed by tie-breaks on lower warm p95 encode latency at the frozen batch size, then on lower peak VRAM. An exact residual tie is a joint selection. +- Model repositories get no Scorecard score. +- The selected model serves every configurable system and the common diagnostics (3.4). + +### 8.3 Document-RAG head-to-head + +- **Candidates**, all ACTIVE (section 1): + - [qmd](https://github.com/tobi/qmd) v2.8.3 (`facd35e01359e59d938bc9418e93fb9318addee3`), through `qmd query` and `qmd search --json` (`README.md:54-56,76-80`); + - [LightRAG](https://github.com/HKUDS/LightRAG) v1.5.7 (`28ff1b05f2ac3f3e6fa14dd2cd33656579bd0c9c`), through `QueryParam(only_need_context=True)` with its default `mix` mode ([`lightrag/base.py:90-117`](https://github.com/HKUDS/LightRAG/blob/28ff1b05f2ac3f3e6fa14dd2cd33656579bd0c9c/lightrag/base.py#L90-L117)); + - a [llama_index](https://github.com/run-llama/llama_index) v0.14.25 (`f12d46acab73f5b2243ef49c2f00101617b38ce4`) hybrid pipeline: BM25 plus a dense retriever, fused by `QueryFusionRetriever` in `reciprocal_rerank` mode with `num_queries=1`, so that no LLM rewrites the query ([`llama-index-core/llama_index/core/retrievers/fusion_retriever.py:24-41`](https://github.com/run-llama/llama_index/blob/f12d46acab73f5b2243ef49c2f00101617b38ce4/llama-index-core/llama_index/core/retrievers/fusion_retriever.py#L24-L41)); + - OpenViking v0.4.22, through `find` (`README.md:59,144` at `e8716760`); + - memsearch v0.4.21, through its hybrid dense, BM25 and RRF search (`README.md:46,386,465` at `2a4652fa`); + - cognee v1.6.2, through `SearchType.CHUNKS` (`cognee/modules/search/types/SearchType.py:4-16` at `ba3631f2`). + - [HippoRAG 2](https://github.com/OSU-NLP-Group/HippoRAG), default-branch commit [`398bfdc388692795c105c1b903db60031975b669`](https://github.com/OSU-NLP-Group/HippoRAG/commit/398bfdc388692795c105c1b903db60031975b669) (2026-09-29; package 2.0.0a5, `OSU-NLP-Group/HippoRAG@398bfdc3:pyproject.toml:7`), through `hipporag.index(docs=docs)` (`README.md:110`; `src/hipporag/HippoRAG.py:499`) and retrieval-only `hipporag.retrieve(queries=[query], num_to_retrieve=top_k)` (`src/hipporag/HippoRAG.py:704,773,794,798`, all at `398bfdc3`). The wrapper consumes ranked `.docs`, `.doc_scores` and `.doc_metadata` directly, without answer generation; `QuerySolution.to_dict()` truncates to five (`src/hipporag/utils/misc_utils.py:70`). +- **HippoRAG 2 pin exception (2026-09-30).** This is a document-RAG SDK candidate outside the memory family: no Claude Code/Codex integration, MCP or HTTP service was found in the pinned README, examples and source. The GitHub release [v1.0.0](https://github.com/OSU-NLP-Group/HippoRAG/releases/tag/v1.0.0), commit `b144c46df14cabe5f5822d8caded4bec5f709461`, commit dated 2025-02-25 (published 2025-02-27 in the gate), is legacy HippoRAG 1, citing arXiv 2405.14831 (`OSU-NLP-Group/HippoRAG@b144c46d:README.md:5`), with no `hipporag` package directory in its tree. PyPI's newest `hipporag`, 2.0.0a4 (uploaded 2025-06-24, recorded in the gate file's `pypi` entry), predates maintained fixes, including [`142725c`](https://github.com/OSU-NLP-Group/HippoRAG/commit/142725c) (2026-09-02), "custom OpenAI embedding endpoint". Re-check the head pin at freeze and install from that commit with the README's `uv pip install -e .` recipe (`OSU-NLP-Group/HippoRAG@398bfdc3:README.md:63-65`). The wrapper uses `llm_model_name`, `llm_base_url` and `llm_api_key_env` for the GPT-6 gateway and `embedding_provider="openai"`, `embedding_model_name`, `embedding_base_url` and `embedding_api_key_env` for 8.2's selected local document embedder (`README.md:121-127`). Keep OpenIE online so extraction and recognition-memory reranking use that configured LLM (`src/hipporag/HippoRAG.py:227-234`; `src/hipporag/rerank.py:35`), with fresh `save_dir` state and local Parquet vectors (`README.md:90,214`, all at `398bfdc3`). Endpoint usage/envelope acceptance, role output budgets, dependency/artifact freeze and native indexing/retrieval acceptance remain pending; source review is not a deployment result. +- **Document-RAG reserve:** [infiniflow/ragflow](https://github.com/infiniflow/ragflow) [v1.0.0-rc1](https://github.com/infiniflow/ragflow/releases/tag/v1.0.0-rc1) (`14bb02ee4584c1ab18f0c722c48beecd6bbf9860`, published 2026-09-29), ACTIVE in the [pre-freeze gate](inputs/r7/landscape-foldin-gate-20260930.json). It remains outside the seven-candidate comparison unless admitted before its family is frozen; any later admission needs a dated amendment and its own frozen comparison. +- **Wrapper.** Each system is wrapped as an MTEB `SearchProtocol` model ([`mteb/models/models_protocols.py:23-84`](https://github.com/embeddings-benchmark/mteb/blob/7fac921e8672cfb6799d7c6a9a542832bc10cbb6/mteb/models/models_protocols.py#L23-L84)). `index(corpus, …)` ingests through the system's native path, and `search(queries, …, top_k)` returns `{query_id: {doc_id: score}}`. `mteb.evaluate` runs the wrappers on E_doc. Returned artifacts must map to MTEB document IDs through a mapping frozen per wrapper; unmappable output is `N/A`, never a constructed score. +- **Configuration.** + - Each system runs its shipped retrieval path, with every active LLM role on GPT-6 (2.1). + - Configurable embedders and rerankers use the 8.2 document-role selection. S_doc chose it, so nothing leaks into E_doc. Otherwise the shipped model is used and labelled. + - qmd's query expansion uses its own local fine-tuned GGUF (`README.md:562,731`), and its model overrides accept only GGUF URIs (`README.md:752`). qmd therefore runs with a recorded fixed-provider shipped default, not GPT-6. +- **Metrics:** nDCG@10 (primary; MTEB's main score), recall@10, warm p95 query latency (5.3's schedule), tokens per query (o200k of the returned payload plus native LLM usage at query time) and ingest tokens (5.2's ledgers). +- **Decision.** Section 7's merit and quality rules apply: + - the primary is the unweighted mean nDCG@10 over E_doc; + - tests use a paired bootstrap over queries stratified by task, with Holm over all 21 pairs; + - then the 0.02 band, 7.3 and 7.4; + - 7.3 uses C_RAG (repair round 1, finding F4): the gross priced cost of ingesting E_doc and answering every E_doc query through the wrapper, summed over every model call at ingest and at query time from ledger (a), with 5.2's price table and exact arithmetic. No client session exists here. A system with no priced model call has C_RAG = 0, which 7.3's equal-minimum rule handles. + + The memory-specific lifecycle/usefulness/coding items (5.4, 5.5 and 8.4) do not apply; the selected-repository maintenance, dependency SBOM/OSV, network, cost-readiness, observation and latency rules do. +- **Feasibility.** Before freeze, project the GPT-6 ingest tokens and wall-clock of the LLM-extraction systems on E_doc. If the projection exceeds the RAG window, drop E_doc tasks prospectively, largest corpus first, before any result. The coordinator or user sets that window before freeze; it defaults to 96 h per host, as in 5.2. + +### 8.4 Code retrieval and the S8 code-task stage + +- **S8: short-code do-no-harm only (ACCEPTED 2026-09-30).** The separate Harbor end-to-end stage checks each client's resolve rate for non-inferiority to the same client's no-memory control within a frozen margin and reports gross/signed token overhead. These short single-session tasks cannot exclude a system **on usefulness**; long-workflow usefulness is decided by LongMemEval-V2 (5.5). S8's pending design no longer makes every host's memory selection pending. Once its amendment is frozen, its coding do-no-harm result is required in that coding cutover/retention scope, reported separately from positive usefulness. +- **Route (repair round 1, finding F1).** S8 uses SWE-bench Verified tasks regenerated by Harbor's maintained adapter, never task bundles from the stale `harbor-datasets` (section 1). + - *Harness and adapter:* `harbor-framework/harbor` v0.23.0 (tag object `3c305dc5`, commit `1e5c5c6db929a10a140d05e606882c671ae20729`) and its `adapters/swebench`, ACTIVE. + - *Data:* the adapter loads `princeton-nlp/SWE-bench_Verified` by name, with no revision ([`adapters/swebench/src/swebench_adapter/adapter.py:46`](https://github.com/harbor-framework/harbor/blob/1e5c5c6db929a10a140d05e606882c671ae20729/adapters/swebench/src/swebench_adapter/adapter.py#L46)). That is a separate Hugging Face dataset ([revision `c104f840cc67f8b6eec6f759ebc8b2693d585d4a`](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified/tree/c104f840cc67f8b6eec6f759ebc8b2693d585d4a), last modified 2025-02-18), not an alias of the maintained [`SWE-bench/SWE-bench_Verified` at `78f471bf655a3137b2e8a75af1501690ec009ec3`](https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified/tree/78f471bf655a3137b2e8a75af1501690ec009ec3), its main head on 2026-09-30 (last modified 2026-08-16). Both are static data pinned by the sha256 of the downloaded parquet. Every generated task's source row must be field-identical to the maintained revision's row for the same instance, and a task with any differing field is excluded before any result. + - *Executable parts and their sources:* + - the Dockerfile, from the adapter's template: `FROM` the SWE-bench instance image that swebench's test specification names ([`task-template/environment/Dockerfile:18`](https://github.com/harbor-framework/harbor/blob/1e5c5c6db929a10a140d05e606882c671ae20729/adapters/swebench/src/swebench_adapter/task-template/environment/Dockerfile#L18); `utils.py:55`), then the uv 0.7.13 installer fetched with `curl | sh` (`Dockerfile:40`); + - `tests/test.sh`, generated from swebench's test specifications ([`utils.py:6-8,55`](https://github.com/harbor-framework/harbor/blob/1e5c5c6db929a10a140d05e606882c671ae20729/adapters/swebench/src/swebench_adapter/utils.py#L6-L55); the adapter requires `swebench>=4.1.0`, `pyproject.toml:12`). It runs the subject repository's own tests ([`task-template/tests/test.sh:4`](https://github.com/harbor-framework/harbor/blob/1e5c5c6db929a10a140d05e606882c671ae20729/adapters/swebench/src/swebench_adapter/task-template/tests/test.sh#L4)), then a grading `parser.py` whose inline dependencies `swebench==4.0.3`, `datasets==2.16.1` and `fastcore<1.11` resolve when the verifier runs (`:10,82`), on a public verifier network (`task.toml:18-19`); + - Harbor's verifier, which uploads and runs that script (`verifier.py:175-232`). No LLM judge is involved. + + At the repair-round gate, harbor, SWE-bench/SWE-bench, huggingface/datasets, AnswerDotAI/fastcore and astral-sh/uv were ACTIVE (section 1). + - *Enforcement:* section 1's task-level provenance manifest. Image tags are mutable, so the built task image is frozen by digest; that covers the base image and the installer. The verifier's resolved dependency set is frozen and recorded on every trial, and a trial whose resolved set differs does not score. Every selected task's subject repository must pass the gate at freeze, and a task whose subject repository is stale is excluded (a task-selection filter). +- **Relayed prior check.** The coordinator relays that the separate token-tool E2E verified 36 SWE-bench Verified tasks as field-identical to `SWE-bench/SWE-bench_Verified@78f471bf`, with `tests/test.sh` being Harbor's adapter-generated script. This revision has not seen that receipt, nor which task bundles it compared. The check is cited as relayed, and its receipt is required before freeze. +- **Status: `pending`.** S8 runs only after a dated amendment freezes task/provenance manifests, each primary system and each client's no-memory control, task count, seeds, native resolve scoring, non-inferiority margin and paired test (including client-wise multiplicity), token-overhead accounting and failure policy. Pending S8 means coding non-inferiority is not yet claimed; it does **not** veto the main usefulness result or host selection. +- **Code-retrieval tools are outside S3 (2026-09-30).** S3 has no code-retrieval tool arm. The coordinator relays that the separate Harbor token-tool E2E covered Serena and did not include [abhigyanpatwari/GitNexus](https://github.com/abhigyanpatwari/GitNexus) [v1.6.12](https://github.com/abhigyanpatwari/GitNexus/releases/tag/v1.6.12) (`c4ecf398de637cf779ab2238e69958cfd5d601ce`, published 2026-09-12), ACTIVE in the [pre-freeze gate](inputs/r7/landscape-foldin-gate-20260930.json). No receipt for that run exists in this checkout; the Harbor receipt is required before freeze. GitNexus is a candidate for its next round; it has no S3 arm. +- codebase-memory-mcp v0.11.0 (`8972ea69c6ad94b1ef1d4ffbf0a92d78d2db1798`) refused `/testbed`. That refusal is recorded as a fit limit, not as a quality result. + - Upstream refuses any indexing root with fewer path components than the platform minimum ([`src/foundation/workspace.h:28-29`](https://github.com/DeusData/codebase-memory-mcp/blob/8972ea69c6ad94b1ef1d4ffbf0a92d78d2db1798/src/foundation/workspace.h#L28-L29); `docs/CONFIGURATION.md:196-199`), and Harbor tasks mount the repository at `/testbed`. + - The refusal itself is relayed by the brief. No receipt for it exists in this checkout, so the Harbor lane's receipt must be cited before freeze. + +## 9. Hosts (r6, retained with r7 edits) + +Each host selects **separately**, with primary retrieval, main LongMemEval-V2 usefulness, hard gates and section 7's rule applied to its own measurements. Secondary/diagnostic lanes and S8's separately qualified do-no-harm scope keep their stated boundaries. +- **Order of acceptance testing on each host:** by primary-lane point estimate, descending. Continue while any untested eligible configuration could still change that host's selection. +- **Cross-host differences** are a diagnostic, not a gate. +- **CUDA timing, VRAM behaviour, model-serving compatibility and lifecycle acceptance do not transfer to macOS**, and Mac results do not transfer back. +- **Workstation GPU window.** r4 recorded production vLLM embeddings on 127.0.0.1:18231 and llama.cpp generation on 127.0.0.1:18232. r7's model selection runs within free VRAM without stopping them (8.2). A model that does not fit still needs **a scheduled production stop and restore that the user approves**; no S3 or AMB-driven process stops a production unit itself. +- **Per-host qualification.** Before its confirmatory run, each host records the RSS, latency and co-residency of every serving model, and runs 7.5's preflight for every system with a silent-fallback stage. Mac qualification is tracked by issue #379, and the bundle must cite its resulting measurement receipt. +- **Mac Hindsight (r6): reported runnable, host receipt required.** **Coordinator brief B:I** relays that Apple Container **1.4.1** was installed by the user on **2026-09-27**, with **Postgres 18.6 and pgvector 0.8.6**. These are the coordinator's supplied host observations, conditional on the **Mac's own sanitized receipts in the bundle before freeze**. Use Hindsight 0.10.2's supported external-PostgreSQL route (`installation.md:37-39` at `5fc4ce20`) through that host's runtime with fresh arm resources and a frozen digest. Reported hosting does not pre-pass 5.4 or the early durability/output-cap checks. +- **Both GPT-6 backbones (r6).** Each host's gateway identity comes from its own sanitized build/patch receipt before freeze (2.2's gateway identity note). The workstation uses `http://127.0.0.1:20128/v1`. **Coordinator brief B:A/I** relays that the Mac has **its own LaunchAgent gateway on the same tree/patch IDs, 12/12 verified**; include **the Mac's own 12/12 and effective-build receipts** before freeze; the Mac does not call the workstation's gateway. Keep reported readiness separate from new arm-level wire/effort/usage and lifecycle acceptance. No sign-in or quota recheck is inferred from an older blocker. + +## 10. What this does not claim + +- Local-harness scores are not official leaderboard results. +- Vendor-published numbers, AMB's leaderboard included, are motivation only. +- A “SOTA” discovery shortlist is not a measured universal winner. GPT-6 wiring, model-card scores, source review, recorded setup smoke, this host's new acceptance and another host's acceptance are different evidence levels. Neither GBrain's hosted 449/470 nor Hindsight's recorded wire probes decides S3. +- Passing the maintenance gate shows activity, not quality. A Scorecard score is a heuristic, and RQ only breaks ties. +- Registry metadata, such as Hugging Face LFS oids and image digests, is not a hash of downloaded bytes. +- A retrieval score says nothing about capture or deletion safety; 5.4 covers those. +- Non-inferiority to no memory on S8 (8.4) shows only that resolve rate is within the frozen do-no-harm margin; it is not positive long-workflow usefulness. A 7.6 deployment fallback without main usefulness qualification is explicitly labelled as such. +- Agreement among reviewers, and blinding, do not validate a measurement. +- Nothing here changes a host until the winner passes a cold-copy cutover with rollback. + +## 11. Freeze procedure + +1. Build the experiment bundle: + - this document and the decision record; + - the inputs under `inputs/` with sha256: the r4 Mac sweeps, five r6 captures (r6 §14), r7 live capture, all three repair-source captures and GPT-6 review capture; current maintenance receipts for selected repositories and complete per-arm dependency SBOM/OSV receipts, including harness import paths and stale transitive flags; + - the pre-freeze discovery capture `inputs/r7/landscape-memory-20260930.json`, sha256 `8f877d85ea57421f1c50704b6a2f9e88b0f8e03e8f281e9f165e03b78cfd75b8`, and maintenance capture/admission screens `inputs/r7/landscape-foldin-gate-20260930.json`, sha256 `bed16fa91e45d116d0ee968bb86b5e9c7fcd503eff820b1c0a92bc33c5a4ac65`; + - the downloaded dataset's sha256 and byte count, the full-population 470, official 419 and QA 500 manifests, any prospectively selected subset with its recomputed cluster and official-intersection counts, the development split hashes with the no-overlap check, and all 20 query and 5 continuation task specifications with their expected evidence and the manual-audit sampling rule; + - **AMB `03c1d0f1d27da63034f0931121c858faba512383` verified byte-identical**, each arm's executed-environment hash, both S3 dataset adapter variants with the no-judge and zero-Google/Groq wire proof, the S3 scorer and its tests, the dispatch wrapper, every thin provider client (Hindsight's included), each system's frozen effective store settings (for Hindsight, the bank-configuration hash), the common depth mapping and retry policy, the parity checklist, and the GPT-6 adversarial audit with its resolutions; + - the **main LongMemEval-V2** small-tier/subset, full-tier haystack-component manifest and downloaded-file verification of at least 20 selected components (or pending lane status), per-domain runtime files/ID lists, native registration/config/bootstrap hashes and the same direct-harness argv for all arms/control, upstream no-retrieval config hash, shared-state/context policy, reader/evaluator request-body wire proofs, five paired replications, arm/baseline failure thresholds/ledgers, prospective cost/power/window estimate and separate k/m_U Holm family; native MemoryAgentBench configs/commands for applicable secondary modes only, with the per-system applicability table and no vendored candidate execution; MemoryArena excluded from execution; + - the Harbor token-tool receipt for the relayed Serena measurement and GitNexus absence; the S8 amendment with task/provenance manifests, resolve-rate margin/test and relayed 36-task receipt, or its recorded `pending` coding do-no-harm status; neither option reintroduces a usefulness or host-selection coupling; + - the A17 supersession amendment (`blueprints/memory-stack/longmemeval/A17-SUPERSESSION.md`), revised to match r7. Its lines 16 and 19–21 still describe r4–r6's reference arm, and that file lies outside this revision's paths; + - the analysis code bound to **SciPy 1.18.1 / statsmodels 0.15.0**, locked NumPy version, frozen 7.2–7.3 functions and 4.5/7.1/7.5/7.6 examples passing with their negative controls, both pairwise families (k, m/m_U, placeholders), 10,000-resample and linear-quantile settings, verified usefulness component minimum, diagnostic sign-flip and achieved-power reports (including prospective subsets); open-QA prefix/context-token evidence, output ownership/exact native filenames and score-manifest reconciliation with retained zero-score failures; + - client and plugin versions; + - the 8.2 selection receipts with every model's full revision, quantization and native serving pin, and the E_doc/S_doc partition; + - the RAG wrappers and their mapping rules; + - the GPT-6 per-role model/effort matrix, the R02 and optional-stage decisions with per-role or coupled-set n, metric, α and test and per-host outcomes, the fixed Astra-max answerer and judge, and no stale fallback models; + - per-host gateway build receipts, container-to-loopback transport, denied-egress negative controls, conversation affinity, cache/dedup exclusion and effective-effort wire receipts for all call paths, AMB/LongMemEval-V2 answerer/evaluator included; every header-limited client's verified exemption/per-arm native route, or its explicit pending status; + - each system's upstream install/test/lifecycle recipe and immutable image/lock pins, fresh container/volume/home identities, effective role overrides and early lifecycle-screen results, both sanitized live deploy receipts, and the Mac's own hosting and 12-of-12 gateway receipts; + - prompts, seeds, fixtures, configurations, **development-selected K**, supported concurrency knobs and observed call concurrency, block-randomized dispatch, measured development tokens/s and **projected wall-clock per host against the 96 h window**, prospective full/subset/reuse decisions, joint serving-model co-residency receipts, 7.5 preflights with full stage-outcome coverage and a successful-gateway invalid-score native control, non-overlapping ledgers, authoritative price mappings or declared unpriced coverage and cost limits, explicit gross-C/local-cost sensitivity boundary, Scorecard/RQ inputs with pinned-commit check/status captures (7.4), and the supported scratch-home sign-in route for 5.2(c). +2. Verify that every referenced file and link exists. +3. Get a GPT-6 cross-family review of the **bundle**, and resolve every required finding. +4. **Obtain the user's explicit acceptance** of the reviewed bundle, including the coordinator decisions listed in "r6 → r7 changes", the repair-round decisions in the decision record's "Decisions for acceptance" and the A17 supersession. Freeze only after both. +5. Record the immutable commit and the bundle's sha256 list outside the worktree, rerun the maintenance gate, then run. +6. Keep per-case outputs, failures and each arm's actual execution status. + +## Appendix A. r1–r6 review history (unchanged) + +The sections below are r6's sections 10–16 exactly as they stand at `f7e5e228` (PR #390's head `d161f691` merged with `main`), except that each heading gains the prefix "r6 §". Their section numbers, "Fixed in" pointers, source keys (I, R, M, W, V, U and B, defined in r6 §14) and locators refer to the r6 text, not to r7. Where they name LongMemEval's scorer or judge prompt for execution, or ai-memory v2.4.0 as the reference, r7 supersedes them ("r6 → r7 changes"). + +## r6 §10. r1 findings and their fixes + +| r1 finding | Fixed in | +|---|---| +| high 1: incumbent privileged, no cheaper route, retention implies qualification | §1 eligibility applies to all; §7 two routes; §7 retention label | +| high 2: comparator, arms and test selectable after results; 2.5-pre vs 2.4.x | §1 frozen reference C4′ on v2.4.0; §4.1 one primary arm; §6 test and cluster bootstrap; §1 historical C3 descriptive | +| high 3: dataset and adapter contract | §3 | +| high 4: token units and accounting | §4.2 | +| high 5: CI overlap is not a tie | §7 qmax − ε selection | +| high 6: Mac subordinate, no Mac costs | §5 per-host selection with all lanes | +| medium 7: freeze binds the whole bundle; accept required | §9 | +| medium 8: latency definition | §4.3 | +| medium 9: usefulness lane | §4.5 | +| medium 10: native Codex control | §2 | +| low 11: reserve admission and engram | §2 | + +## r6 §11. r2 re-check findings and their fixes (r3) + +| r2 re-check finding | Fixed in | +|---|---| +| partly resolved r1-2: cluster assignment, weighting, seed aggregation, RNG, C4′ preflight | §6: frozen preflight, cluster manifest, seed averaging, question weighting, PCG64 seed | +| partly resolved r1-4: payload ledger promoted to complete cost; aggregation undefined | §4.2 decision ledger; §7 routes use it | +| new medium 1: "lower 95% bound" ambiguous | §6: one-sided 5th percentile, `numpy.percentile(method="linear")` | +| new medium 2: non-inferiority escapes Holm | §6: per-host family holds both hypotheses; Holm at α 0.05 | +| new medium 3: payload fallback changes the winner | §4.2: payload counts support only a payload-size claim; incomplete ledger means `pending` | +| new high 4: superiority bypasses usefulness | §7: usefulness floor before either route | + +## r6 §12. Relationship to the A1–A16.3 protocol and A17 (r4, proposed) + +- **What A17 was.** VelaNext was retired, so the confirmatory rerun R1 of the frozen memory-stack protocol (A1–A16.3, v3 #380 and v4 #386) needed another host and a new amendment naming it before any run (`docs/decisions/2026-09-25-retire-vela-velanext.md:36-44`). The workstation took R1 as "A17" (`docs/decisions/2026-09-25-workstation-sota-refresh.md:542-546`; issue #274). No A17 text was ever written, and no session holds a draft (checked 2026-09-27). +- **Why S3 supersedes it without spending a look.** No confirmatory arm of A1–A16.3 has ever run on any host: no C3′, C4′, C4 or D2h result exists (`retire-vela-velanext.md:43-44`; `catalogs/foundation/memory-stack-20260925.json:1318`; `docs/decisions/2026-09-27-mac-single-writer-staged.md:48`). Retiring R1 in favour of S3 therefore consumes no confirmatory look and selects nothing after seeing results. +- **What carries over.** The user's 2026-09-25 choice of official ai-memory v2.4.0 as the control (issue #274; `workstation-sota-refresh.md:228`) becomes S3's reference, and the A-protocol's C3/C4/D2h configurations are the starting point for S3's C3′, C4′ and agentmemory configuration. There are two stated differences (r5): + - C3′/C4′ use Qwen3-Embedding-4B's native, unprefixed input, because v2.4.0 has no prefix setting; + - agentmemory's adapter must prove shipped-hook equivalence (sections 1 and 2). The historical Mac C3 on `19b6429` stays descriptive. +- **Additional r6 differences:** the GPT-6 LLM backbone (including C4′ and agentmemory), development-selected configurable local retrieval models, Astra-max QA/judging, isolated native deployments and research-memory usefulness scope in sections 1–5. The A-protocol's local generation pins and historical zero-LLM results are not carried over as r6 scored configurations. +- **What does not carry over.** The A-protocol's statistics and decision rules (section 6), its v4 = v3 reproduction gate and its VelaNext-only drivers. S3's gates, unmodified AMB orchestration with frozen provider modules, and section 6's upstream-bound statistics replace them. +- **The amendment.** `blueprints/memory-stack/longmemeval/A17-SUPERSESSION.md` records this as a dated amendment in the protocol's own lineage, a new file, since frozen text is never edited. It is **proposed**. It takes effect only when the user confirms it at the S3 freeze (section 9). Until then nothing runs under either protocol. + +## r6 §13. r4 review findings and their fixes (r5) + +| r4 finding | Fixed in | +|---|---| +| high 1: v2.4.0 has no embedding-prefix setting; C3′/C4′ cannot reproduce prefixed C3/C4 | §1 definitions: native unprefixed input; S3 controls, not reproductions; embedding-request bytes verified before scoring; §12 | +| high 2: session-ID clustering misses shared conversation content | §3 canonical `(role, content)` rule, 452/401 clusters reproduced with frozen v4 code; §6 manifest | +| medium 3: upstream does not supply both corpus constructions | §3 official track upstream; full-session track the A2 local extension | +| medium 4: D2h is not the shipped default, and hook replay omits transcripts | §2 explicit local-MiniLM configuration; hook equivalence required | +| medium 5: C4′ fallbacks not connected to the decision rule | §7 reference fallbacks: comparisons inconclusive, selection unresolved, no post-hoc C3′ comparator | +| low 6: pending provenance written as present | §2 inputs "will be committed"; §5 #379 is an issue, and the bundle cites its receipt | + +## r6 §14. r6 changes and their sources + +**Source convention.** Repository `path:line` citations refer to the input checkout at **`75e30ea4`** unless a different pin is named. `embed.md`, `rerank.md`, `memsys.md`, `memllm.md` and `gen.md` mean files under [`inputs/sweep-20260927-mac/`](inputs/sweep-20260927-mac/README.md). Upstream links specify the inspected tag/commit. **I, R, M, W and V now resolve exclusively to the repository copies below**, captured **2026-09-27**. Each has one added source/date line and the required personal-home prefix → `~` and session scratch-prefix → `` substitutions; all other bytes, including EOF newline state, are preserved. Hashes cover **these sanitized copies**, and line ranges count the added first line. No mutable scratchpad or workflow journal is an evidentiary locator for this revision. Former J/G leads are replaced with the pinned upstream citations in 1.2, 2/2.1 and 4.4; host/image observations still require their own sanitized native receipts before freeze (5/9), not raw worker conversations or credential stores. + +| Key | Repository capture, line range and sha256 | Evidence boundary | +|---|---|---| +| **I** | [`inputs/r6/s3-r6-inputs.md:1–58`](inputs/r6/s3-r6-inputs.md); sha256 `22389274ad57b42b4ad365a9495ab630331cc2ba4e7f76cfe9a10c3d52901e31` | Gateway rules **4–14**; cognee smoke **16–35**; live hosting **37–54**; Hindsight **56–58**. Supersedes the stale `b0a99fe4…` capture. I:13/31's arm-wide affinity, I:53's effort knob and the broad omission claim at I:48 are corrected in 1.2/2.1. I:34's manifest registration is future integration work outside this repair. | +| **R** | [`inputs/r6/gateway-ab-designs.md:1–101`](inputs/r6/gateway-ab-designs.md); sha256 `d38e9973eda2ae1c84fb2f635959a2068a4358267bb4df51586460959a0ee73b` | Arm rules **7–11**, upstream-harness/statistics rule **15–29**, R02 **31–53**. Supersedes the stale `27e45209…` capture. Its promptfoo/li26 mechanics are leads for their own lane; S3 uses AMB and its own frozen metric. No R02 line supports the removed one-request pacing rule. | +| **M** | [`inputs/r6/memory-merit-synth.md:1–175`](inputs/r6/memory-merit-synth.md); sha256 `89da63f6197d03384e7b716270cb04130d4b3f534dc2a88a247791d05824688f` | Merit/admission/lifecycle leads. Unverified code-level claims remain leads (**167–173**); selected facts use pinned upstream citations in 2 and 4.4. | +| **W** | [`inputs/r6/eval-frameworks-report.md:1–131`](inputs/r6/eval-frameworks-report.md); sha256 `c5f25772de014b945a20b02c2e4b464d34c59f90e9f09060500600ebff0c7f55` | Upstream framework research: AMB **10,53–71**, MTEB **73–81**, SciPy/statsmodels **83–97**. Migration instructions and source review, not executed S3 acceptance. | +| **V** | [`inputs/r6/r6-claude-review.md:1–125`](inputs/r6/r6-claude-review.md); sha256 `aac90a9b092c7a1ea38cb42c455bd00768cabc7a8696b6035001c458924a4824` | Full cross-family review: H1–H4 **11–52**, M1–M5 **56–98**, L1–L4 **102–109**, verification gaps **119–123**. Its P/I/R/M locators describe the pre-repair revision; the active locators in this document use the captures above. Its claims are leads until checked against pinned sources; corrections are recorded in 15. | +| **U** | User's **2026-09-27 directions 1–4**, reproduced below | Normative directions, distinct from the coordinator's detailed requirements A–I. | +| **B** | **Coordinator's r6 brief, requirements A–I**, mapped below; attribution checked against V:103 | Coordinator-authored specification under delegation. **B:I** relays Apple Container 1.4.1 / Postgres 18.6 / pgvector 0.8.6 and the Mac's 12/12 gateway claim; the Mac's own receipts remain required before freeze. The **96 h** window is the coordinator's repair decision 2, overturnable by the user before freeze. | + +**U:1–4, verbatim user directions; A–I are not user quotations:** + +1. “make sure the memory lane using the best of the best repos, congee, hindsight or other evl via gpt6, via runtime workers hosting the sota repos powered by gpt6” +2. “never use staled models, keep using the real SOTA models” +3. “deploy hindsight and cognee now with gpt6 via omni when ready, and make sure our architecture layers resolute with the best candidates live natively, with the best repos of full lifecycle and upstream commands e2e” +4. “make sure they are the best in its layer, with evl of the sotaness, features and best suitable for our ecosystem and pave the way to north star” + +| Change | Landed in | Source and resulting contract | +|---|---|---| +| **A — GPT-6 backbone and gateway discipline** | **1.1–1.2, 2, 3, 4.1, 5, 9, 12** | U:1–3; B:A; I:4–14,18–58; R:7–11,31–53; pinned upstream links in 1.2. Every active LLM role is GPT-6; choose/freeze model and effort on the separate development split, use `-max` where needed, conversation affinity, no-cache plus a dedup-ineligible stream/resolved temperature, fresh per-call IDs where supported or verified omission, compression off and observed effort. C4′'s old local LLM and agentmemory's zero-LLM primary are superseded explicitly. | +| **B — Answerer and judge** | **4.5, 9** | B:B; I:5–12; R:50; [OmniRoute max suffix](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/executors/codex.ts#L1451-L1456). Fixed GPT-6 Astra max for QA answerer/blinded judge; retained stratified manual audit, with prompts and judge independence frozen. | +| **C — Native hosting and corrected configurations** | **2.1, 3, 9** | B:C; I:16–58; Pinned key-precedence and adapter sources in 2/2.1; W:53–71. Live deployments are hosting evidence only; preserve both deploy receipts and the failed unguarded Hindsight start. Fresh container/volume/home per arm, upstream image/lock pins, no scored persistent cognee, instructor `tool_call`, per-role GPT-6 overrides and corrected embedding key precedence. | +| **D — Candidates and plain retrieval diagnostics** | **2, 4.1, 6** | B:D; M:9–13,59–67,71–117; [GBrain pinned session metric and 449/470](https://github.com/garrytan/gbrain/blob/e78f1c38b947b053f3a46881340f74f316be855a/CHANGELOG.md#L2039-L2061); pinned Mem0/Attemory admission sources in 2. Keep seven primaries/C4′, exclude Mem0 OSS/Attemory on fit, screen GBrain through the late-candidate rule; add BM25/dense Qwen3 outside the family. | +| **E — Current local retrieval challengers** | **2, 3, 4.1, 9** | U:2; B:E; `embed.md:9–17,28–41,64–75`; `rerank.md:9–14,44–73`. LoCoMo chooses configurable local embedders/dedicated rerankers; LongMemEval never selects them. MiniLM/BGE-small are labelled controls/diagnostics or documented fixed-provider defaults, with agentmemory's explicitly directed B:C MiniLM exception. MTEB 2.21.8 supplies the registered task and two-stage runner (W:73–81). `memllm.md:136–199` and `gen.md:23–85` remain discovery inputs; their local generation rankings do not override the mandated GPT-6 LLM backbone. | +| **F — Research-agent usefulness** | **4.5(a), 7, 9** | B:F; `blueprints/us-equities/north-star.md:33–35,57–60,74–79`; `catalogs/us-equities/README.md:52–58`; decision-index and four domain READMEs at the RM01–RM08 locations. Keep 20 queries + 5 continuations; at least eight research-memory queries have frozen expected evidence and count toward the same minus-one usefulness floor. | +| **G — Cost, sample and pooled-account load** | **4.2, 6, 9** | B:G and coordinator repair decision 2; approximately 115K history scale in [pinned benchmark overview](https://github.com/AttemorySystem/attemory/blob/603c03afa9a04e48b778922eae156a0024752f37/README.md#L125); I:35. Plan **all 470** (452/401 full-population clusters), with only a **prospective stratified subset** if measured projection exceeds **96 h per host**. Preserve stochastic paired seeds, 10,000 resamples, Holm α=0.05 over 14 and +0.05/−0.02 rules. Freeze development-selected K for **full parallel utilization**, measured tokens/s and wall-clock projections; a deterministic arm is ingested/counted once. Never gate on a recorded quota. | +| **H — Probe lifecycle blockers early** | **4.4, 5, 9** | B:H; M:45,76–77,128,138–142,172; agentmemory issues [1273](https://github.com/rohitg00/agentmemory/issues/1273), [1190](https://github.com/rohitg00/agentmemory/issues/1190), [1389](https://github.com/rohitg00/agentmemory/issues/1389); MemPalace [1581](https://github.com/MemPalace/mempalace/issues/1581); Hindsight [models.mdx:154–169](https://github.com/vectorize-io/hindsight/blob/f8950b0c07d9e34c76493dba802bb309f0ce60fd/hindsight-docs/docs/developer/models.mdx#L154-L169) and I:58. Development checks target delete, complete empty-instance restore, sustained-load OOM, concurrent-writer corruption, actual database durability settings and output-limit compatibility. | +| **I — Both hosts ready for their own qualification** | **1.1, 5, 9** | **B:A/I**, the coordinator's relayed Apple Container/Postgres/pgvector and 12/12 gateway observations, not U's verbatim words. The Mac's own sanitized hosting, gateway and build/patch receipts are mandatory before freeze; keep each host's decision and scored-arm qualification separate. | + +**Corrections recorded in r6, with verification paths.** In addition to the table: (1) R:8 supersedes per-arm affinity in I:13/31; (2) the pinned upstream implementation supersedes the research claim that agentmemory's separate embedding key wins, confirmed in the pinned `embedding/index.ts:37–38` and `embedding/openai.ts:64–68`; (3) I:25–35's measured instructor success/JSON failures supersede cognee's research-only structured-output inference, with the pinned LiteLLM/config corrections in 2.1 retained; (4) GBrain's pinned `CHANGELOG.md:2042–2061` supersedes `memsys.md:36`'s chunks/session dismissal; (5) the coordinator's B:I report conditionally supersedes the old Mac owner-install blocker. The sweep summary's broad “no Mac path” for Nemotron (`README.md:16`) is also narrowed to its unregistered llama.cpp class: the detailed `embed.md:12,40–41,71` leaves PyTorch/MPS as a development qualification path. These corrections do not rewrite historical source captures or claim unexecuted acceptance. + +## r6 §15. r6 cross-family review findings and their fixes + +The complete review is **V**, captured above. All thirteen findings are applied in this repair; the table records the correction and its verification path. This is a **protocol/source repair**, not runtime acceptance or a freeze. Measured throughput, provider parity, latency/fallbacks and native host receipts remain bundle prerequisites. + +| Finding | Fixed in | Applied change and source verification | +|---|---|---| +| **H1 — stale sources** | **1.1–2.1, 4.4, 9, 14** | Captured all five requested inputs; hashes and every active I/R/M/W/V line locator refer to the sanitized repository copies. R02 is R:31–53; live hosting is I:37–54; Hindsight is I:56–58. Removed J/G scratchpad/journal locators in favor of the existing pinned upstream citations and explicit host-receipt requirements. Original bytes differ only by the required header/sanitization, including preserved EOF newlines. | +| **H2 — standalone harness** | **Status, 3, 9, 12** | Unmodified **AMB 03c1d0f1**, frozen/hashed MemoryProvider modules, unchanged **LongMemEval @9e0b455f** scoring, pinned loader path, Gemini-key/.env handling and equal-depth parity audit. **Correction to V:24 and V:27:** the shipped cognee provider is an in-process SDK wrapper, not an HTTP client, and hard-codes an old LLM/BGE-small plus top_k=50. Its separate frozen provider adaptation is required; AMB orchestration stays unmodified. Verified at [cognee.py:66–86,118–137](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/cognee.py#L66-L86), [Hindsight HTTP:1242–1255](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/hindsight.py#L1242-L1255), [CLI:12–30](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/cli.py#L12-L30) and W:53–71. | +| **H3 — pacing/feasibility** | **1, 3, 4.2, 6, 9, 14** | Removed unsourced single-call/one-second limits. Freeze development-selected **K** for the user's **full parallel utilization**, native concurrency controls, block-randomized order, measured tokens/s and per-host projection. **96 h per host** is the coordinator's decision; a prospective stratified subset with power calculation is allowed only before results. Deterministic arms ingest/count once. The review's canary projection is illustrative, not full-run throughput (I:35; V:29–41). | +| **H4 — query latency/fallbacks** | **1, 4.3, 6, 7, 9** | GPT-6 query calls remain inside **warm p95 ≤1.0 s**, with **10 s** timeouts. Isolate latency from other arms' calls; report queue wait separately. Development preflight now measures C4′ fallback rate and warm p95; **>5% fallbacks selects C3′ before confirmatory execution**. Later failure still leaves the host unresolved. Source: [ai-memory v2.4.0 reranker behavior](https://github.com/akitaonrails/ai-memory/blob/v2.4.0/docs/llm-providers.md#L185-L195). No GPT-6 latency has been measured by this repair. | +| **M1 — statistics implementation** | **6, 7, 9** | Bound the unchanged weighted-cluster estimand to **SciPy 1.18.1 / statsmodels 0.15.0**, explicit **10,000** paired resamples, **PCG64(20260927)**, linear fifth percentile and **14** Holm hypotheses with missing-arm p=1.0. Sign-flip is diagnostic only. Removed the undefined adjusted-bound equivalence. Verified [SciPy signature/quantile path](https://github.com/scipy/scipy/blob/v1.18.1/scipy/stats/_resampling.py#L300-L303) and [Holm branch](https://github.com/statsmodels/statsmodels/blob/v0.15.0/statsmodels/stats/multitest.py#L233-L247); W:83–97. | +| **M2 — envelope/static headers** | **1.2, 2.1, 9, 14** | Wire-verified stream or absent/>0.1 resolved temperature; explicit 1.0 is permitted. Defined question × seed × arm conversation, fresh-process route and conditional Hindsight affinity route. Static idempotency keys are never called fresh: omit both key headers on static-only adapters and prove no replay. Read [idempotency.ts:74–110](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/handlers/chatCore/idempotency.ts#L74-L110) and [header fallback:42–55](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/src/lib/idempotencyLayer.ts#L42-L55), closing V's stated verification gap. An initial source lookup under services returned 404; [chatCore's import:33](https://github.com/diegosouzapw/OmniRoute/blob/a58000c7685f4091c7a6fd8ddf3ebce7d2ec67c3/open-sse/handlers/chatCore.ts#L33) resolves the actual path used here. | +| **M3 — live deployments** | **2.1, 3, 4.4, 9** | Recorded I:37–54's hosting evidence; protect ports **3800/3710/5433**, archive both sanitized deploy receipts, recheck **10.0.2.2:20128**, preserve the guard modification and first failure, and keep live services unscored/untouched. Corrected operation **LLM_MODEL/LLM_REASONING_EFFORT** names and global-none/unset-operation temperature resolution. Verified [config.py:210–245,383–462](https://github.com/vectorize-io/hindsight/blob/f8950b0c07d9e34c76493dba802bb309f0ce60fd/hindsight-api-slim/hindsight_api/config.py#L210-L245); live claims are input receipts, not newly reproduced execution. | +| **M4 — R02 selection** | **1.1, 3, 6, 9** | Before development calls freeze full-system LoCoMo supporting-evidence recall@5, schema validity, **n**, cluster manifest and one-sided α=.05 bootstrap; coupled roles select jointly, on each host. Unresolved roles leave arms pending and require a dated amendment before their first confirmatory run. R:31–53 motivates the comparison; section 6 supplies its statistical implementation. | +| **M5 — judge/model checks** | **4.1, 4.5(b), 9** | Unchanged LongMemEval **get_anscheck_prompt**, including abstention, under the Astra-max envelope rather than the temperature-0 script call path; MTEB **2.21.8 LoCoMo** and upstream two-stage reranking with recall@5 frozen before selection. Verified [judge prompt/call path](https://github.com/xiaowu0162/LongMemEval/blob/9e0b455f4ef0e2ab8f2e582289761153549043fc/src/evaluation/evaluate_qa.py#L24-L43), [LoCoMo registration](https://github.com/embeddings-benchmark/mteb/blob/2.21.8/mteb/tasks/retrieval/eng/lmeb_retrieval.py#L996-L1038) and W:73–81. MTEB's default ndcg_at_10 is not silently substituted. | +| **L1 — GPU consent** | **5** | Restored **“a scheduled production stop and restore that the user approves”**; the runner never stops production itself (V:102; accepted r5 source at 75e30ea4, this file:221). | +| **L2 — attribution/Mac receipts** | **Status, 1–5, 9, 14** | U contains only verbatim directions **1–4**; **B** is the coordinator's **A–I** brief. Host-install and 12/12 statements are relayed observations; require the Mac's own receipts before freeze (V:103). | +| **L3 — MiniLM exception** | **2, 4.1, 14** | MiniLM remains primary as the explicitly directed **B:C exception to B:E**, despite agentmemory's configurable OpenAI-compatible embedder. **Attribution correction to V:106:** requirement C is coordinator-authored, reaffirmed by this delegated repair request; it is not one of U's verbatim directions. Upstream [local provider](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/providers/embedding/local.ts#L8-L46) and [OpenAI-compatible provider](https://github.com/rohitg00/agentmemory/blob/2d38dafede67d0d4ed920cde94d2106e98825b8a/src/providers/embedding/openai.ts#L64-L68). | +| **L4 — achieved power** | **6, 9** | Require both routes' achieved-power report from frozen analysis over stated discordance **0.10–0.30**, with cluster/seed/family assumptions and simulation uncertainty. Informational, not a new gate or margin; prospective subset power remains required. V:108–109's normal approximation stays illustrative. | + +**Verification boundary.** This repair read the pinned AMB registry/CLI/runner/loader/providers, OmniRoute dedup/idempotency code, Hindsight config/affinity, ai-memory reranker documentation, SciPy/statsmodels implementations, LongMemEval judge prompt and MTEB LoCoMo/two-stage sources. It did not execute provider/model runs, inspect live services or credential stores, remeasure the Mac, or treat archived worker answers as upstream acceptance. The byte/hash/locator checks verify the document repair; they do not satisfy the experiment's pre-freeze execution gates. + +**Repair validation.** Sanitized-copy byte equality (including EOF state), all five copy hashes, active source-locator bounds, all 13 finding rows, analysis-snippet syntax and `git diff --check` pass. The coordinator committed the repair and merged `main` under the hot-file protocol, re-registering this document and the five `inputs/r6/` copies in `manifests/evidence.json`; `python3 scripts/validate.py` then passed (merge commit `f968acf8`). The section 16 fixes change this document again, so it is re-registered and re-validated in the same commit. + +## r6 §16. r6 re-check findings and their fixes (2026-09-27) + +A one-round Claude re-check compared the repaired draft with pinned AMB `03c1d0f1`, LongMemEval `9e0b455f`, SciPy 1.18.1, statsmodels 0.15.0 and MTEB 2.21.8. It confirmed the 13 section 15 rows and found the defects below. N1 and N2 sat in the AMB text that coordinator decision 3 had adopted from the review, so their fix is a coordinator decision. + +**Decision (coordinator, 2026-09-27; alternatives and overturn condition in section 3):** one frozen, hashed LongMemEval dataset adapter in AMB's retrieval mode. The coordinator verified the premises in the AMB source at `03c1d0f1`: +- `EvalRunner.__init__` constructs `GeminiJudge()` (runner.py:42); +- AMB's LongMemEval dataset has `task_type = "open"` (longmemeval.py:67); +- its documents carry `"{question_id}_{session_id}"` IDs in the ID and the context (longmemeval.py:329–349); +- the retrieval branch makes no judge call and passes the provider's ranked documents to `score_retrieval` (runner.py:222–227; modes/retrieval.py); +- `EvalRunner.run` takes a `Dataset` instance (runner.py:51–71); +- the lock pins cognee 0.5.4 and hindsight-client 0.9.2 (uv.lock:854–855, 2077–2078). + +| Finding | Severity | Fixed in | Fix | +|---|---|---|---| +| **N1 — Gemini judge on every LongMemEval query** | High | 3, 9 | Retrieval mode with the adapter's `task_type = "retrieval"` removes every judge call. A placeholder is used only to construct the client. The wire proof shows no Google or judge call. | +| **N2 — gold-revealing raw session IDs** | High | 3 | The adapter emits opaque IDs, an ID-free context and the frozen track, and exports each ranked opaque-ID list through `score_retrieval`. The parity audit verifies at the wire that no provider receives a raw ID. | +| **N3 — cognee lock conflict** | Medium | 3, 9 | The cognee arm runs cognee v1.6.1's tag lock plus unmodified AMB source. The bundle hashes each arm's executed environment. hindsight-client 0.9.2 versus the 0.10.1 service is added to the parity audit. | +| **N4 — sequential runner versus K and route (a)** | Medium | 3, 9 | A frozen dispatch wrapper with no benchmark logic launches unmodified per-unit `EvalRunner.run` processes in the frozen order and concurrency. | +| **N5 — AMB judge prompts and headers** | Medium | 4.5(b) | AMB's judge is not used. The S3 answerer and judge run outside AMB under 1.2, and 4.5(b) is `pending` until the answerer integration passes the wire check. | +| **N6 — `validate.py` on the pushed tree** | Medium | Repair validation | The coordinator re-registered the document and the five `inputs/r6/` copies. `validate.py` passed at `f968acf8` and is re-run for this commit. | +| **N7 — V:28 locator** | Low | 15 (H2 row) | "Correction to V:24 and V:27". | +| **N8 — user attribution** | Low | 4.2 (K rule) | V:32, the memory locator, and "historical evidence, not authority"; the K rule is attributed to coordinator decision 2. | +| **N9 — removed runner still named** | Low | 2 (pins), 5 | "the Mac's own sanitized host receipt"; "no S3 or AMB-driven process". | +| **N10 — precision** | Low | 4.4, 4.5(b), 14, 15 | `M:139,172`; `167–173`; `evaluate_qa.py#L24-L43`; section 5 removed from H4's "Fixed in". `embed.md:31–41` (the challenger shortlist, section 4.1) and `28–41` (controls plus shortlist, E row) stay distinct by design, after checking the committed sweep file: rows 28–30 are the MiniLM control, mxbai and LFM2-ColBERT. | + +**Residual verification gaps**, named by the re-check and not checkable from its snapshot, all left for the section 9 bundle review: +- OmniRoute `codex.ts`, `chatCore.ts`, `idempotency.ts`, `idempotencyLayer.ts` and `requestDedup.ts`; +- Hindsight `config.py`, `cache_affinity.py` and its installation and model docs; +- cognee config and Dockerfile, and agentmemory sources; +- the GBrain, Mem0 and Attemory links, and the `akitaonrails/ai-memory` URL; +- MTEB `two_stage_reranking.md`, which is not in the wheel; +- the sweep-file and RM01–RM08 locators. diff --git a/blueprints/memory-layer-s3/inputs/r6/eval-frameworks-report.md b/blueprints/memory-layer-s3/inputs/r6/eval-frameworks-report.md new file mode 100644 index 000000000..4bbfd69bd --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r6/eval-frameworks-report.md @@ -0,0 +1,131 @@ +Source: /eval-frameworks/report.md; capture date: 2026-09-27. +**Use upstream frameworks for orchestration and statistics, while retaining explicit task contracts. W1 and W3 have no verified complete replacement meeting every requirement unchanged.** + +Read-only research as of **2026-09-27**; no installations, service/model runs, edits, commits, or credential-store reads. Maintenance counts below came from paginated `gh api repos/OWNER/REPO/commits?since=2026-06-29T00:00:00Z&per_page=100`, including default-branch merges. Licenses are informational. + +| Workflow | Primary | Runner-up | Recommended pin / latest release | Maintained | Fit | Remaining gaps | What it replaces | +|---|---|---|---|---|---|---|---| +| **W1** | [Harbor](https://github.com/harbor-framework/harbor) | Inspect AI + inspect_swe | **v0.23.0**, Sep 12; Apache-2.0 | 533 commits | Native Codex/Claude transcripts, repetitions, concurrency, sandbox tasks | Exact `stack-worker` profile, existing worktree binding, same-call success/sentinel assertion, Loki reconciliation | Orchestration in `m13_probe.py` and `matrix_scan.py`, conditionally; retain `loki_reconcile.py` | +| **W2** | [Promptfoo](https://github.com/promptfoo/promptfoo) | Inspect AI | **0.123.1**, Sep 18; MIT | 844 commits | Gateway providers, headers, structured output, repeated comparisons, usage/latency | Dynamic request headers, aggregate micro-F1, paired inference, affinity verification | [eval_arm.py](~/code/native-agent-stack/blueprints/convergence-practice/local-inference-latest-20260926/eval_arm.py) request/repetition/reporting code | +| **W3** | [AMB](https://github.com/vectorize-io/agent-memory-benchmark), **conditional**, plus original LongMemEval scorer | Native LongMemEval retrieval harness | **03c1d0f1d27da63034f0931121c858faba512383**; no release; last push Sep 22; license undeclared | 12 commits | LongMemEval-S orchestration; Hindsight/cognee adapters | Six adapters, unchanged metric integration, backbone configuration, lifecycle acceptance; CLI defect | Planned S3 runner and [lme_harness.py](~/code/native-agent-stack/blueprints/memory-stack/longmemeval/v4/lme_harness.py), only after gaps close | +| **W4** | [MTEB](https://github.com/embeddings-benchmark/mteb) | BEIR | **2.21.8**, Sep 23; Apache-2.0 | 406 commits | Registered LoCoMo, retrieval and two-stage reranking | Dataset/model choices still required; no memory lifecycle evaluation | W4 use of [mteb_lmeb.py](~/code/native-agent-stack/blueprints/memory-stack/longmemeval/v4/mteb_lmeb.py) and `rerank_stage.py` | +| **W5** | [SciPy](https://github.com/scipy/scipy) + [statsmodels](https://github.com/statsmodels/statsmodels) | arch | **1.18.1**, Aug 21 / **0.15.0**, Aug 27; BSD-3-Clause | 571 / 1,265 commits | Paired bootstrap, permutation tests, Holm, statistical inference | Cluster unit, estimand, non-inferiority margin remain study definitions | [analyze.py:85](~/code/native-agent-stack/blueprints/convergence-practice/local-inference-latest-20260926/analyze.py:85) `paired_bootstrap`; proposed S3 statistics | +| **W6** | Each tool’s upstream suite, below | Inspect AI for shared task-quality comparison | Seven pins below | All show recent activity | Native compression/retrieval evidence | No suite establishes all seven tools’ agent-task quality unchanged | Replace fixture-only E2E claims; retain accounting/integration files | +| **W7** | [Anthropic skill-creator](https://github.com/anthropics/skills) paired benchmark procedure | Superpowers TDD/verification | **33375500bcea98d610eb30ce10ac4e59b89c390d**, Sep 24; no release; selected skills Apache-2.0 | 14 commits | Paired skill/baseline quality, time and tokens | Procedure requires actual runs; not W5 statistical inference | Ad hoc evaluation procedures; no separate harness file identified | + +Commands below are **migration instructions, not executed acceptance**. Angle-bracket inputs denote case artifacts still requiring preparation. + +**W1 — choose Harbor for native transcripts.** Its pinned adapters invoke [Codex `--json` and preserve native sessions](https://github.com/harbor-framework/harbor/blob/v0.23.0/src/harbor/agents/installed/codex.py#L1436), and [Claude `stream-json`](https://github.com/harbor-framework/harbor/blob/v0.23.0/src/harbor/agents/installed/claude_code.py#L1895). Rewardkit provides [tool-used/tool-not-used criteria](https://github.com/harbor-framework/harbor/blob/v0.23.0/docs/content/docs/rewardkit/built-in-criteria.mdx#L82), but these alone do not prove that the same successful call returned the correct worktree sentinel. + +```bash +uv tool install 'harbor==0.23.0' +harbor run -p '' -a codex -m '' --ak version=0.157.1 -k 20 -n 2 +harbor run -p '' -a claude-code -m '' --ak version=2.1.283 -k 20 -n 2 +``` + +These flags establish repetitions/concurrency, **not the required two existing worktrees**. Harbor uses temporary client homes; unchanged `-p stack-worker` support was not found in its inspected options/source. Retain the native gate until positive, disabled-server and wrong-root cases satisfy binding and Loki reconciliation. + +Inspect **0.3.271** (Sep 26; 1,504 commits; MIT) with inspect_swe **0.2.71** (Sep 17; 76 commits; MIT) is the runner-up: its [bridge proxies model calls](https://github.com/meridianlabs-ai/inspect_swe/blob/0.2.71/docs/_snippets/agent_intro.md#L3), and [Codex transcripts are reconstructed from bridge events](https://github.com/meridianlabs-ai/inspect_swe/blob/0.2.71/src/inspect_swe/_codex_cli/_events/consumer.py#L3). + +Installed Claude 2.1.283 independently offers: + +```bash +claude plugin eval '' --runs 20 --concurrency 2 --mocks off --ablation none --no-publish --json '' +``` + +Installed help confirms `--mocks record|off` and `--allow-real-servers`; recorded mocks require the latter to contact missing real servers. This is a useful Claude-only lane. + +**W2 — retain Promptfoo, with explicit metric and header integration.** Its pinned documentation covers [base URL, headers, schema/tool output and effort](https://github.com/promptfoo/promptfoo/blob/0.123.1/site/docs/providers/openai.md#L182) and [session hooks](https://github.com/promptfoo/promptfoo/blob/0.123.1/site/docs/configuration/reference.md#L365). + +Configure the two model IDs against `http://127.0.0.1:20128/v1`; enumerate singleton combinations explicitly. Preserve one session key per conversation while generating fresh request IDs. Static headers alone are insufficient. + +```bash +npm install -g promptfoo@0.123.1 +promptfoo eval -c '' --repeat 20 --max-concurrency 1 --no-cache --no-share -o '' +``` + +Use its [Python scorer interface](https://github.com/promptfoo/promptfoo/blob/0.123.1/site/docs/configuration/expected-outputs/python.md#L15) with frozen li26 semantics: **aggregate TP/FP/FN before micro-F1; do not average filing F1**. Pair observations by case/repetition and analyze with W5. + +Promptfoo’s response-cache switch differs from `X-OmniRoute-No-Cache`. Provider cache-read tokens are [separate accounting fields](https://github.com/promptfoo/promptfoo/blob/0.123.1/src/providers/openai/util.ts#L997); calculate cached-input share from returned provider counts, not `tokenUsage.cached`. Affinity still requires gateway observation. + +OpenAI Evals had zero commits in the window; that does not describe hosted API capabilities. [lm-evaluation-harness supports endpoint/header configuration](https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.13/docs/API_guide.md#L104), but offers a less direct fit than Promptfoo’s case/provider comparison workflow. + +**W3 — do not commission a new eight-arm harness yet.** AMB’s [registry](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/__init__.py#L15) covers **Hindsight and cognee**. Coverage among the eight requested systems is: + +- AMB: **2/8**. +- [Supermemory MemoryBench](https://github.com/supermemoryai/memorybench/blob/94e2af54b661d90e77dddbd8fa4fa5b28c07a24e/src/providers/index.ts#L9): **0/8**; adapts Supermemory, Mem0, Zep, filesystem and RAG. +- [THUIR MemoryBench](https://github.com/THUIR/MemoryBench/blob/247ba8c1e327fa297c0d57aaad0e3cbd66bac656/README.md#L166), [mem0’s suite](https://github.com/mem0ai/memory-benchmarks/tree/4b61c5d31b9c668a12b4f5e78064248a02c82d2b), and [Letta Evals](https://github.com/letta-ai/letta-evals/tree/688e099a70295a281ea3667398fb53b4d7af89d4): **0/8** in reviewed adapters. +- Native LongMemEval: **0/8 service adapters**, but the required scorer. + +Retain [LongMemEval `eval_utils.py:24–46`](https://github.com/xiaowu0162/LongMemEval/blob/9e0b455f4ef0e2ab8f2e582289761153549043fc/src/retrieval/eval_utils.py#L24): `recall_all@5` means **every gold session appears among five unique retrieved sessions**, a binary question-level result—not fractional recall or answer accuracy. AMB’s retrieval-only mode uses different assertions. + +```bash +git clone https://github.com/vectorize-io/agent-memory-benchmark.git +cd agent-memory-benchmark +git checkout 03c1d0f1d27da63034f0931121c858faba512383 +uv run amb providers +uv run amb run --dataset longmemeval --split s --memory hindsight +uv run amb run --dataset longmemeval --split s --memory cognee +``` + +These cover a subset only. OpenAI answer-model configuration exists, but the [CLI unconditionally requires a Gemini key](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/cli.py#L25). GPT-6 through a custom URL for every system’s backbone, unchanged scoring, and lifecycle acceptance remain unresolved. + +**W4 — explicitly select LoCoMo.** Its first verified MTEB release is **2.13.8, May 19**; it remains [registered in 2.21.8](https://github.com/embeddings-benchmark/mteb/blob/2.21.8/mteb/tasks/retrieval/eng/lmeb_retrieval.py#L996). Use the [upstream two-stage example](https://github.com/embeddings-benchmark/mteb/blob/2.21.8/docs/get_started/advanced_usage/two_stage_reranking.md#L23): + +```bash +pip install 'mteb==2.21.8' +mteb run -m sentence-transformers/static-similarity-mrl-multilingual-v1 -t LoCoMo --prediction-folder w4-predictions +python -c 'import mteb; t=mteb.get_task("LoCoMo").convert_to_reranking("w4-predictions",top_k=100); mteb.evaluate(mteb.get_model("cross-encoder/ms-marco-TinyBERT-L2-v2"),t)' +``` + +These models reproduce the upstream example; substitute frozen evaluation arms. BEIR **v2.2.0**, June 4, 2025, has zero commits since cutoff and needs LoCoMo conversion. MTEB’s LoCoMo dataset license is separately CC-BY-NC-4.0. + +**W5 — replace statistical algorithms, preserve the estimand.** + +```bash +pip install 'scipy==1.18.1' 'statsmodels==0.15.0' +``` + +There is no upstream CLI consuming the local li26/S3 format. Bind existing data/scoring to these supported APIs: + +- [`bootstrap`](https://github.com/scipy/scipy/blob/v1.18.1/scipy/stats/_resampling.py#L363): `paired=True, method="percentile", confidence_level=.95, alternative="greater", n_resamples=10000`. +- Resample **question-cluster IDs**, retaining both arms and all observations within each sampled cluster. Recompute aggregate metrics. +- [`permutation_test`](https://github.com/scipy/scipy/blob/v1.18.1/scipy/stats/_resampling.py#L1705): `permutation_type="samples"` for paired swaps, respecting cluster boundaries. +- [`multipletests`](https://github.com/statsmodels/statsmodels/blob/v0.15.0/statsmodels/stats/multitest.py#L233): `method="holm"` on valid family p-values. +- Predeclare non-inferiority margin δ; compare the one-sided lower bound with `−δ`, with the multiplicity policy specified separately. + +Cluster-robust covariance is not cluster bootstrap. arch is the bootstrap-focused runner-up; Pingouin adds no necessary primitive here. + +**W6 — run upstream suites, then add task-quality acceptance where absent.** Source-checkout commands below require the linked tag. + +- **[RTK v0.50.0](https://github.com/rtk-ai/rtk/blob/v0.50.0/scripts/benchmark.sh#L31)** — Sep 24; 881 commits; Apache-2.0. Install: `cargo install --git https://github.com/rtk-ai/rtk --tag v0.50.0 --locked`. Run `rtk verify --require-all`; `bash scripts/benchmark.sh`. Benchmark tokens are characters/4 estimates; no agent-quality gate. +- **[context-mode v1.0.169](https://github.com/mksglu/context-mode/blob/v1.0.169/tests/ecosystem-benchmark.ts#L449)** — Jun 29; 146; Elastic-2.0. Install `npm install -g context-mode@1.0.169`; checkout: `npm install && npm run test:ecosystem`. Measures execution/bytes, not answer quality. +- **[Headroom v0.39.1](https://github.com/headroomlabs-ai/headroom/blob/v0.39.1/headroom/evals/README.md#L108)** — Sep 26; 1,154; Apache-2.0. `pip install 'headroom-ai[evals]==0.39.1'`; `python -m headroom.evals suite --tier 1 --ci`. Provides before/after quality evaluations; still needs the actual workflow. +- **[jCodeMunch v1.108.319](https://github.com/jgravelle/jcodemunch-mcp/blob/v1.108.319/benchmarks/competitive/run.py#L1)** — Sep 16; 1,080; Dual-Use 1.1. `pip install 'jcodemunch-mcp==1.108.319' tiktoken`; `python benchmarks/competitive/run.py --adapters null_readall,null_grep,jcodemunch --runs 20 --sandbox none`. Citation F1 and baselines; not completed coding-task quality. +- **[Serena v1.7.0](https://github.com/oraios/serena/blob/v1.7.0/docs/04-evaluation/020_prompts/000_prompts.md#L3)** — Aug 9; 442; **MIT at this tag**. `uv tool install -p 3.13 'serena-agent==1.7.0'`. Use upstream evaluation prompts; no standalone deterministic benchmark command found. Self-evaluation is insufficient for independent acceptance. +- **[QMD v2.8.3](https://github.com/tobi/qmd/blob/v2.8.3/README.md#L1058)** — Aug 16; 110; MIT. `npm install -g @tobilu/qmd@2.8.3`; then `qmd collection add test/eval-docs --name eval-docs`, `qmd embed -c eval-docs`, `qmd bench src/bench/fixtures/example.json --json`. Retrieval quality/latency, not downstream answer quality. +- **[TOON v4.1.1](https://github.com/toon-format/toon/blob/v4.1.1/benchmarks/README.md#L8)** — Aug 5; 135; MIT. `npm install -g @toon-format/cli@4.1.1`; checkout `pnpm install`; in `benchmarks/`, run `pnpm benchmark:tokens` and `pnpm benchmark:accuracy`. Deterministic answer scoring covers comprehension, not generation quality. + +Keep [token_manifest.py](~/code/native-agent-stack/tools/token-report/token_manifest.py) for accounting and [native_token_ci.py](~/code/native-agent-stack/scripts/native_token_ci.py:2) for integration fixtures; its existing documentation already distinguishes these evidence levels. + +**W7 — load targeted upstream skills.** Installed cached **Vercel skills 1.7.0** discovery was actually run for skill-creator, E2E, testing and property-based testing; no installation occurred. + +Load the existing Anthropic skill-creator copy recorded by adoption, rather than installing a duplicate. Its [paired benchmark procedure](https://github.com/anthropics/skills/blob/33375500bcea98d610eb30ce10ac4e59b89c390d/skills/skill-creator/SKILL.md#L169) consumes completed runs with: + +```bash +python -m scripts.aggregate_benchmark /iteration-N --skill-name +``` + +For a clean host: + +```bash +npx skills@1.7.0 add https://github.com/anthropics/skills/tree/33375500bcea98d610eb30ce10ac4e59b89c390d/skills/skill-creator -a codex claude-code -y +``` + +Load Superpowers verification for W1–W6 and TDD for integration changes: **v6.4.2**, Sep 25, 74 commits, MIT. Load Trail of Bits property-based-testing for parser/scorer invariants: **0cc1c73a5e96749ab32d7ea5e14892fafa6972ae**, Sep 24, no release, 81 commits, CC-BY-SA-4.0. Harbor’s [Rewardkit skill](https://github.com/harbor-framework/harbor/blob/v0.23.0/docs/content/docs/rewardkit/index.mdx#L9) specifically fits W1 verifier construction. `mcp-builder` fits MCP implementation changes; webapp-testing has no W1–W6 browser requirement. + +**Records retained or overturned:** retain Promptfoo 0.123.1 for W2 and MLflow’s conditional tracking status. Narrow Inspect’s default for W1; correct its “unversioned” metadata using [0.3.271’s tag/changelog](https://github.com/UKGovernmentBEIS/inspect_ai/blob/0.3.271/CHANGELOG.md#L1). Update MTEB 2.21.0 to 2.21.8 and replace W4’s LongMemEval selection with LoCoMo. Correct li26’s **2.5th-percentile** lower endpoint to the requested **5th percentile**. Superpowers [renamed `testing-anti-patterns.md` to `writing-good-tests.md`](https://github.com/obra/superpowers/blob/v6.4.2/RELEASE-NOTES.md#L130). Preserve community-sweep R18’s rejection of blanket `networkidle`; the current Anthropic skill still contains it. No source review here establishes new acceptance. + +**Not found:** S3 `PREREGISTRATION.md` in this checkout; a complete unchanged W1 gate/Loki reconciler; an eight-system S3 command preserving the required metric; or a turnkey li26 statistical CLI. Those gaps should not be filled by inventing another harness. + +TOKEN TOOLS USED: RTK: bounded CLI output, context-mode: API/source processing, Serena: exact statistical symbol inspection, QMD: recorded-document retrieval, ai-memory: historical decision leads. \ No newline at end of file diff --git a/blueprints/memory-layer-s3/inputs/r6/gateway-ab-designs.md b/blueprints/memory-layer-s3/inputs/r6/gateway-ab-designs.md new file mode 100644 index 000000000..48a1a20d5 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r6/gateway-ab-designs.md @@ -0,0 +1,101 @@ +Source: /capability-gate/gateway-ab-designs.md; capture date: 2026-09-27. +# Gateway A/B designs: R02, L02 and D03/D04 (reviewed by token-save-manifest-html 2026-09-27; its two findings are fixed here) + +Built on the gateway owner's `ab_design` for each feature in `omniroute-feature-verdict.interim.json`, and hosted in +this session's capability-gate harness. + +Rules for every arm: +- one `x-omniroute-session` key per CONVERSATION, never one per arm. An accepted explicit header overrides the body identifiers for affinity, so a per-arm constant pins the whole arm to one account (the owner's GPT-6 RH6/L01 finding; measured 2026-09-27: 8 distinct headers spread over 3 accounts, while 7 calls under one header stayed on 1). The arm is tagged separately, by the `otel.environment` value or an Idempotency-Key prefix; +- a fresh `Idempotency-Key` per call; +- `X-OmniRoute-No-Cache: true` with `stream: true` on FW calls, so neither the legacy exact semantic cache nor in-flight dedup can serve a sample; +- the effective effort is read back from call_logs (`scratchpad/effort_window.py`, effort columns only, read-only). + +Each design is frozen before its first scored call, with the list of hashes, and gets a GPT-6 review. + +## Harness: upstream, not self-written (user direction 2026-09-27; eval-frameworks/report.md W2 and W5) + +- **Runner:** promptfoo 0.123.1, the catalog's quality-evaluation pin. + - Providers: the OpenAI chat provider with `apiBaseUrl: http://127.0.0.1:20128/v1`, one per arm model (docs site/docs/providers/openai.md#L182 at 0.123.1: base URL, headers, response_format and tools, effort). + - Session hooks (configuration/reference.md#L365) set a session key per conversation and a fresh Idempotency-Key per request. Static headers alone cannot do this. + - Run with `promptfoo eval -c --repeat N --max-concurrency 1 --no-cache --no-share -o `. promptfoo's own `--no-cache` is its response cache, separate from `X-OmniRoute-No-Cache`, so both are set. +- **Quality:** + - a promptfoo Python assertion (configuration/expected-outputs/python.md#L15) that calls li26's own `parse_items` and `filing_scores`, and returns per-filing TP/FP/FN as named metrics; + - micro-F1 is aggregated from summed TP/FP/FN, never as an average of filing F1. +- **Cached-input share:** taken from the provider's returned cached-token fields (src/providers/openai/util.ts#L997), not promptfoo's `tokenUsage.cached`. Affinity is still observed in call_logs. +- **Statistics:** + - scipy 1.18.1 `stats.bootstrap(..., paired=True, method="percentile", alternative="greater", n_resamples=10000)`, over cluster-level aggregates where clusters apply; + - `stats.permutation_test(permutation_type="samples")` for paired swaps; + - statsmodels 0.15.0 `multipletests(method="holm")` for any family. + - li26's own `paired_bootstrap` is not reused. + +## R02: FW effort (cx/gpt-6-astra, medium by default, versus cx/gpt-6-astra-max) + +- **Question:** does max beat medium for FW (chat/completions) roles by enough to pay for its reasoning tokens? +- **Tasks, two parts:** + 1. **Wire and cost on a deterministic task.** + - Items: a frozen, seeded subset of li26's 8-K filings. Take 60 filings, stratified by item count, from the li26 acquisition, pinned by the sha256 in `plan.json`. + - Requests: built with li26's `build_request` and prompt template, with the llama.cpp-only fields (`chat_template_kwargs`) dropped. The gateway's strict `json_schema` needs `additionalProperties: false` on every object: the GPT-6 Responses backend rejected cognee's KnowledgeGraph without it (400). So first send ONE li26 request with the converted schema through the gateway. If it fails, add `additionalProperties: false` to li26's schema in BOTH arms, or use tool calling. Never fall back to `json_object`: through the gateway it needs the word "json" in the input messages, because toResponses.ts L120-136 moves the system prompt into `instructions`. + - Session keys: one per filing. + - Scoring: li26's `parse_items`, `filing_scores` and `micro_f1`, plus `paired_bootstrap` (seed 20260926). + - li26 sits near ceiling for GPT-6 (#359: 0.992–0.995 for every tier). So this part is a cost and wire test with a non-inferiority bound, not a quality ranking. + 2. **Quality where it matters.** The FW lanes are the S3 memory systems' LLM roles: extraction, retain/reflect and consolidation. That comparison runs as S3 r6 development runs on S3's separate development question split, never the confirmatory set, and S3 r6 freezes each role's effort from it. +- **Arms:** A = `cx/gpt-6-astra` (no `reasoning_effort`, so medium); B = `cx/gpt-6-astra-max`. Prompts, schema and order are identical. The two arms are interleaved per filing, with the order alternating. +- **Metrics:** + - micro-F1; + - strict-schema validity; + - reasoning and output tokens from usage; + - latency p50 and p95; + - timeouts and 429s; + - cache-read share (call_logs and `/api/usage/cache-health`, read-only). +- **Wire check** (must pass before scoring): a `-max` call logs `reasoning_effort_upstream = max`, and an effort-less call logs `medium`. These columns are filled only on rows with encrypted reasoning. +- **Decision:** + - B is adopted for a role only if its quality gain is significant: on the S3 development runs for memory roles, or on li26 by a paired-bootstrap lower bound above 0. + - Otherwise, where A is non-inferior (lower bound ≥ −0.02) and cheaper, the role keeps medium or moves to `cx/gpt-6-sol` medium (#359's mechanical tier). + +## L02: Codex context, compaction and tool-output limits (the omniroute profile, cx/gpt-6-*) + +- **Question:** does raising `model_context_window` and `model_auto_compact_token_limit` preserve long-task quality and the prompt cache better than the default? +- **Arms:** + - control: 272000 / 244800 / 10000; + - candidate: 872000 / 784800 / 10000; + - both set with `-c` per run, and `model_max_output_tokens` omitted. +- **Models:** Astra/max and Sol/medium, evaluated independently. +- **Tasks:** matched multi-hour Codex tasks that cross at least one control compaction boundary. Two sources: + - replays of this repository's real GPT-6 review tasks, meaning PR reviews with frozen heads and frozen expected findings (the findings GPT-6 already confirmed on #404, #410 and #411); + - one long synthetic task with planted early decisions that must be recalled after compaction. +- **Metrics:** + - correctness: the recall of expected findings, and of each planted decision; + - exact evidence recovery; + - tool-result truncation events; + - compaction count and trigger; + - cached versus uncached input and total tokens (from Codex `turn.completed` usage); + - latency; + - resolved model metadata. +- **Decision:** the candidate is adopted per model only if correctness is non-inferior AND total billed tokens fall. +- **Known divergence:** both arms sit at Codex's 90% compaction clamp (244800/272000 and 784800/872000; GPT-6's catalog has a 272000 default and an 872000 ceiling, per RH6/L02). OmniRoute's own guide (docs/guides/CODEX-CLI-CONFIGURATION.md:146) recommends 85-88%, "never above 90%". An optional third arm at 88% (767360) tests that advice; otherwise record the divergence. +- **What it measures:** behind the gateway, compaction runs locally, with no remote compaction V2 for a non-OpenAI provider (RH6/L03). So L02 measures local summarization. +- **Cost warning:** this is the costliest A/B, measured in hours per arm. Run it after R02 and D03/D04. + +## D03/D04: prompt-cache affinity and singleton combos + +- **Question:** can a lane move to a singleton combo (for example `lane-judgment-astra-max`) without losing prompt-cache continuity or changing the resolved model and effort? +- **When:** only before a lane adopts a combo. +- **Arms:** + - (a) the direct model name; + - (b) a single-model priority combo; + - (c) the combo with `promptCacheAffinityEnabled` toggled. This arm is the owner's to apply and restore. +- **Tasks:** the same frozen R02 li26 subset, sent as matched multi-turn conversations. There are 10 conversations of 6 turns, and each turn reuses the growing prefix, so the cache has something to hit. Each conversation gets its own session key, so 10 keys per arm. +- **The singleton combo** needs canonical `codex/` targets (RH2/D04 correction), universal handoff off (D05) and an explicit max tier (G5/D03 correction). +- **Metrics:** + - the resolved upstream model and effort per call (call_logs); + - cache-read share per turn; + - account spread across the 4 accounts; + - latency; + - quality: li26 micro-F1, as a non-regression check. +- **Decision:** a lane may adopt its singleton combo if the resolved model and effort are identical, the cache-read share is non-inferior (per-turn paired, lower bound ≥ −5 points), and quality is non-inferior. + +## Evidence and receipts + +- Each run records its arm identity tags (`otel.environment` or request headers), the call_logs row counts per arm, and the frozen hashes. +- Receipts go through `scripts/host_receipts.py record` at a published main commit. +- Results keep the li26 cost test, the S3 development-run quality, and vendor claims as separate evidence classes. diff --git a/blueprints/memory-layer-s3/inputs/r6/memory-merit-synth.md b/blueprints/memory-layer-s3/inputs/r6/memory-merit-synth.md new file mode 100644 index 000000000..eec1cd1b5 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r6/memory-merit-synth.md @@ -0,0 +1,175 @@ +Source: /memory-merit-synth.md; capture date: 2026-09-27. +## Is Hindsight the best memory system for this stack? + +**Provenance.** This draws on two Claude research lanes and one refuter pass, all dated 2026-09-27. Every claim was confirmed by the refuter against its cited source, except those marked *(not verified)*. +- The GPT-6 cross-family research job did not run. Its lane refused the network step and returned only a read-only readback of the repository catalogs, so this answer has no cross-family research vote. +- Both research inputs arrived truncated. Nothing from the missing parts is used. +- "S3" means `blueprints/memory-layer-s3/PREREGISTRATION.md`, DRAFT r5 (not frozen), on `origin/claude/memory-layer-s3-prereg-20260927` @ 75e30ea4. I read it directly with `git show`. + +### 1. Direct answer + +**The evidence does not show Hindsight is the best, and it does not show Hindsight is below its peers.** The refuter ran 18 dominance checks, in both directions, and every one returned `holds=false`. Hindsight has earned a place in S3; it has not earned a verdict. + +On your second point: no system in this set is clearly worse than its peers. Two should get no further effort, but because they don't fit this stack, not because they lose on merit: Mem0 OSS and Attemory (see §3). + +**The first caveat: none of Hindsight's published numbers is for the pinned v0.10.1.** +- The paper ran v0.1.0 (vendor blog `2026-03-23-agent-memory-benchmark.mdx:92` @ v0.10.1). +- The vendor-harness (AMB) 94.6% ran 0.4.x. Its `uv.lock` pins hindsight-api 0.4.15 and hindsight-all/embed 0.4.17; the blog says 0.4.19. +- The independent full rerun ran 0.4.17. +- The only same-judge comparison (OAMB) ran 0.9.2. + +**Dimension by dimension** + +1. **Published performance** + - For: + - It has the highest score in the only same-model, same-judge comparison of several systems. OAMB used 60 balanced LongMemEval-S questions with DeepSeek V4.1 Flash in every role: Hindsight 57/60, Mem0 OSS 52, OpenViking 51 (github.com/rocke2020/open-agent-memory-benchmark @03509051, 2026-09-13; third party, affiliation not verified). + - An independent full-500 rerun scored 448/500 (89.6%) on a DeepSeek stack with API 0.4.17, with about 2 pp run-to-run drift. The author calls it a provider variant, not a reproduction (rocke2020/agent-memory-benchmark, branch `deepseek-provider` @b6eac3db, 2026-08-22). + - Vendor paper, LongMemEval-S QA with a GPT-OSS-120B judge: 83.6% with GPT-OSS-20B, 89.0% with GPT-OSS-120B, 91.4% with Gemini-3 Pro answering. Full-context GPT-OSS-20B scores 39.0% (arXiv 2512.12818v1 Table 3, 2025-12-14; vendor plus co-authors). + - Vendor harness AMB: LongMemEval-S 94.6% (473/500), LoCoMo10 92.0% (1417/1540), BEAM-10M mean rubric score 0.6408 with 147/200 passing (github.com/vectorize-io/agent-memory-benchmark @03c1d0f1). + - Against: + - The OAMB lead is not significant: exact McNemar p = 0.125 against Mem0 and 0.070 against OpenViking. The report's own status is `no_clear_accuracy_leader`. + - No retrieval recall has been published for LongMemEval, LoCoMo or BEAM, so Hindsight cannot be placed on S3's metric at all. + - Two README v0.10.1 claims are refuted: "most accurate agent memory system ever tested" (line 44), and "independently reproduced" by Virginia Tech and The Washington Post (line 50), since both institutions are co-author affiliations. + - Paper Table 3 mixes judges: its Zep, Supermemory and full-context GPT-4o columns are copied from Supermemory's GPT-4o-judged report. + - The AMB 94.6% run used 43,624.5 context tokens per question. That is 1.88× the vendor's own hybrid-search control, which scored 74.0% with 23,221.7 tokens under the same answerer and judge. + - The 91.4% and 94.6% results both depend on hosted Gemini answerers. +2. **Local by default: against.** The default LLM provider is `openai`, falling back to gpt-4o-mini. Only the embedder (bge-small-en-v1.5), the reranker (ms-marco-MiniLM-L-6-v2) and the pg0 database are local (`hindsight_api/config.py` @ v0.10.1, lines 1082, 1084, 1119, 1204-1205, 1264-1266). agentmemory (in keyless mode) and MemPalace are local by default. +3. **Both-client integration: tie.** Hindsight has official hooks plus MCP for both Claude Code and Codex, as agentmemory and MemPalace do. That puts it ahead of Mem0, whose plugins need a hosted key. The refuter did not re-read Hindsight's installer or hook files. +4. **Write cost and serving weight: against by design, but not the worst measured.** + - Every retain runs LLM extraction, plus Postgres and a worker. agentmemory and MemPalace write with zero LLM calls. + - In OAMB, however, Hindsight indexed fastest of the three: median 761 s from indexing to ready per history, against 842 s for Mem0 OSS and 1,190 s for OpenViking. It used 58.44M indexing tokens for 60 histories (only partly counted). + - No full LongMemEval-S ingest time exists (see misquote c). The catalog's ~86 h is an unverified projection. +5. **Maintenance: middle of the pack.** Releases since 2026-07-29: Hindsight 6 core releases; agentmemory 1, MemPalace 5, Honcho 1, Mem0 9 (Python), OpenViking 23, ai-memory 43, GBrain 109 (`gh api …/releases`, 2026-09-27). + - Open issue #4696 is not a default-configuration hang. The reporter retracted the 1-document observation (recall takes about 8 s there) and traced the problem to a qwen3-reranker-8b they had chosen, running on CPU (https://github.com/vectorize-io/hindsight/issues/4696). +6. **Lifecycle: for.** It has the richest set of lifecycle features here: supersession on re-retain, reversible invalidation, bank export and import in restore or merge mode, and cloning. None of these was exercised. + - The peers' gaps are confirmed. agentmemory has #1273 open (governance delete deletes nothing) and #1190 open (restore is additive). MemPalace has no verified restore, and its "contradiction detection" is exact-match deduplication (arXiv 2604.21284). + +**How S3 decides, and where that leaves Hindsight.** S3 chooses on full-track `recall_all@5` and workload cost. Candidates must first pass eligibility gates (the §4.4 lifecycle acceptance items and a 1.0 s warm-p95 latency limit) and a usefulness floor (§7). End-to-end QA, where all of Hindsight's published strength lies, is not the deciding lane. Its lifecycle strength counts at the §4.4 gate. + +**Why these numbers cannot be compared** +- Metric: QA accuracy (Hindsight, Mem0, Honcho, Zep, OAMB) is not session `recall_all@5` (S3's metric), and neither is `recall_any@5` (the agentmemory and MemPalace vendor figures). `recall_any@5` counts a hit when any one gold session is in the top 5, so it is easier. +- Judge and answerer: + - GPT-OSS-120B in the paper; + - gemini-2.5-flash-lite judging gemini-3.1-pro-preview in AMB; + - GPT-5 for Mem0 OSS; + - GPT-4o for Honcho, Zep, Supermemory, Mastra, GBrain's QA, and the baselines copied into Hindsight's Table 3; + - DeepSeek for OAMB and the rocke2020 rerun. +- Question set: the paper, AMB and Mem0 use the original 500 questions, including 30 abstention questions. S3 uses the cleaned 470-question full track. +- Context budget: AMB Hindsight used 43.6K tokens per question against 23.2K for its control. Mem0 Platform used the top 200 memories. In OAMB, Hindsight used 15.91k tokens and Mem0 6.54k. +- Aggregation and hosted parts: Mastra's 94.87% is an unweighted mean of category scores (468/500 = 93.6% per question). GBrain's 95.53% used hosted OpenAI embeddings and a Voyage reranker. + +**The only comparisons that hold up** +- Our own harness (Mac, descriptive only): + - agentmemory with local MiniLM scored 0.821 `recall_all@5` against the historical ai-memory C3 at 0.570: +0.251, 95% CI [+0.203, +0.299], Holm p 0.0001. + - agentmemory's keyless default scored 0.617 (+0.047, p 0.52). + - Plain BM25 (0.747) and dense Qwen3 (0.753) also beat C3 in the same harness (`~/code/native-agent-stack/evidence/artifacts/memory-stack-20260925/experiment.json`, observations 16-20). + - Ingestion went through REST, not the shipped hooks. C3 ran on a 2.5-pre build, not S3's v2.4.0 reference (S3 line 57). +- OAMB, as above: same model and same judge, no clear leader. Mem0 ran with OAMB's own temporal extraction prompt. +- Roughly comparable: MemPalace 96.6% against agentmemory 95.2% vendor `recall_any@5`. Both use the same dataset file, metric and MiniLM embedder, but how MemPalace built its corpus is unclear. +- Roughly comparable: Zep 71.2% against Mastra OM 84.23%, EmergenceMem 82.4% and Supermemory 81.6%, all with a gpt-4o answerer and judge (arXiv 2501.13956v1). These were run by different vendors about a year apart. + +### 2. Systems that remain competitive + +| System | Retrieval evidence (metric) | Local by default | Both clients | LLM calls on write | Serving weight | Releases since 07-29; confirmed open bugs | Lifecycle | +|---|---|---|---|---|---|---|---| +| Hindsight v0.10.1 | None; QA only. OAMB 57/60 (on v0.9.2) | No (`openai`) | Hooks + MCP (tie) | Extraction on every retain; OAMB 761 s per history | API + worker + pg0 Postgres + local embedder and reranker + LLM | 6; #4696 (seen with a non-default 8B reranker) | Richest; not exercised | +| agentmemory v0.9.29 | `recall_all@5` 0.821, our measurement, 470-question full track (local MiniLM, REST path); keyless default 0.617 | Yes, keyless; 0.821 needs `EMBEDDING_PROVIDER=local` | Hooks + MCP (tie); shipped-hook equivalence unproven | None | iii engine + server, no Postgres | 1; #1389 (iii engine OOM) | Delete (#1273) and restore (#1190) broken, both open | +| MemPalace v3.10.0 | Vendor `recall_any@5`: 96.6% raw, 98.4% hybrid v4 (450 held-out); palace modes 84.2% / 89.4% R@5 (maintainer-reported); our `recall_all@5` re-score 0.857 / 0.887 *(not verified)* | Yes | Hooks + MCP (tie) | None | Lighter than Hindsight (ChromaDB) | 5; P0 #1581 (HNSW corruption) | No verified restore; contradiction detection is exact-match dedup | +| OpenViking v0.4.21 | None; OAMB 51/60 (on v0.4.19) | Uncited | Hooks + MCP *(peer lane, not verified)* | VLM/LLM extraction; OAMB 1,190 s per history, 58.22M tokens for 60 histories | OAMB median retrieval 0.79 s (Hindsight 0.17 s) | 23 | Uncited | +| GBrain v0.59.0.0 | Vendor `recall_all@5` 95.53% (449/470) on S3's exact metric and split, using hosted embeddings and reranker; cannot be ranked against 0.821 | Not shown; local path unmeasured | Plugins for both *(peer lane; Codex hooks not verified)* | *not verified* | *not verified* | 109 | *not verified* | + +- **Mem0 OSS v2.2.1** is competitive on QA: 52/60 in OAMB, and 91.0% vendor-reported under a GPT-5 judge. But it defaults to OpenAI, and its official Claude Code and Codex plugins (hooks plus MCP) require a hosted Mem0 Platform key (`docs/integrations/codex.mdx` and `claude-code.mdx` @ v2.2.1). It therefore fails this stack's local, both-client requirement. The 73.8% re-run tested the hosted Platform, not OSS, and was run by a competitor (Maximem, 2026-05-27). +- **Reference, ai-memory:** 43 releases since 07-29. S3's reference is the official v2.4.0 C4′ configuration, which has never run (`~/code/native-agent-stack/docs/decisions/2026-09-27-mac-single-writer-staged.md:48`). + +### 3. Clearly dominated systems: none + +All 18 dominance checks returned `holds=false`, so nothing is dropped on merit. The catalog's rejects are not merit findings either: +- Honcho, OpenViking and Basic Memory were rejected on AGPL grounds (license is excluded under your rule), and all three share one copied rationale (`~/code/native-agent-stack/catalogs/foundation/memory-stack-20260925.json` lines 401, 438 and 475). +- The Letta reject names no peer that beats it. + +**Insufficient evidence, not dropped** +- cognee (S3 primary): no confirmed figure; the peer lane's entry was truncated. +- Basic Memory (S3 primary): internal vendor figures only (recall@5 0.951 on a 60-question subset; LoCoMo recall@5 0.745, then 0.823 after a fix). +- memsearch (S3 primary) and engram (reserve 1): no published benchmark. +- Honcho (reserve 5): vendor QA only. LongMemEval-S 90.4% under a GPT-4o judge, where Gemini 3 Pro with full context alone scores 92.0% (plasticlabs.ai, 2025-12-19). +- Graphiti/Zep (reserve 2): its 71.2% trails peers using a gpt-4o answerer, but those peers' other dimensions are uncited. It is a poor fit, not a dominated system. +- Supermemory (reserve 3): cross-judge figures only; its page's gpt-4o QA row repeats its Recall@20 value (97%). +- MCP Memory Service (reserve 7): R@5 of 86.0%, without saying whether that counts any or all gold sessions. +- Letta server: retired. + +**Ineligible for this stack, not dominated** +- Attemory: vendor `recall_all@5` of 0.9638, the highest in the set. It has no capture hooks, no Codex plugin, and takes about 21 s per query (vendor figure), against S3's 1.0 s warm-p95 gate (line 254). +- Mem0 OSS: its plugins need a hosted key. + +**Not assessed in this wave:** MemMachine, SimpleMem, claude-mem, MemOS, byterover-cli, deja-vu, total-agent-memory. + +### 4. S3 arm set and screening order (the S3 owner decides) + +S3's frozen protocol picks each host's winner, not any number above. S3 treats vendor numbers as "motivation only" (line 286) and never as S3 evidence (line 92). S3 is still DRAFT r5 and freezes only after a GPT-6 `accept` on the whole bundle (lines 3 and 304), so arm changes can still go into r6. + +**What the evidence supports for the arm set** +1. Keep all seven primary candidates (agentmemory, Hindsight, OpenViking, cognee, MemPalace, basic-memory, memsearch) and the v2.4.0 C4′ reference. The evidence gives no reason to drop any of them. +2. Don't add Mem0 OSS or Attemory. Both fail S3's admission rule, which requires an official local install and official integration with both clients (line 115). That is a fit problem, not a merit one. +3. Screen GBrain for admission. + - It is the only system that publishes S3's exact metric on S3's exact split. + - The catalog's reason for setting it aside, "Chunks are not sessions" (`memory-stack-20260925.json:280`), is contradicted by GBrain's CHANGELOG [0.48.4.0], lines 2042-2045, which define `recall_all@5` at session level. + - Its figure depends on hosted parts and its local path is unmeasured, so it must pass admission before it can be run. That can happen in r6 or through a late-candidate amendment (lines 94-98). +4. Add plain BM25, and possibly dense Qwen3, as diagnostics outside the decision family. In the historical harness both beat C3 and agentmemory's keyless default. Running them shows whether a candidate adds anything over a plain retriever. +5. For the Hindsight arm: + - Freeze how its recalled facts map back to source sessions. Output that cannot be mapped scores retrieval `N/A` (line 136), and no run has ever reported session recall for Hindsight. + - Pin both the core (0.10.1) and the capture package, `integrations/coding-agents/v0.7.0` @ 0c0869b7. The Mac sweep notes that `releases/latest` does not show that package. + - The Mac arm stays `pending`: pg0-embedded is linked against Homebrew OpenSSL (line 227). That comes from the Mac session. + +**Screening order.** S3 §5 already orders acceptance testing by the primary-lane point estimate (line 218). For the order of execution, the evidence supports: +1. The §6 reference preflight: C4′ on 5 frozen development questions (line 231). +2. The zero-LLM arms: + - agentmemory with `EMBEDDING_PROVIDER=local`, plus the shipped-hook equivalence proof S3 requires (lines 81-82); + - MemPalace, after freezing which mode its shipped hooks actually use, since the published figures are for raw and hybrid modes. + - Probe their §4.4 deletion, restore and durability items early, either as development checks (line 164) or through an r6 order change. agentmemory's #1273, #1190 and #1389, and MemPalace's P0 #1581, hit those items directly and could disqualify an arm whatever its recall. +3. basic-memory and memsearch. There is no comparable evidence for either, and their write cost is not verified. +4. The LLM-extraction arms in the drained-GPU window (line 221): Hindsight on the workstation, OpenViking and cognee. + - They use the frozen Qwen3-8B Q4_K_M as their extraction model (lines 163 and 211). + - Measure each one's local ingest rate on development questions first, because no full LongMemEval-S ingest time exists for any of them. +5. Reserves in the frozen order (lines 106-117). + - Graphiti and Supermemory will probably fail admission, because the lanes report no official plugins for both clients *(not verified)*. + - Honcho's status is disputed: the peer lane reports first-party plugins for both clients, while the Mac sweep says "SDK-based". The admission screen will settle it. + +**Check before the freeze (lane claims the refuter did not check)** +- The peer lane reports that Hindsight needs an LLM supporting at least 65,000 output tokens (`models.mdx:156`). Check this against the frozen Qwen3-8B Q4_K_M. +- The Hindsight lane reports that pg0 0.15.2 starts Postgres with fsync off. That matters for the §4.4 durability items. +- The Hindsight lane reports three settings that could break a frozen arm: coding-agents `autoUpdate` defaults to true, an unset `embedVersion` runs the latest hindsight-embed, and the installer pre-selects Hindsight Cloud. +- agentmemory reportedly switches to a hosted embedder when provider keys are in the environment (`src/config.ts:265-278`). S3's `env -i` rule (line 151) already covers this. +- Other open issues the lanes cite: Hindsight #4529 (pg0 index corruption blocks retain; the reporter suspects hardware), plus #4702, #4687 and #4493 on the Codex path; agentmemory #1321; MemPalace #1924, #2326 and #2403. + +### Misquotes the refuter found (7) +a. **The date of AMB's 94.6%.** The catalog calls it "undated" (`memory-stack-20260925.json:201`, relayed by the GPT-6 compile), and the peer lane gave "repo commit 2026-09-22". In fact it was committed and announced on **2026-03-23** (commits decbb07f and 7021c59c). 2026-09-22 is the date of the AMB HEAD commit. +b. **The version and LLM behind the 94.6%.** The Hindsight lane said they were "not recorded". AMB's `catalog.json` names gemini-2.5-flash-lite as the extraction model, and the lock file pins 0.4.15 and 0.4.17 (the blog says 0.4.19). So the run was on 0.4.x, not v0.10.1. +c. **The 8.4 h ingest.** Both Claude lanes said 30,090,034 ms (8.4 h) for 11,303 documents. 11,303 is only the count for questions 264-500, out of 23,867 document occurrences, and a resumed run does not restore its time or counts. The 8.4 h covers roughly the resumed half, and the full ingest time is not recorded. +d. **BEAM 100K.** The peer lane gave 75%. The correct figure is **73.4%** (mean score 0.7337; 81.5% pass). The vendor's landing page shows 75%, which contradicts the vendor's own data file. +e. **AMB `mean_recall`.** The Hindsight lane said it is null for every Hindsight run. It is null for every LongMemEval, LoCoMo, BEAM, LifeBench and PersonaMem row, but the hindsight-cloud PrecisionMemBench row shows recall 1.0 and precision 0.8481 over 77 queries. The underlying point, that no LongMemEval or LoCoMo recall exists, still stands. +f. **Supermemory's judge.** The peer lane said the page does not state one. It does: "gpt-4o judges the answers". +g. **Honcho, OpenViking and Basic Memory.** The GPT-6 compile said their numbers are on LoCoMo, a different benchmark from Hindsight's. Honcho's 90.4% is LongMemEval-S, and Hindsight publishes LoCoMo results too. The real mismatch is the judge and the model. + +Related corrections: +- The GBrain catalog row (line 280), covered in §4. +- The Mac sweep calls Attemory's figure "undated"; it is dated by its commit, 2026-08-13. + +### Refuted claims (8) +- README v0.10.1: "independently reproduced" (line 50). +- README v0.10.1: "most accurate … ever tested" (line 44). +- Hindsight lane: "no independent reproduction exists". +- Hindsight lane: #4696 "hangs forever, even on a 1-document bank". +- Peer lane: "Mastra's 94.87% and Mem0 Platform's 94.4% are higher". Mastra's figure is 93.6% per question, Mem0's 472/500 is below Hindsight's 473/500, and the numbers span four judge setups. +- Peer lane: "Mem0 has no official Codex capture". It has one, but it needs a hosted key. +- Hindsight lane: "the dominance rule cannot be met for Mem0". OAMB is a same-judge comparison, and Mem0 is still not dominated. +- Peer lane: "Hindsight is frontier on QA". Its lead is unproven, not disproven. + +### Unverifiable, not used +- The Gemini 3 Pro full-context 92.0% as a ceiling for Hindsight (it used a different judge). +- The MemPalace `recall_all@5` re-score. +- Basic Memory LoCoMo R@5 76.4% (absent from the vendor's summaries). +- The ~86 h Hindsight projection. +- Hindsight's code-level claims: installer behaviour, fact extraction, http.py routes, pg0 fsync, Codex hook status and the AMB recall budget. +- Cognee BEAM 100K 0.79. + +Token tools used: `ctx_execute` once, to copy S3's PREREGISTRATION.md into the session scratchpad and list its section headers. Everything else was focused Read calls and rg/git in Bash. No compressors were stacked, and no token savings are claimed. diff --git a/blueprints/memory-layer-s3/inputs/r6/r6-claude-review.md b/blueprints/memory-layer-s3/inputs/r6/r6-claude-review.md new file mode 100644 index 000000000..68abc9e29 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r6/r6-claude-review.md @@ -0,0 +1,125 @@ +Source: /s3-r6/repair/claude-review.md; capture date: 2026-09-27. +I found four high-severity defects: two need new text (the self-written harness and the GPT-6 pacing), and two are pre-freeze blockers (stale sources and the latency gate). r6 should not freeze as written. + +- **G (full 470 questions):** honest but not feasible. The arithmetic is exact (54.05M, 270.25M, 2.162B and 4.324B, recomputed in code and labelled as scenarios). But r6 gives no wall-clock projection and adds a pacing limit that no source supports. +- **F and H (research-memory tasks and lifecycle probes):** genuinely sourced. Every RM01–RM08 number sits at its cited README line, and every decision-index anchor falls inside the right record: edgartools (6330, 6352), LEAN (17233, 17248), alpaca-py (1592) and skfolio (19176). My first attribution pass suggested otherwise, but that came from a bad lookup method, not the file. + +`P:n` below means line n of `/wt-s3/blueprints/memory-layer-s3/PREREGISTRATION.md`. + +## High + +**H1. Source keys I and R are stale, and R changed too (§14, P:444–445).** +- **Defect:** + - s3-r6-inputs.md now has sha256 `30eb565f…` and 58 lines; r6 records `b0a99fe4…`, lines 1–38. + - gateway-ab-designs.md now has sha256 `7deb3208…` and 101 lines; r6 records `27e45209…`, lines 1–84. You flagged only the inputs file; this second change is new. + - The captured versions no longer exist at those paths, so §9 step 2 ("every referenced file exists") fails. + - R gained a block, "Harness: upstream, not self-written" (R:14–28), which r6 never absorbs. + - Old R:14–36 appears to have moved to R:30–52 (inferred from content; please verify). So P:67's "R:14–36, especially 24–36" now points at the statistics block. + - I's Hindsight lines moved from 36–38 to 55–57; r6 cites them at P:149, P:268 and P:460. + - The M, J and G hashes still match. +- **Replacement:** "I: s3-r6-inputs.md:1–58, sha256 `30eb565fd95d78570ae7e499ebc540a415a79cbba0fb75d1b40796fd17289555`. It supersedes the overwritten `b0a99fe4…` capture. Live deployments are I:36–53; Hindsight is I:55–57. R: gateway-ab-designs.md:1–101, sha256 `7deb3208e7717005e6b9d2d40cfde72d4f16d0e1cfc24827684f8db718e2ea09`. Arm rules are R:6–10, the upstream-harness rule R:14–28, and R02 R:30–52." Then remap every I: and R: locator. + +**H2. The runner is self-written (§3 P:181; §9 P:375).** +- **Defect:** "S3's runner is **new code**" contradicts the user's direction ("…rather than self written test"). It also contradicts W3: "Planned S3 runner… replaced"; "should not be filled by inventing another harness" (report.md:9, 52–70, 128). +- **Source:** AMB's registry at `03c1d0f1` ships cognee, Hindsight (including an HTTP provider) and bm25 providers. AMB's `cli.py` exits without GEMINI_API_KEY or GOOGLE_API_KEY (`_resolve_gemini_key`), and loads `.env` with `override=True`. +- **Replacement:** + + > "S3 writes no standalone runner. Orchestration is AMB `03c1d0f1d27da63034f0931121c858faba512383`, unmodified. Its cognee and Hindsight HTTP providers target the isolated containers of §2.1. The other five primaries and the reference are added only as frozen, hashed AMB MemoryProvider modules. AMB's answer assertions and retrieval-mode scores are not S3 results. Providers export ranked opaque session IDs, scored by unchanged LongMemEval `eval_utils.py` @9e0b455f. AMB's loader must consume the pinned file (sha256 `d6f21ea9…`); the official user-text track is a provider ingest parameter. Recorded upstream defects: (1) the Gemini-key exit — use AMB's Python runner, or a non-secret placeholder with Google egress blocked plus wire proof that no Gemini call occurs; (2) no `.env` in the checkout. AMB is maintained by Hindsight's vendor (vectorize-io), so a pre-freeze parity audit shows every provider uses its native ingest and recall at equal depth. The §3 isolation rules bind every process AMB drives." + +**H3. The G pacing is unsourced and the run may not be feasible (§4.2 P:237–239).** +- **Defect:** + - "One in-flight GPT-6 request per host, two across S3, one per second" cites R:26–32. Those lines are R02's arms and metrics and say nothing about pacing. + - It contradicts the user's recorded 2026-09-27 decision: "gpt6 can run with full parallel utilization… run them parallel" and "do not adopt… throttles" (memory `provider-limits-and-logins.md:17–25`; historical evidence, not authority). + - The harness cannot enforce it on concurrency inside the systems themselves (Hindsight `*_LLM_MAX_CONCURRENT`; ai-memory's reranker caps at four). + - There is no throughput or wall-clock projection. +- **Illustration only (one 7-call canary at ~1K tokens per call; larger chunks change it):** + - I:34's canary gives about 145 input tokens/s at one call in flight. + - That is about 104 h per ingest pass and about 22 days for five passes, per GPT-6-ingesting system per host. + - Combined with "no cost-driven subset… never shorten the run" (P:306, P:239), this invites post-hoc truncation. +- **Replacement:** + + > "Concurrency is a frozen throughput parameter, not a quota gate. Use K in-flight GPT-6 calls per host, with K chosen on the development split from measured throughput and fresh 429s. Set it via each system's supported knobs, record it as a departure, and verify it from call_logs. Use a block-randomized dispatch order, identical across arms. Before the freeze, state measured development tokens/s and the projected wall-clock per host. If the projection exceeds [user-set window], adopt G's stratified subset with its prospective power calculation, decided before any result. Seeds replicate only stochastic components; an arm that is deterministic by source review is ingested once and counted once." + +**H4. GPT-6 in the query path versus unchanged latency gates (P:42, P:239, §4.3, P:307, P:331, P:350).** +- **Defect:** + - r6 moves C4′'s reranker, and any query-rewriting or LLM-reranking role, to gateway GPT-6. + - It does not restate the consequence under unchanged gates: warm p95 ≤ 1.0 s for every configuration including the reference, the 10 s timeout, and >5% rerank fallbacks making the host unresolved. + - ai-memory v2.4.0 documents the reranker as "an LLM call on the search hot path… On any error or timeout… preserves the fused order" and "saturated queries keep their local ranking without waiting." + - "Measure client-observed queueing" (P:239) lets other arms' calls holding the host slot leak into an arm's latency. + - The exact GPT-6 rerank latency is unmeasured (my inference), but the incoherence stands regardless of the number. +- **Replacement:** + + > "§4.3: query-path GPT-6 calls are inside the gated warm p95. The 1.0 s bar and 10 s timeout are unchanged, and a native query path requiring GPT-6 may be latency-ineligible; this is declared before execution. The latency lane runs with no other arm's call in flight on that host, and queue wait is reported separately. §6 preflight (a justified change to r5's 'settles only whether C4′ can run'): because r6 replaces C4′'s local LLM, the preflight also measures C4′'s fallback rate and warm p95 on the development split under the frozen envelope. If fallbacks exceed 5%, the reference is C3′ before any confirmatory run." + +## Medium + +**M1. Bind the statistics to scipy/statsmodels (§6; §7 P:339).** +- **Source:** W5 (report.md:82–96); R:24–28. In scipy v1.18.1, `n_resamples` defaults to 9999, and the confidence interval goes through `stats.quantile` (`_resampling.py:663`). +- **Replacement:** + + > "Estimand unchanged, using scipy==1.18.1 and statsmodels==0.15.0. Use per-cluster arrays: s = the sum of seed-averaged question Δ, and n = the question count (452 clusters). + > `res = scipy.stats.bootstrap((s, n), lambda s, n, axis=-1: s.sum(axis)/n.sum(axis), paired=True, vectorized=True, n_resamples=10000, method='percentile', confidence_level=0.95, alternative='greater', rng=numpy.random.Generator(numpy.random.PCG64(20260927)))`. + > Bound = `numpy.percentile(res.bootstrap_distribution, 5, method='linear')`. p_sup = share ≤ 0; p_NI = share ≤ −0.02. + > Holm = `multipletests(p, alpha=0.05, method='holm')` over all 14 planned hypotheses, with p = 1.0 for missing arms so m stays 14. + > A cluster sign-flip `permutation_test(permutation_type='samples')` for superiority is reported but decides nothing." + + In §7, delete "equivalent to the adjusted one-sided lower bound", which is undefined. + +**M2. The call envelope is over-strict in one place and unimplementable in another (§1.2 P:73–74; P:152).** +- **Temperature rule:** + - The pinned `shouldDeduplicate` (requestDedup.ts @a58000c7) returns false when `stream === true`. It treats an absent temperature as 1.0 and deduplicates only at ≤ `maxTemperatureForDedup`. + - So an explicit 1.0 is gated exactly like omission, as I:30–31 already say. This corrects the brief's rule A as well as r6's "Temperature 1.0 alone does not replace this rule." + - In-flight dedup also needs concurrent identical requests, which r6's own pacing rules out. As written, the rule can push cognee to `pending`. +- **Static headers:** cognee's `LLM_ARGS` and Hindsight's `HINDSIGHT_API_LLM_DEFAULT_HEADERS` (config.py:176) are process-static. + - Nothing implements r6's "resolved at conversation/call scope". + - cognee-live's 13 calls on one account (I:44) confirm static headers pin affinity, consistent with R:7. +- **Undefined unit:** "Conversation" is not defined for ingest. +- **Replacement:** + + > "Every call sends `X-OmniRoute-No-Cache: true` and is dedup-ineligible: `stream: true`, or a resolved temperature that is absent or > 0.1, verified at the wire. Idempotency-Key is fresh where per-call headers exist; a static key is not fresh. Before choosing between an absent and a constant key, read the pinned idempotency.ts key composition (chatCore.ts:763–765 shows messages feed the digest), and prove no replay in call_logs. A conversation is one question × seed × arm run. Route (a): one fresh arm process per run, carrying that run's key (about 18,800 starts per host). Route (b), Hindsight only: no static session header, and `HINDSIGHT_API_{OP}_LLM_CACHE_AFFINITY=openai_prompt_cache_key` (config.py:180, 403), pending a wire check of the value sent." + +**M3. The live deployments are not absorbed (§2.1, §3; I:36–53).** +- **Replacement:** + + > "Live services are hosting evidence only; they are never scored or touched. cognee-live runs from the pinned digest on rootless Docker 29.8.1 at 127.0.0.1:3800, auth on. hindsight-live is a systemd --user unit on 3710/5433 (pg0). Archive sanitized copies of both deploy receipts. Add 3800, 3710 and 5433 to the fail-closed port list. The path 10.0.2.2:20128 is the candidate container-to-host transport; re-verify it per arm. Live envelopes use static session headers, so they are not scored envelopes. hindsight-live's stale-`postmaster.pid` guard is a local lifecycle modification: a scored arm does not inherit it silently, and the unguarded first failure counts as H evidence. Use upstream knob names `HINDSIGHT_API_{RETAIN,REFLECT,CONSOLIDATION,MENTAL_MODEL_REFRESH}_LLM_{MODEL,REASONING_EFFORT}` (config.py:383, 401, 426, 440, 460), not the input's `{…}_REASONING_EFFORT`. Set `HINDSIGHT_API_LLM_TEMPERATURE=none` and leave `HINDSIGHT_API_LLM_TEMPERATURE_{RETAIN,REFLECT,CONSOLIDATION,VERIFICATION}` unset: per-operation values override the global (config.py:212), and the shipped defaults are 0.1/0.9/0.0/0.0 (config.py:242–245)." + + This replaces P:149's "temperatures are explicitly frozen". + +**M4. The R02 effort-selection rule is underspecified (P:67).** +- **Defect:** "Role quality", "significant" and "paired lower bound" have no stated metric, sample size, test or alpha. It does not say whether coupled roles are selected jointly, whether selection runs per host, or what happens to an unresolved role after the freeze. +- **Replacement:** + + > "Before the first development call, freeze, per role or coupled role set: LoCoMo full-system supporting-evidence recall@5 plus schema validity; n; a one-sided cluster bootstrap at 0.05 (the M1 binding); joint selection for coupled roles; and runs on each host. An arm with an unresolved role is pending, and needs a dated amendment before its first confirmatory run." + +**M5. Other self-written evaluation pieces (4.5(b) P:272; 4.1 P:200).** +- **Source:** LongMemEval `evaluate_qa.py:24` `get_anscheck_prompt`, which includes the abstention case. Its call path runs at temperature 0 (L108), which is cache- and dedup-eligible at the gateway. W4: MTEB 2.21.8 registers LoCoMo and documents two-stage reranking (report.md:72–80). +- **Replacement:** + + > "The judge prompt is `get_anscheck_prompt`, unchanged, sent to Astra max under §1.2, not through evaluate_qa.py's call path. The embedder and reranker check runs MTEB 2.21.8's LoCoMo task, with the metric frozen first." + +## Low + +- **L1. User approval dropped from the GPU window (§5 P:295).** r5 said "a scheduled production stop and restore that the user approves"; r6 dropped "that the user approves". Restore it. +- **L2. Coordinator text attributed to the user (§14 P:449; §5 P:301–302).** Requirements A–I are the coordinator's brief; only directions 1–4 are the user's words. Split U into U (directions 1–4) and a separate key for the brief, which relays the Apple Container 1.4.1 / Postgres 18.6 / pgvector 0.8.6 and 12/12 claims. Require the Mac's own receipts in the bundle before freeze. +- **L3. agentmemory's MiniLM exception is mis-framed (P:93, P:199).** + - **Defect:** r6 frames it as "no alternative" at provider level, but §4.1 says "wherever upstream supports configuration". + - **Replacement:** "MiniLM stays primary by user requirement C — a user-directed exception to E, although agentmemory supports an OpenAI-compatible embedder." +- **L4. Achieved power is not stated (P:306).** + - **Illustration** (normal approximation, n=470, 10–30% discordant questions, seeds ignored): an exactly equivalent arm clears the −0.02 non-inferiority bar with probability 0.20–0.39 at α 0.05, and 0.03–0.09 at Holm's first step (0.05/14). + - **Replacement:** "Report achieved power for both routes from the frozen analysis code over a stated discordance range; this is information, not a gate or a margin change." + +## What is correct +- Requirements A–I each landed: status line, GPT-6 role contract, `-max`, compression off, effort columns (§1.1–1.2), Astra-max answerer and judge with the manual audit kept, per-arm isolation, seven primaries and C4′ kept, Mem0 and Attemory ineligible on fit, the GBrain screen, BM25 and dense Qwen3 diagnostics, and the Mac "runnable" note. +- F's numbers match the cited READMEs: data-readiness 12, 15, 17–18, 38–40; simulation-research 22–23, 41–42; identity-readiness 50, 79, 90, 97, 114–120; catalyst-provenance 3–6, 11, 69–70. North-star 33–35, 57–60 and 74–79 support the justification. +- The early lifecycle checks are backed by the merit synthesis (M:44, 127, 137–138). At the Hindsight pin, models.mdx says "at least 65,000 output tokens" and requires `RETAIN_MAX_COMPLETION_TOKENS` > `RETAIN_CHUNK_SIZE` (3000). +- The statistics are preserved: 452/401 clusters, PCG64 seed 20260927, 10,000 resamples, a linear 5th-percentile bound, Holm at α 0.05 over 14 hypotheses, +0.05/−0.02 thresholds, and the minus-one usefulness floor. r5's accepted fixes (§10, §11, §13) are all present. +- The development split is LoCoMo with overlap checks, the answerer and judge are fixed rather than selected, and the two hosts decide separately. +- Spot-checked at their pins and true: Hindsight's per-operation env names and its `none` temperature setting, cognee's "Keys given directly in LLM_ARGS win", and ai-memory's reranker behaviour. + +## Verification gaps +- I did not parse the workflow journal (J). Claims resting only on J are not re-checked: agentmemory's key precedence, the image digests, the GBrain changelog lines and the Mem0/Attemory lines. +- I did not spot-check the embed.md and rerank.md locators. +- I did not read OmniRoute's `idempotency.ts`. +- GPT-6 rerank latency and throughput are unmeasured. + +TOKEN TOOLS USED: ctx_batch_execute/ctx_execute: hashing inputs, the r5→r6 diff, decision-index and README line checks, pinned upstream greps via curl, arithmetic; Read: artifact, brief, inputs, report, memory file; Grep: ai-memory repository URL. \ No newline at end of file diff --git a/blueprints/memory-layer-s3/inputs/r6/s3-r6-inputs.md b/blueprints/memory-layer-s3/inputs/r6/s3-r6-inputs.md new file mode 100644 index 000000000..ff749aedc --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r6/s3-r6-inputs.md @@ -0,0 +1,58 @@ +Source: /s3-r6-inputs.md; capture date: 2026-09-27. +# S3 r6 inputs collected 2026-09-27 (for the r6 revision of blueprints/memory-layer-s3/PREREGISTRATION.md) + +## Gateway requirements for every GPT-6 FW role (token-save-manifest-html's verified gateway verdict) +- **Effort:** + - chat/completions without `reasoning_effort` runs at MEDIUM; + - `cx/gpt-6-astra-max` pins max (codex.ts L1451-1456; the suffix wins over the body); + - per-role effort is frozen from the R02 development runs (see capability-gate/gateway-ab-designs.md). +- **Cache and dedup:** + - the legacy exact semantic cache is armed (temperature 0 is eligible); + - in-flight dedup covers temperature ≤0.1 non-stream requests; + - so scored calls send `X-OmniRoute-No-Cache: true` plus `stream: true` (or omit temperature), with a fresh Idempotency-Key per call, sample and arm. +- **Affinity:** a stable `x-omniroute-session` per arm or conversation. Without it, calls scatter across the 4 accounts and lose the prompt cache. +- **Compression:** all OmniRoute compression engines stay OFF, and `codex/*` stays excluded, so the byte-identical path is kept. + +## cognee 1.6.1 (setup-only smoke passed, token-save-manifest-html) +- Receipt: `~/.local/state/native-agent-stack/s3-arms/receipts/cognee-install-smoke-20260927.json`. +- **Install:** bare metal from tag v1.6.1 eb90d037's uv.lock (`uv sync --python 3.12 --frozen --no-dev --no-editable`, mirroring the upstream Dockerfile); lane-owned CPython 3.12.14; `env -i`; CPU only. +- **Config:** + - `LLM_PROVIDER=custom`, `LLM_MODEL=openai/cx/gpt-6-astra-max`, the gateway endpoint and a placeholder key; + - fastembed `BAAI/bge-small-en-v1.5` (upstream's local default); + - shipped sqlite, lancedb and ladybug stores; + - `ENABLE_BACKEND_ACCESS_CONTROL=true`, `TELEMETRY_DISABLED=1`; + - `DataItem(data_id=uuid5)` pinned per session. +- **Structured output: use instructor `tool_call` mode.** + - The shipped litellm_native json_object fallback fails at the gateway with 400. The chat→Responses translator moves the system message into `instructions` (OmniRoute toResponses.ts L120-136), and the backend requires "json" in the input messages. + - Strict `json_schema` also fails with 400, because cognee's KnowledgeGraph lacks `additionalProperties: false`. +- **Per-role models:** `LLM_EXTRACTION_MODEL`, `LLM_SUMMARIZATION_MODEL` and `LLM_QUERY_MODEL`. +- **Hygiene:** + - cognee folds `llm_temperature=0.0` into every call (config.py L271-298), which makes it cache- and dedup-eligible; + - fix with `LLM_ARGS={"temperature":1.0,"extra_headers":{"x-omniroute-session":"","X-OmniRoute-No-Cache":"true"}}` (LLM_ARGS wins, L287); + - the gateway strips temperature for codex models, so this changes cache eligibility only; + - cognee CAN send headers. +- **Registration:** cognee is not in manifests/stack.json, so `host_receipts.py` can't file its receipt until r6 registers the candidate. Do that in the r6 PR. +- **Measured:** add 5.2 s and cognify 50.0 s on the canary; recall persisted across a fresh process; 7 calls all at effort_upstream=max; 7,237 tokens in and 2,045 reasoning. + +## Live deployments (token-save-manifest-html, 2026-09-27 ~18:15Z) +These are hosting evidence for r6. They are never used as scored arms. +- **Receipts:** `~/.local/state/native-agent-stack/s3-arms/receipts/cognee-live-deploy-20260927.json` and `hindsight-live-deploy-20260927.json`. +- **cognee-live:** + - runs from the pinned digest on rootless Docker 29.8.1, at 127.0.0.1:3800, with auth on; + - reaches GPT-6 at max through the container gateway address 10.0.2.2:20128; + - uses instructor tool_call, LLM_ARGS temperature 1.0 with the session and No-Cache headers, and fastembed on CPU; + - its lifecycle checks passed: install, use, restart, recovery and persistence; + - it made 13 gateway calls, all at max, all on 1 account. +- **hindsight-live:** + - runs the existing lane install as an enabled systemd --user unit on 3710/5433, with LLM `cx/gpt-6-astra-max`; + - `HINDSIGHT_API_LLM_TEMPERATURE=none` omits temperature everywhere; + - its default headers carry the session key and No-Cache, and cache affinity uses `openai_prompt_cache_key`; + - its lifecycle checks passed: install, use, restart, persistence, and recovery after a guard fix (a same-boot stale postmaster.pid with a dead PID is cleared). The failed first attempt is kept. +- **Hindsight per-role knobs, for r6's arm:** + - `HINDSIGHT_API_{RETAIN,REFLECT,CONSOLIDATION}_LLM_MODEL`; + - `HINDSIGHT_API_{RETAIN,REFLECT,CONSOLIDATION}_REASONING_EFFORT`; + - per-operation temperatures. + +## Hindsight 0.10.1 (token-stack-e2e-proof, installed earlier today) +- Bare metal from the tag's uv.lock with pg0-embedded, the GPT-6 backbone via OmniRoute; retain worked end to end, and recall persisted across a restart. +- Its request shapes (temperature, max_tokens up to 64000, json_object with the schema in the system message, tools with tool_choice required) returned 9/9 at 200 on sol, luna and astra (this session's probe at 16:05Z). diff --git a/blueprints/memory-layer-s3/inputs/r7/gpt6-review-20260930.md b/blueprints/memory-layer-s3/inputs/r7/gpt6-review-20260930.md new file mode 100644 index 000000000..75297cb8e --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r7/gpt6-review-20260930.md @@ -0,0 +1,39 @@ +# GPT-6 cross-family review of the S3 preregistration, DRAFT r7 (2026-09-30) + +Added source line: the findings section below is copied byte for byte from the coordinator's r7 repair-round-1 brief of 2026-09-30, which relays it verbatim from the GPT-6 review (gateway `cx/gpt-6-astra`, effort max). Verdict: **needs_changes**, seven findings (three high, four medium). The coordinator reports that all 12 of the review's quoted source excerpts matched their sources (12/12); this capture did not repeat that check. Line numbers such as `PREREGISTRATION.md:65` and `decision record:84` refer to the reviewed text, which is identical at `a146c419` and `a6ca5072`. The resolutions are in the preregistration's "r7 repair round 1" subsection and in the decision record's "Review and repair" section. + +## Findings (verbatim from the GPT-6 review, gateway cx/gpt-6-astra effort max, 2026-09-30) +### F1 [high] PREREGISTRATION.md §§1, 8.4; maintenance gate +- Issue: Calling Harbor task bundles hashed data does not prevent execution of code from the stale harbor-datasets repository. The design cannot currently substantiate its universal no-stale-code claim. +- Evidence: PREREGISTRATION.md:65 prohibits “scripts, evaluators, prompt code or packages built from it,” but :583 permits tasks “only as hashed data.” The permitted pinned repository contains executable task tests, for example “python /tests/llm_judge.py” in [harbor-datasets@37db1088, tests/test.sh:4](https://github.com/harbor-framework/harbor-datasets/blob/37db108843a49bb31a592e37a75e2c40dc3f9749/datasets/aa-lcr/aa-lcr-1/tests/test.sh#L4). The active [Harbor verifier@1e5c5c6d:175–232](https://github.com/harbor-framework/harbor/blob/1e5c5c6db929a10a140d05e606882c671ae20729/src/harbor/verifier/verifier.py#L175-L232) uploads task tests and executes their command. No selected-task manifest establishes that such code is excluded. +- Fix: Freeze a task-level provenance manifest distinguishing inert inputs from Dockerfiles, setup scripts, tests and judges. Permit execution only from gate-passing sources, with explicit enforcement. Replace the executable task components through maintained upstream sources or leave the Harbor lane pending. + +### F2 [high] PREREGISTRATION.md §§2, 4.3–4.5; Hindsight parity +- Issue: The reused vendor provider silently changes Hindsight's native pipeline. Judge-free scoring and an audit promise do not resolve this concrete configuration mismatch. +- Evidence: PREREGISTRATION.md:254 promises providers operate “without changing ingest or recall semantics,” and :256 selects hindsight-http. Its inherited [bank configuration@03c1d0f1:218–243](https://github.com/vectorize-io/agent-memory-benchmark/blob/03c1d0f1d27da63034f0931121c858faba512383/src/memory_bench/memory/hindsight.py#L218-L243) sets “kwargs: dict = dict(enable_observations=False)”; the asynchronous bank-creation path also uses these kwargs (:425–433). Hindsight's pinned [native configuration@5fc4ce20:1711–1712](https://github.com/vectorize-io/hindsight/blob/5fc4ce20917b916240cef27c212c387a177f115b/hindsight-api-slim/hindsight_api/config.py#L1711-L1712) defaults observations and automatic consolidation to True. This changes retrieval, ingest work and cost. Additionally, the provider's asynchronous recall ignores k and uses token budgets, so the draft's “equal candidate depth” requirement (:284) needs an operational definition. +- Fix: Preserve the selected native bank configuration through supported configuration or a thin local provider, keeping AMB unchanged. Freeze and verify effective bank settings before ingestion. Resolve every provider's depth policy and test the actual asynchronous ingest/recall paths, including failure handling, in the parity audit. + +### F3 [high] PREREGISTRATION.md §7.2; winner and tie-break set +- Issue: Membership in the tie-break set is undefined when several systems share the highest point estimate. This can change the selected system. +- Evidence: PREREGISTRATION.md:463 defines L as “on an exact tie, every tied system,” but :464 uses the singular condition “whose Holm-adjusted test against L ... does not reject.” Equal point estimates need not produce equal paired-test results against a third system because their question-level differences can have different variances. A candidate within 0.02 could therefore be separated from one co-leader but not another. +- Fix: Define the maximizer set explicitly and specify the quantifier. For example, admit a non-maximizer only when it is within 0.02 and is not significantly worse than every co-leader. Freeze executable examples covering tied leaders with conflicting pairwise decisions, and multiple equal minimum costs. + +### F4 [medium] PREREGISTRATION.md §§5.2, 7.3; decision record:84 +- Issue: The cost scalar does not clearly distinguish gross priced usage from usage differences after subtracting the no-memory control. The distinction changes the 0.80 ratio and can produce zero or negative costs. +- Evidence: PREREGISTRATION.md:326 says “The same client's no-memory control on the same workload is subtracted pairwise.” At :334, C is “summed over every model invocation in ledgers (a) and (c).” Gross costs of 120 and 130 do not meet the 0.80 threshold; subtracting a common baseline of 100 produces 20 and 30, which do. Equal zero or negative minima also make the singular selection instruction at :473 ambiguous. +- Fix: Define C explicitly as nonnegative gross workload cost and report control-subtracted differences separately, or replace the ratio with a prespecified rule suitable for signed incremental costs. Specify equal-minimum handling and the corresponding RAG workload. + +### F5 [medium] PREREGISTRATION.md §§5.3, 7.5; decision record:81 +- Issue: The generalized fallback preflight combines incompatible concurrency requirements. +- Evidence: PREREGISTRATION.md:504 requires the preflight to run “at the confirmatory concurrency K and on 5.3's warm schedule,” while :349 freezes that schedule at “concurrency 1.” This matters for saturation-triggered fallbacks, including ai-memory's four-call cap. +- Fix: Specify separate measurements: fallback behavior under confirmatory load K, and gated warm latency at concurrency 1. State which workload and denominator determine the 5% fallback threshold and which result controls stage disabling. + +### F6 [medium] PREREGISTRATION.md §7.4; decision record:85 +- Issue: The pinned-commit CI term can award full credit while relevant jobs remain unfinished. +- Evidence: PREREGISTRATION.md:491 awards K=10 when “every completed check run” has an allowed conclusion and the combined status “is not failure or error.” One successful completed lint check plus an in-progress test run and pending combined status satisfies that wording. +- Fix: Require the frozen relevant check/status set to be complete, with no queued, in-progress or pending entries, before awarding K=10. Specify pagination and handling of missing or inconclusive results. Retain the explicit description of release cadence and its weights as coordinator heuristics. + +### F7 [medium] PREREGISTRATION.md §7.1; decision record:83 +- Issue: The symmetric usefulness floor establishes relative ranking but no minimum usefulness for deployment. +- Evidence: PREREGISTRATION.md:459 requires successes to be “at least the highest success count among the systems meeting items 1–5, minus one.” If all systems achieve 0/25, every otherwise qualifying system passes this floor; a zero-success system also passes when the best achieves 1/25. +- Fix: Add a common absolute usefulness requirement or a requirement relative to the same no-memory control, while retaining symmetry. Alternatively, obtain explicit acceptance that this is solely a relative filter and that selection provides no absolute usefulness qualification. diff --git a/blueprints/memory-layer-s3/inputs/r7/landscape-foldin-gate-20260930.json b/blueprints/memory-layer-s3/inputs/r7/landscape-foldin-gate-20260930.json new file mode 100644 index 000000000..5458fb509 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r7/landscape-foldin-gate-20260930.json @@ -0,0 +1,364 @@ +{ + "checked_at_utc": "2026-09-30T16:30:31Z", + "window_start_utc": "2026-07-02T16:30:31Z", + "commands": [ + "gh api repos/OWNER/REPO", + "gh api repos/OWNER/REPO/commits?since=SINCE&per_page=100&page=N", + "gh api repos/OWNER/REPO/releases/latest", + "gh api repos/OWNER/REPO/commits/TAG" + ], + "discovery_input": "inputs/r7/landscape-memory-20260930.json", + "repositories": [ + { + "repository": "thedotmack/claude-mem", + "archived": false, + "default_branch": "main", + "head": "a077975ebd41643902f53e538e9bdf36c8367a0a", + "last_commit_utc": "2026-09-30T11:09:49Z", + "commits_in_window": "300+", + "latest_release": { + "tag": "v13.28.0", + "commit": "57c38303734482a596b3e2a052b969b30584be0d", + "published_utc": "2026-09-26T18:37:19Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "primary (3.1)", + "basis": "official local install and official integrations with both clients", + "locators": [ + "thedotmack/claude-mem@57c38303:docs/public/installation.mdx:27", + "thedotmack/claude-mem@57c38303:docs/public/installation.mdx:55", + "thedotmack/claude-mem@57c38303:src/services/integrations/CodexCliInstaller.ts:221", + "thedotmack/claude-mem@57c38303:src/services/integrations/CodexCliInstaller.ts:507" + ] + } + }, + { + "repository": "doobidoo/mcp-memory-service", + "archived": false, + "default_branch": "main", + "head": "9079737b8e6da0b3e9d4d3f8e24979b56ec00079", + "last_commit_utc": "2026-09-30T13:32:51Z", + "commits_in_window": "300+", + "latest_release": { + "tag": "v11.14.0", + "commit": "bed24ef7214192bfe59ad46f6852165678735a77", + "published_utc": "2026-09-25T09:44:09Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "primary (3.1)", + "basis": "official local install; Claude Code stdio registration; actual Codex recipe uses third-party mcp-proxy, which must be pinned and pass section 1's maintenance gate if selected. Alternative: labelled adaptation with Codex 0.159.2's native streamable-HTTP MCP client (codex mcp add … --url with --bearer-token-env-var MCP_API_KEY) to the owned loopback /mcp endpoint with its Authorization: Bearer header. Either Codex path must be pinned and gated before freeze; the recipe's no-native-HTTP claim is contradicted by pinned Codex source, and its referenced examples/codex-mcp-config.json is absent at bed24ef7", + "locators": [ + "doobidoo/mcp-memory-service@bed24ef7:README.md:111", + "doobidoo/mcp-memory-service@bed24ef7:docs/integration/multi-client.md:450-477", + "doobidoo/mcp-memory-service@bed24ef7:docs/integration/multi-client.md:447-448", + "openai/codex@ff6aec96:codex-rs/cli/src/mcp_cmd.rs:145-157,343-359", + "openai/codex@ff6aec96:codex-rs/config/src/mcp_types.rs:640-644" + ] + } + }, + { + "repository": "MemMachine/MemMachine", + "archived": false, + "default_branch": "main", + "head": "d57f5cb36a357c01085f571a0cdd2dfbc9882f89", + "last_commit_utc": "2026-09-25T19:15:26Z", + "commits_in_window": 28, + "latest_release": { + "tag": "v0.3.9", + "commit": "1d3e0ce2782b83571e1974198c4d317e601cb263", + "published_utc": "2026-05-18T02:22:52Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "primary (3.1), at the latest release v0.3.9; the deployment spec was read at main 899ace1a, 31 commits ahead, and its locators were re-read at 1d3e0ce2 (2026-09-30)", + "basis": "official local install; Claude Code MCP through .mcp.json; Codex skill from the pinned packages/skills/memmachine-memory tree or a pinned, section-1-gated installer (npx skills add / pipx run shskills install --agent codex). The skill drives mem-cli from memmachine-client, which must be pinned in the arm's SBOM; the Codex path must be pinned and gated before freeze", + "locators": [ + "MemMachine/MemMachine@1d3e0ce2:docs/install_guide/integrate/claude_code_mcp.mdx:62", + "MemMachine/MemMachine@1d3e0ce2:packages/skills/README.md:13-24", + "MemMachine/MemMachine@1d3e0ce2:packages/skills/memmachine-memory/SKILL.md:31-33", + "MemMachine/MemMachine@1d3e0ce2:packages/client/pyproject.toml:6,38" + ] + } + }, + { + "repository": "agentscope-ai/ReMe", + "archived": false, + "default_branch": "main", + "head": "81c16d3e2c1010c2ec3ed6eede114d3c3ac882d1", + "last_commit_utc": "2026-09-30T10:30:36Z", + "commits_in_window": 174, + "latest_release": { + "tag": "v0.4.1.12", + "commit": "4f7c8786e3975b6ff1b20685ed56c3ccbaa4ed9f", + "published_utc": "2026-09-14T05:03:54Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "primary (3.1)", + "basis": "official local install; Claude Code plugin to a shared HTTP MCP server; Codex through the official ReMe Memory skill", + "locators": [ + "agentscope-ai/ReMe@4f7c8786:integrations/claude_code/README.md:23", + "agentscope-ai/ReMe@4f7c8786:README.md:188" + ] + } + }, + { + "repository": "aiming-lab/SimpleMem", + "archived": false, + "default_branch": "main", + "head": "db80b6a7c591e0ea730a058e9f5fc4eb06572299", + "last_commit_utc": "2026-07-24T07:40:38Z", + "commits_in_window": 14, + "latest_release": { + "tag": "v0.3.0", + "commit": "c830efa3fd2614bc4eb0b4a80bc5836ece9ded96", + "published_utc": "2026-05-21T16:02:50Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "out on fit (3.3): no official Codex integration", + "basis": "no Codex setup in any Markdown document at the latest release c830efa3 or at main db80b6a7; the Claude path is a skill copy", + "locators": [ + "aiming-lab/SimpleMem@c830efa3:SKILL/README.md:13" + ] + } + }, + { + "repository": "mem0ai/mem0", + "archived": false, + "default_branch": "main", + "head": "94c3fe9f238f3dbf29c9ce98643bd71eb13077cd", + "last_commit_utc": "2026-09-25T17:34:12Z", + "commits_in_window": 234, + "latest_release": { + "tag": "ts-v3.3.1", + "commit": "94c3fe9f238f3dbf29c9ce98643bd71eb13077cd", + "published_utc": "2026-09-25T17:36:35Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "3.3 exclusion retained; the landscape's add-primary recommendation is not adopted", + "basis": "the official Codex plugin core posts to Platform routes; the pinned tree has no self-hosted OpenMemory directory, and OpenMemory remains only in changelog entries and documentation redirects", + "locators": [ + "mem0ai/mem0@94c3fe9f:integrations/codex-plugin/core/memory_core.py:1996", + "mem0ai/mem0@94c3fe9f:docs/docs.json:1182" + ] + } + }, + { + "repository": "OSU-NLP-Group/HippoRAG", + "archived": false, + "default_branch": "main", + "head": "398bfdc388692795c105c1b903db60031975b669", + "last_commit_utc": "2026-09-29T14:00:48Z", + "commits_in_window": 21, + "latest_release": { + "tag": "v1.0.0", + "commit": "b144c46df14cabe5f5822d8caded4bec5f709461", + "published_utc": "2025-02-27T17:29:48Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "document-RAG candidate (8.3) at a dated default-branch pin exception, not the memory family", + "basis": "Python SDK only, with no Claude Code or Codex integration, MCP or HTTP service. The latest GitHub release v1.0.0 (b144c46d) is the legacy HippoRAG 1 code (README cites arXiv 2405.14831; no hipporag package in its tree); HippoRAG 2 is the default branch, package 2.0.0a5, unreleased; PyPI's newest hipporag is in 'pypi' below", + "locators": [ + "OSU-NLP-Group/HippoRAG@398bfdc3:README.md:105", + "OSU-NLP-Group/HippoRAG@398bfdc3:README.md:110", + "OSU-NLP-Group/HippoRAG@398bfdc3:pyproject.toml:7", + "OSU-NLP-Group/HippoRAG@b144c46d:README.md:5" + ] + } + }, + { + "repository": "qualixar/superlocalmemory", + "archived": false, + "default_branch": "main", + "head": "5bfa47c941b6329b23f93b17f3dce7f6d70a2b9a", + "last_commit_utc": "2026-09-12T19:07:09Z", + "commits_in_window": "300+", + "latest_release": { + "tag": "v4.1.17", + "commit": "5bfa47c941b6329b23f93b17f3dce7f6d70a2b9a", + "published_utc": "2026-09-12T19:12:11Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "memory reserve 4 (3.2)", + "basis": "landscape-reported integrations with both clients; release present" + } + }, + { + "repository": "NevaMind-AI/memU", + "archived": false, + "default_branch": "main", + "head": "2c050bc9681a4c0aff1af211a000e73d14f33356", + "last_commit_utc": "2026-09-21T11:45:36Z", + "commits_in_window": 133, + "latest_release": { + "tag": "v1.5.1", + "commit": "357aefc8012705bfde723f141d52f675fe712bed", + "published_utc": "2026-03-23T07:38:28Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "memory reserve 5 (3.2)", + "basis": "landscape-reported integrations with both clients; release present" + } + }, + { + "repository": "JaceHo/AgentMem", + "archived": false, + "default_branch": "main", + "head": "0770f5d861142486a6e8d57599ae36db7ddd93be", + "last_commit_utc": "2026-08-31T04:12:09Z", + "commits_in_window": 3, + "latest_release": { + "tag": null, + "commit": null, + "published_utc": null, + "prerelease": null + }, + "verdict": "ACTIVE", + "screen": { + "route": "memory reserve 6 (3.2)", + "basis": "landscape-reported integrations with both clients; no GitHub release to pin" + } + }, + { + "repository": "MemTensor/MemOS", + "archived": false, + "default_branch": "main", + "head": "a7367d07e55db61099f7b4e2c1108bc5831a24f3", + "last_commit_utc": "2026-09-22T01:51:03Z", + "commits_in_window": 264, + "latest_release": { + "tag": "v2.0.34", + "commit": "41bf5c7fa89ee08ebedf3c662638b06fc29aca8d", + "published_utc": "2026-09-23T05:24:20Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "memory reserve 8 (3.2)", + "basis": "local adapters target other agents; Claude Code/Codex local path unverified" + } + }, + { + "repository": "EverMind-AI/EverOS", + "archived": false, + "default_branch": "main", + "head": "6aff7351bc4aebf92aa92217858ac4bf736eef28", + "last_commit_utc": "2026-09-30T12:02:41Z", + "commits_in_window": 83, + "latest_release": { + "tag": "v1.4.1", + "commit": "462ebf9fd59b55c03fefb8eec855c62500f2a3cf", + "published_utc": "2026-09-24T10:33:18Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "memory reserve 7 (3.2)", + "basis": "Claude Code/Codex capture adapter unverified" + } + }, + { + "repository": "BAI-LAB/MemoryOS", + "archived": false, + "default_branch": "main", + "head": "587ed7755c7aed179965792830ff1b5ad9a6fa92", + "last_commit_utc": "2026-07-07T12:32:18Z", + "commits_in_window": 2, + "latest_release": { + "tag": "V1.2", + "commit": "cbb4c74c7b8005f29d9bb363f93d966ed7f82452", + "published_utc": "2025-07-18T07:21:44Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "memory reserve 9 (3.2)", + "basis": "MCP implementation; capture hooks unverified; oldest release" + } + }, + { + "repository": "infiniflow/ragflow", + "archived": false, + "default_branch": "main", + "head": "9766c8c5483b10d923acbbf2ba5e25b14819be19", + "last_commit_utc": "2026-09-30T12:08:16Z", + "commits_in_window": "300+", + "latest_release": { + "tag": "v1.0.0-rc1", + "commit": "14bb02ee4584c1ab18f0c722c48beecd6bbf9860", + "published_utc": "2026-09-29T05:52:57Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "document-RAG reserve (8.3)", + "basis": "self-hosted document RAG platform; no coding-client integration" + } + }, + { + "repository": "abhigyanpatwari/GitNexus", + "archived": false, + "default_branch": "main", + "head": "acb65b95b62647b19781caa60afd17d3bed23d4a", + "last_commit_utc": "2026-09-29T13:49:44Z", + "commits_in_window": "300+", + "latest_release": { + "tag": "v1.6.12", + "commit": "c4ecf398de637cf779ab2238e69958cfd5d601ce", + "published_utc": "2026-09-12T21:40:04Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "outside S3: a candidate for the Harbor token-tool E2E's next round; no S3 arm", + "basis": "local code graph with MCP for both clients (landscape report, unverified at a pin); absence from the Harbor run is relayed, with no receipt in this checkout; the Harbor receipt is required before freeze; S3 has no code-retrieval tool arm" + } + }, + { + "repository": "oraios/serena", + "archived": false, + "default_branch": "main", + "head": "8a3ce35cae29a93748842ba463231d6c78d181e7", + "last_commit_utc": "2026-09-29T10:49:50Z", + "commits_in_window": "300+", + "latest_release": { + "tag": "v1.7.0", + "commit": "949a27ef1e5fda1a6e7b561e777bcece345c6ffd", + "published_utc": "2026-08-09T18:38:06Z", + "prerelease": false + }, + "verdict": "ACTIVE", + "screen": { + "route": "outside S3: measurement as a token tool in the separate Harbor E2E is relayed; no S3 change", + "basis": "the landscape listed it as a code-retrieval reserve; S3 has no code-retrieval tool arm; no receipt for the relayed Harbor run exists in this checkout, and the Harbor receipt is required before freeze" + } + } + ], + "pypi": [ + { + "package": "hipporag", + "latest_version": "2.0.0a4", + "latest_upload_utc": "2025-06-24T21:02:54.727514Z", + "source": "https://pypi.org/pypi/hipporag/json", + "checked_utc": "2026-09-30T17:03:07Z" + } + ], + "screen_revised_utc": "2026-09-30T18:03:21Z", + "screen_revision_note": "screen annotations repaired after cross-family review: actual MCP Memory Service Codex recipe and native-HTTP adaptation, pending pinned/gated Codex paths for it and MemMachine, completed MemMachine release-source re-read, and relayed Serena/GitNexus Harbor facts requiring a receipt before freeze; prior screen corrections retained; captured gate data unchanged" +} diff --git a/blueprints/memory-layer-s3/inputs/r7/landscape-memory-20260930.json b/blueprints/memory-layer-s3/inputs/r7/landscape-memory-20260930.json new file mode 100644 index 000000000..7bb039b44 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r7/landscape-memory-20260930.json @@ -0,0 +1,648 @@ +{ + "checked_at_utc": "2026-09-30T15:09:53Z", + "candidates": [ + { + "name": "agentmemory", + "repo": "https://github.com/rohitg00/agentmemory", + "latest_release": "v0.9.29 observed in changelog; latest release tag not independently confirmed", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Native Claude Code and Codex plugins, lifecycle hooks, skills and MCP.", + "published_eval": "Committed LongMemEval-S retrieval evaluation: R@5 95.2%, R@10 98.6%; also coding-agent-life-v1. Retrieval results are not answer accuracy.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Direct coding-agent integration and local lexical/vector retrieval. Provisional until default-branch activity is verified.", + "sources": [ + "https://github.com/rohitg00/agentmemory" + ] + }, + { + "name": "Hindsight", + "repo": "https://github.com/vectorize-io/hindsight", + "latest_release": "v0.10.2", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "hindsight-coding-agents supports Claude Code and Codex; bank-scoped MCP endpoint.", + "published_eval": "Published LongMemEval results; hindsight-system-evals provides evaluation code. Published configurations must be distinguished from the proposed local configuration.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Strong evaluated memory architecture; local PostgreSQL, embeddings and compatible LLM providers supported. Maintenance admission remains provisional.", + "sources": [ + "https://github.com/vectorize-io/hindsight" + ] + }, + { + "name": "OpenViking", + "repo": "https://github.com/volcengine/OpenViking", + "latest_release": "v0.4.22", + "last_commit_utc": "2026-09-29", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Claude Code and Codex plugins/hooks and MCP integration.", + "published_eval": "Repository benchmark material covers LoCoMo and tau2-bench. Reported improvements used an earlier version and Doubao models; they do not establish local-model performance.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Maintained filesystem-oriented memory and document RAG contender. Count once, with separate memory and document workloads.", + "sources": [ + "https://github.com/volcengine/OpenViking" + ] + }, + { + "name": "cognee", + "repo": "https://github.com/topoteretes/cognee", + "latest_release": "GitHub tag unverified; PyPI 1.6.2 observed", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "MCP integration; exact Claude Code/Codex lifecycle parity not verified.", + "published_eval": "Published knowledge-graph memory research. A reproducible current-version result on the requested memory benchmarks was not verified.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Current keyless local extraction/search changes materially improve suitability. Pin graph, vector and optional LLM stages; maintenance admission provisional.", + "sources": [ + "https://github.com/topoteretes/cognee" + ] + }, + { + "name": "MemPalace", + "repo": "https://github.com/MemPalace/mempalace", + "latest_release": "v3.10.0", + "last_commit_utc": "unverified; default branch observed as develop", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Claude Code and Codex plugin directories, MCP and agent transcript ingestion.", + "published_eval": "Committed LongMemEval retrieval results and benchmark documentation. Headline near-perfect recall involves different reranking/held-out protocols; it is not end-to-end QA accuracy.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Useful local retrieval baseline. Preregister the exact untouched test split and retrieval configuration; verify activity on develop.", + "sources": [ + "https://github.com/MemPalace/mempalace" + ] + }, + { + "name": "basic-memory", + "repo": "https://github.com/basicmachines-co/basic-memory", + "latest_release": "v0.23.2", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "MCP and Claude Code/Codex plugin integrations.", + "published_eval": "Benchmark code exists; a published reproducible result on the requested memory QA benchmarks was not verified.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Markdown/SQLite control with optional local semantic search. Maintenance admission provisional.", + "sources": [ + "https://github.com/basicmachines-co/basic-memory" + ] + }, + { + "name": "memsearch", + "repo": "https://github.com/zilliztech/memsearch", + "latest_release": "unverified; supplied v0.4.21 exceeds stale indexed release page", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Claude Code plugin; Codex integration through supported hooks/skills.", + "published_eval": "Published bilingual optional-reranker evaluation; requested longitudinal memory benchmark reproduction not verified.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Local embeddings and Markdown-backed retrieval are relevant. Disable optional hosted reranking for the local arm; verify release and activity.", + "sources": [ + "https://github.com/zilliztech/memsearch" + ] + }, + { + "name": "ai-memory", + "repo": "https://github.com/akitaonrails/ai-memory", + "latest_release": "unverified; supplied v2.4.2, indexed release page still v2.4.1", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "First-party Claude Code/Codex support, MCP and capture hooks.", + "published_eval": "Repository evaluation material exists; a comparable requested-benchmark score was not verified.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Coding handoffs, Git-backed memory and local search provide a useful distinct baseline. Release and maintenance checks remain provisional.", + "sources": [ + "https://github.com/akitaonrails/ai-memory" + ] + }, + { + "name": "engram", + "repo": "https://github.com/Gentleman-Programming/engram", + "latest_release": "v2.2.1", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Claude Code plugin and Codex setup command/MCP.", + "published_eval": "Requested-benchmark evaluation and reproducible result not verified.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Small SQLite/FTS memory control requiring neither embeddings nor a memory LLM. Maintenance admission provisional.", + "sources": [ + "https://github.com/Gentleman-Programming/engram" + ] + }, + { + "name": "Graphiti", + "repo": "https://github.com/getzep/graphiti", + "latest_release": "v0.30.2", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Repository MCP server usable by Claude Code and Codex.", + "published_eval": "Temporal-graph/Zep research is published; managed Zep results do not establish performance of this exact Graphiti release and local stack.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Temporal graph architecture merits comparison. Use maintained supported graph storage and compatible local models; verify default-branch activity.", + "sources": [ + "https://github.com/getzep/graphiti" + ] + }, + { + "name": "supermemory local", + "repo": "https://github.com/supermemoryai/supermemory", + "latest_release": "server-v0.0.8", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Claude Code/Codex integrations advertised; their routing to the local server needs verification.", + "published_eval": "MemoryBench harness linked; reported LongMemEval retrieval recall must be separated from answer accuracy.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Local server supports local embeddings and compatible LLM endpoints. Before admission, verify source-build reproducibility, local plugin routing and activity.", + "sources": [ + "https://github.com/supermemoryai/supermemory" + ] + }, + { + "name": "qmd", + "repo": "https://github.com/tobi/qmd", + "latest_release": "v2.8.3", + "last_commit_utc": "unverified", + "archived": false, + "category": "rag-document", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Local CLI, MCP server and Claude Code integration; Codex can use MCP.", + "published_eval": "Requested memory-benchmark evaluation not verified. Evaluate as a document-retrieval baseline.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Fully local BM25, dense retrieval, expansion and reranking. Freeze model files and retrieval settings; maintenance admission provisional.", + "sources": [ + "https://github.com/tobi/qmd" + ] + }, + { + "name": "LightRAG", + "repo": "https://github.com/HKUDS/LightRAG", + "latest_release": "v1.5.7", + "last_commit_utc": "unverified", + "archived": false, + "category": "rag-document", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Local server/API; a first-party Claude Code/Codex capture plugin was not verified.", + "published_eval": "Published LightRAG paper, reproduction material and RAGAS evaluation support; document QA rather than demonstrated coding-agent longitudinal memory.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Relevant graph/document RAG architecture with local compatible models. Fix the adapter and pipeline recipe; verify activity.", + "sources": [ + "https://github.com/HKUDS/LightRAG" + ] + }, + { + "name": "llama_index hybrid", + "repo": "https://github.com/run-llama/llama_index", + "latest_release": "v0.14.25", + "last_commit_utc": "unverified", + "archived": false, + "category": "rag-document", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Library/API integration; exact coding-agent MCP adapter not verified.", + "published_eval": "Numerous retrieval/evaluation components; 'hybrid' does not identify a single reproducible submitted system.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Preregister one upstream-supported BM25/vector/fusion/reranking recipe, component versions and storage backend. Maintenance admission provisional.", + "sources": [ + "https://github.com/run-llama/llama_index" + ] + }, + { + "name": "agent-memory-benchmark", + "repo": "https://github.com/vectorize-io/agent-memory-benchmark", + "latest_release": "unverified", + "last_commit_utc": "unverified", + "archived": false, + "category": "benchmark-harness", + "self_hostable_local": false, + "claude_code_or_codex_integration": "Evaluation harness, not a coding-agent integration.", + "published_eval": "Published benchmark code, prompts and evaluation workflow. Documented setup uses Gemini credentials; fully local reader/judge routing was not verified.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Retain conditionally as a harness; admission requires activity verification and a supported local/gateway reader and judge configuration.", + "sources": [ + "https://github.com/vectorize-io/agent-memory-benchmark" + ] + }, + { + "name": "LongMemEval-V2", + "repo": "https://github.com/xiaowu0162/LongMemEval-V2", + "latest_release": "unverified", + "last_commit_utc": "2026-08-09", + "archived": false, + "category": "benchmark-harness", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Includes Codex and AgentRunbook-C adapters; not a drop-in memory layer.", + "published_eval": "2026 paper and released evaluation code/data; repository documents local Qwen reader/embedding configurations and compatible endpoints.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Maintained recent benchmark with longer agent trajectories. Preserve chronological ingestion and prevent question/gold leakage.", + "sources": [ + "https://github.com/xiaowu0162/LongMemEval-V2/commits" + ] + }, + { + "name": "MemoryAgentBench", + "repo": "https://github.com/HUST-AI-HYZ/MemoryAgentBench", + "latest_release": "unverified", + "last_commit_utc": "2026-09-25", + "archived": false, + "category": "benchmark-harness", + "self_hostable_local": false, + "claude_code_or_codex_integration": "Memory-agent evaluation implementations; no native coding-agent plugin.", + "published_eval": "ICLR 2026 benchmark and reproducible repository covering retrieval, test-time learning, long-range understanding and selective forgetting.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Maintained capability coverage beyond recall. Fully local execution of every reader/judge path was not verified.", + "sources": [ + "https://github.com/HUST-AI-HYZ/MemoryAgentBench/commits" + ] + }, + { + "name": "MTEB", + "repo": "https://github.com/embeddings-benchmark/mteb", + "latest_release": "2.21.9", + "last_commit_utc": "2026-09-30", + "archived": false, + "category": "benchmark-harness", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Embedding evaluation only.", + "published_eval": "Published benchmark, datasets, task implementations and reproducible local-model evaluation.", + "in_current_list": true, + "recommendation": "keep", + "reason": "Use to choose and freeze embeddings; its scores do not measure longitudinal agent memory or coding outcomes.", + "sources": [ + "https://github.com/embeddings-benchmark/mteb/commits" + ] + }, + { + "name": "SimpleMem", + "repo": "https://github.com/aiming-lab/SimpleMem", + "latest_release": "v0.3.0, latest indexed release", + "last_commit_utc": "2026-07-24", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Self-hosted MCP server; Docker configuration supports Ollama and compatible endpoints.", + "published_eval": "2026 SimpleMem paper; LoCoMo F1 43.24% under its protocol, with repository benchmark/reproduction scripts. This is not comparable to LLM-judge accuracy percentages.", + "in_current_list": false, + "recommendation": "add_primary", + "reason": "Maintained compression-and-planning architecture with reproducible research and a local deployment path; substantial omission.", + "sources": [ + "https://github.com/aiming-lab/SimpleMem/commits" + ] + }, + { + "name": "ReMe", + "repo": "https://github.com/agentscope-ai/ReMe", + "latest_release": "v0.4.1.4, latest indexed release", + "last_commit_utc": "2026-09-30", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Claude Code MCP/skills and capture hook; Codex MCP/skills, with capture lifecycle requiring separate verification.", + "published_eval": "Repository LongMemEval harness reports 89.4% on cleaned S500; BEAM results cover specified 100K/1M subsets, not automatically the entire benchmark.", + "in_current_list": false, + "recommendation": "add_primary", + "reason": "Recent maintained entrant combining episodic/procedural reuse and local file memory. Normalize its comparatively large reader-token budgets.", + "sources": [ + "https://github.com/agentscope-ai/ReMe/commits" + ] + }, + { + "name": "MemMachine", + "repo": "https://github.com/MemMachine/MemMachine", + "latest_release": "v0.3.9, latest indexed release", + "last_commit_utc": "2026-09-25", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "MCP server and REST/SDK interfaces usable by both agents.", + "published_eval": "April 2026 paper reports LoCoMo and LongMemEval-S results; evaluation directory documents reproduction and legacy/current API differences.", + "in_current_list": false, + "recommendation": "add_primary", + "reason": "Evaluated episodic/profile memory architecture; Docker setup explicitly supports Ollama embeddings and OpenAI-compatible LLMs.", + "sources": [ + "https://github.com/MemMachine/MemMachine/commits" + ] + }, + { + "name": "Mem0 OSS core", + "repo": "https://github.com/mem0ai/mem0", + "latest_release": "v2.2.1", + "last_commit_utc": "2026-09-25", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Self-hosted core SDK/API; do not substitute the cloud-only coding-agent plugin. Exact supported local capture adapter requires qualification.", + "published_eval": "Published Mem0 research; upstream memory-benchmarks documents self-hosted Mem0/Qdrant evaluation on LoCoMo, LongMemEval and BEAM.", + "in_current_list": true, + "recommendation": "add_primary", + "reason": "Correct the exclusion: the plugin's cloud dependency does not exclude the maintained OSS core, which supports local models and storage.", + "sources": [ + "https://github.com/mem0ai/mem0/commits" + ] + }, + { + "name": "claude-mem", + "repo": "https://github.com/thedotmack/claude-mem", + "latest_release": "v13.24.23, latest indexed release", + "last_commit_utc": "2026-09-30", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Native Claude Code lifecycle capture; current repository also ships Codex integration.", + "published_eval": "Comparable published LongMemEval/LoCoMo/BEAM result with a reproducible harness was not verified.", + "in_current_list": false, + "recommendation": "add_primary", + "reason": "Highly relevant coding-session capture contender despite missing verified benchmark results. Source exposes a configurable compatible provider base URL.", + "sources": [ + "https://github.com/thedotmack/claude-mem/commits" + ] + }, + { + "name": "mcp-memory-service", + "repo": "https://github.com/doobidoo/mcp-memory-service", + "latest_release": "v11.13.0 observed in upstream documentation", + "last_commit_utc": "2026-09-30", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Local MCP/REST service with Claude Code and Codex integration documentation.", + "published_eval": "Checked-in LongMemEval, LoCoMo and DevBench retrieval scripts; LongMemEval R@5 80.4%, R@10 90.4%. These are retrieval metrics.", + "in_current_list": false, + "recommendation": "add_primary", + "reason": "Maintained local SQLite/vector/embedding service with direct agent relevance and reproducible retrieval evaluation.", + "sources": [ + "https://github.com/doobidoo/mcp-memory-service/commits" + ] + }, + { + "name": "MemOS", + "repo": "https://github.com/MemTensor/MemOS", + "latest_release": "v2.0.33 announced upstream; latest GitHub release tag unverified", + "last_commit_utc": "2026-09-22", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Verified local plugin adapters target OpenClaw, Hermes and DeepSeek Harness. Claude Code/Codex Cloud CLI exists; equivalent local adapter not verified.", + "published_eval": "Published LoCoMo/LongMemEval results and OmniMemEval framework. Cloud/core results cannot be assigned to the local plugin automatically.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Major omission and maintained local implementation, but target-agent adapter and benchmark-to-implementation identity need qualification.", + "sources": [ + "https://github.com/MemTensor/MemOS/commits" + ] + }, + { + "name": "EverOS / EverMemOS lineage", + "repo": "https://github.com/EverMind-AI/EverOS", + "latest_release": "unverified for current rewrite; v1.2.1 indexed for the lineage", + "last_commit_utc": "2026-09-30", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Coding-assistant integrations/community MCP ecosystem; exact current Claude Code/Codex capture adapter not verified.", + "published_eval": "EverMemOS 2026 paper reports strong LoCoMo results. Current EverOS has benchmark/run-spec material, but is a rewritten Markdown/SQLite/LanceDB runtime.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Important maintained research lineage. Reproduce the current implementation before transferring predecessor scores or promoting to primary.", + "sources": [ + "https://github.com/EverMind-AI/EverOS/commits" + ] + }, + { + "name": "SuperLocalMemory", + "repo": "https://github.com/qualixar/superlocalmemory", + "latest_release": "v4.0.1, latest indexed release", + "last_commit_utc": "2026-09-12", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Claude Code plugin and Codex/MCP/skills integrations.", + "published_eval": "August 2026 preprint; reproducible current-version head-to-head on the requested benchmark suite not verified.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Recent local memory-control architecture with no-LLM and Ollama modes; promising, but evaluation evidence needs qualification.", + "sources": [ + "https://github.com/qualixar/superlocalmemory/commits" + ] + }, + { + "name": "memU", + "repo": "https://github.com/NevaMind-AI/memU", + "latest_release": "unverified; changelog documents 2.0.0-beta.0", + "last_commit_utc": "2026-09-21", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Current host adapters/skills support Claude Code and Codex; verify version-specific capture behavior.", + "published_eval": "Comparable reproducible current-version result on the requested benchmarks not verified.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Maintained file/skill memory with local storage and configurable embedding endpoints. Significant implementation changes complicate score transfer.", + "sources": [ + "https://github.com/NevaMind-AI/memU/commits" + ] + }, + { + "name": "Memori", + "repo": "https://github.com/MemoriLabs/Memori", + "latest_release": "unverified", + "last_commit_utc": "2026-09-18", + "archived": false, + "category": "agent-memory", + "self_hostable_local": false, + "claude_code_or_codex_integration": "Documented Claude Code/Codex MCP setup uses api.memorilabs.ai and a Memori API key.", + "published_eval": "March 2026 research reports LoCoMo results; full OSS self-hosted augmentation reproduction was not verified.", + "in_current_list": false, + "recommendation": "exclude", + "reason": "Maintained, but the verified current SDK/MCP path depends on hosted services. BYODB and commercial on-premises claims do not establish a fully local OSS pipeline.", + "sources": [ + "https://github.com/MemoriLabs/Memori/commits" + ] + }, + { + "name": "Memobase", + "repo": "https://github.com/memodb-io/memobase", + "latest_release": "unverified", + "last_commit_utc": "2026-01-11", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Self-hosted API/SDK; target-agent plugin parity not verified.", + "published_eval": "Current reproducible evaluation was not qualified because the maintenance exclusion already applies.", + "in_current_list": false, + "recommendation": "exclude", + "reason": "Default-branch head is older than the 2026-07-02 cutoff, violating the specified 90-day rule.", + "sources": [ + "https://github.com/memodb-io/memobase/commits" + ] + }, + { + "name": "MemoryOS", + "repo": "https://github.com/BAI-LAB/MemoryOS", + "latest_release": "unverified", + "last_commit_utc": "unverified; cached page shows 2025-07-29 but was crawled 1.2 years ago", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Repository MCP implementation; target-agent capture hooks not verified.", + "published_eval": "EMNLP 2025 oral paper with LoCoMo evaluation scripts in eval/.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Research-backed candidate, but not admitted as maintained: stale commit evidence cannot establish its present head or justify a definitive exclusion.", + "sources": [ + "https://github.com/BAI-LAB/MemoryOS/commits" + ] + }, + { + "name": "AgentMem", + "repo": "https://github.com/JaceHo/AgentMem", + "latest_release": "unverified", + "last_commit_utc": "unverified", + "archived": false, + "category": "agent-memory", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Claude Code/Codex MCP and hook integrations advertised.", + "published_eval": "Repository advertises LongMemEval retrieval and answer-evaluation scripts; protocol equivalence and reported results were not independently qualified.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Relevant local Redis-backed entrant. Requires license, release, maintenance and benchmark qualification before admission.", + "sources": [ + "https://github.com/JaceHo/AgentMem" + ] + }, + { + "name": "HippoRAG / HippoRAG 2", + "repo": "https://github.com/OSU-NLP-Group/HippoRAG", + "latest_release": "unverified", + "last_commit_utc": "2026-09-03", + "archived": false, + "category": "rag-document", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Python library; native Claude Code/Codex MCP integration not verified.", + "published_eval": "Published HippoRAG/HippoRAG 2 research and reproducible multi-hop document QA evaluation.", + "in_current_list": false, + "recommendation": "add_primary", + "reason": "Maintained graph/PageRank retrieval architecture missing from document RAG coverage. Explicit local LLM and embedding endpoint support.", + "sources": [ + "https://github.com/OSU-NLP-Group/HippoRAG/commits" + ] + }, + { + "name": "RAGFlow", + "repo": "https://github.com/infiniflow/ragflow", + "latest_release": "v1.0.0-rc1 observed; stable v0.27.2", + "last_commit_utc": "2026-09-30", + "archived": false, + "category": "rag-document", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Self-hosted retrieval API; exact first-party coding-agent integration not verified.", + "published_eval": "Comparable reproducible result on the requested conversational memory benchmarks not verified.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Maintained document-processing/RAG platform with local models; larger deployment and parsing scope may exceed the intended lightweight layer comparison.", + "sources": [ + "https://github.com/infiniflow/ragflow/commits" + ] + }, + { + "name": "Serena", + "repo": "https://github.com/oraios/serena", + "latest_release": "unverified", + "last_commit_utc": "2026-09-24", + "archived": false, + "category": "rag-code", + "self_hostable_local": true, + "claude_code_or_codex_integration": "MCP integration explicitly supports Claude Code and Codex.", + "published_eval": "Requested memory-benchmark reproduction not verified; evaluate symbol retrieval and coding outcomes separately.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Important local language-server/symbol retrieval control with persistent project memories, complementary to document/vector RAG.", + "sources": [ + "https://github.com/oraios/serena/commits" + ] + }, + { + "name": "GitNexus", + "repo": "https://github.com/abhigyanpatwari/GitNexus", + "latest_release": "unverified", + "last_commit_utc": "unverified for canonical upstream", + "archived": false, + "category": "rag-code", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Canonical README search describes local code knowledge graphs and MCP support for Claude Code/Codex.", + "published_eval": "Reproducible coding-outcome or requested memory-benchmark result not verified.", + "in_current_list": false, + "recommendation": "add_reserve", + "reason": "Search-only upstream lead, not admitted as maintained. Correction: nxpatterns/gitnexus is a fork; its September 9 date was not transferred to upstream. Canonical page could not be opened within budget.", + "sources": [] + }, + { + "name": "Letta, current Letta Code runtime", + "repo": "https://github.com/letta-ai/letta-code", + "latest_release": "unverified", + "last_commit_utc": "2026-09-30", + "archived": false, + "category": "other", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Own coding-agent runtime/App Server; a drop-in memory layer for Claude Code/Codex was not verified.", + "published_eval": "Relevant memory-agent research exists; exact current-runtime benchmark reproduction was not qualified.", + "in_current_list": true, + "recommendation": "exclude", + "reason": "Scope exclusion, not blanket retirement: V1 is retired, but the maintained successor supports a local backend. Compare separately because it changes the agent runtime.", + "sources": [ + "https://github.com/letta-ai/letta-code/commits" + ] + }, + { + "name": "beads", + "repo": "https://github.com/gastownhall/beads", + "latest_release": "v1.3.0 stable; v1.3.1-rc.1 prerelease observed in search", + "last_commit_utc": "unverified", + "archived": false, + "category": "other", + "self_hostable_local": true, + "claude_code_or_codex_integration": "Coding-agent task/dependency tracking integrations.", + "published_eval": "Long-term memory/RAG benchmark reproduction not qualified.", + "in_current_list": true, + "recommendation": "exclude", + "reason": "Retain task-tracker scope exclusion. Search-only metadata; canonical project moved from steveyegge/beads to gastownhall/beads, and no page budget remained.", + "sources": [] + } + ], + "missing_from_list_summary": "The list has substantial omissions, but this bounded audit does not certify exhaustive completeness or all admission metadata. Add_primary: SimpleMem, ReMe, MemMachine, Mem0 OSS core, claude-mem, mcp-memory-service, and HippoRAG. Add_reserve: MemOS, EverOS/EverMemOS lineage, SuperLocalMemory, memU, MemoryOS, AgentMem, RAGFlow, Serena, and the unqualified GitNexus lead. Memobase definitively fails the 90-day rule; Memori's verified current integration fails the no-mandatory-cloud requirement. Reconsidering Mem0 corrects the cloud-plugin/core conflation. Letta remains excluded by comparison scope, while its current local runtime is active. in_current_list includes names explicitly listed as excluded. Forty distinct URLs were attempted; no load exceeded one minute. The GitHub API endpoint and incorrectly guessed LadybugDB/GitNexus URL failed. GitHub CLI authentication and shell networking were unavailable. Many repository root pages omitted dynamic commit metadata, so their keep recommendations are provisional, not verified maintenance admissions. Commit dates have day precision, without verified UTC timestamps; tags are latest observed primary-source/indexed releases, not a guarantee against indexing lag. archived=false reflects inspected repository presentation, not authenticated API verification. Local capability is documentation/source-based and was not installation-tested; false for harnesses means a fully local reader/judge path was not verified. Search-only GitNexus and beads have empty sources rather than unopened URLs. Remaining unqualified discovery frontier includes LangMem, LightMem, MIRIX, A-MEM, Haystack, RAG-Anything, PageIndex, Code-RAG, and broader RAG applications. Additional harness leads include mem0ai/memory-benchmarks, supermemoryai/memorybench, MemTensor/OmniMemEval, MemoryArena, and EverMemBench; their release/activity metadata was not qualified within the page budget. Before freezing the preregistration, resolve unknown activity against the 2026-07-02 cutoff, pin exact implementations/adapters, separate retrieval recall from answer accuracy, and normalize reader/judge models, ingestion and retrieval token budgets. Current local implementations must be rerun rather than inheriting scores from cloud products, predecessors or different benchmark subsets." +} \ No newline at end of file diff --git a/blueprints/memory-layer-s3/inputs/r7/live-verification-20260930.json b/blueprints/memory-layer-s3/inputs/r7/live-verification-20260930.json new file mode 100644 index 000000000..dd8b4e028 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r7/live-verification-20260930.json @@ -0,0 +1,140 @@ +{ + "captured_utc": "2026-09-30T02:52:42Z", + "purpose": "Live verification behind blueprints/memory-layer-s3/PREREGISTRATION.md DRAFT r7: maintenance gate, release pins, changed cited files, image digests, static-data metadata, GPU budget and client versions.", + "scope": "Metadata and source reads only (GitHub REST through gh 2.101.0, PyPI JSON, Hugging Face API, GHCR and Docker Hub manifests, nvidia-smi, --version). No provider call, deployment or scored run.", + "gate": { + "rule": "stale = archived, or zero default-branch commits since window_start; the floor of the OpenSSF Scorecard Maintained check (ossf/scorecard v5.5.0 checks/evaluation/maintained.go:31-32,88-100)", + "window_start_utc": "2026-07-02T02:52:42Z", + "commands": ["gh api repos/OWNER/REPO --jq .archived", "gh api \"repos/OWNER/REPO/commits?since=WINDOW_START&per_page=1\" --jq length"], + "count_note": "commits_in_window pages the commits endpoint to at most 300; count_capped marks a count that reached the cap", + "repos": { + "columns": ["repo", "role", "archived", "default_branch", "head", "head_utc", "commits_in_window", "count_capped", "latest_release", "latest_release_utc", "verdict"], + "rows": [ + ["vectorize-io/agent-memory-benchmark", "harness", false, "main", "03c1d0f1d27d", "2026-09-22T12:23:44Z", 12, false, null, null, "ACTIVE"], + ["embeddings-benchmark/mteb", "harness", false, "main", "7fac921e8672", "2026-09-29T21:16:07Z", 300, true, "2.21.10", "2026-09-29T21:16:12Z", "ACTIVE"], + ["ossf/scorecard", "harness", false, "main", "bd19d486f4a3", "2026-09-29T20:35:19Z", 22, false, "v5.5.0", "2026-04-23T21:41:36Z", "ACTIVE"], + ["snap-research/locomo", "data", false, "main", "3eb6f2c585f5", "2024-08-13T03:21:21Z", 0, false, null, null, "STALE"], + ["xiaowu0162/LongMemEval", "data", false, "main", "9e0b455f4ef0", "2026-05-11T22:49:24Z", 0, false, null, null, "STALE"], + ["xiaowu0162/LongMemEval-V2", "diag", false, "main", "2cc8c540bdb8", "2026-08-09T22:31:22Z", 3, false, null, null, "ACTIVE"], + ["harbor-framework/harbor-datasets", "data", false, "main", "37db108843a4", "2026-05-16T19:48:38Z", 0, false, null, null, "STALE"], + ["harbor-framework/harbor", "harness", false, "main", "9b168361ebe9", "2026-09-30T02:33:35Z", 300, true, "v0.23.0", "2026-09-12T04:55:17Z", "ACTIVE"], + ["CoIR-team/coir", "data", false, "main", "89d0e769c18f", "2025-06-30T02:01:36Z", 0, false, null, null, "STALE"], + ["niieani/gpt-tokenizer", "harness", false, "main", "9182f2e56735", "2026-08-16T04:03:31Z", 11, false, "4.0.0", "2026-08-16T02:29:41Z", "ACTIVE"], + ["scipy/scipy", "stats", false, "main", "c71fcecc1bdd", "2026-09-29T17:15:49Z", 300, true, "v1.18.1", "2026-08-21T23:52:53Z", "ACTIVE"], + ["statsmodels/statsmodels", "stats", false, "main", "1767776211ac", "2026-09-29T14:19:51Z", 300, true, "v0.15.0", "2026-08-27T06:44:51Z", "ACTIVE"], + ["numpy/numpy", "stats", false, "main", "f99b1a0446b0", "2026-09-30T00:21:57Z", 300, true, "v2.5.3", "2026-09-06T16:37:15Z", "ACTIVE"], + ["diegosouzapw/OmniRoute", "gateway", false, "release/v3.8.52", "a1a2dce1a64a", "2026-09-29T22:07:49Z", 300, true, "v3.8.51", "2026-09-30T01:41:50Z", "ACTIVE"], + ["rohitg00/agentmemory", "primary", false, "main", "ab3e4efd2826", "2026-09-28T15:48:39Z", 39, false, "v0.9.29", "2026-08-16T13:56:45Z", "ACTIVE"], + ["vectorize-io/hindsight", "primary", false, "main", "eb021da3b250", "2026-09-29T17:05:35Z", 300, true, "v0.10.2", "2026-09-29T10:06:40Z", "ACTIVE"], + ["volcengine/OpenViking", "primary", false, "main", "b20f192e6139", "2026-09-29T12:29:14Z", 300, true, "v0.4.22", "2026-09-28T05:56:50Z", "ACTIVE"], + ["topoteretes/cognee", "primary", false, "main", "ba3631f2ed36", "2026-09-29T21:58:03Z", 300, true, "v1.6.2", "2026-09-29T22:00:46Z", "ACTIVE"], + ["MemPalace/mempalace", "primary", false, "develop", "5ea40d30caf0", "2026-09-29T11:22:34Z", 300, true, "v3.10.0", "2026-09-16T00:24:11Z", "ACTIVE"], + ["basicmachines-co/basic-memory", "primary", false, "main", "88c3990c9a17", "2026-09-30T02:36:10Z", 300, true, "v0.23.2", "2026-08-25T20:44:29Z", "ACTIVE"], + ["zilliztech/memsearch", "primary", false, "main", "2a4652fa086f", "2026-09-24T08:07:41Z", 79, false, "v0.4.21", "2026-09-24T08:08:42Z", "ACTIVE"], + ["akitaonrails/ai-memory", "primary", false, "main", "a0ca8d1a5fbd", "2026-09-29T16:27:08Z", 300, true, "v2.4.2", "2026-09-29T17:39:11Z", "ACTIVE"], + ["Gentleman-Programming/engram", "reserve", false, "main", "96e009abcd9a", "2026-09-30T01:47:06Z", 300, true, "v2.2.1", "2026-09-25T23:19:07Z", "ACTIVE"], + ["getzep/graphiti", "reserve", false, "main", "852ca401d89f", "2026-09-29T15:40:47Z", 116, false, "v0.30.2", "2026-09-08T20:38:41Z", "ACTIVE"], + ["supermemoryai/supermemory", "reserve", false, "main", "ce4facf662b2", "2026-09-30T01:01:29Z", 183, false, "server-v0.0.8", "2026-08-17T18:39:22Z", "ACTIVE"], + ["mem0ai/mem0", "out", false, "main", "94c3fe9f238f", "2026-09-25T17:34:12Z", 236, false, "ts-v3.3.1", "2026-09-25T17:36:35Z", "ACTIVE"], + ["letta-ai/letta", "out", false, "main", "5bcdd177d70f", "2026-09-10T17:59:06Z", 8, false, "0.16.8", "2026-05-14T17:14:24Z", "ACTIVE"], + ["gastownhall/beads", "out", false, "main", "3f9561db2bd1", "2026-09-30T01:17:57Z", 300, true, "v1.3.0", "2026-09-15T19:33:18Z", "ACTIVE"], + ["garrytan/gbrain", "late", false, "master", "608a174dcfa1", "2026-09-29T18:39:41Z", 300, true, "v0.60.10.0", "2026-09-29T18:41:54Z", "ACTIVE"], + ["tobi/qmd", "rag", false, "main", "04e4dbd8245c", "2026-09-09T20:51:54Z", 110, false, "v2.8.3", "2026-08-16T22:30:34Z", "ACTIVE"], + ["HKUDS/LightRAG", "rag", false, "main", "453dce83d6d0", "2026-09-26T08:33:35Z", 300, true, "v1.5.7", "2026-09-02T07:23:55Z", "ACTIVE"], + ["run-llama/llama_index", "rag", false, "main", "7e2c60a78ec2", "2026-09-29T04:49:06Z", 102, false, "v0.14.25", "2026-09-21T16:22:27Z", "ACTIVE"], + ["DeusData/codebase-memory-mcp", "code", false, "main", "80eb92a7017d", "2026-09-28T22:32:27Z", 300, true, "v0.11.0", "2026-09-15T23:40:39Z", "ACTIVE"] + ] + } + }, + "pins": { + "columns": ["repo", "ref", "commit", "commit_utc"], + "rows": [ + ["rohitg00/agentmemory", "v0.9.29", "2d38dafede67d0d4ed920cde94d2106e98825b8a", "2026-08-16T13:54:32Z"], + ["vectorize-io/hindsight", "v0.10.2", "5fc4ce20917b916240cef27c212c387a177f115b", "2026-09-29T09:40:54Z"], + ["vectorize-io/hindsight", "v0.10.1", "f8950b0c07d9e34c76493dba802bb309f0ce60fd", "2026-09-21T14:59:59Z"], + ["volcengine/OpenViking", "v0.4.22", "e8716760934d95ef73414484a45bfe475b9296e2", "2026-09-28T05:55:14Z"], + ["topoteretes/cognee", "v1.6.2", "ba3631f2ed363a6ea50d649c34c56885af6b36fe", "2026-09-29T21:58:03Z"], + ["topoteretes/cognee", "v1.6.1", "eb90d03740755f5252b8b12cce91fd09970f2d81", "2026-09-24T17:49:18Z"], + ["MemPalace/mempalace", "v3.10.0", "22fd87f09c19d5ffb2d6966486483353937931c0", "2026-09-16T00:22:41Z"], + ["basicmachines-co/basic-memory", "v0.23.2", "c0bd87c6d5a4a58034b1d6c8c5018e443b0bd048", "2026-08-25T20:43:45Z"], + ["zilliztech/memsearch", "v0.4.21", "2a4652fa086fbd45e92bfd8da7781ebe1642baa7", "2026-09-24T08:07:41Z"], + ["akitaonrails/ai-memory", "v2.4.2", "a0ca8d1a5fbd5920799411fa891fe6d49c90efc1", "2026-09-29T16:27:08Z"], + ["akitaonrails/ai-memory", "v2.4.0", "b1b25219b507cf56cb7334ac1a408dc9b09eaa50", "2026-09-21T22:22:36Z"], + ["akitaonrails/ai-memory", "v2.4.1", "433a19f3d54dea287571b1423591db2a89965fa9", "2026-09-25T21:01:14Z"], + ["Gentleman-Programming/engram", "v2.2.1", "818be842f57a95063f62f1b745297a96c407c143", "2026-09-25T23:04:24Z"], + ["getzep/graphiti", "v0.30.2", "eaa4128681bc53487138a4bbc22d58336ebe70d2", "2026-09-08T20:37:55Z"], + ["supermemoryai/supermemory", "server-v0.0.8", "5d2b5855fe492a3682a1cde4a255e2db0c4db595", "2026-08-16T23:20:38Z"], + ["letta-ai/letta", "0.16.8", "1131535716e8a31c9a437f8695e25ac98f203a24", "2026-05-14T16:58:54Z"], + ["gastownhall/beads", "v1.3.0", "f45b249ce6b40ba62aecc03949e6371e8f7c79d8", "2026-09-15T19:13:59Z"], + ["garrytan/gbrain", "v0.60.10.0", "608a174dcfa1d39d5ea2d8fb5b296122b1cc78c5", "2026-09-29T18:39:41Z"], + ["tobi/qmd", "v2.8.3", "facd35e01359e59d938bc9418e93fb9318addee3", "2026-08-16T22:28:32Z"], + ["HKUDS/LightRAG", "v1.5.7", "28ff1b05f2ac3f3e6fa14dd2cd33656579bd0c9c", "2026-09-02T07:21:51Z"], + ["run-llama/llama_index", "v0.14.25", "f12d46acab73f5b2243ef49c2f00101617b38ce4", "2026-09-21T16:20:40Z"], + ["embeddings-benchmark/mteb", "2.21.10", "7fac921e8672cfb6799d7c6a9a542832bc10cbb6", "2026-09-29T21:16:07Z"], + ["ossf/scorecard", "v5.5.0", "c395761df6afe1a69e476bc60a013a94bcbc153f", "2026-04-23T21:26:29Z"], + ["niieani/gpt-tokenizer", "4.0.0", "fb04ebca53f662200e737caefe9a5ef372a5e41a", "2026-08-16T02:27:39Z"], + ["scipy/scipy", "v1.18.1", "e4e854eaa8f18d807cd3496028e257e36caa93cc", "2026-08-21T17:54:36Z"], + ["statsmodels/statsmodels", "v0.15.0", "278ff9950636cdd4939b4055e339a8e681d79cab", "2026-08-27T06:30:23Z"], + ["numpy/numpy", "v2.5.3", "dd88c0c19b54ad9ed3533224221285bf0873249a", "2026-09-05T22:47:25Z"], + ["diegosouzapw/OmniRoute", "v3.8.51", "c1e30b7676975feb298b49eff6ff58923c04b89e", "2026-09-29T22:58:13Z"], + ["harbor-framework/harbor", "v0.23.0", "1e5c5c6db929a10a140d05e606882c671ae20729", "2026-09-12T04:55:13Z"], + ["DeusData/codebase-memory-mcp", "v0.11.0", "8972ea69c6ad94b1ef1d4ffbf0a92d78d2db1798", "2026-09-15T17:16:24Z"], + ["vectorize-io/agent-memory-benchmark", "03c1d0f1", "03c1d0f1d27da63034f0931121c858faba512383", "2026-09-22T12:23:44Z"], + ["xiaowu0162/LongMemEval", "9e0b455f", "9e0b455f4ef0e2ab8f2e582289761153549043fc", "2026-05-11T22:49:24Z"], + ["xiaowu0162/LongMemEval-V2", "main", "2cc8c540bdb87fe6761629b585e727e1c4704520", "2026-08-09T22:31:22Z"], + ["snap-research/locomo", "main", "3eb6f2c585f5e1699204e3c3bdf7adc5c28cb376", "2024-08-13T03:21:21Z"], + ["harbor-framework/harbor-datasets", "main", "37db108843a49bb31a592e37a75e2c40dc3f9749", "2026-05-16T19:48:38Z"], + ["CoIR-team/coir", "main", "89d0e769c18f0576a766072ba1071d4e04cca3dd", "2025-06-30T02:01:36Z"] + ] + }, + "changed_between_releases": { + "columns": ["repo", "from", "to", "path", "changed"], + "rows": [ + ["vectorize-io/hindsight", "v0.10.1", "v0.10.2", "hindsight-api-slim/hindsight_api/config.py", true], + ["vectorize-io/hindsight", "v0.10.1", "v0.10.2", "hindsight-api-slim/hindsight_api/engine/cache_affinity.py", true], + ["vectorize-io/hindsight", "v0.10.1", "v0.10.2", "hindsight-docs/docs/developer/installation.md", true], + ["vectorize-io/hindsight", "v0.10.1", "v0.10.2", "hindsight-docs/docs/developer/models.mdx", true], + ["vectorize-io/hindsight", "v0.10.1", "v0.10.2", "docker/docker-compose/claude-code/docker-compose.yaml", false], + ["topoteretes/cognee", "v1.6.1", "v1.6.2", "Dockerfile", false], + ["topoteretes/cognee", "v1.6.1", "v1.6.2", "cognee/infrastructure/llm/config.py", true], + ["topoteretes/cognee", "v1.6.1", "v1.6.2", "cognee/tasks/ingestion/data_item.py", false], + ["akitaonrails/ai-memory", "v2.4.0", "v2.4.2", "crates/ai-memory-cli/src/config.rs", true], + ["akitaonrails/ai-memory", "v2.4.0", "v2.4.2", "crates/ai-memory-llm/src/embedding.rs", false], + ["akitaonrails/ai-memory", "v2.4.0", "v2.4.2", "crates/ai-memory-llm/src/reranker.rs", false], + ["akitaonrails/ai-memory", "v2.4.0", "v2.4.2", "docs/llm-providers.md", true] + ] + }, + "images": [ + {"image": "ghcr.io/vectorize-io/hindsight:0.10.2", "digest": "sha256:d1840062a5b79940ab7a9f4809ceb90fc776d4ad737cd9329e9b5836cc64ab70"}, + {"image": "ghcr.io/vectorize-io/hindsight:0.10.1", "digest": "sha256:b4d3b76f363aa40cf348450e7f8f52a50653008731b99824b14623a196182e73", "note": "equals the r6 record (control)"}, + {"image": "cognee/cognee:1.6.2", "digest": "sha256:41c06180608a053f07fffc4da56dd1efba62967dd3d4a1023b33939abee5d45d"}, + {"image": "cognee/cognee:1.6.1", "digest": "sha256:db0973f4b913d73daa4061bc19362cde6edc59d1be8243b6667fade364b06428", "note": "equals the r6 record (control)"} + ], + "huggingface": [ + {"id": "datasets/xiaowu0162/longmemeval-cleaned", "revision": "98d7416c24c778c2fee6e6f3006e7a073259d48f", "file": "longmemeval_s_cleaned.json", "bytes": 277383467, "lfs_sha256": "d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442", "note": "registry metadata; the freeze hashes the downloaded file"}, + {"id": "datasets/mteb/LoCoMo", "revision": "02e2c3dea15d9fdfd1cd7a0f65f5f8ae2ed4c1ac", "last_modified": "2026-05-11T15:15:34.000Z", "license": "cc-by-nc-4.0"}, + {"id": "nvidia/Nemotron-3-Embed-1B-BF16", "revision": "c0c9fea93ea424587517f2c59e20db9f1d6bf615", "last_modified": "2026-08-27T23:53:03.000Z"}, + {"id": "google/embeddinggemma-300m", "revision": "57c266a740f537b4dc058e1b0cda161fd15afa75", "last_modified": "2025-09-25T15:11:17.000Z", "gated": true}, + {"id": "ggml-org/embeddinggemma-300M-GGUF", "revision": "0f741b5a6585bd53aeb15cd1372c56f2a0f65e12", "last_modified": "2026-04-29T11:39:35.000Z"}, + {"id": "Qwen/Qwen3-Reranker-0.6B", "revision": "e61197ed45024b0ed8a2d74b80b4d909f1255473", "last_modified": "2026-04-16T08:55:59.000Z"}, + {"id": "ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF", "revision": "a02f48bb4f057028298c21fa033da2b30d7742d5", "last_modified": "2025-10-03T10:19:38.000Z"}, + {"id": "sentence-transformers/all-MiniLM-L6-v2", "revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41", "last_modified": "2026-06-01T06:29:13.000Z"} + ], + "gpu": { + "captured_utc": "2026-09-30T03:13:24Z", + "name": "NVIDIA GeForce RTX 4090", + "total_mib": 24564, + "used_mib": 12138, + "free_mib": 12005, + "earlier_reading": { + "free_mib": 12726, + "captured_utc": null, + "note": "earlier reading in the same session; its time was not recorded" + } + }, + "clients": { + "captured_utc": "2026-09-30T02:50:03Z", + "claude_code": "2.1.285", + "codex_cli": "0.157.1" + } +} diff --git a/blueprints/memory-layer-s3/inputs/r7/repair1-verification-20260930.json b/blueprints/memory-layer-s3/inputs/r7/repair1-verification-20260930.json new file mode 100644 index 000000000..fff210921 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r7/repair1-verification-20260930.json @@ -0,0 +1,333 @@ +{ + "captured_utc": "2026-09-30T04:23:48Z", + "purpose": "Live verification behind the r7 repair round 1 of blueprints/memory-layer-s3/PREREGISTRATION.md (DRAFT r7): maintenance gate for the sources of the S8 code-task stage, the Harbor tag resolution, the SWE-bench Verified dataset revisions, GitHub's REST description for the CI term, the locked hindsight-all wheel, and every upstream locator the repair relies on.", + "scope": "Metadata and source reads only. GitHub reads used `gh api graphql` (gh 2.101.0) because the account's shared REST core quota refused requests at 2026-09-30T04:16Z; GraphQL returns the same archived flag, default-branch commit history and blob text at a pinned commit. Hugging Face Hub API, PyPI JSON and one raw.githubusercontent.com read at a pinned commit. No deployment, provider call or scored run.", + "gate": { + "rule": "PREREGISTRATION section 1: stale if archived or no default-branch commit in the 90 days before the check", + "window_start_utc": "2026-07-02T04:23:48Z", + "query": "repository(owner,name){isArchived defaultBranchRef{name target{... on Commit{oid committedDate history(since: WINDOW_START){totalCount}}}}}", + "repos": [ + { + "repo": "harbor-framework/harbor", + "role": "S8 harness and SWE-bench adapter (8.4)", + "archived": false, + "default_branch": "main", + "head": "9b168361ebe973113640b9183b03cbd7884d720e", + "last_commit_utc": "2026-09-30T02:33:35Z", + "commits_in_window": 526, + "verdict": "ACTIVE" + }, + { + "repo": "harbor-framework/harbor-datasets", + "role": "supplies nothing to S3 (1, 8.4)", + "archived": false, + "default_branch": "main", + "head": "37db108843a49bb31a592e37a75e2c40dc3f9749", + "last_commit_utc": "2026-05-16T19:48:38Z", + "commits_in_window": 0, + "verdict": "STALE" + }, + { + "repo": "SWE-bench/SWE-bench", + "role": "swebench package: test specs at task generation (4.1.0), the verifier's parser (4.0.3), instance images", + "archived": false, + "default_branch": "main", + "head": "02e7a74ffd0b707aab73d203fe87bdc7c76afc8e", + "last_commit_utc": "2026-09-02T01:50:34Z", + "commits_in_window": 113, + "verdict": "ACTIVE" + }, + { + "repo": "huggingface/datasets", + "role": "adapter's dataset loader; verifier parser dependency (2.16.1)", + "archived": false, + "default_branch": "main", + "head": "fa995bdaea171d50d086743364f23e9db6b557d8", + "last_commit_utc": "2026-09-28T10:04:31Z", + "commits_in_window": 62, + "verdict": "ACTIVE" + }, + { + "repo": "AnswerDotAI/fastcore", + "role": "verifier parser dependency (<1.11)", + "archived": false, + "default_branch": "main", + "head": "ebbe03da6a711f56755568ac0d13d823088dc2ea", + "last_commit_utc": "2026-09-28T07:18:32Z", + "commits_in_window": 262, + "verdict": "ACTIVE" + }, + { + "repo": "astral-sh/uv", + "role": "runs the verifier parser; installer 0.7.13 in the task image", + "archived": false, + "default_branch": "main", + "head": "06fdf0cdb591adae479feb5b769b0213eabb229c", + "last_commit_utc": "2026-09-30T02:07:14Z", + "commits_in_window": 1218, + "verdict": "ACTIVE" + } + ] + }, + "tags": [ + { + "repo": "harbor-framework/harbor", + "tag": "v0.23.0", + "tag_object": "3c305dc5611e3600afc3c818735a82322f0264f8", + "commit": "1e5c5c6db929a10a140d05e606882c671ae20729" + } + ], + "huggingface": [ + { + "id": "SWE-bench/SWE-bench_Verified", + "revision": "78f471bf655a3137b2e8a75af1501690ec009ec3", + "last_modified_utc": "2026-08-16T04:23:43.000Z", + "files": [ + ".gitattributes", + "README.md", + "data/test-00000-of-00001.parquet", + "eval.yaml" + ], + "note": "maintained dataset; this revision was also its main head at capture" + }, + { + "id": "princeton-nlp/SWE-bench_Verified", + "revision": "c104f840cc67f8b6eec6f759ebc8b2693d585d4a", + "last_modified_utc": "2025-02-18T23:48:55.000Z", + "files": [ + ".gitattributes", + "README.md", + "data/test-00000-of-00001.parquet" + ], + "note": "a separate dataset repository, not an alias; Harbor v0.23.0's swebench adapter loads it by name without a revision (adapter.py:46)" + } + ], + "pypi": [ + { + "package": "hindsight-all", + "version": "0.4.17", + "wheel": "hindsight_all-0.4.17-py3-none-any.whl", + "sha256": "480bcdb04049fc8aeeeb5b55390e63c90803ba1e52cb2c8b7a3aa270557e7a99", + "note": "AMB 03c1d0f1's locked version; the downloaded wheel matched this digest; it provides the top-level module `hindsight`, whose __init__.py:60 imports HindsightClient from client_wrapper.py:185 (class HindsightClient(Hindsight))" + } + ], + "github_rest_description": { + "repo": "github/rest-api-description", + "commit": "af2c102501766f14fe4edc688e449427a971e66a", + "committed_utc": "2026-09-29T22:35:20Z", + "file": "descriptions/api.github.com/api.github.com.json", + "bytes": 13008045, + "sha256": "bab647359529118583d35867f0fdb42ea7ea1b8a2f712590b61ba45b6d4d9b37", + "facts": [ + { + "lines": "55478-55482", + "fact": "GET /repos/{owner}/{repo}/commits/{ref}/check-runs (checks/list-for-ref): above 1000 check suites on a ref, check runs are limited to the 1000 most recent suites; parameters status (queued, in_progress, completed), filter (latest default, all), per_page (default 30, max 100), page" + }, + { + "lines": "55667-55671", + "fact": "GET /repos/{owner}/{repo}/commits/{ref}/status (repos/get-combined-status-for-ref): combined state is failure if any context reports error or failure, pending if there are no statuses or a context is pending, success if the latest status for all contexts is success; per_page (default 30, max 100), page" + }, + { + "lines": "55584", + "fact": "GET /repos/{owner}/{repo}/commits/{ref}/check-suites (checks/list-suites-for-ref): the response carries total_count and check_suites" + }, + { + "lines": "145900-145911", + "fact": "check-run status enum: queued, in_progress, completed, waiting, requested, pending" + }, + { + "lines": "145913-145925", + "fact": "check-run conclusion enum: success, failure, neutral, cancelled, skipped, timed_out, action_required; nullable" + } + ] + }, + "source_locators": [ + { + "repo": "harbor-framework/harbor-datasets", + "commit": "37db108843a49bb31a592e37a75e2c40dc3f9749", + "path": "datasets/aa-lcr/aa-lcr-1/tests/test.sh", + "lines": { + "4": "python /tests/llm_judge.py" + } + }, + { + "repo": "harbor-framework/harbor", + "commit": "1e5c5c6db929a10a140d05e606882c671ae20729", + "path": "src/harbor/verifier/verifier.py", + "lines": { + "175": "if not self._skip_tests_upload:", + "180": "await self.environment.upload_dir(", + "229": "await self.environment.exec(" + } + }, + { + "repo": "harbor-framework/harbor", + "commit": "1e5c5c6db929a10a140d05e606882c671ae20729", + "path": "adapters/swebench/src/swebench_adapter/adapter.py", + "lines": { + "46": "ds = load_dataset(\"princeton-nlp/SWE-bench_Verified\")[\"test\"]" + } + }, + { + "repo": "harbor-framework/harbor", + "commit": "1e5c5c6db929a10a140d05e606882c671ae20729", + "path": "adapters/swebench/src/swebench_adapter/utils.py", + "lines": { + "6": "from swebench.harness.constants import LATEST, MAP_REPO_VERSION_TO_SPECS", + "8": "from swebench.harness.test_spec.test_spec import make_test_spec", + "10": "extra_build_instance_images_kwargs = {\"tag\": LATEST}", + "55": "spec = make_test_spec(sample, namespace=\"swebench\")" + } + }, + { + "repo": "harbor-framework/harbor", + "commit": "1e5c5c6db929a10a140d05e606882c671ae20729", + "path": "adapters/swebench/src/swebench_adapter/task-template/tests/test.sh", + "lines": { + "4": "{test_commands}", + "10": "# dependencies = [\"swebench==4.0.3\", \"datasets==2.16.1\", \"fastcore<1.11\"]", + "82": "uv run parser.py | tee -a \"$LOG_FILE\"" + } + }, + { + "repo": "harbor-framework/harbor", + "commit": "1e5c5c6db929a10a140d05e606882c671ae20729", + "path": "adapters/swebench/src/swebench_adapter/task-template/environment/Dockerfile", + "lines": { + "18": "FROM {docker_image}", + "40": "RUN curl -LsSf https://astral.sh/uv/0.7.13/install.sh | sh" + } + }, + { + "repo": "harbor-framework/harbor", + "commit": "1e5c5c6db929a10a140d05e606882c671ae20729", + "path": "adapters/swebench/src/swebench_adapter/task-template/task.toml", + "lines": { + "18": "[verifier]", + "19": "network_mode = \"public\"" + } + }, + { + "repo": "harbor-framework/harbor", + "commit": "1e5c5c6db929a10a140d05e606882c671ae20729", + "path": "adapters/swebench/pyproject.toml", + "lines": { + "12": "\"swebench>=4.1.0\"," + } + }, + { + "repo": "vectorize-io/agent-memory-benchmark", + "commit": "03c1d0f1d27da63034f0931121c858faba512383", + "path": "src/memory_bench/memory/hindsight.py", + "lines": { + "219": "kwargs: dict = dict(enable_observations=False)", + "225": "env_mission = os.environ.get(\"AMB_RETAIN_MISSION\")", + "231": "extraction_mode = os.environ.get(\"AMB_RETAIN_EXTRACTION_MODE\")", + "243": "self._client.create_bank(bank_id=bank_id, name=f\"Benchmark Bank ({bank_id})\", **kwargs)", + "433": "await client.acreate_bank(bank_id=bank_id, name=f\"Benchmark Bank ({bank_id})\", **kwargs)", + "728": "max_tokens = 32768", + "743": "\"budget\": \"high\",", + "744": "\"max_tokens\": max_tokens,", + "1087": "retain_async=True,", + "1146": "await self._await_bank_ingest(client, bank_id)", + "1159": "response = await asyncio.wait_for(_call, timeout=300)", + "1175": "return [], None", + "1180": "return [], None", + "1242": "class HindsightHTTPMemoryProvider(HindsightCloudMemoryProvider):", + "1252": "from hindsight import HindsightClient", + "1255": "self._client = HindsightClient(base_url=self._cloud_base_url, api_key=self._cloud_api_key)" + } + }, + { + "repo": "vectorize-io/agent-memory-benchmark", + "commit": "03c1d0f1d27da63034f0931121c858faba512383", + "path": "src/memory_bench/modes/retrieval.py", + "lines": { + "34": "k = int(meta.get(\"retrieval_limit\") or 20)", + "45": "docs, raw = await memory.async_retrieve(", + "46": "query, k=k, user_id=user_id, query_timestamp=meta.get(\"query_timestamp\"), filters=filters" + } + }, + { + "repo": "vectorize-io/hindsight", + "commit": "5fc4ce20917b916240cef27c212c387a177f115b", + "path": "hindsight-api-slim/hindsight_api/config.py", + "lines": { + "1711": "DEFAULT_ENABLE_OBSERVATIONS = True # Observations enabled by default", + "1712": "DEFAULT_ENABLE_AUTO_CONSOLIDATION = True # Auto-consolidation after retain enabled by default" + } + }, + { + "repo": "vectorize-io/hindsight", + "commit": "5fc4ce20917b916240cef27c212c387a177f115b", + "path": "hindsight-docs/docs/developer/api/memory-banks.mdx", + "lines": { + "35": "You don't need to pre-create a bank. Hindsight will automatically create it with default settings when you first use it.", + "242": "Toggles observation consolidation on or off. When `false`, no consolidation runs for this bank — neither automatic nor manual. Defaults to `true` when the observations feature is enabled on the server.", + "246": "Controls whether consolidation runs automatically after retain, delete, and update operations. When `false`, consolidation only runs when explicitly triggered via the [consolidate endpoint](/developer/observations#trigger-consolidation). Defaults to `true`.", + "508": "Bank configuration fields (retain mission, extraction mode, observations mission, etc.) are managed via a **separate config API**, not the `create_bank` call. This lets you change operational settings independently from the bank's identity and disposition.", + "547": "- **`config`** — the fully resolved configuration (server defaults merged with bank overrides)", + "548": "- **`overrides`** — only the fields explicitly overridden for this bank" + } + }, + { + "repo": "vectorize-io/hindsight", + "commit": "5fc4ce20917b916240cef27c212c387a177f115b", + "path": "hindsight-docs/docs/developer/configuration.mdx", + "lines": { + "1630": "The recall request takes a `budget` parameter (`low` / `mid` / `high`, default `mid`) that maps to an integer `thinking_budget` used by every retrieval method (semantic, BM25, graph, temporal). These knobs control that mapping. They are hierarchical — overridable per bank via the [config API](#hierarchical-configuration).", + "2257": "| `HINDSIGHT_API_ENABLE_OBSERVATIONS` | Enable observation consolidation | `true` |", + "2258": "| `HINDSIGHT_API_ENABLE_AUTO_CONSOLIDATION` | Automatically trigger consolidation after retain, delete, and update operations. When `false`, consolidation only runs when explicitly triggered via the [consolidate endpoint](/developer/api/operations#consolidation). Configurable per bank. | `true` |" + } + }, + { + "repo": "vectorize-io/hindsight", + "commit": "5fc4ce20917b916240cef27c212c387a177f115b", + "path": "hindsight-api-slim/hindsight_api/api/http.py", + "lines": { + "413": "class RecallRequest(BaseModel):", + "448": "budget: Budget = Budget.MID", + "449": "max_tokens: int = 4096", + "1951": "enable_observations: bool | None = Field(", + "1989": "def get_config_updates(self) -> dict[str, Any]:", + "2020": "\"enable_observations\",", + "8365": "\"/v1/default/banks/{bank_id}\",", + "8380": "config_updates = request.get_config_updates()", + "8384": "config_updates=config_updates or None,", + "9381": "\"/v1/default/banks/{bank_id}/config\"," + } + }, + { + "repo": "vectorize-io/hindsight", + "commit": "5fc4ce20917b916240cef27c212c387a177f115b", + "path": "hindsight-docs/docs/developer/api/operations.mdx", + "lines": { + "67": "### `consolidation`", + "73": "- After every retain that added world/experience facts (gated by per-bank `enable_auto_consolidation` and `enable_observations`).", + "109": "GET /v1/default/banks/{bank_id}/operations", + "116": "| `status` | Filter by `pending`, `processing`, `completed`, `failed`, `cancelled`. |", + "117": "| `type` | Filter by `retain`, `file_convert_retain`, `consolidation`, `refresh_mental_model`, `graph_maintenance`, `webhook_delivery`. |" + } + }, + { + "repo": "vectorize-io/hindsight", + "commit": "5fc4ce20917b916240cef27c212c387a177f115b", + "path": "hindsight-docs/docs/developer/api/recall.mdx", + "lines": { + "112": "The maximum number of tokens the returned facts can collectively occupy. Defaults to `4096`. Only the `text` field of each fact is counted toward this budget — metadata, tags, entities, and other fields are not included. After reranking, facts are included in relevance order until this budget is exhausted — so you always get the most relevant memories that fit. A fact too long for the remaining budget is skipped rather than ending the selection, so shorter facts ranked behind it still come back. Hindsight is designed for agents, which think in tokens rather than result counts: set `max_tokens` to however much of your context window you want to allocate to memories.", + "163": "When enabled and `types` includes `observation`, each observation result is accompanied by the original contributing facts it was synthesized from. Source facts are returned in a top-level `source_facts` dict keyed by fact ID, and each observation result carries a `source_fact_ids` list for cross-referencing. Facts are deduplicated across observations. The `max_tokens` sub-option (default `4096`) limits the total token budget for source facts." + } + }, + { + "repo": "akitaonrails/ai-memory", + "commit": "a0ca8d1a5fbd5920799411fa891fe6d49c90efc1", + "path": "docs/llm-providers.md", + "lines": { + "338": "untrusted data. A timeout, provider error, or incomplete/invalid score set", + "339": "preserves the normal order. `global=true` and supplemental global-preference", + "340": "hits keep their existing non-RRF ranking. Concurrent provider calls are capped", + "341": "at four; saturated queries keep their local ranking without waiting." + } + } + ] +} diff --git a/blueprints/memory-layer-s3/inputs/r7/repair2-verification-20260930.json b/blueprints/memory-layer-s3/inputs/r7/repair2-verification-20260930.json new file mode 100644 index 000000000..eef73bf43 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r7/repair2-verification-20260930.json @@ -0,0 +1,116 @@ +{ + "schema_version": 1, + "date": "2026-09-30", + "base_commit": "a266a3831c32de3fb9ee907a0a6d7092548182c9", + "evidence_classes": ["local pinned-source review", "synthetic discrete examples", "synthetic negative controls"], + "scope": "Repair-round-2 protocol and decision-record changes; no installation, deployment, model run or fresh online maintenance gate.", + "network_requests": 0, + "source_observation": { + "method": "Read the supplied local upstream checkouts; verify each HEAD and commit date with rtk git show -s --format='%H %cI'; inspect the cited source at that checkout. No upstream checkout was edited or executed.", + "pins": [ + {"repo": "Gentleman-Programming/engram", "commit": "818be842f57a95063f62f1b745297a96c407c143", "commit_date": "2026-09-25T23:04:24+00:00"}, + {"repo": "HUST-AI-HYZ/MemoryAgentBench", "commit": "538026089d1a8a8eff05121d0db89b388f360eba", "commit_date": "2026-09-25T02:11:33-04:00"}, + {"repo": "MemPalace/mempalace", "commit": "22fd87f09c19d5ffb2d6966486483353937931c0", "commit_date": "2026-09-15T21:22:41-03:00"}, + {"repo": "ZexueHe/MemoryArena", "commit": "6cd9de14b71915e39ac742a20dc33785e14b6aab", "commit_date": "2026-05-31T20:10:42-04:00"}, + {"repo": "akitaonrails/ai-memory", "commit": "a0ca8d1a5fbd5920799411fa891fe6d49c90efc1", "commit_date": "2026-09-29T13:27:08-03:00"}, + {"repo": "basicmachines-co/basic-memory", "commit": "c0bd87c6d5a4a58034b1d6c8c5018e443b0bd048", "commit_date": "2026-08-25T15:43:45-05:00"}, + {"repo": "getzep/graphiti", "commit": "eaa4128681bc53487138a4bbc22d58336ebe70d2", "commit_date": "2026-09-08T16:37:55-04:00"}, + {"repo": "rohitg00/agentmemory", "commit": "2d38dafede67d0d4ed920cde94d2106e98825b8a", "commit_date": "2026-08-16T14:54:32+01:00"}, + {"repo": "supermemoryai/supermemory", "commit": "5d2b5855fe492a3682a1cde4a255e2db0c4db595", "commit_date": "2026-08-16T23:20:38+00:00"}, + {"repo": "topoteretes/cognee", "commit": "ba3631f2ed363a6ea50d649c34c56885af6b36fe", "commit_date": "2026-09-29T23:58:03+02:00"}, + {"repo": "vectorize-io/agent-memory-benchmark", "commit": "03c1d0f1d27da63034f0931121c858faba512383", "commit_date": "2026-09-22T14:23:44+02:00"}, + {"repo": "vectorize-io/hindsight", "commit": "5fc4ce20917b916240cef27c212c387a177f115b", "commit_date": "2026-09-29T11:40:54+02:00"}, + {"repo": "volcengine/OpenViking", "commit": "e8716760934d95ef73414484a45bfe475b9296e2", "commit_date": "2026-09-28T13:55:14+08:00"}, + {"repo": "xiaowu0162/LongMemEval-V2", "commit": "2cc8c540bdb87fe6761629b585e727e1c4704520", "commit_date": "2026-08-09T15:31:22-07:00"}, + {"repo": "zilliztech/memsearch", "commit": "2a4652fa086fbd45e92bfd8da7781ebe1642baa7", "commit_date": "2026-09-24T16:07:41+08:00"} + ], + "maintenance_boundary": "Commit dates of supplied pins are observed. The round-2 brief supplies active LongMemEval-V2/MemoryAgentBench and stale MemoryArena classifications; that repair did not refresh archival/default-branch metadata. Round 2 required maintenance receipts for executing dependencies; ACCEPTED user decision 3 in round 3 supersedes that scope: gate selected repositories, require dependency SBOM/OSV acceptance and flag stale dependencies without blocking their harness. MemoryArena code is never run.", + "observations": [ + { + "finding": "H1/M4", + "sources": ["vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:363", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:385", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:459", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:474", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:492"], + "observed": "Per-unit async gather may abort before a final row/save. Native output path is output_dir/dataset/run_name/mode/split.json; final saves merge/filter/write the file. The S3 collector must reconcile failures to the frozen denominator, and each tuple/attempt must own its output_dir." + }, + { + "finding": "M2", + "sources": ["vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/__init__.py:1", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/bm25.py:1", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/mem0.py:5", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/hybrid_search.py:5", "topoteretes/cognee@ba3631f2:cognee/infrastructure/databases/vector/embeddings/config.py:19", "zilliztech/memsearch@2a4652fa:src/memsearch/core.py:62", "zilliztech/memsearch@2a4652fa:src/memsearch/core.py:274", "rohitg00/agentmemory@2d38dafe:src/providers/_openai-shared.ts:25", "MemPalace/mempalace@22fd87f0:mempalace/__init__.py:59"], + "observed": "Provider imports execute BM25/Mem0/Qdrant/sentence-transformers code even when those providers implement no arm. Default OpenAI endpoints/providers and logger-only telemetry suppression cannot establish an egress boundary." + }, + { + "finding": "M3/L3", + "sources": ["vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/modes/rag.py:49", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/base.py:54", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/dataset/longmemeval.py:131", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/dataset/longmemeval.py:296", "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:228"], + "observed": "RAG omits k; base async k defaults to 10; the answer prompt JSON-serializes truthy raw_response and otherwise uses context. Stock adapter exposes question_id as user_id. Empty open-QA context is scored incorrect before judge dispatch." + }, + { + "finding": "M5/M6", + "sources": ["vectorize-io/hindsight@5fc4ce20:hindsight-api-slim/hindsight_api/api/http.py:448", "rohitg00/agentmemory@2d38dafe:src/providers/_openai-shared.ts:148", "MemPalace/mempalace@22fd87f0:mempalace/llm_client.py:347"], + "observed": "Hindsight default recall budget is mid, token limit 4096. Selected agentmemory auth-header path supplies native auth headers; MemPalace's compatible client sends temperature 0.1 and auth-only headers. Header exemptions/routes have not been measured." + }, + { + "finding": "H2/F7/L4", + "sources": ["xiaowu0162/LongMemEval-V2@2cc8c540:README.md:41", "xiaowu0162/LongMemEval-V2@2cc8c540:README.md:50", "xiaowu0162/LongMemEval-V2@2cc8c540:README.md:114", "xiaowu0162/LongMemEval-V2@2cc8c540:README.md:191", "xiaowu0162/LongMemEval-V2@2cc8c540:README.md:219-223", "xiaowu0162/LongMemEval-V2@2cc8c540:README.md:236", "xiaowu0162/LongMemEval-V2@2cc8c540:memory_modules/memory.py:25", "xiaowu0162/LongMemEval-V2@2cc8c540:memory_modules/memory.py:146", "xiaowu0162/LongMemEval-V2@2cc8c540:memory_modules/memory.py:170", "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:128", "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:190", "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:193-196", "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:70", "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:126", "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:335", "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/qa_eval_metrics.py:491", "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/scripts/run_no_retrieval.sh:20", "HUST-AI-HYZ/MemoryAgentBench@538026089:README.md:178", "HUST-AI-HYZ/MemoryAgentBench@538026089:bash_files/sh/run_memagent_rag_agents.sh:18"], + "observed": "LongMemEval-V2 supports registered Memory backends and direct memory-config harness use. JSON alone requires an already registered type. Baseline scripts own domain/tier/output arguments and invoke run_eval. That wrapper does not forward evaluator base URL; direct harness does. Evaluator effort choices are low/medium/high. Native no-base evaluator constructs the OpenAI SDK client. TTL/CR metrics and the actual MemoryAgentBench sh-directory script are verified." + }, + { + "finding": "N1", + "sources": ["akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-mcp/src/server.rs:2114", "akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-mcp/src/server.rs:2133", "akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-mcp/src/server.rs:2157", "akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-mcp/src/server.rs:2176", "akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-mcp/src/server.rs:6004"], + "observed": "After provider success, the stage rejects a missing, duplicate, unknown-ID, nonfinite or out-of-range score set; it emits 'reranker returned incomplete or invalid scores; keeping pre-rerank order'. Saturation/error/timeout have separate signals. The referenced upstream unit test is read, not run." + } + ], + "deployment_review": { + "lead": "Coordinator-supplied deployment-spec review; only upstream locators, not the private input location, are published.", + "recipes_source_supported_as_specified": ["agentmemory", "hindsight", "memsearch", "basic-memory", "openviking", "engram"], + "corrections_verified_in_pinned_source": [ + {"system": "cognee", "sources": ["topoteretes/cognee@ba3631f2:cognee/api/client.py:132", "topoteretes/cognee@ba3631f2:cognee/api/client.py:297", "topoteretes/cognee@ba3631f2:cognee-mcp/src/cognee_client.py:84", "topoteretes/cognee@ba3631f2:cognee/shared/utils.py:400", "topoteretes/cognee@ba3631f2:cognee/version.py:22"], "correction": "Explicitly provision auth for the selected authenticated deployment, configure x-api-key MCP auth, disable API and MCP telemetry, and expect the source-build -local version suffix."}, + {"system": "supermemory-local", "sources": ["supermemoryai/supermemory@5d2b5855:apps/docs/concepts/how-it-works.mdx:109", "supermemoryai/supermemory@5d2b5855:apps/docs/ingestion/add-memories.mdx:73"], "correction": "Native status includes failed; irrecoverable errors may delete a document. Polls must terminate on failed/404/deadline and verify searchable evidence. Supplied review's isolated bind/auth gap remains pending."}, + {"system": "mempalace", "sources": ["MemPalace/mempalace@22fd87f0:deploy/docker-compose.server.yml:47", "MemPalace/mempalace@22fd87f0:mempalace/palace/backend.py:41", "MemPalace/mempalace@22fd87f0:mempalace/cli/cmd_serve.py:145", "MemPalace/mempalace@22fd87f0:mempalace/cli/cmd_init.py:110", "MemPalace/mempalace@22fd87f0:mempalace/cli/cmd_init.py:178"], "correction": "Init/mine and server must share compose project, store, backend and writable corpus files; helpers require the explicit backend. Retain the native token guard."}, + {"system": "ai-memory", "sources": ["akitaonrails/ai-memory@a0ca8d1a:docker/docker-compose.yml:13", "akitaonrails/ai-memory@a0ca8d1a:docker/docker-compose.yml:25", "akitaonrails/ai-memory@a0ca8d1a:docker/docker-compose.yml:26", "akitaonrails/ai-memory@a0ca8d1a:README.md:288", "akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-cli/src/commands/serve.rs:2497"], "correction": "Build the local pinned image and use upstream docker run env-file wiring with a new name/volume/free port; avoid occupied 49374 and live 49474."}, + {"system": "graphiti", "sources": ["getzep/graphiti@eaa41286:mcp_server/docker/Dockerfile.standalone:46", "getzep/graphiti@eaa41286:mcp_server/docker/README.md:92", "getzep/graphiti@eaa41286:mcp_server/docker/docker-compose-neo4j.yml:3", "getzep/graphiti@eaa41286:mcp_server/docker/docker-compose-neo4j.yml:38", "getzep/graphiti@eaa41286:mcp_server/src/graphiti_mcp_server.py:1188"], "correction": "Pin core build version and dependency/image identities, use in-network bolt://neo4j:7687, pull and pin Neo4j before pull-never, and use the actual mounted config path."} + ], + "boundary": "Recipes and corrections are source-reviewed only; no installation, readiness, lifecycle, network or gateway acceptance is claimed. The eleven-system deployment section retains source locators for the six available recipes as well." + } + }, + "synthetic_checks": { + "source": "blueprints/memory-layer-s3/PREREGISTRATION.md", + "source_revision": "0d027e06902f25d6448efbcc19ddbe9431b571f8", + "method": "Extract each named python fenced block without its fence and trailing newline; execute verbatim with rtk python3 -c BLOCK. Compare the 7.2-7.3 block to rtk proxy git show HEAD:blueprints/memory-layer-s3/PREREGISTRATION.md. Each negative control changes only the named rule and must raise AssertionError.", + "positive": [ + {"section": "7.1", "sha256": "b4c82874cf3e08ec6f50090e35c91135b07525ccdbc793b64b294badad3af91f", "exit_code": 0, "stdout": "7.1 usefulness examples: all assertions hold", "stderr": ""}, + {"section": "7.2-7.3", "sha256": "541b59b2161848ef21dcf7e89ef46d6f85d722bdf390847c26727625b3b406bf", "byte_identical_to_base": true, "exit_code": 0, "stdout": "7.2-7.3 frozen examples: all assertions hold", "stderr": ""}, + {"section": "7.5", "sha256": "5e801ce8666ae6145bea1a2e9a07862d1b42db17c226883238671f5299c1b7ef", "exit_code": 0, "stdout": "7.5 successful-transport invalid-score example: fallback observed; unknown is pending", "stderr": ""} + ], + "negative": [ + {"mutation": "7.2 all leaders changed to any leader", "exit_code": 1, "observed_error": "AssertionError"}, + {"mutation": "7.3 strict tied-cost inequality changed to inclusive", "exit_code": 1, "observed_error": "AssertionError"}, + {"mutation": "7.1 not-worse-than-all-best changed to any best", "exit_code": 1, "observed_error": "AssertionError"}, + {"mutation": "7.5 successful gateway completion falsely implies accepted stage output", "exit_code": 1, "observed_error": "AssertionError"} + ], + "boundary": "Synthetic examples verify discrete protocol logic. They are not unchanged upstream tests, bootstrap/Holm execution, actual gateway successes, native provider fallback observation, benchmark scores or model runs." + }, + "publication_check_correction": { + "evidence_class": "local structural validation; retained failed attempt", + "command": "rtk python3 scripts/validate.py", + "exit_code": 1, + "stdout": "Publication validation failed:\nblueprints/memory-layer-s3/PREREGISTRATION.md: contains possible personal home path\n", + "stderr": "", + "cause": "The publication scanner matched an upstream relative documentation directory named home inside a source locator; it was not an absolute private host path. An earlier substring-only privacy check made the same false-positive classification.", + "correction": "Use the equivalent named-provider configuration table at zilliztech/memsearch@2a4652fa:docs/cli.md:303, verified directly in the pinned source. No source claim or private-path policy was relaxed; validation is rerun after re-registration." + }, + "residuals": [ + "Native provider/collector/registration-bootstrap implementations and parity/failure/depth/output-ownership acceptance are not built/run in this protocol repair.", + "Current selected-repository maintenance gates, dependency SBOM/OSV acceptance, all-wire/outcome coverage and network-isolation receipts must be supplied before scored execution; AMB remains pending for missing acceptance, not dependency staleness alone (accepted round-3 scope).", + "agentmemory and MemPalace lack the required verified header exemption/per-arm route; supermemory-local's isolated bind/auth boundary remains unresolved.", + "The N1 actual successful-gateway invalid-score native fault control remains pending; only the source-backed synthetic rule has run.", + "LongMemEval-V2 data/subset, preflight cost/power, native backend and five-replication results are pending; secondary MemoryAgentBench has not run.", + "Joint embedder/reranker/local-LLM serving residency has not been measured.", + "cx/GPT-6 price mappings are unpriced; any cost-dependent tie is unresolved, without a zero or partial-total substitution.", + "S8's separately frozen do-no-harm design and relayed 36-task receipt remain pending; A17 supersession is outside the allowed paths." + ], + "round3_locator_corrections": { + "evidence_class": "local pinned-source re-read; correction of source locators, not a replay of round-2 checks", + "judge_effort": {"previous": "evaluation/harness.py:188 or :190", "corrected": "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:193-196", "observed": "evaluator reasoning-effort choices are low, medium, high; :188 is the section comment and :190 is evaluator base URL"}, + "privacy": {"previous": "README.md:191 used for the query privacy claim", "corrected": "xiaowu0162/LongMemEval-V2@2cc8c540:README.md:219-223", "observed": "query context has only random run-local query_invocation_id; evaluator keeps question metadata, gold and configuration private"}, + "question_ids": {"previous": "evaluation/run_eval.py:69", "corrected": "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:70", "observed": ":69 is --limit; :70 is --question-ids"}, + "historical_checks": "Round-2 synthetic results/hashes are retained at source_revision; round-3 changed examples and actual outputs are recorded separately in repair3-verification-20260930.json." + } +} diff --git a/blueprints/memory-layer-s3/inputs/r7/repair3-verification-20260930.json b/blueprints/memory-layer-s3/inputs/r7/repair3-verification-20260930.json new file mode 100644 index 000000000..161e0aacd --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/r7/repair3-verification-20260930.json @@ -0,0 +1,323 @@ +{ + "schema_version": 1, + "date": "2026-09-30", + "base_commit": "0d027e06902f25d6448efbcc19ddbe9431b571f8", + "evidence_classes": [ + "local pinned-source review", + "synthetic discrete examples", + "synthetic negative controls", + "synthetic native source-function/CLI-template check" + ], + "scope": "Repair-round-3 protocol, decision-record and locator corrections; landscape completeness deferred by the current instruction. No installation, deployment, downloaded-data verification, model run or fresh online maintenance/vulnerability gate.", + "network_requests": 0, + "source_observation": { + "method": "Read the supplied local upstream checkouts; verify HEAD/commit dates with rtk git show -s --format='%H %cI'; inspect original source at those pins. The template check extracts unchanged native parser/request-function segments into a synthetic namespace. No full upstream runner or provider is executed, and no upstream checkout is edited.", + "pins": [ + { + "repo": "xiaowu0162/LongMemEval-V2", + "commit": "2cc8c540bdb87fe6761629b585e727e1c4704520", + "commit_date": "2026-08-09T15:31:22-07:00" + }, + { + "repo": "HUST-AI-HYZ/MemoryAgentBench", + "commit": "538026089d1a8a8eff05121d0db89b388f360eba", + "commit_date": "2026-09-25T02:11:33-04:00" + }, + { + "repo": "vectorize-io/agent-memory-benchmark", + "commit": "03c1d0f1d27da63034f0931121c858faba512383", + "commit_date": "2026-09-22T14:23:44+02:00" + }, + { + "repo": "dorianbrown/rank_bm25", + "commit": "47aa3ddf8dc1ebeb7ef4e65f2b4536af44594099", + "commit_date": "2024-10-08T14:30:42+02:00" + } + ], + "observations": [ + { + "finding": "1: LME2 control/arm invocation parity", + "sources": [ + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:75-78", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:188-189", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:297-354", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:153-199", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:853-885", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/qa_eval_metrics.py:517-530", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/memory_configs/no_retrieval.json:1-4", + "xiaowu0162/LongMemEval-V2@2cc8c540:data/public_data.py:79-124" + ], + "observed": "The wrapper forwards reader temperature 0.6, top-p 0.95, top-k 20 and concurrency 16, but not reader reasoning effort or evaluator base URL. The native no_retrieval JSON matches run_eval's built configuration. Per-domain question selection rejects IDs absent in that domain. All arms/control now use one registration bootstrap calling the unchanged direct harness with identical reader/judge argv, omitted sampling overrides, explicit evaluator base URL and a mandatory actual request-body wire check." + }, + { + "finding": "2: independent resampling unit", + "sources": [ + "xiaowu0162/LongMemEval-V2@2cc8c540:data/public_data.py:51-64", + "xiaowu0162/LongMemEval-V2@2cc8c540:data/public_data.py:110-124", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:1169-1170", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:1180-1213", + "xiaowu0162/LongMemEval-V2@2cc8c540:leaderboard/README.md:149-150" + ], + "observed": "The loader maps each question to trajectory IDs; the native harness shares one memory for a common haystack. Leaderboard points preserve the domain haystacks. The protocol now fixes full-tier connected components of shared trajectories/content/native memory before subsets/results, verifies them against downloaded bytes, and requires at least 20 represented independent components. Under the minimum the lane stays pending; no question-level substitute or seed multiplication is allowed.", + "boundary": "The minimum 20 is a preregistered design safeguard, not an upstream requirement or certified statistical sufficiency. No downloaded haystacks or actual cluster count were verified in this source-only repair; the likely two-domain shared-haystack design remains pending." + }, + { + "finding": "3: open-QA context cap", + "sources": [ + "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/modes/rag.py:49-54", + "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/dataset/longmemeval.py:131-132" + ], + "observed": "RAG omits k and formats every returned document, so retrieval escalation can enlarge the answer prompt. Open QA now returns the shortest prefix of the final native ranking covering five distinct sessions, capped at 20 items, with raw_response=None. It reports exact formatted context tokens per question/seed/arm separately from native reader input tokens. Retrieval-only scoring keeps the full final ranking." + }, + { + "finding": "4: MemoryAgentBench applicability", + "sources": [ + "HUST-AI-HYZ/MemoryAgentBench@538026089:agent.py:64-89", + "HUST-AI-HYZ/MemoryAgentBench@538026089:agent.py:236-255", + "HUST-AI-HYZ/MemoryAgentBench@538026089:agent.py:518-531", + "HUST-AI-HYZ/MemoryAgentBench@538026089:methods/agentmemory.py:3-17", + "HUST-AI-HYZ/MemoryAgentBench@538026089:methods/agentmemory.py:57-82", + "HUST-AI-HYZ/MemoryAgentBench@538026089:methods/agentmemory.py:111-190", + "HUST-AI-HYZ/MemoryAgentBench@538026089:README.md:178-180" + ], + "observed": "Dispatch supports letta, mem0, cognee, zep, knowl, agentmemory, total_agent_memory, rag_* and long-context controls. The agentmemory path uses the external REST service and normalized numbered facts for CR; TTL validity is not established for that parser. Cognee uses the benchmark's vendored directory, not the selected 1.6.2 release; Zep constructs the cloud client, not Graphiti. The per-system table restricts secondary results to applicable native modes and prohibits vendored candidate copies. No MAB run or unsupported-system adapter is claimed." + }, + { + "finding": "5-7: discrete decision corrections", + "sources": [ + "blueprints/memory-layer-s3/PREREGISTRATION.md:section 4.5", + "blueprints/memory-layer-s3/PREREGISTRATION.md:section 6", + "blueprints/memory-layer-s3/PREREGISTRATION.md:section 7.1", + "blueprints/memory-layer-s3/PREREGISTRATION.md:section 7.6" + ], + "observed": "Best is computed only over operationally eligible, completely observed memory systems. Baseline final failures above 5% make the whole usefulness family pending, with p=1 placeholders. Insufficient clusters/comparison coverage and pending contenders that could change qualification or primary selection also leave the decision pending/unresolved. Synthetic examples/negative controls exercise eligible-best, cluster/failure boundaries, pending higher/equal/unknown scorers and pending qualification." + }, + { + "finding": "8: corrected locators", + "sources": [ + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/harness.py:193-196", + "xiaowu0162/LongMemEval-V2@2cc8c540:README.md:219-223", + "xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:69-70" + ], + "observed": "Judge reasoning-effort choices are low/medium/high at harness.py:193-196. Query privacy is README.md:219-223. run_eval.py:69 is --limit; :70 is --question-ids. The preregistration, decision record and round-2 receipt are corrected; historical round-2 synthetic outputs remain bound to source_revision." + }, + { + "finding": "User decision 3: dependency maintenance scope", + "sources": [ + "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/__init__.py:2", + "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/bm25.py:1", + "dorianbrown/rank_bm25@47aa3ddf8dc1ebeb7ef4e65f2b4536af44594099", + "native-agent-stack@0d027e06:.github/workflows/security-scan.yml:46-86" + ], + "observed": "Accepted user decision 3 replaces the executing-transitive maintenance gate: selected repositories pass the 90-day Maintained rule; transitive dependencies require a clean OSV scan and per-arm SBOM. A stale dependency is flagged without blocking its harness. AMB imports rank_bm25 0.2.2; supplied source last commit 2024-10-08 is a stale flag. Other dependency versions are not declared stale from age alone. Repository CI pins OSV-Scanner 2.6.0 and scans lockfiles with scan source --no-resolve.", + "scanner_discovery": { + "command": "rtk osv-scanner --version", + "exit_code": 127, + "stdout": "", + "stderr": "[rtk: No such file or directory (os error 2)]\n" + }, + "boundary": "The scanner command was not found in this worker environment. No online gate, advisory download, dependency scan, clean vulnerability result or installation is claimed. AMB is pending for missing dependency SBOM/OSV acceptance, not the rank_bm25 stale flag." + } + ] + }, + "accepted_decision": { + "id": 3, + "date": "2026-09-30", + "status": "ACCEPTED", + "relayed_user_words": "with best sota practice you recommend, make sure all repos candidates are latest sota", + "scope": "90-day Maintained for repositories selected to run; no-known-vulnerability OSV acceptance and per-arm SBOM for transitive dependencies; stale dependencies flagged without blocking the harness", + "stale_transitive_flags": [ + { + "package": "rank_bm25", + "version": "0.2.2", + "repo": "dorianbrown/rank_bm25", + "commit": "47aa3ddf8dc1ebeb7ef4e65f2b4536af44594099", + "last_observed_commit": "2024-10-08", + "importer": "vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/bm25.py:1", + "blocks_harness_by_staleness": false + } + ] + }, + "synthetic_checks": { + "command": "rtk python3 -c BLOCK (verbatim fenced examples; one mutation per negative control)", + "positive": [ + { + "section": "4.5", + "sha256": "3d458a5c1d74faa0aa2c44111ff15ae0f3033fc99d8abb1689b3eddabb1804f0", + "exit_code": 0, + "stdout": "4.5 open-QA prefix examples: all assertions hold\n", + "stderr": "" + }, + { + "section": "7.1", + "sha256": "6b7778260d09f852653a9f9ea93b8bf13af009f7aa986ce0981943e9692dc1fd", + "exit_code": 0, + "stdout": "7.1 usefulness examples: all assertions hold\n", + "stderr": "" + }, + { + "section": "7.2-7.3", + "sha256": "541b59b2161848ef21dcf7e89ef46d6f85d722bdf390847c26727625b3b406bf", + "exit_code": 0, + "stdout": "7.2-7.3 frozen examples: all assertions hold\n", + "stderr": "", + "byte_identical_to_base": true + }, + { + "section": "7.5", + "sha256": "5e801ce8666ae6145bea1a2e9a07862d1b42db17c226883238671f5299c1b7ef", + "exit_code": 0, + "stdout": "7.5 successful-transport invalid-score example: fallback observed; unknown is pending\n", + "stderr": "", + "byte_identical_to_base": true + }, + { + "section": "7.6", + "sha256": "48ab67635c85abc791eda43f93f6401bc51b22461736fec9de77c3c1ec9b9f53", + "exit_code": 0, + "stdout": "7.6 pending-contender examples: all assertions hold\n", + "stderr": "" + } + ], + "negative": [ + { + "section": "7.2-7.3", + "mutation": "all leaders changed to any leader", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 34, in \n assert tie_break_set(q, {(\"C\", \"A\")}) == ({\"A\", \"B\"}, {\"A\", \"B\", \"D\"})\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.2-7.3", + "mutation": "strict cost inequality changed to inclusive", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 47, in \n assert cost_tied({\"A\": F(8), \"B\": F(10)}) == {\"A\"}\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.1", + "mutation": "not-worse-than-all-best changed to any best", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 30, in \n assert usefulness_status(\"C\", tied, {(\"no_retrieval\", \"C\"), (\"C\", \"B\")}, **tied_ready) == \"unqualified\"\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.1", + "mutation": "eligible-best filter removed", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 33, in \n assert usefulness_status(\"A\", ineligible_high, losses | {(\"A\", \"X\")}, **ready) == \"qualified\"\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.1", + "mutation": "minimum cluster guard removed", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 40, in \n assert usefulness_status(\"A\", scores, losses, eligible={\"A\", \"B\"}, cluster_count=19) == \"pending\"\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.1", + "mutation": "baseline failure guard removed", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 39, in \n assert usefulness_status(system, scores, losses, baseline_failure_rate=F(51, 1000), **ready) == \"pending\"\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "4.5", + "mutation": "20-item prompt cap removed", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 16, in \n assert len(prefix) == 20 and short\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "4.5", + "mutation": "shortest-prefix stopping removed", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 12, in \n assert [item[\"id\"] for item in prefix] == list(range(13)) and not short\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.5", + "mutation": "gateway completion falsely implies accepted stage output", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 21, in \n assert observed_stage_outcome(ids, invalid, gateway_completed=True) == \"fallback:invalid_output\"\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.6", + "mutation": "pending-contender guard removed", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 20, in \n assert fallback_status(scores, {\"A\", \"B\"}, {\"P\"}) == (\"pending\", set())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.6", + "mutation": "pending exact tie ignored", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 21, in \n assert fallback_status({**scores, \"P\": F(8, 10)}, {\"A\", \"B\"}, {\"P\"}) == (\"pending\", set())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.6", + "mutation": "unknown pending primary ignored", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 22, in \n assert fallback_status(scores, {\"A\", \"B\"}, {\"unknown\"}) == (\"pending\", set())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + }, + { + "section": "7.6", + "mutation": "pending qualification ignored", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 24, in \n assert fallback_status({**scores, \"P\": F(7, 10)}, {\"A\", \"B\"}, {\"P\"},\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n pending_qualification=True) == (\"pending\", set())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError\n" + } + ], + "result": "5 positive blocks passed; 13 negative controls failed as expected; frozen 7.2-7.3 and 7.5 blocks byte-identical to base" + }, + "native_source_function_check": { + "evidence_class": "synthetic source-function and CLI-template check; not an unchanged upstream test or native wire acceptance", + "source": "xiaowu0162/LongMemEval-V2@2cc8c540bdb87fe6761629b585e727e1c4704520", + "functions": [ + { + "name": "parse_args", + "sha256": "427ff2eb610b241ffb58de5943f34fe860bf33bef6027695d5a3163458df0246" + }, + { + "name": "build_extra_body", + "sha256": "b37b51dec8d791d912e5a9907b045c74d2964ff980890028f4b5c2463dc9ad95" + }, + { + "name": "build_reader_request", + "sha256": "2aea3e2cb89d8acf1a75d838882e5cbba6bc514dfa222b737f9dc2d49c2af4f2" + }, + { + "name": "_call_chat_completion", + "sha256": "f6a6bff328ed69c61d7ba9e850761d871ec394a6aba81d7e1e2d9b2b2e48a970" + } + ], + "template_sha256": "b3d3f354d948a0c90e54b7473370d011f015431d5c2f12263ccb1a5c1517e109", + "command": "rtk python3 -c CODE (unchanged source function segments, native parser argv from 5.5 template, synthetic client)", + "positive": { + "exit_code": 0, + "stdout": "5.5 native parser/request-function check: common control/arm parameters accepted; no wire or model call\n", + "stderr": "" + }, + "negative": { + "mutation": "inject wrapper temperature 0.6 and top-k 20 into common argv", + "exit_code": 1, + "stdout": "", + "stderr": "Traceback (most recent call last):\n File \"\", line 171, in \n assert params == expected, params\n ^^^^^^^^^^^^^^^^^^\nAssertionError: {'model': 'cx/gpt-6-astra-max', 'timeout': 43200.0, 'max_tokens': 20000, 'reasoning_effort': 'high', 'temperature': 0.6, 'extra_body': {'top_k': 20}}\n" + }, + "boundary": "Functions are extracted from pinned source into a synthetic namespace; a fake SDK client captures kwargs. No native backend, upstream full runner, gateway request, provider effort or model was run." + }, + "evidence_boundary": "Synthetic outputs verify source-derived argv and discrete protocol decisions only. They are not unchanged upstream tests, a bootstrap/Holm run, downloaded-file cluster verification, real gateway request-body/effective-effort observation, benchmark outcomes or model runs.", + "residuals": [ + "Landscape completeness is deferred as instructed; no candidates are added or removed by a new landscape sweep.", + "Downloaded LongMemEval-V2 haystack/trajectory files and the outcome-independent component-count preflight are pending. A small-tier design with about two shared haystacks cannot qualify under the minimum 20; no alternative question-level test is authorized.", + "Native backend registration/bootstrap/collector and open-QA provider-prefix implementations, per-domain materialization, actual request-body parity, retry/failure accounting and paired main-usefulness results remain required before scoring.", + "Fresh selected-repository maintenance receipts and per-arm dependency SBOM/OSV acceptance are pending; no clean scan is claimed.", + "MemoryAgentBench has not run; agentmemory is applicable only to normalized-fact CR pending native acceptance. Unsupported/vendored modes do not supply selected-system secondary evidence.", + "Header-limited gateway routes, supermemory-local isolated bind/auth, native successful-gateway invalid-score fault control and joint serving-model residency remain pending.", + "cx/GPT-6 prices remain unpriced; cost-dependent ties are unresolved. S8's amendment/36-task receipt and the out-of-scope A17 update remain outstanding." + ] +} diff --git a/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/README.md b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/README.md new file mode 100644 index 000000000..9a9f1d9a9 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/README.md @@ -0,0 +1,29 @@ + + +**S3 section 2 input from `mac-coordinator-64gb-20260925` (2026-09-27): five verified sweep tables** + +The five tables follow as separate comments. Each suggests a path under `inputs/sweep-20260927-mac/`. Every table came from a Sonnet sweep against primary sources: the Hugging Face API, the model cards and LICENSE files, the llama.cpp source at b11146 and b11214, registry JSON, and the GitHub releases API. An independent Opus pass then checked the claims that decide each ranking, and the tables were corrected. Each table's "Verification" section lists the corrections and their sources. Per the S3 merit rules, the published and self-reported scores here (LMEB, AA-LCR, vendor cards, and the MemReranker paper's Table 6) are discovery evidence, not S3 evidence. + +What changes inputs: + +- **Memory LLM (consolidation/rerank).** The AA-LCR v1.1 scores are recoverable from Artificial Analysis's page data. The mapping reproduces the baseline's own figures exactly (Qwen3.5-9B 46.0/70.0, Qwen3.6-35B-A3B 64.3/71.7, Nemotron 60.3). + - Non-reasoning: Qwen3.8-27B 69.3, trial 64.3, control 46.0, Gemma-4-26B-A4B 42.3. + - Reasoning: Qwen3.8-27B 82.0, trial 71.7, control 70.0, Gemma 65.7, granite-4.2-30b 49.0, granite-4.2-8b 45.0. + - Qwen3.8-27B's open risk is ai-memory's 20 s reranker gate. This Mac's measured timing goes into #379. +- **Gemma-4 license.** `license_link` resolves to plain Apache-2.0. +- **LFM2.5-8B-A1B.** It has an Ollama tag, `lfm2.5:8b`. Its LFM Open License v1.0 withholds commercial use from entities with US$10M or more in annual revenue. +- **Embedders.** No new dense text embedder in the last 4-6 weeks. + - Nemotron-3-Embed-1B beats F2LLM-v2-4B on every LMEB figure, but has no llama.cpp or Mac path (`Ministral3Model` is unregistered at b11146). + - F2LLM-v2-4B leads only for a llama.cpp deployment: Apache-2.0, `Qwen3Model` registered, the same path as production. + - "LMEB rank" is the leaderboard's Borda rank. +- **Rerankers.** MemReranker-4B has the best LongMemEval NDCG@10/MAP (0.8354/0.8043, self-reported, community MLX only). Qwen3-Reranker-0.6B is the only one with a maintainer-demonstrated stock GGUF (b11057 untested). BGE-v2-m3 vs Qwen3-Reranker-0.6B is effectively a tie. +- **ai-memory lineage.** `19b6429` is 46 commits ahead of and 62 behind v2.4.1 (merge base `8ee81bed`), and v2.4.0 is its ancestor. This goes into the reference-arm rationale. +- **Generation.** No in-window model has task-level evidence against Qwen3.8-27B. + - granite-4.2-30b: official GGUF, MLX and Ollama, about 27.8 GB at 32K. + - Nemotron-3.5-Lightning-30B-A3B: about 27.0 GB at 32K, KV 0.20 GB. + - Either one would need the li26 protocol. +- **Memory systems.** Trial pins unchanged: agentmemory 0.9.29, MemPalace 3.10.0, Hindsight 0.10.1. For Hindsight's capture path, pin `integrations/coding-agents/v0.7.0`. + - Mac Hindsight is `pending` (Apple Container, the owner's install). + - claude-mem's interactive installer pre-selects hosted CMEM Pro. + - GBrain's hang issues #5284 and #5449 are still open. + - Memorix v1.9.6 and mcp-memory-service v11.14.0 are Apache-2.0 defer candidates. diff --git a/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/embed.md b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/embed.md new file mode 100644 index 000000000..3917dcfd0 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/embed.md @@ -0,0 +1,77 @@ + + +**S3 input from mac-coordinator-64gb-20260925, 2026-09-27: `embed` sweep table** (suggested path: `inputs/sweep-20260927-mac/embed.md`). Sonnet sweep, independently verified by an Opus pass against primary sources; its "Verification" section lists every correction applied. The published and self-reported numbers here are discovery evidence, not S3 evidence. + +
Table + +# Embedding model sweep for local memory/RAG on 64GB Apple Silicon Mac (M5 Pro) + +## Verification (2026-09-27, independent Opus pass) +- nvidia/Nemotron-3-Embed-1B-BF16's base model corrected to mistralai/Ministral-3-3B-Instruct-2512 (not the 8B sibling). Source: embed/raw/nvidia.json (`base_model:mistralai/Ministral-3-3B-Instruct-2512` / `base_model:finetune:mistralai/Ministral-3-3B-Instruct-2512` tags on the 1B repo, distinct from the 8B repo's own `Ministral-3-8B-Instruct-2512` tags). +- Qwen3-Embedding-4B's LongMemEval 50.21 is 93rd of 105 on LMEB, not the weakest; the weakest is sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 at 28.58. "LMEB rank" is redefined below as the leaderboard's own Borda rank over 22 tasks (not a re-sort by the displayed mean), and each of the four picks now also states its position by mean and by LongMemEval. Source: embed/lmeb_all105.txt (105-row leaderboard dump; positions independently recomputed by sorting this file by its `mean` and `LongMemEval` columns: F2LLM-v2-4B 6th/5th, Nemotron-1B 2nd/3rd, harrier-0.6b 8th/46th, Qwen3-Embedding-4B 51st/93rd — all four match exactly). +- This file has no dedicated ranking/rationale section, so the Nemotron-1B vs. F2LLM-v2-4B comparison and the llama.cpp-vs-PyTorch/MPS framing are stated here rather than invented as a new section: on every LMEB figure (Borda rank 3 vs 10, mean 61.50 vs 58.68, LoCoMo 50.77 vs 43.32, LongMemEval 81.58 vs 81.42) Nemotron-1B beats F2LLM-v2-4B outright. F2LLM-v2-4B's edge is deployment-shaped, not evidence-shaped: it is `Qwen3Model` (registered in llama.cpp's `conversion/qwen.py:159`, apache-2.0, same size/path as production's own Qwen3-Embedding-4B row), whereas Nemotron-1B's bare `Ministral3Model` is not registered there (see the architecture-registry facts below) — so for a llama.cpp deployment target F2LLM-v2-4B is the pragmatic #1, but for a PyTorch/MPS target the evidence favors Nemotron-1B on every figure. harrier-oss-v1-0.6b's LongMemEval position is only 46th (of 105), well behind both. +- Added repos created in the window that are not text embedders, so the "no new dense text embedder" conclusion stands unchanged: tencent/EVIE-Preview-4.5B, tencent/EVIE-4.5B, tencent/EVIE-8B (image-document retrievers), tencent/WeVisDoc-2B/4B (image-document retrievers), voyageai/rerank-3 (a reranker). None of these appear in embed/raw/tencent.json or embed/raw/voyageai.json (the original sweep's own org listings), confirming they were not in scope for this sweep's HF org pulls; added here on the independent pass. Source: https://huggingface.co/tencent/EVIE-Preview-4.5B, https://huggingface.co/tencent/WeVisDoc-2B, https://huggingface.co/voyageai/rerank-3. + +Sweep date: 2026-09-27. Sources: Hugging Face API (`/api/models`, `/api/models/{org}/{name}`, per-model `raw/main/config.json`, `README.md`, `LICENSE`), MTEB/RTEB leaderboard backend (`https://mteb-leaderboard-backend.hf.space/v1/benchmarks/{LMEB,RTEB(eng, beta),LongEmbed}/scores`, snapshot this session), `ggml-org/llama.cpp` `conversion/*.py` at commit `59657a613ab0fa4ab327d6c790123dff30bfbd67` (catalog tag b11057) and `7fe450e19305b828c199d602c23a8337aaa1f03b` (v0.5.0/b11146) — `convert_hf_to_gguf.py` is byte-identical between the two, `ollama.com/library/{name}` HTTP status, `gh api repos/embeddings-benchmark/results/commits` for submitter attribution. Baseline: `catalogs/foundation/memory-stack-20260925.json` (non_repository_decisions, role Embedder, lines 937-1309) and `catalogs/us-equities/models.md` (lines 19-25). + +**Correction from an earlier draft of this file**: LongEmbed's API returns `meanTask: null` (not 0) for any model missing one of its 6 sub-tasks. A first pass rendered `null` as `0.00` for Qwen3-Embedding-4B, embeddinggemma-300m, harrier-oss-v1-0.6b/27b and others — this looked like "fails long-document retrieval" but is actually "only the synthetic Passkey-Retrieval sub-task is scored; the other 5 are not evaluated for this model, so no overall mean exists." Verified directly against `longembed.json` `scoresByTask` (raw, not the derived mean) for each row below. Do not read a blank LongEmbed mean as a zero. + +Legend: NC = non-commercial-restricted license (flagged). Arch = HF `config.json` `architectures[0]`, and whether that exact class string is registered in llama.cpp's `conversion/` package. LMEB evidence = leaderboard-hosted (KaLM/MTEB `embeddings-benchmark/results` repo), zero-shot (no train-set contamination flag) unless noted; submitter accounts checked via `gh api .../commits` show no visible vendor affiliation for either Nemotron or F2LLM-v2 (same maintainer accounts, e.g. `ItsukiFujii`, merge both) — treat as **leaderboard/community-submitted**, distinct from a vendor's own blog claim (DV). RTEB(eng,beta): `meanTask/meanPublic/meanPrivate` are null for 264/274 rows (a private-holdout design); only `rank` (of 274) is usable there. **LMEB rank** = the leaderboard's own Borda rank over its 22 constituent tasks, not a re-sort by the displayed `mean` column — the two orders diverge (e.g. Nemotron-1B is Borda rank 3 but 2nd by mean; F2LLM-v2-4B is Borda rank 10 but 6th by mean); each pick below states its position by mean and by LongMemEval alongside its Borda rank for that reason. + +## Full candidate table + +| Model (HF repo) | rev (sha, short) | Release/lastModified | Params | Dim | Max tok | License | Mac artifact | llama.cpp arch (b11057+) | Mem Q8/BF16 | Key evidence (source) | +|---|---|---|---|---|---|---|---|---|---|---| +| **google/embeddinggemma-300m** (current Mac pin) | 57c266a7 | created 2025-07-17, lastMod 2025-09-25 | 302.9M | 768 (MRL to 128) | 2048 | gemma (gated=manual, ToS not plain OSS) | Official: `ggml-org/embeddinggemma-300M-GGUF`, `-qat-q8_0-GGUF`, `-qat-q4_0-GGUF`; Ollama `embeddinggemma` (HTTP 200) | `Gemma3TextModel` — registered (`conversion/gemma.py:182`) | 0.30/0.61 GB | No LMEB row (LongMemEval/LoCoMo: unknown). LongEmbed: only Passkey-Retrieval scored (60.75%); other 5 long-doc tasks not evaluated, no overall mean (not "fails," genuinely unverified) | +| **microsoft/bitnet-embedding-0.6b** | 16176d10 | created/lastMod 2026-07-15/17 | 596M (1.58-bit native) | 1024 | 32768 | MIT | Official GGUF `bitnet-embeddings-0.6b-bf16-i2_s.gguf` shipped pre-quantized by publisher | GGUF-only repo, no config.json; base is BitNet b1.58 — `BitnetForCausalLM`/`BitNetForCausalLM` registered (`conversion/bitnet.py:11`); `i2_s` is a native llama.cpp quant type | ~0.12GB native 1.58-bit (Q8-equiv 0.60/BF16-equiv 1.19) | DV only (publisher README table, MTEB v2 mean 67.49 — below harrier-oss-v1-0.6b's 69.0 at the same size). No LMEB/RTEB/LongEmbed row → LongMemEval/LoCoMo unknown | +| microsoft/bitnet-embedding-270m | 5f1c2fd1 | created/lastMod 2026-07-15/17 | 268M (1.58-bit) | 640 | 32768 | MIT | same as above | same | ~0.05GB native (Q8-equiv 0.27/BF16-equiv 0.54) | DV: MTEB v2 mean 66.26. No board row | +| **sentence-transformers/all-MiniLM-L6-v2** (agentmemory control, baseline) | 1110a243 | lastMod 2026-06-01 | 22.7M | 384 | 256 | apache-2.0 | agentmemory ships `Xenova/all-MiniLM-L6-v2` ONNX (community org, not sentence-transformers itself — community-only per task rule) | n/a (BERT-family; not individually re-checked) | 0.02/0.05 GB | LMEB rank 99, LongMemEval 67.94, LoCoMo 25.86. LongEmbed: full 6-task mean 29.81 (rank 136/201, genuinely weak on long docs) | +| mixedbread-ai/mxbai-embed-large-v1 | — | lastMod 2026-01-23 | 335M | 1024 | 512 | apache-2.0 | Official ONNX in repo (`onnx` tag) | not individually checked | 0.34/0.67 GB | LMEB rank 71, LongMemEval 74.53, mean 50.24. RTEB rank 70. LongEmbed full mean 38.92 (rank 91) | +| LiquidAI/LFM2.5-ColBERT-350M-GGUF / LFM2-ColBERT-350M-GGUF | — | lastMod 2026-06-22 / 2026-06-29 | 353M | late-interaction (multi-vector) | 128000 | LFM Open License v1.0 (verified via LICENSE text: free if annual revenue <$10M, Sections 1 & 5) | Official GGUF from publisher | `Lfm2Model`/`Lfm2BidirectionalModel` registered (`conversion/lfm2.py:68`) | 0.35/0.71 GB | Dense sibling LiquidAI/LFM2.5-Embedding-350M: LMEB rank 72, LongMemEval 62.79, mean 51.05. ColBERT variant needs late-interaction (multi-vector) retrieval infra, not a drop-in for single-vector cosine RAG | +| Qwen/Qwen3-Embedding-4B (production baseline) | 5cf2132a | lastMod 2025-06-20 (unchanged, cross-checked this session — sha matches baseline JSON exactly) | 4.02B | 2560 | 32768 | apache-2.0 | Official GGUF `Qwen/Qwen3-Embedding-4B-GGUF`; Ollama `qwen3-embedding:4b` `df5bd2e3c74c` | `Qwen3ForCausalLM`/`Qwen3Model` registered (`conversion/qwen.py:159`) | 4.02/8.04 GB (Ollama tag is Q4_K_M, smaller) | LMEB rank 47 (Borda; 51st by mean, 93rd by LongMemEval), LongMemEval **50.21** (93rd of 105 by LongMemEval, not the weakest scored model on the board — that is sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 at 28.58), LoCoMo 30.08, mean 53.30. RTEB rank 194/274. LongEmbed: only Passkey-Retrieval scored (84.25%), no full mean | +| **codefuse-ai/F2LLM-v2-4B** | 3e565254 | created 2026-03-09, lastMod 2026-09-03 (metadata touch only — not a new release) | 4.02B | 2560 | 40960 | apache-2.0 | None official (safetensors only) | `Qwen3Model` — registered, same class as Qwen3-Embedding | 4.02/8.04 GB | LMEB rank 10 (Borda; 6th by mean, 5th by LongMemEval), LongMemEval **81.42**, LoCoMo 43.32, mean 58.68, zeroShotPct=100/trainedOnTasks=[] (no contamination flag, same as Nemotron). RTEB rank 23/274. LongEmbed **full 6-task mean 75.36** (rank 7/201) | +| codefuse-ai/F2LLM-v2-1.7B | c5650fe0 | 2026-03-09 / 2026-09-03 | 1.72B | 2048 | 40960 | apache-2.0 | none official | `Qwen3Model` — registered | 1.72/3.44 GB | LMEB rank 14, LME 78.82, LoCoMo 45.65, mean 57.60. LongEmbed full mean 71.43 (rank 11) | +| codefuse-ai/F2LLM-v2-8B | 2f1beedb | 2026-03-09 / 2026-09-03 | 7.57B | 4096 | 40960 | apache-2.0 | none official | `Qwen3Model` — registered | 7.57/15.14 GB | LMEB rank 8, LME 81.46, LoCoMo 42.29, mean 58.64. RTEB rank 20. LongEmbed full mean 76.64 (rank 5, best of the family) | +| **codefuse-ai/F2LLM-v2-0.6B** | 8786315a | 2026-03-02 / 2026-09-03 | 596M | 1024 | 40960 | apache-2.0 | **Official ONNX** (`onnx/model.onnx` shipped by codefuse-ai itself — counts as official per task rule) | `Qwen3Model` — registered | 0.60/1.19 GB | LMEB rank 37, LME 76.22, LoCoMo 39.54, mean 55.78. LongEmbed full mean 70.81 (rank 21) | +| codefuse-ai/F2LLM-v2-14B | — | 2026-03-09 / 2026-09-03 | 13.99B | 5120 | 40960 | apache-2.0 | none official | `Qwen3Model` — registered | 13.99/27.98 GB | LMEB rank 30, LME 63.04, LoCoMo 43.76, mean 54.68. RTEB rank 19. LongEmbed full mean 75.57 (rank 6) | +| **microsoft/harrier-oss-v1-0.6b** (baseline trial) | f9b9dc8d | lastMod 2026-03-30 | 596M | 1024 | 32768 | MIT | none official (onnx-community conversion = community-only per baseline) | `Qwen3Model` (HF tag confirms `qwen3`) — **registered** — this updates the baseline row, which said "community GGUFs unverified"; unlike Nemotron, harrier's bare class IS in the registry, so a self-conversion has a real precedent (Qwen3-Embedding, F2LLM-v2) | 0.60/1.19 GB | LMEB rank 5 (Borda; 8th by mean, 46th by LongMemEval), LME 72.46, LoCoMo 48.05, mean 58.43 (baseline). LongEmbed: only Passkey scored (76.75%), no full mean | +| microsoft/harrier-oss-v1-270m | — | lastMod 2026-03-30 | 268M | 640 | 32768 | MIT | none official | `Gemma3TextModel` (HF tag `gemma3_text`) — registered | 0.27/0.54 GB | RTEB rank 258. No LMEB row found for 270m specifically | +| microsoft/harrier-oss-v1-27b | — | lastMod 2026-03-30 | 27.0B | — | 131072 | MIT | none official | `Gemma3TextModel` — registered | 27/54 GB (too large to run comfortably alongside a memory LLM on 64GB) | RTEB rank 240. LongEmbed: only Passkey scored (97.5%), no full mean | +| **nvidia/Nemotron-3-Embed-1B-BF16** (baseline trial — the size the baseline itself flags as MPS-deployable) | c0c9fea9 | created 2026-07-14, lastMod 2026-08-27 | 1.14B | 2048 | 32768 | openmdw-1.1 (notice-preserving + patent-suit termination; not OSI-standard) | none official | `Ministral3Model` (bare, base = Ministral-3-3B-Instruct) — **NOT registered**; only `Ministral3ForCausalLM`/`Mistral3ForConditionalGeneration` (causal variants) are (`conversion/mistral3.py:14`) | 1.14/2.28 GB | LMEB rank 3 (Borda; 2nd by mean, 3rd by LongMemEval), LME 81.58, LoCoMo 50.77, mean 61.50. RTEB rank 14 (tied w/ Qwen3-Embedding-8B) | +| nvidia/Nemotron-3-Embed-8B-BF16 (baseline trial) | d1f2f257 | created 2026-07-14, lastMod 2026-08-28 | 7.95B | 4096 | 32768 | openmdw-1.1 | none official; community GGUFs unverified for bidirectional attention (baseline) | same `Ministral3Model` gap | 7.95/15.91 GB | LMEB **rank 1**, LongMemEval **85.70** (best found this sweep), LoCoMo **56.51** (best found), mean 64.36. RTEB rank 2/274 | +| nvidia/Nemotron-3-Embed-1B-NVFP4 (baseline trial) | f6301282 | 2026-07-14 / 2026-08-28 | 705M | — | — | openmdw-1.1 | none official | same gap | 0.70/1.41 GB | Not found on LMEB (only BF16 siblings scored) — unknown | +| Alibaba-NLP/UEmbed-2B / -4B / -9B | d02beb2d (2B), 2982eab2 (4B) | created 2026-07-29, lastMod 2026-08-18 | 2.21B/4.54B/8.39B | not verified (nested VL config; hidden_size not at top level) | not verified | cc-by-4.0 | none | `Qwen3_5ForConditionalGeneration` nominally registered (`conversion/qwen.py:637`, text-only class), but repo ships custom `src/models/qwen35_embedding.py` (trust_remote_code) + separate `sparse_weights.pt` for a dense+sparse hybrid head the generic class does not model — **conversion correctness unverified** | 2.21/4.43 — 8.39/16.79 GB | No LMEB/RTEB/LongEmbed row found. models.md: "conditional," no verified eval | +| Alibaba-NLP/core-emb-2b / core-emb-8b | — | created 2026-08-30/31, lastMod 2026-09-04 | not verified | not verified | not verified | cc-by-4.0 | none | not checked | unknown | No board row; genuinely new (this month). models.md: "watch," paper date only, not a confirmed weight-release announcement | +| tencent/WeMM-Embedding-2B / -4B / -9B | 41a4fafd (4B) | created 2026-08-25, lastMod 2026-09-03 | 2.72B/5.17B/9.41B | not verified | not verified | **apache-2.0** (confirmed by reading the `LICENSE` file itself — the HF metadata tag shows "other" but the license text states Apache-2.0 with third-party-component carve-outs) | none | same `Qwen3_5ForConditionalGeneration` caveat as UEmbed | 2.72/5.44 — 9.41/18.82 GB | No LMEB/RTEB/LongEmbed row — genuinely new (this month) | +| tencent/R3-embedding-0.6b | 9df7e112 | created/lastMod 2026-07-08 | 596M | not verified | not verified | apache-2.0 | none | `Qwen3Model` — registered (base_model Qwen3-Embedding-0.6B per README) | 0.60/1.19 GB | **Fine-tuned specifically for agent-skill routing** (bi-encoder recall stage of R3-Skill, paired with R3-Rerank-0.6B) — not general passage/memory retrieval; applicability to LongMemEval-style RAG is out-of-domain/unverified. No board row | +| voyageai/voyage-4-nano | — | createdAt 2026-01-06 | 346M | not verified | not verified | apache-2.0 (HF tag; LICENSE.txt file itself not read) | none (safetensors + custom `modeling_qwen3_bidirectional.py`, trust_remote_code) | `qwen3`-tagged bidirectional variant; not individually verified against the registry string | 0.35/0.69 GB | No RTEB row found for the nano tier specifically (mongodb/voyage-4-large, a separate hosted-only tier, is RTEB(eng,beta) **rank 1** overall — not the open-weight model) | +| jinaai/jina-embeddings-v5-text-small / -omni-small (+nano variants) | dd76d535 (text-small) | created 2026-01-22/Mar-31, lastMod 2026-04-15/Aug-27 | 596M–1.63B | — | 8192–32768 | **cc-by-nc-4.0 — NON-COMMERCIAL** | none | `JinaEmbeddingsV5Model` — custom arch, no `jina*.py` file in `conversion/`; **not registered** | 0.6–1.6 / 1.2–3.3 GB | LMEB rank 28 (text-nano, LME 57.60), rank 50 (text-small, LME 74.70). Double-blocked: NC license + unsupported arch | +| Linq-AI-Research/Linq-Embed-Mistral | — | lastMod 2024-06-05 (no change since) | 7.11B | 4096 | 32768 | **cc-by-nc-4.0 — NON-COMMERCIAL** | none | Mistral base, not verified for this bidirectional variant | 7.11/14.22 GB | LMEB rank 19 by mean but LongMemEval only 38.52 (weak). RTEB rank 105/274. LongEmbed full mean 45.82 (rank 51) | +| NovaSearch/stella_en_400M_v5 (+1.5B_v5) | — | lastMod 2025-07-28 (no change since) | 435M (/1.5B) | 4096 | 8192 | MIT | not checked this session | not checked | 0.44/0.87 GB | LMEB rank 12, LME 77.70, LoCoMo 49.03, mean 57.88 | +| Snowflake/snowflake-arctic-embed-l-v2.0 (+m-v2.0) | — | lastMod 2025-07-28 (l) / 2025-04-24 (m) (no change since) | 568M (/305M) | 1024 (/768) | 8192 | apache-2.0 | not checked this session | not checked | 0.57/1.14 GB (l) | LMEB rank 15 (l, LME 71.21), rank 13 (m, LME 70.99) | +| ibm-granite/granite-embedding-311m-multilingual-r2 (+97m, small-english-r2, english-r2) | — | lastMod 2026-05-18 (r2 pair) / 2026-01-21 (small-en) | 312M/97M/48M/149M | 768/384/384/768 | 8192 | apache-2.0 | not checked this session | not checked | 0.10–0.31 / 0.19–0.62 GB | LMEB ranks 57-91 range (LME 64-69). LongEmbed full mean 71.72 (311m-r2, rank 17) — best-performing tiny official-org model on long-doc after F2LLM-v2 | + +## Rerankers found in the sweep (out of scope for the Embedder role, noted for completeness) +Alibaba-NLP/core-reranker-2b / -8b (cc-by-4.0, created 2026-08-30) — not evaluated as an embedder. voyageai/rerank-3 (created 2026-09-23) likewise — added on verification, see Recency below. + +## Recency, precisely +- **Genuinely new HF repos created in the last 4-6 weeks (Aug 16 – Sep 27, 2026):** tencent/WeMM-Embedding-2B/4B/9B (created Aug 25) and Alibaba-NLP/core-emb-2b/8b (created Aug 30). Neither has any LMEB/RTEB/LongEmbed row yet. +- **Added on verification, not text embedders (excluded from the counts above):** tencent/EVIE-Preview-4.5B, tencent/EVIE-4.5B, tencent/EVIE-8B (image-document retrievers) and tencent/WeVisDoc-2B/4B (image-document retrievers); voyageai/rerank-3 (a reranker, see the Rerankers section above). None of these change the "no new dense text embedder" conclusion below. +- **New since 2026-07-01 but created before the 6-week window:** microsoft/bitnet-embedding-0.6b/270m (Jul 15), tencent/R3-embedding-0.6b (Jul 8), Alibaba-NLP/UEmbed-2B/4B/9B (Jul 29), nvidia/Nemotron-3-Embed-8B/1B-BF16/1B-NVFP4 (Jul 14, already the baseline's own trial rows). +- **Everything else that shows an Aug/Sep `lastModified`** (Nemotron Aug 27-28, jina-v5 Aug 27, UEmbed Aug 18, F2LLM-v2 Sep 3, mixedbread-ai Aug 19, LiquidAI ColBERT Aug 17-18) **is a metadata/README revision touch on a model released earlier** (F2LLM-v2's actual release was 2026-03-09; the Sep 3 date is not a new drop). Confirmed by comparing each row's `createdAt` to its `lastModified` in the raw HF sweep files under `raw/*.json`. +- No target-publisher org (Qwen, nvidia, google, microsoft, BAAI, jinaai, Snowflake, ibm-granite, Alibaba-NLP, intfloat, mixedbread-ai, nomic-ai, NovaSearch, Linq-AI-Research, codefuse-ai, tencent, LiquidAI, sentence-transformers, voyageai) released a **new dense text embedder** in the last 4-6 weeks; the two truly-new repos (WeMM-Embedding, core-emb) are the closest fit and are multimodal/hybrid, unbenchmarked. + +## llama.cpp architecture registry facts (primary source, this session) +Fetched `conversion/*.py` from `ggml-org/llama.cpp` at commit `59657a613ab0fa4ab327d6c790123dff30bfbd67` (catalog b11057) and `7fe450e19305b828c199d602c23a8337aaa1f03b` (v0.5.0/b11146); `convert_hf_to_gguf.py` itself is byte-identical between the two (`diff` = no output), so "b11057 or later" is one fact set here. +- `conversion/qwen.py:159` — `@ModelBase.register("Qwen3ForCausalLM", "Qwen3Model")` — covers Qwen3-Embedding, F2LLM-v2 (all sizes), tencent/R3-embedding-0.6b, harrier-oss-v1-0.6b. +- `conversion/qwen.py:637` — `@ModelBase.register("Qwen3_5ForConditionalGeneration", "Qwen3_5ForCausalLM")` (`model_arch=QWEN35`, text-only) — covers the base LM class Alibaba-NLP/UEmbed-* and tencent/WeMM-Embedding-* declare, but both ship custom trust_remote_code embedding/sparse heads the generic class does not model; conversion correctness unverified. +- `conversion/gemma.py:182` — `@ModelBase.register("Gemma3TextModel")` — covers embeddinggemma-300m and harrier-oss-v1-270m/27b. +- `conversion/bitnet.py:11` — `@ModelBase.register("BitnetForCausalLM", "BitNetForCausalLM")` — consistent with microsoft/bitnet-embedding's native i2_s GGUF. +- `conversion/lfm2.py:68` — `@ModelBase.register("Lfm2Model", "Lfm2BidirectionalModel")` — covers LiquidAI's embedding/ColBERT lines. +- `conversion/mistral3.py:14` — `@ModelBase.register("Mistral3ForConditionalGeneration", "Ministral3ForCausalLM")` — does **not** include bare `Ministral3Model`, the exact class nvidia/Nemotron-3-Embed-* declares. Primary-source confirmation of the baseline's caution ("community GGUFs are unverified for its bidirectional attention"). +- No `conversion/jina*.py` file exists — jinaai's v5 custom architecture has no converter. + +## Ollama library check (this session, HTTP status on ollama.com/library/{name}, guessed tag names) +embeddinggemma=200, granite-embedding=200, qwen3-embedding=200, all-minilm=200; nemotron-embed=404, f2llm=404, bitnet-embedding=404, harrier=404 (a 404 on a guessed name does not rule out a differently-named official tag; not exhaustively searched). + +
diff --git a/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/gen.md b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/gen.md new file mode 100644 index 000000000..c68478e73 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/gen.md @@ -0,0 +1,134 @@ + + +**S3 input from mac-coordinator-64gb-20260925, 2026-09-27: `gen` sweep table** (suggested path: `inputs/sweep-20260927-mac/gen.md`). Sonnet sweep, independently verified by an Opus pass against primary sources; its "Verification" section lists every correction applied. The published and self-reported numbers here are discovery evidence, not S3 evidence. + +
Table + +# Local-generation model sweep, 2026-08-01 to 2026-09-27 (revised after review) + +## Verification (2026-09-27, independent Opus pass) +- Nemotron-3.5-Lightning-30B-A3B KV at 32K recomputed from the real layer split (6 attention / 23 mamba / 23 moe of 52 layers, 2 kv heads x head_dim 128) instead of an all-52-layers-as-attention upper bound: 0.20GB, not <=1.74GB; total revised to approx 27.0GB (25.27 + 0.20 + 1.49), below granite-4.2-30b's 27.8GB, so the two rows are swapped into ascending-fit order and the "KV split not resolved" unknown is removed. Source: verify/gen/cfg_nemotron_bf16.json (`layers_block_type`). +- Corrected the two HF slugs that 401 under their short form to the slugs that actually resolve: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 and prism-ml/Ternary-Bonsai-2-27B-gguf. Source: verify/gen/api_nvidia_NVIDIA-Nemotron-3.5-Lightning-30B-A3B.json and api_prism-ml_Ternary-Bonsai-2-27B.json (both `{"error":"Invalid username or password."}`) vs. api_nvidia_NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.json / api_prism-ml_Ternary-Bonsai-2-27B-gguf.json (both resolve). +- Bonsai 2 27B: stock llama.cpp rejects the shipped PQ2_0/PTQ1_0 files outright (unknown types); "garbage output" is specific to the separate, generic Q2_0 type it silently loads with no Hadamard runtime — these are not the same claim. Added the card's own M5 Pro throughput (approx 28 tok/s), the Mac target chip, ahead of the M5 Max aside. Source: verify/gen/readme_bonsai2_27b_gguf.md lines 139-141 (rejection wording), 191 and 206 (M5 Pro tok/s). +- Qwen3.8-27B's "smallest 32K footprint" claim qualified to "among rows with a worked-out KV" — Muse-Glimmer-30B's KV was never resolved in this sweep, so it is not a counted comparison (see this table's own Unknowns). +- Newest llama.cpp tag is b11214 (2026-09-27T12:45:21Z); reconfirmed Xing4_0ForCausalLM absent at both b11146 and b11214 via a fresh architecture-registry scan (previously only checked at b11057/b11214). Source: verify/gen/archs_b11146.txt, archs_b11214.txt, scan2_b11146.txt, scan2_b11214.txt (0 files contain `xing4` at either tag). +- Hosted models: confirmed claude-opus-5-5 (releasedOn 2026-09-22) and claude-fable-5-1 (releasedOn 2026-09-01) directly against the Anthropic docs' embedded page JSON; sharpened the OpenAI entry to state the docs page itself carries no release-date field for gpt-6-sol/gpt-6-luna (only knowledge-cutoff dates), and that 2026-09-22 is CNBC/secondary-source only. Source: verify/gen/anth_models_opus-5-5_overview.html, anth_about-claude_models_overview.html (`\"releasedOn\":\"2026-09-22\"`, `\"releasedOn\":\"2026-09-01\"`); verify/gen/oai_developers.openai.com_api_docs_models.html. + +Mac target: M5 Pro, 64GB unified memory, about 53 GiB Metal-available, budget about 40GB at Q4 including KV for 32K context. +llama.cpp floor: b11057. Diffed against b11214 (2026-09-27T12:45:21Z, newest tag at sweep time); only 2 new HF-arch registrations appeared between them (BailingMoeV3VL, Gemma4DSparkModel), neither relevant here. +Method: HF API createdAt-desc sweep of 18 orgs (plus corrected slugs stepfun-ai, prism-ml, XingChen-AGI), then a second lastModified-desc sweep of the same 18 to catch updates to pre-window repos (44 hits reviewed, listed below). +Evidence labels: [doc]=publisher documentation/card, [obs]=observed this session (HF API/config/tree/Ollama fetch), [self]=publisher-reported benchmark, [vendor-harness]=publisher-run but against third-party baselines, [unk]=not verified this session. +Release date vs createdAt: HF repo createdAt often precedes the publisher-stated public Release Date on the card. Where the card states one, both are given. + +## Sorted by Mac fit, ascending estimated total footprint within each bucket + +| Model | Org (in 18-list?) | Repo created [obs] | Card Release Date [doc] | Revision (sha, 12-hex) | Total / Active params [obs] | License [obs+doc] | Mac artifact | Ollama tag [obs] | llama.cpp arch @ b11057 [obs] | Q4 weight | KV f16 @32K | Total est | Fit | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| ibm-granite/granite-4.2-3b | ibm-granite (yes) | 2026-08-07 | unk | e459acceac81 | 3.66B dense | apache-2.0 | Official GGUF + official MLX | granite4.2:3b, 2.2GB | GraniteForCausalLM: yes | 2.24GB [obs] | 2.68GB | 6.4GB | Fits comfortably | +| tencent/ContextPilot-E4B | tencent (yes) | 2026-08-27 | unk | a7ba41c18ddc | 7.94B (Gemma4 MatFormer effective-4B) | Tencent custom LICENSE file (HF tag: other) | mradermacher GGUF (i1 imatrix) | none found | Gemma4ForConditionalGeneration: yes | 5.30GB [obs] | not resolved, nested config | approx 8-9GB | Fits comfortably | +| tencent/ContextPilot-8B | tencent (yes) | 2026-08-27 | unk | 595abeaed72e | 8.19B dense (Qwen3-8B base) | Tencent custom LICENSE file (HF tag: other) | mradermacher GGUF (i1 imatrix) | none found | Qwen3ForCausalLM: yes | 5.03GB [obs] | 4.83GB | 11.4GB | Fits comfortably | +| XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B | XiaomiMiMo (yes) | 2026-09-21 | unk | 2367e865d009 | 9.41B (Qwen3.5-9B distill) | mit | bartowski+ggml-org GGUF, mlx-community 4bit | none found | Qwen3_5ForConditionalGeneration: yes | approx 5.6GB (Q4_K_M est; Q8_0 used in baseline test) | approx 4.8GB | approx 11-12GB | Fits comfortably; ALREADY TESTED, see note | +| ibm-granite/granite-4.2-8b | ibm-granite (yes) | 2026-08-07 | unk | f8de16cdcdbc | 8.79B dense | apache-2.0 | Official GGUF + official MLX | granite4.2:8b, 5.3GB | GraniteForCausalLM: yes | 5.35GB [obs] | 5.37GB | 12.2GB | Fits comfortably | +| tencent/ContextPilot-14B | tencent (yes) | 2026-08-27 | unk | 8eafd8356c5c | 14.77B dense (Qwen3-14B base) | Tencent custom LICENSE file (HF tag: other) | mradermacher + bartowski GGUF | none found | Qwen3ForCausalLM: yes | 9.00GB [obs] | 5.37GB | 15.9GB | Fits comfortably | +| meta-models/Muse-Glimmer-30B (NOT in 18-org list) | meta-models | 2026-08-09 | unk | a4e59da52a7b | 29.78B (nested config) | apache-2.0 | unsloth GGUF, mlx-community 8bit | none found | MuseGlimmerForConditionalGeneration: yes | 15.88GB [obs, UD-Q4_K_XL] | not resolved | approx 18-20GB | Fits comfortably; NO benchmark gathered | +| Qwen/Qwen3.8-27B (REFERENCE: existing default, in-window release) | Qwen (yes) | 2026-08-05 | unk | 1d4bf0f2ff60 | 27.78B, HYBRID attention (16 of 64 layers full-attention, interval 4; rest linear/SSM-style with O(1) state) | apache-2.0 | Official; unsloth GGUF (proven quant) | qwen3.8:27b, 18GB | Qwen3_5ForConditionalGeneration: yes (proven) | 16.46GB [obs, unsloth UD-Q4_K_M] | approx 2.15GB (only 16 full-attn layers, 4 kv heads, head_dim 256; linear layers add a small constant state, not context-scaling) | approx 20.1GB | Fits comfortably; smallest 32K footprint among rows with a worked-out KV in the approx 27-30B class here (Muse-Glimmer-30B's KV is not worked out, so it is not a counted comparison) | +| nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 | nvidia (yes) | 2026-08-01 | 2026-08-11 [doc, card Release Date field] | a9904d24bcc1 | 31.58B total / 3B active, hybrid Mamba+MoE (NemotronH) | OpenMDW-1.1 | unsloth/bartowski/ggml-org/lmstudio-community GGUF (no official) | nemotron-3.5-lightning:30b-a3b, 25GB | NemotronHForCausalLM: yes | 25.27GB [obs, unsloth UD-Q4_K_M] | 0.20GB [obs, config `layers_block_type`: 6 attention / 23 mamba / 23 moe of 52 layers; 2 kv heads x head_dim 128, only the 6 attention layers scale with context] | approx 27.0GB (25.27 + 0.20 + 1.49) | Fits comfortably; below granite-4.2-30b's 27.8GB at 32K, but still 6.9GB above Qwen/Qwen3.8-27B's 20.1GB two rows up - not the smallest of the class | +| ibm-granite/granite-4.2-30b | ibm-granite (yes) | 2026-08-07 | 2026-08-25 [doc, card Release Date field] | 9e668ce1c538 | 29.28B dense | apache-2.0 | Official GGUF + official MLX, also lmstudio-community/bartowski/mradermacher | granite4.2:30b, 18GB | GraniteForCausalLM: yes | 17.72GB [obs] | 8.59GB (plain GQA, all 64 layers full-attention; no hybrid discount) | 27.8GB | Fits comfortably; costs approx 7.7GB MORE than Qwen3.8-27B at 32K despite similar weight size | + +### runtime_unsupported (mainline llama.cpp b11057 or b11214) + +| Model | Org | Created [obs] | Revision | Params | License | Why unsupported | Note | +|---|---|---|---|---|---|---|---| +| XingChen-AGI/Xing4.0-29B-A4B (NOT in 18-org list) | XingChen-AGI | 2026-09-16 | baae3c3e813c | 31.2B total/4B active MoE, full MHA (32 kv=32 attn heads, no GQA) | apache-2.0 | Xing4_0ForCausalLM not registered at b11057; reconfirmed absent at b11146 and b11214 (2026-09-27T12:45:21Z, the newest tag) via a fresh architecture-registry scan | Even if supported, full-MHA 32K KV alone is approx 18.79GB, worst of this table | +| prism-ml/Ternary-Bonsai-2-27B-gguf (base Qwen/Qwen3.8-27B) | prism-ml (closest match to task's PrismML) | 2026-09-16 | b072e1d3b35a | approx 27B ternary, 1.72-2.13 bits/weight | apache-2.0 | Custom ternary hybrid-attention kernel; own card confirms stock llama.cpp rejects the shipped PQ2_0/PTQ1_0 files outright as unknown types (does not run at all) and silently loads the unrelated generic Q2_0 type with no Hadamard activation runtime, producing garbage output — "produces garbage" applies only to that Q2_0 fallback, not to PQ2_0/PTQ1_0; needs the PrismML-Eng/llama.cpp FORK | approx 5.9-7.2GB per card [self]; card measures approx 28 tok/s TG128 on an Apple M5 Pro (this Mac's target chip: 28.1 tok/s PQ2_0 current build, 28.7 tok/s on an earlier pre-rotation 7.2GB build pending re-measurement — readme_bonsai2_27b_gguf.md lines 191/206); a separate figure of approx 47 tok/s is claimed for the wider Apple M5 Max, publisher-claimed and not independently verified | + +## Too large for this Mac regardless of quant (total safetensors element counts [obs]; MoE = total, not active) + +New since 2026-08-01: deepseek-ai/DeepSeek-V4-Pro-0813 1650.5B mit (approx 990GB@Q4); XiaomiMiMo/MiMo-V2.6-Pro-RL 1024.2B mit (approx 615GB); deepseek-ai/DeepSeek-V4.1-Flash 763.2B mit (approx 458GB, baseline watch); zai-org/GLM-5.3 753.3B glm-5.3-custom (approx 452GB, baseline watch); tencent/Hy4-preview 780.0B apache-2.0 (approx 468GB, new Hunyuan successor); nvidia/Nemotron-3-Labs-Ultra-Math-SFT+RL and nvidia/NVIDIA-Nemotron-Labs-Teacher x5 (STEM/General-Reasoning/Instruction-Following/Competition-Coding/Chat), all 560.5B, license other (approx 336GB each); XiaomiMiMo/MiMo-V2.6-Flash-RL 310.8B mit (approx 186GB); zai-org/GLM-5.3-Flash 321.3B mit (approx 193GB, baseline watch); deepseek-ai/DeepSeek-V4-Flash-Vision-Exp 304.6B mit (approx 183GB). +Unchanged from baseline watch list, still too large: Qwen/Qwen3.8-2.4T-A95B (2.4T/95B active); moonshotai/Kimi-K3 (2.8T/104B active; lastModified moved to 2026-09-02, no new repo -- card-level change only); Qwen/Qwen3.8-Flash-Next (approx 180B stored/6B active). + +## Official alt-quants of the existing default (not new models) + +- Qwen/Qwen3.8-27B-FP8 (created 2026-08-13, 27.78B, apache-2.0): official FP8 of the same Qwen3.8-27B; approx 27.8GB at FP8, larger than the proven UD-Q4_K_M. nvidia/Qwen3.8-27B-NVFP4 (created 2026-09-04) is a CUDA-kernel-specific mirror, not a Metal/llama.cpp path. + +## lastModified sweep (catches update-not-create; 44 in-window lastModified hits reviewed across the 18 orgs, createdAt before 2026-08-01) + +No previously-unknown Mac-fit foundation model surfaced this way; all 44 hits were either (a) already-covered too-large siblings getting card/file touches (GLM-5.2, Nemotron-3-Ultra-550B-A55B, Nemotron-3-Super-120B-A12B, Kimi-K2-Thinking-NVFP4, Qwen3.6-35B-A3B-NVFP4, GLM-5.2-NVFP4, DeepSeek-V4-* NVFP4 mirrors), (b) non-generation models (OCR, guardian/safety, parse, calibration, audio), or (c) small older items just touched (Phi-4-reasoning-vision-15B, Ternary-Bonsai-27B-gguf original). Two items worth naming: deepseek-ai/DeepSeek-V4-Flash-0731 was created 2026-07-31 (one day before the window) and only touched 2026-08-01, a boundary case not otherwise counted here. nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning (created April 2026, 30B/3B active, touched 2026-08-24/25) is an older same-size-class sibling of the Lightning-30B-A3B row above. +Cross-reference: the Nemotron-3.5-Lightning card benchmark table names Qwen-3.6-35B-A3B and Nemotron 3 Super as stronger peers and Nemotron 3 Nano / GPT-OSS-20B as weaker; nvidia/NVIDIA-Nemotron-3-Super-120B-A12B (120B/12B active) and nvidia/Qwen3.6-35B-A3B-NVFP4 (mirror) confirm those identities but both are pre-window and/or too large for this Mac at Q4. + +## Orgs checked with no new generation-model release in the window + +| Org | Newest relevant repo observed | Created [obs] | +|---|---|---| +| meta-llama | Llama-Guard-4-12B | 2025-04-23, nothing since | +| openai (HF weights) | gpt-oss-safeguard-20b/120b | 2025-09-18 | +| mistralai | Magistral-Small-2507-GGUF | 2025-07-23 | +| google (generation family) | gemma-4 QAT variants | 2026-06-05 (timesfm-3.0 2026-08-24 is forecasting, non-commercial, baseline) | +| baidu | ERNIE-4.5-VL-28B-A3B-Thinking | 2025-11-07 | +| MiniMaxAI | MiniMax-M3 / M3-MXFP8 | 2026-06-02 | +| microsoft (chat/text LLM) | Fara1.5-27B/4B agent VLM | 2026-07-17 (Phi newest Phi-Ground-Any 2026-05-07) | +| allenai | OLMo-3 family | 2026-02-19 to 02-28 (Sept HF OLMo-3 activity is third-party fine-tunes) | +| stepfun-ai | Step-3.7-Flash family | 2026-05-23 to 05-28 | +| PrismML / prism-ml | ternary GGUF/MLX/AWQ only, no plain safetensors base | newest gguf-dev 2026-09-17 | + +## Hosted models (release dates only, per task; primary source fetched 2026-09-27) + +- OpenAI, https://developers.openai.com/api/docs/models (fetched 2026-09-27): gpt-6-astra still the flagship, same model ID, knowledge cutoff Apr 30 2026 shown; the page has no version/snapshot field so unchanged-vs-refreshed cannot be fully confirmed from it alone. Two new siblings are live on the same index: gpt-6-sol (knowledge cutoff Apr 20 2026) and gpt-6-luna (May 18 2026), both absent from the baseline record. The OpenAI docs page itself carries no release-date field for either — only the knowledge-cutoff dates above are on-page; the 2026-09-22 expansion date is secondary-source only (WebSearch, CNBC), not corroborated by developers.openai.com. +- Anthropic, https://platform.claude.com/docs/en/models/overview and .../opus-5-5/overview (fetched 2026-09-27; releasedOn fields read directly from the pages' embedded JSON, confirmed against the raw page source): claude-opus-5-5, releasedOn 2026-09-22, lifecycle active, latest true -- newer than the baseline-pinned claude-opus-5[1m] (2026-07-24). claude-fable-5-1, releasedOn 2026-09-01, matches baseline exactly, remains user-excluded (not re-argued here). claude-sonnet-5, releasedOn 2026-06-30. The same release wave also names claude-mythos-5-1 (paired with Fable 5.1 in the announcement link); not independently investigated. + +## Unknowns + +- No independent (non-publisher) benchmark located this session for granite-4.2-30b, Nemotron-3.5-Lightning-30B-A3B, ContextPilot, or Muse-Glimmer-30B; all quality evidence above is publisher self-reported or publisher-run-harness (Nemotron explicitly: NeMo Gym/Evaluator, publisher-run against third-party baselines). +- ContextPilot-E4B and Muse-Glimmer-30B use nested multimodal configs; exact per-layer KV math not resolved this session. +- Whether Xing4.0-29B-A4B or Ternary-Bonsai-2-27B work on any mainline llama.cpp branch newer than b11214, or only on forks/custom builds, was not re-verified beyond the architecture-registry check. +- Exact Q4_K_M file size for the current MiMo-V2.6-Distill-Qwen-9B revision was not refetched (baseline receipt used Q8_0); figure above is an estimate. +- meta-models (Muse-Glimmer) and XingChen-AGI are outside the mandated 18-org list; included because they surfaced via the nvidia/prism-ml mirror trail. +- Publisher Release Date fields were found only on the granite-4.2-30b and Nemotron-3.5-Lightning cards; other rows show HF createdAt only, which can precede the public announcement. + +## Revisions (40-hex), exact repo id : sha + +- deepseek-ai/DeepSeek-V4-Flash-Vision-Exp: 6821d6ad3681a4b137b066b76094fa82ebd0a380 +- deepseek-ai/DeepSeek-V4-Pro-0813: 72e1d3230f6c080a530b0a1d46f8eb4602340597 +- deepseek-ai/DeepSeek-V4.1-Flash: dba1be0a40aa45a94ad051997016db3960a90277 +- ibm-granite/granite-4.2-30b-GGUF: 27b350a791e81d9a4d1ddca1c49282e9ec533768 +- ibm-granite/granite-4.2-30b-q4-mlx: d3f143b440062060eb05f7b8677ced5b4d9a8946 +- ibm-granite/granite-4.2-30b: 9e668ce1c538387ef24d3644e9b0606647762636 +- ibm-granite/granite-4.2-3b-q4-mlx: 0c6f39b1827afd5eb2c1c3b13751929857434953 +- ibm-granite/granite-4.2-3b: e459acceac81e5fe67c07d9cfc72329a332e7eb1 +- ibm-granite/granite-4.2-8b-q4-mlx: 9047f073e46c527f70dade07bb55b1700d728e36 +- ibm-granite/granite-4.2-8b: f8de16cdcdbc6c779ca517604e050d82cc119e44 +- nvidia/Muse-Glimmer-30B-NVFP4: 47818374517751c48c55cde2621594926b1888b6 +- nvidia/Nemotron-3-Labs-Ultra-Math-SFT: c6337c397cc7ea2cf7b594e0b880bc874558e20d +- nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16: 434456c9a6753f29d24e23c95d622aaf17111b3b +- nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16: a9904d24bcc1d289a1950fa9d2b978c47cf903b9 +- nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4: bee7596271d1495f6992ae224aefde4410e816b8 +- nvidia/NVIDIA-Nemotron-Labs-Teacher-Chat: 7fbf767583dfd7f18f11734626ce1fb311d6acb1 +- prism-ml/Ternary-Bonsai-2-27B-gguf: b072e1d3b35a0a630cece372c2127528e0994386 +- Qwen/Qwen3.8-27B-FP8: 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a +- tencent/ContextPilot-14B: 8eafd8356c5c5e713e92e623c8f1d346de6d0f95 +- tencent/ContextPilot-8B: 595abeaed72e4a4d9da94d04917f0618989a5553 +- tencent/ContextPilot-E4B: a7ba41c18ddcd334f23f63ef7a608ca46bfe3aa4 +- tencent/Hy4-preview-FP8: 4215ec29de873a998e849cee902654490c7ff4d1 +- tencent/Hy4-preview: 705d81ee51566a186d645b74c974d642ef2828fe +- tencent/UI-Mate-27B: 3ade2378fc84032d5017c1a9c93c4eaa77d65e57 +- tencent/UI-Mate-9B: 05dd5f2975195a5bb03d4363e8767f12158c8421 +- XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B: 2367e865d009c13ac81713a2878291d33ab28177 +- XiaomiMiMo/MiMo-V2.6-Flash-RL: 5711b268169967567844e1e560e8a3966da959b1 +- XiaomiMiMo/MiMo-V2.6-Pro-RL: 73875d00b30a89ef8cc353a0b60b0e9f9561952d +- XingChen-AGI/Xing4.0-29B-A4B: baae3c3e813cad5f888f1f485cfff659c89076c5 +- zai-org/GLM-5.3-Flash: eb9eb208eb0d988989d07a6a12d0fdeb5f52574a +- zai-org/GLM-5.3: aca966e4e02791568aa6a4ced368624b3d897f42 +- Qwen/Qwen3.8-27B: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 (base card) +- unsloth/Qwen3.8-27B-GGUF: 4ca720788d1e01f1bff70c033e0d0028fd02e502 (proven quant, per baseline) +- meta-models/Muse-Glimmer-30B: a4e59da52a7bc87ae7251dd5545c0dd437c44b68 +- moonshotai/Kimi-K3: f831ab66814297da540d832a5235f8e904f29d06 (per baseline, unchanged) +- Qwen/Qwen3.8-Flash-Next: de4b8e4d43b917e7706784d8bb445c9af86a3540 (per baseline, unchanged) +- Qwen/Qwen3.8-2.4T-A95B: 207bd685a7e3696cfaff12ded7c6a7ea0f88c996 (per baseline, unchanged) + +## Protocol caveats (added after review) + +- granite-4.2-30b README line 347 states thinking is enabled by default; the card benchmarks quoted above (AIME25 89.17, GPQA 66.41, MMLU-Pro 77.60) are THINKING-MODE scores. The baseline proven extraction protocol for Qwen3.8-27B runs thinking DISABLED at about 17 output tokens per filing. These numbers do not transfer to that protocol; granite-4.2-30b has no thinking-off, task-matched evidence. +- Ternary-Bonsai-2-27B self-reported 84.78 average is likewise across 14 THINKING-MODE benchmarks per its own card; same non-transfer caveat, on top of it being runtime_unsupported on mainline. +- tencent/ContextPilot-E4B license was read from the sibling ContextPilot-8B repo LICENSE file (Tencent custom terms); E4B itself is Gemma4-based and was not individually read this session. Mark E4B license as [unk: LICENSE file present, not opened] rather than assuming it matches the 8B terms. + +
diff --git a/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/memllm.md b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/memllm.md new file mode 100644 index 000000000..1b56c9981 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/memllm.md @@ -0,0 +1,201 @@ + + +**S3 input from mac-coordinator-64gb-20260925, 2026-09-27: `memllm` sweep table** (suggested path: `inputs/sweep-20260927-mac/memllm.md`). Sonnet sweep, independently verified by an Opus pass against primary sources; its "Verification" section lists every correction applied. The published and self-reported numbers here are discovery evidence, not S3 evidence. + +
Table + +# Local memory-LLM sweep, 2026-09-27 (since 2026-07-01, emphasis last 4-6 weeks) + +## Verification (2026-09-27, independent Opus pass) +- Gemma-4 license: removed the "custom Gemma terms" caveat everywhere it appeared (rows 2 and 6, Unknowns, ranking (a) items 2/3) - `license_link` redirects to a plain "Apache License 2.0" page, and the card body itself says "License: Apache 2.0". Source: verify/memllm/raw/gemma_4_license.html (canonical `ai.google.dev/gemma/apache_2`), verify/memllm/raw/README_google_gemma-4-26B-A4B-it.md, verify/memllm/raw/api_google_gemma-4-26B-A4B-it.json (`cardData.license_link`). +- LFM2.5-8B-A1B does have an Ollama tag (row 1, ranking (a) item 1): `lfm2.5:8b` and `lfm2.5:8b-a1b-q4_K_M` both resolve; added the LFM Open License v1.0 terms (US$10M revenue threshold incl. >=50%-controlled affiliates, free non-commercial/local use, automatic termination on breach). Source: verify/memllm/raw/ollama_lfm2.5_tags.html, ollama_lfm2.5_8b_manifest.json; verify/memllm/raw/LICENSE_LiquidAI_LFM2.5-8B-A1B (Sections 1, 5, 11). +- granite-4.2-8b BFCL v4 corrected to 52.39 at the pinned revision f8de16cdcdbc6c779ca517604e050d82cc119e44 (row 3, Sources, ranking (a) item 3) - 50.29 was a superseded 2026-08-18 card. Source: verify/memllm/raw/commits_granite8b.json (commit dated 2026-08-18T19:55:56Z superseded by 2026-09-04T21:02:25Z), verify/memllm/raw/src_eesel.html ("8B is close enough (52.39 BFCL...)"). +- Granite speed (row 4, Unknowns, ranking (b) item 3): granite-4.2-30b is 76.6 tok/s on Artificial Analysis's hosted API (not Mac), AA's own label "slower than average" - not "notably slow, 28 tok/s"; granite-4.2-8b is 84.9 tok/s (Unknowns). "Notably slow" is AA's wording for Qwen3.8 (xhigh) at 45.5 tok/s. Source: verify/memllm/raw/src_aa_gr30.html, src_aa_gr8.html, src_aa_qwen38.html (each page's own stat tile). +- Dropped "the only hard instruction-following number" (ranking (a) item 1): Liquid's own comparison card gives Gemma-4-26B-A4B-it IFEval 91.40/IFBench 47.25; AA independently records Gemma IFBench 72.4/45.4 and LFM2.5-8B-A1B approx 55.7. Source: verify/memllm/raw/README_LiquidAI_LFM2.5-8B-A1B.md (Knowledge-and-instruction-following table); verify/memllm/raw/src_aa_gemma26.html (embedded chart JSON, `ifbench` field). +- Credited "Qwen3.8 beats all medium models / ties DeepSeek-V4-Flash" (row 5, ranking (b) item 1) to a Hacker News comment on an older Intelligence Index scale, not to Artificial Analysis. Source: verify/memllm/raw/src_hn.html (https://news.ycombinator.com/item?id=49334544). +- LFM2.5-8B-A1B (public 2026-05-28, row 1) and Gemma-4-26B-A4B-it (public by 2026-04-01, row 2) flagged as released before this sweep's since-2026-07-01 window; only their later touches (Aug/Jul) fall inside it. Source: verify/memllm/raw/api_LiquidAI_LFM2.5-8B-A1B.json, api_google_gemma-4-26B-A4B-it.json (`createdAt`). +- Replaced the "AA-LCR unrecoverable" unknown with the recovered AA-LCR v1.1 scores, reproducing the baseline's Qwen3.5-9B 46.0/70.0, Qwen3.6-35B-A3B 64.3/71.7 and Nemotron 60.3 exactly as a cross-check (Unknowns; rows 2, 4, 5, 6; ranking (b) fully rewritten on this metric). Source: verify/memllm/aa_lcr_parsed.tsv. +- Rewrote ranking (b) in full on AA-LCR (the table's own deciding metric for that role): new order Qwen3.8-27B > Gemma-4-26B-A4B-it > granite-4.2-30b; only Qwen3.8-27B beats both the control and the trial. Re-decided ranking (a)'s #2/#3 now that Gemma's license caveat is gone: Gemma-4-26B-A4B-it moves to #2 on this role's own stated discriminator (throughput: 3.8B active vs granite-8b's 8.8B fully dense), granite-4.2-8b to #3; reasons stated inline. Source: verify/memllm/aa_lcr_parsed.tsv plus the row-level sources above. + +Method: Hugging Face API (`/api/models?author=...&sort=createdAt`, then per-repo `/api/models/{id}` for +sha/lastModified/license/gated/safetensors.total, then `resolve/main/config.json`) for architecture; +llama.cpp b11057 = commit `59657a613ab0fa4ab327d6c790123dff30bfbd67` (GitHub tree + raw conversion/*.py at that +commit, matches the catalog pin exactly); Ollama support = `HEAD registry.ollama.ai/v2/library//manifests/` +(200/404, read-only, no pull); WebSearch for AA/IFEval/BFCL; primary LICENSE/README/commit-history reads for the +flagged claims below. All revisions are 40-hex shas observed now via the HF API on 2026-09-27 (see Revisions). +Q4/Q8 sizes are exact GGUF blob bytes from `?blobs=true` unless marked otherwise. + +**Correction after review:** `config.json` shows the *production control* Qwen3.5-9B is itself +`Qwen3_5ForConditionalGeneration` / pipeline_tag `image-text-to-text` (same wrapper class as Qwen3.8-27B). "VL-wrapped" +is dropped as a loss reason for Qwen3.8-27B and softened for Gemma-4-26B-A4B - not a differentiator from the running control. + +## Baseline (catalogs/foundation/memory-stack-20260925.json, non_repository_decisions, role "Memory LLM") + +| Model | Repo | Date | Params | License | Mac artifact | Evidence | +|---|---|---|---|---|---|---| +| Qwen3.5-9B (**retain**, control) | Qwen/Qwen3.5-9B | 2026-03-02 | 9B, `qwen3_5`, image-text-to-text tag | Apache-2.0 | Ollama qwen3.5:9b-mlx `203e30078279` / prod tag qwen3.5-9b-64k `5efc0dd55d4a` | D: AA-LCR 46.0 / 70.0(reason). DV LongBench-v2 55.2 | +| Qwen3.6-35B-A3B (**trial**) | Qwen/Qwen3.6-35B-A3B | 2026-04-24 | 35B/A3B MoE | Apache-2.0 | Ollama qwen3.6:35b-a3b `096fdbd02fe6`(GGUF 22.6GB)/-nvfp4 `e92a3e94bbca`(MLX 23.6GB) | D: AA-LCR 64.3/71.7 (+18.3 vs control). No LongMemEval | +| LFM2.5-2.6B (**trial**) | LiquidAI/LFM2.5-2.6B-GGUF | 2026-09-22 | 2.6B dense | LFM1.0 (custom, flag) | no Ollama tag (404) | DV IFStruct 85.49; 220 tok/s M5 Max CPU | +| Nemotron-3.5-Lightning-30B-A3B (**defer**) | nvidia/...-BF16 | 2026-08-24 | 30B/A3B MoE | OpenMDW-1.1 | Ollama nemotron-3.5-lightning:30b-a3b `e7a64ff15fb1` | D: AA-LCR 60.3(reason only). DV 52.0. **In-window artifacts**: Base-BF16 (08-05), NVFP4/DSpark/DFlash (08-04/05) - Blackwell NVFP4 only, no new Mac evidence, decision unchanged | +| MemReader-4B-thinking (**reject**) | IAAR-Shanghai/MemReader-4B-thinking | 2026-04-08 | 4B | Apache-2.0 | - | DV LongMemEval 83.0% (4-tool pipeline, not standalone) | + +## New finds since 2026-07-01, in 3-35B class, text-capable (sorted by suitability) + +| # | Model | Date (created/lastMod) | Params | License | Mac artifact | llama.cpp b11057 | Q4/Q8 (exact bytes) | Ctx | Key evidence | +|---|---|---|---|---|---|---|---|---|---| +| 1 | **LFM2.5-8B-A1B** | 2026-05-28/08-24 (public 2026-05-28, before this sweep's since-2026-07-01 window; only the Aug touches fall inside it) (DSpark 08-10, DSpark-GGUF 08-19) | 8.47B/~1.5B active MoE (`Lfm2MoeForCausalLM`, 32 experts×top4) | **LFM Open License v1.0**: Commercial Use is unlicensed for a Legal Entity (including affiliates it controls, or that control it, at >=50% ownership) at or above US$10,000,000 annual revenue; non-commercial or local use is free regardless of revenue; any breach terminates the licence automatically | official GGUF+MLX(4/5/6/8bit,bf16)+ONNX+unsloth mirror; **Ollama tag confirmed: `lfm2.5:8b` and `lfm2.5:8b-a1b-q4_K_M` both resolve (200) — corrects the earlier 404 finding** | `Lfm2MoeForCausalLM` registered, conversion/lfm2.py:97 | 5.16GB/9.01GB | 128k | D/DV: IFEval 91.84 (+12.4 over LFM2-8B-A1B's 79.44); vendor claims IFEval parity w/ Gemma-4-26B-A4B. No LongMemEval. AA-LCR: 0 recorded on Artificial Analysis (a literal zero, not a missing value — genuineness unverifiable) | +| 2 | **Gemma-4-26B-A4B-it** | 2026-03-11/07-20 (public by 2026-04-01, before this sweep's since-2026-07-01 window; only the 07-20 touch falls inside it) | 25.2B/3.8B active MoE (`Gemma4ForConditionalGeneration`, 128 experts×top8) | Apache-2.0, plain: the README's `license_link` (`ai.google.dev/gemma/docs/gemma_4_license`) redirects to `ai.google.dev/gemma/apache_2` ("Apache License 2.0"), and the card's own body text reads "License: Apache 2.0" - **corrects the earlier custom-Gemma-terms caveat; no restriction beyond Apache-2.0** | official Google QAT Q4_0 GGUF (14.44GB text+1.19GB mmproj); **ggml-org/gemma-4-26B-A4B-it-GGUF**; **lmstudio-community** GGUF+MLX-4bit; **Ollama gemma4:26b (200)** | `Gemma4ForConditionalGeneration` registered, conversion/gemma.py:631 | Q4(official QAT Q4_0)=14.44GB+1.19GB mmproj; Q8(ggml-org, exact)=26.86GB+0.81GB mmproj | 260k | D: AA Intelligence Index v4.3.2 (reasoning, est.) ~17 - different benchmark version than baseline's plain AA-LCR. **AA-LCR v1.1 recovered on verification: 65.7 (reasoning) / 42.3 (non-reasoning)** - see Unknowns. D-third-party (dev.to/aurigait): "a July 2026 refresh" reportedly cut JSON/parameter errors (Tau2 Telecom +10.1%, TB2 +4.5% on sibling 31B) - **checked the repo's own commit log: the 07-20 commit is "Add response_template to tokenizer_config.json," 07-15 is a chat-template fix ("null handling, reasoning preservation, turn-tag balance, input validation"); these are template/config commits consistent with, but not confirmed as, the specific change the third-party figures describe**. Sibling Gemma-4-31B: 0.798 Value-Accuracy, independent Structured-Output-Benchmark (arXiv 2604.25359). Liquid's own comparison card also lists this model's IFEval 91.40 / IFBench 47.25; Artificial Analysis independently records IFBench 72.4 (reasoning) / 45.4 (non-reasoning) | +| 3 | **granite-4.2-8b** | 2026-08-07/09-04 | 8.79B dense (`GraniteForCausalLM`) | Apache-2.0 | official ibm-granite GGUF+MLX(4/6/8bit); **lmstudio-community** GGUF+MLX; **Ollama granite4.2:8b (200)** | `GraniteForCausalLM` registered, conversion/granite.py:17 (long-supported) | 5.35GB/9.35GB | 128k | DV: BFCL v4 tool-calling **52.39** at the pinned revision (f8de16cdcdbc6c779ca517604e050d82cc119e44, 2026-09-04) - corrects 50.29, which was a superseded 2026-08-18 card; eesel.ai's independent review reports 52.39 too; native "reason-before-call" tool design; thinking on/off switch | +| 4 | **granite-4.2-30b** | 2026-08-07/09-04 | 29.28B dense | Apache-2.0 | same coverage; **Ollama granite4.2:30b (200)** | registered, same class | 17.72GB/31.11GB | 128k | DV: BFCL v4 **61.39** (best of new finds). D (Artificial Analysis hosted API, not a Mac measurement): 76.6 tok/s, AA's own tier label is "slower than average" - corrects an earlier "notably slow, 28 tok/s" misreading; AA reserves "notably slow" for Qwen3.8 (xhigh) at 45.5 tok/s, not this model. **AA-LCR v1.1 recovered on verification: 49.0** - see Unknowns. Latency risk vs the 20s gate remains (fully dense, no MoE discount), just less severe than 28 tok/s implied | +| 5 | **Qwen3.8-27B** | 2026-08-05/08-14 | 27.78B dense, `Qwen3_5ForConditionalGeneration` (**same wrapper class as the production control**) | Apache-2.0 | unsloth+lmstudio-community GGUF(Q4_K_M-UD 16.46GB/Q8_0 29.05GB, exact)+MLX(4/5/6/8bit); **Ollama qwen3.8:27b (200)** | `Qwen3_5ForConditionalGeneration`/`Qwen3_5MoeForCausalLM` registered, conversion/qwen.py:637,643 | 16.46GB/29.05GB (exact) | 260k | D: AA Intelligence Index (xhigh) 34 - composite index, not plain AA-LCR. **AA-LCR v1.1 recovered on verification: 82.0 (xhigh) / 79.7 (medium) / 77.3 (low reasoning), 69.3 (non-reasoning)** - the only new find that beats both the control (46.0/70.0) and the trial (64.3/71.7) on AA-LCR, on both axes; see Unknowns. "Beats all medium 40-150B models, ties DeepSeek-V4-Flash-0731(304B)" is a Hacker News comment's reading of an older Intelligence Index scale (news.ycombinator.com/item?id=49334544), not an Artificial Analysis claim | +| 6 | **Gemma-4-12B-it** | 2026-05-23/07-20 | 11.96B dense, omni (`Gemma4UnifiedForConditionalGeneration`) | Apache-2.0, plain (same correction as row 2 - the license_link caveat is resolved, not a restriction) | official Google QAT Q4_0 GGUF (6.98GB+0.175GB mmproj); **ggml-org/gemma-4-12B-it-GGUF**; **lmstudio-community** GGUF+MLX-4bit+QAT-GGUF; **Ollama gemma4:12b (200)** | `Gemma4UnifiedForConditionalGeneration` registered, conversion/gemma.py:812,933 | Q4(official QAT)=6.98GB+0.175GB; Q8(ggml-org,exact)=12.67GB+0.159GB | 260k | D: AA Intelligence Index non-reasoning 20 / reasoning 14 (v4.3.2, not plain AA-LCR). AA has a direct release-comparison page vs Qwen3.5-9B (Sources). Closest in-class size analogue to the control. **AA-LCR v1.1 recovered on verification: 63.7 (reasoning) / 35.0 (non-reasoning)** - see Unknowns | +| - | granite-swash-3b-a600m (excluded: base-only, short ctx) | card date 2026-07-07 | 3.02B/0.598B active MoE (`GraniteMoeSWAForCausalLM`, 48×top4, SWA+sinks) | Apache-2.0 | none found | `GraniteMoeSWAForCausalLM` registered, conversion/granite.py:124 | n/a | **8192 only** | README states verbatim: "early exploration...small-scale preview for upcoming Granite series," explicitly a **base model, not instruct** - JSON/tool reliability unproven | +| - | ContextPilot-14B/8B/E4B (excluded: license) | 2026-08-27/08-31 | 14.77B dense (`Qwen3ForCausalLM`) | **LICENSE file read directly (primary source, not a paraphrase): body is Apache License 2.0 text with an added "Section 0": "ContextPilot-14B is made available solely for the purpose of scientific research and development. You shall not use it for any other purpose." Hard research-only restriction, confirmed, blocks adoption** | community-only: bartowski/mradermacher GGUF (not ggml-org/unsloth/lmstudio-community); no Ollama tag (404) | `Qwen3ForCausalLM` long-supported | - | 40960 native | DV, likely self-reported (arXiv 2608.28476, Tencent authors): 72.20 avg on 4 long-context benchmarks. On-theme (proactive context mgmt, structured memory, retrieval, "soft context offloading") but license-blocked | +| - | Nemotron-Labs-Audex-30B-A3B (excluded: wrong modality) | created 2026-07-06 | ~30B (shards) | other | - | - | - | - | File tree = whisper/audiogen/VAE/speech-decoder shards - **audio S2S system, not a text memory LLM** despite MoE-like name | +| - | granite-4.2-3b (in-class, not ranked - too small for consolidation, notable for extraction speed) | 2026-08-07/09-02 | 3.66B dense | Apache-2.0 | official+lmstudio-community GGUF/MLX; Ollama tag not probed | registered | 2.24GB/3.89GB | 128k | Same family as rows 3-4, no separate benchmark found | + +## Oversized "Flash"-branded releases this window (excluded on total-footprint alone; 64GB unified memory) + +| Model | Date | Total params (safetensors, exact) | License tag | +|---|---|---|---| +| Qwen/Qwen3.8-Flash-Next | 2026-08-24/27 | 180.0B | other | +| zai-org/GLM-5.3-Flash | 2026-08-25/09-07 | 321.3B | mit (tag) | +| zai-org/GLM-5.3 | 2026-08-25/09-04 | 753.3B | other | +| deepseek-ai/DeepSeek-V4-Flash-0731 | 2026-07-31/08-01 | 304.2B | mit (tag) | +| XiaomiMiMo/MiMo-V2.6-Flash-RL | 2026-09-21/22 | 310.8B | mit (tag) | + +## Memory-specialised releases: dedicated sweep + +Searched: (1) `IAAR-Shanghai` (MemReader-4B-thinking's org) full listing; (2) `search=MemReader`; (3) `search=memory&pipeline_tag=text-generation`, +filtered `createdAt>=2026-07-01`, limit 100. Results: IAAR-Shanghai's own follow-up to MemReader is **MemPrivacy-4B/1.7B-RL/SFT** (2026-05-08/09, +privacy-focused, not extraction/consolidation) and **MemReranker-4B** (2026-04-27, a `text-classification` cross-encoder, not a generative LLM) - +both predate the window; IAAR-Shanghai's only in-window releases are Metis-4B/9B/27B (VL, 2026-07-16, no memory framing). The generic memory-keyword +search surfaced ~35 in-window hits, all small independent/research repos with no vendor backing or benchmark evidence meeting the "serious" bar: +domain-siloed SFT variants (`Jiarui-Wang/MemSFT-Qwen3-{Bio,Law,OpenSWI}-Memory-{1.7B..8B}`, 2026-08-01), architecture probes +(`Rubin-Wei/MemoryDecoder-{OLMo,Qwen3,Pythia}-*`, 2026-07-23), and single-author projects (`Jepoxy/LFM2.5-350M-Memory-Extractor`, +`vtava/Laya-MemoryFusion-*`, `OLAResearchX/memoryathena-*`). None outranks ContextPilot (above, license-excluded) or the generalist rows 1-6 on +any documented metric. **No serious vendor-backed memory-specialised LLM release found in-window beyond ContextPilot.** + +## Orgs checked with nothing new/relevant since 2026-07-01 + +- **openai**: gpt-oss-20b/120b unchanged since 2025-08-26 (lastModified); no successor in any window. +- **moonshotai**: only Kimi-K3 touched (lastMod 2026-09-02, createdAt 2026-06-13, pre-window, VL, oversized). +- **meta-llama**: nothing since Llama-4/Llama-Guard-4 (2025-04/05); stalest org checked. +- **baidu**: nothing since ERNIE-4.5 updates (2025-11-26); latest any touch is an OCR model (2026-07-29). +- **stepfun-ai** (org is `stepfun-ai`, not `stepfun`): nothing since Step-3.7-Flash (2026-05-23, VL, pre-window). +- **allenai**: only climate/science models in window (ACE2S, SamudrACE) - no LLM. +- **google, tencent, nvidia, zai-org, deepseek-ai, XiaomiMiMo**: covered above; remainder of their Aug-Sep 2026 output is OCR/embedding/rerank/UI-agent/TTS/robotics/vision, not general memory LLMs. + +## Revisions (40-hex, HF API `.sha`, resolved 2026-09-27) + +Baseline: Qwen/Qwen3.5-9B `c202236235762e1c871ad0ccb60c8ee5ba337b9a` · Qwen/Qwen3.6-35B-A3B `995ad96eacd98c81ed38be0c5b274b04031597b0` · +LiquidAI/LFM2.5-2.6B-GGUF `e7caca5d835a3901a8e0d63e94009429bafafdfc` (base LiquidAI/LFM2.5-2.6B `654f9463ce32b05d0429d76fe1f580b27d4c1ac0`) · +nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 `a9904d24bcc1d289a1950fa9d2b978c47cf903b9` · IAAR-Shanghai/MemReader-4B-thinking `3fcb57e5653ec5d733eea4df61f4b339d4ac0e92` + +New finds: LiquidAI/LFM2.5-8B-A1B `5dd22602c2e9f6a097b1de4c4efe0658b605015c` · google/gemma-4-26B-A4B-it `4d7ae4984b7db7de8f8457170b3f1a419ee76d52` · +google/gemma-4-12B-it `707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7` · ibm-granite/granite-4.2-8b `f8de16cdcdbc6c779ca517604e050d82cc119e44` · +ibm-granite/granite-4.2-30b `9e668ce1c538387ef24d3644e9b0606647762636` · ibm-granite/granite-4.2-3b `e459acceac81e5fe67c07d9cfc72329a332e7eb1` · +Qwen/Qwen3.8-27B `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` · ibm-granite/granite-swash-3b-a600m `568a94fcfeff42a8cab9ee2ffea89fde12071282` · +tencent/ContextPilot-14B `8eafd8356c5c5e713e92e623c8f1d346de6d0f95` · nvidia/Nemotron-Labs-Audex-30B-A3B `4e7e342045736382ddf3e2952c313847a08642b8` + +Oversized: Qwen/Qwen3.8-Flash-Next `de4b8e4d43b917e7706784d8bb445c9af86a3540` · zai-org/GLM-5.3-Flash `eb9eb208eb0d988989d07a6a12d0fdeb5f52574a` · +zai-org/GLM-5.3 `aca966e4e02791568aa6a4ced368624b3d897f42` · deepseek-ai/DeepSeek-V4-Flash-0731 `7872f01b1d1fe23eabc4c98b48bffcef5a386062` · +XiaomiMiMo/MiMo-V2.6-Flash-RL `5711b268169967567844e1e560e8a3966da959b1` + +llama.cpp b11057 pin: `https://github.com/ggml-org/llama.cpp/tree/59657a613ab0fa4ab327d6c790123dff30bfbd67` (tag `b11057`, matches catalog). + +## Sources (by row) + +- AA pages: https://artificialanalysis.ai/models/gemma-4-26b-a4b-non-reasoning · https://artificialanalysis.ai/models/gemma-4-26b-a4b · + https://artificialanalysis.ai/models/gemma-4-12b-non-reasoning · https://artificialanalysis.ai/models/gemma-4-12b · + https://artificialanalysis.ai/models/granite-4-2-30b · https://artificialanalysis.ai/models/granite-4-2-8b · + https://artificialanalysis.ai/models/qwen3-8-27b-non-reasoning · https://artificialanalysis.ai/models/qwen3-8-27b · + https://artificialanalysis.ai/models/releases/comparisons/gemma-4-12b-vs-qwen3-5-9b (direct control comparison) · + https://artificialanalysis.ai/articles/gemma-4-everything-you-need-to-know +- LFM2.5-8B-A1B: https://www.liquid.ai/blog/lfm2-5-8b-a1b · https://benchlm.ai/models/lfm2-5-8b-a1b +- Granite 4.2: https://huggingface.co/blog/ibm-granite/granite-4-2 · https://github.com/ibm-granite/granite-4.2-language-models · + https://www.eesel.ai/blog/granite-4-2-review (BFCL v4 52.41/52.39/61.39, IBM-reported; corrects the earlier 50.29 for the 8B, superseded 2026-08-18 card; 52.41 not independently attributed to a specific row in this sweep) +- Gemma-4 JSON refresh (third-party, not confirmed against the primary commit log - see row 2 caveat): + https://dev.to/system_rationale/part-3-making-gemma-4-agents-production-ready-guardrails-structured-outputs-and-self-healing-575n · + https://aurigait.com/blog/gemma-4-features-benchmarks-guide/ +- Structured Output Benchmark (independent, Gemma-4-31B 0.798): https://arxiv.org/pdf/2604.25359 +- ContextPilot: https://huggingface.co/tencent/ContextPilot-14B/raw/main/LICENSE (primary, read directly) · https://arxiv.org/html/2608.28476v1 +- Qwen3.8-27B composite-score corroboration: https://news.ycombinator.com/item?id=49334544 + +## Unknowns / evidence gaps + +- **AA-LCR v1.1 recovered on verification** (the client-side-rendered score the first pass could not extract via curl) - recovered directly from the Artificial Analysis page data, not re-derived: Qwen3.5-9B (control) 46.0/70.0(reason), Qwen3.6-35B-A3B (trial) 64.3/71.7, Nemotron-3.5-Lightning 60.3(reason only) all reproduce the baseline/table exactly, cross-validating the recovery method. New finds: Qwen3.8-27B 82.0 (xhigh) / 79.7 (medium) / 77.3 (low reasoning), 69.3 (non-reasoning); Gemma-4-26B-A4B-it 65.7 (reasoning) / 42.3 (non-reasoning); Gemma-4-12B-it 63.7 (reasoning) / 35.0 (non-reasoning); granite-4.2-30b 49.0; granite-4.2-8b 45.0; LFM2.5-8B-A1B 0 (recorded, a literal zero - genuineness unverifiable). Only Qwen3.8-27B beats both the control and the trial on this metric; Gemma-4-26B-A4B-it and granite-4.2-30b both trail the control itself now that their scores are known. +- No LongMemEval / consolidation-specific / listwise-reranking benchmark found for any new candidate (same gap the + baseline records for Qwen3.6-35B-A3B itself). +- LFM2.5-8B-A1B active params: name implies ~1B, Liquid's blog says "~1.5B" - both reported, not reconciled. +- p95 latency at 4-8K tokens (ai-memory's 20s/5%-fallback gate) was not measured for any candidate; qualitative AA + speed figures (Artificial Analysis's hosted API, not a Mac measurement) are granite-4.2-30b 76.6 tok/s, AA's own + label "slower than average" (not "notably slow" as an earlier draft had it - AA reserves "notably slow" for + Qwen3.8 (xhigh) at 45.5 tok/s), and granite-4.2-8b 84.9 tok/s. + +## Ranked top 3 + +### (a) Hindsight/MemPalace extraction (runs on every retain; throughput + JSON reliability dominate) + +1. **LFM2.5-8B-A1B**, conditional on clearing the LFM1.0 license gate (free for non-commercial/local use; commercial + use unlicensed at/above US$10M annual revenue including >=50%-controlled affiliates; any breach terminates it). + ~1.5B active beats Qwen3.5-9B's 9B-all-active on per-token cost, and IFEval 91.84 is a strong instruction-following + number (neither baseline model has one on record) - though not the only one surfaced: Gemma-4-26B-A4B-it also + carries IFEval 91.40/IFBench 47.25 on Liquid's own comparison card, and AA-LCR is now known for this model too + (0, recorded - genuineness unverifiable). Vs Qwen3.6-35B-A3B: fewer active params still (1.5B vs 3B), a plausible + speed win, but zero memory/consolidation evidence (the trial 35B-A3B at least has the +18.3 AA-LCR proxy); Ollama + now carries a tag (`lfm2.5:8b`, confirmed 200 - corrects the earlier 404 finding). +2. **Gemma-4-26B-A4B-it** - the deciding factor for this role is throughput, and it wins that outright: 3.8B active + vs granite-4.2-8b's 8.8B fully dense (no speed win over the control at all). Now unambiguous Apache-2.0 (the + license_link caveat is resolved, no longer a differentiator against granite). Carries IFEval 91.40/IFBench 47.25 + on Liquid's own card (AA independently records IFBench 72.4/45.4) - a closer proxy to JSON-instruction reliability + than granite's tool-calling BFCL. AA-LCR: this model's 65.7 reasoning / 42.3 non-reasoning vs granite-4.2-8b's + newly-recovered single-mode 45.0 - Gemma ahead on the reasoning comparison, granite slightly ahead against Gemma's + non-reasoning figure; not a clean win for either, so throughput remains the deciding factor above. Caveat: fits + this role only at Q4 (14.44GB+1.19GB mmproj) - the Q8 build (26.86GB+) + exceeds the 35B-A3B trial's own footprint. +3. **granite-4.2-8b** - same size class as Qwen3.5-9B (8.8B vs 9B dense, no speed win over the control), but new + (Sept 2026), unambiguous Apache-2.0, and ships fresh native tool-calling evidence (BFCL v4 52.39 at the pinned + revision) the baseline lacks - the only BFCL number of the two. Vs Qwen3.6-35B-A3B: loses the active-compute race + (8B active > 3B active) - wins only if Mac memory headroom, not latency, is binding (5.35GB vs 22.6GB Q4 on disk). + Demoted behind Gemma-4-26B-A4B-it now that Gemma's license caveat is gone and this role's own stated discriminator + (throughput) favors Gemma's MoE by more than 2x. + +### (b) ai-memory consolidation/reranking (bounded 20s / <=5% fallback at 4-8K; quality-bound) + +**Rewritten on AA-LCR**, which this table calls the deciding metric for this role, now that the v1.1 scores are +recovered (see Unknowns) instead of "not recoverable." Non-reasoning: Qwen3.8-27B 69.3 > trial Qwen3.6-35B-A3B 64.3 > +control Qwen3.5-9B 46.0 > Gemma-4-26B-A4B-it 42.3. Reasoning: Qwen3.8-27B 82.0 > trial 71.7 > control 70.0 > +Gemma-4-26B-A4B-it 65.7 > granite-4.2-30b 49.0. Only Qwen3.8-27B beats both the control and the trial; Gemma and +granite-30b both now measurably trail the control itself, which was hidden while their scores were unknown. + +1. **Qwen3.8-27B** - the only candidate that clears the deciding metric: AA-LCR 82.0 (xhigh) / 79.7 (medium) / 77.3 + (low reasoning), 69.3 (non-reasoning), beating both the control (46.0/70.0) and the trial (64.3/71.7) on every + reasoning tier and non-reasoning. Same qwen3_5 lineage and same `ForConditionalGeneration` wrapper as the + production control (prompt/tokenizer compatibility, not a new liability - this was confirmed, not assumed), 260k + context. "Beats all medium 40-150B models" is a Hacker News reading of an older Intelligence Index scale, not an + AA claim - dropped as evidence here. Open risk: fully dense (27.78B active vs the trial's 3B active), and p95 + latency against the 20s gate is unmeasured for this model on this Mac - the one thing standing between this pick + and displacing the trial. +2. **Gemma-4-26B-A4B-it** - longer native context (260k) than the control, MoE 3.8B active vs the control's + 9B-all-active, and the widest Mac tooling of any new find (official QAT GGUF + ggml-org + lmstudio-community + + Ollama tag, vs the control's single Ollama MLX tag); now unambiguous Apache-2.0. On the deciding metric it loses + to both the control and the trial (65.7/42.3 vs 70.0/46.0 and 71.7/64.3) - a materially weaker position than "no + AA-LCR number was recoverable" implied - but it still clears granite-4.2-30b by a wide reasoning margin (65.7 vs + 49.0). A third-party JSON-reliability-refresh claim exists for this window but is **not confirmed against the + primary commit log** (row 2). +3. **granite-4.2-30b** - best tool-calling score of any new find (BFCL v4 61.39), 128k context, unambiguous + Apache-2.0. On AA-LCR (49.0) it is the weakest of the three role candidates, trailing the control on reasoning by + 21 points. Vs the 35B-A3B trial: similar total disk (30B vs 35B) but **fully dense** (30B active vs 3B active); + AA's hosted-API speed figure is 76.6 tok/s labeled "slower than average" (not "notably slow" - that label is + AA's wording for Qwen3.8 xhigh at 45.5 tok/s) - a real but smaller latency risk against the 20s p95 gate than + the earlier 28 tok/s figure implied, and one the MoE trial does not carry at all. + +**Operating point:** at Q4, every ranked row (1-6) fits well under the Qwen3.6-35B-A3B trial's 22.6GB GGUF footprint +(largest is granite-4.2-30b at 17.72GB); at Q8, rows 2 (26.86GB+), 4 (31.11GB) and 5 (29.05GB) exceed it - this is the +memory-headroom argument behind picks (a)#2 and (b)#2, and it only holds at Q4/QAT-Q4_0, not at Q8. + +Evidence classes follow the catalog's own D (discovery/independent) / DV (discovery/vendor) convention; every score +above is one or the other, never NCE (no native Mac harness run was performed in this sweep). + +
diff --git a/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/memsys.md b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/memsys.md new file mode 100644 index 000000000..0281226d6 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/memsys.md @@ -0,0 +1,67 @@ + + +**S3 input from mac-coordinator-64gb-20260925, 2026-09-27: `memsys` sweep table** (suggested path: `inputs/sweep-20260927-mac/memsys.md`). Sonnet sweep, independently verified by an Opus pass against primary sources; its "Verification" section lists every correction applied. The published and self-reported numbers here are discovery evidence, not S3 evidence. + +
Table + +# Memory-systems sweep — 2026-09-27 + +## Verification (2026-09-27, independent Opus pass) +- ai-memory: `19b6429` vs v2.4.1 ahead/behind counts were swapped (correct: ahead_by 46, behind_by 62, merge base `8ee81bede45b`); added that v2.4.0 is an ancestor of `19b6429` and v2.4.1 is not, and reframed the rollback-pin move as a trade-off rather than a recommendation. No local evidence file for this repo's git compare exists in this session's verify directory; command: `gh api repos/akitaonrails/ai-memory/compare/v2.4.1...19b6429`. +- GBrain: removed "now documented", "weakened" and "now feasible" — the local-provider docs already existed at the baseline pin v0.56.2.0, and embedding/reranking still default to hosted Voyage; #5284/#5449 are still open and #5449's fix series has not shipped in v0.59.0.0. No local evidence file for this repo in this session's verify directory; source as given: github.com/garrytan/gbrain issues #5284, #5449. +- claude-mem v13.28.0: corrected the installer-default quote — the general interactive (`npx`) installer pre-selects hosted "CMEM Pro" (a hosted *observer*, not a sync tier), the non-interactive default is the Anthropic (local) plan, and telemetry is on by default (opt-out). No local evidence file for this repo in this session's verify directory; source as given: thedotmack/claude-mem's installer script/CLI text. +- mcp-memory-service v11.14.0: confirmed Codex CLI is listed among supported MCP clients (client support, not native hooks). Source: verify/memsys/pypi-mcp-memory-service.json (package `description`, "CLI & Terminal AI (MCP)" section). +- Hindsight: added that the newer integration tags (`integrations/coding-agents/v0.7.0` @ `0c0869b7`, 2026-09-24; paperclip, obsidian, hermes) are not visible through `releases/latest`, and recommended pinning `integrations/coding-agents/v0.7.0` for the capture path; recorded the Mac trial blocker (pg0-embedded 0.15.2 / macOS Postgres 18.1.0 / openssl@3 dylibs / DYLD_* stripped under `initdb`'s `popen`/`/bin/sh` call). verify/memsys/pypi-hindsight-all.json confirms only the package version (0.10.1); the integration-tag and Mac-blocker specifics have no local evidence file in this session and are taken as given. +- Cognee: refined the disqualifier correction already in the draft with the specific commit (topoteretes/cognee-integrations @ `4042bcf8`, unpinned) and the running-server requirement, and made explicit that it stays deferred for other reasons (Docker/graph-store weight, no published benchmark). No local evidence file for this repo in this session's verify directory; source as given: topoteretes/cognee-integrations. + +Baseline: `catalogs/foundation/memory-stack-20260925.json` (repository_decisions) + +`evidence/artifacts/memory-stack-20260925/convergence.json`, both in +the repository checkout. All tags/commits below resolved 2026-09-27 via +`gh api repos/{owner}/{repo}/releases/latest` (falls back to `/tags` if no Release object +exists — none needed it) then `git/ref/tags/{tag}` → `git/tags/{sha}` for annotated tags, +i.e. GitHub Releases only, never a main-branch HEAD. Full commands run are in +`gh_fetch.sh` / `gh_raw.tsv` in this directory. Verified byte-identical against the +baseline's own pins for agentmemory, MemPalace, Hindsight, Honcho, OpenViking, Basic +Memory, Letta — the resolution method matches theirs. + +## Baseline systems + +| System | Latest release @commit, date | License | Local-on-Mac (no hosted key) | Claude/Codex integration | Storage/services | Best published benchmark | 2026-09-25 decision | Suggested change | +|---|---|---|---|---|---|---|---|---| +| **ai-memory** (akitaonrails/ai-memory) — production | Running: `19b6429` local 2.5-pre build (reports 2.4.0), not an official artifact. Official: **v2.4.1** @ `433a19f3d54d`, 2026-09-25T21:46 | MIT | Yes — Ollama 0.34.4 + qwen3-embedding:4b + qwen3.5-9b-64k, no hosted key | Production memory server shared by Claude Code and Codex (operational record; hook/plugin detail not re-verified this wave) | Ollama local serving, local store | NCE (Mac, descriptive): C3 aimem-qwen3 0.570 recall_all@5 (470/470), 20/20 on a drift-check reproduction. `evidence/artifacts/memory-stack-20260925/convergence.json#/rows/0` | retain (control) | **Verified**: `19b6429` vs `v2.4.1` = diverged (ahead_by 46, behind_by 62; merge base `8ee81bede45b`) — corrects an earlier swap of these two counts. v2.4.0 is a strict ancestor of the running `19b6429`; v2.4.1 is not. v2.4.1 is a 2.4.x maintenance release (Windows checksum LF fix, +126 commits over v2.4.0), NOT the running 2.5 code and NOT "the first official 2.5 release" the next-test needs. No official 2.5 exists yet. **Trade-off, not a recommendation**: moving the rollback pin to v2.4.1 means falling back to 62 commits of code production never ran, while also losing the 46 commits production did run that v2.4.1 lacks; staying on v2.4.0 loses only the latter, since it is a pure ancestor and introduces nothing untested. Decision (retain, control) unchanged; the rollback-pin choice is left open as a trade-off, not resolved here. | +| **agentmemory** (rohitg00/agentmemory) — trial | v0.9.29 @ `2d38dafede67`, 2026-08-16T13:56 — **unchanged** | Apache-2.0 | Yes — keyless mode (am-keyless) or MiniLM in-process | README confirms: **native plugin + 12 hooks + 54 MCP tools (Claude Code)**; **native plugin + 6 hooks + MCP (Codex CLI)**; also Cursor/Gemini CLI/OpenCode/etc. Note: baseline's Codex caveat (openai/codex#16430) is about **Codex Desktop**, a different surface from the **Codex CLI** hooks this README documents — re-verify which surface D2h targets | Local memory server on :3111, REST+MCP, no external DB required | NCE (Mac): am-minilm 0.821 recall_all@5 (470/470), +25.1pp over ai-memory C3 (Holm p=0.0001). Own bench doc: github.com/rohitg00/agentmemory/blob/main/benchmark/LONGMEMEVAL.md. `convergence.json#/rows/1` | trial | None — main moved (pushed 09-26) but no new tag. Clarify Codex CLI vs Desktop hook-dispatch scope before D2h. | +| **MemPalace** (MemPalace/mempalace) — trial | v3.10.0 @ `22fd87f09c19`, 2026-09-16T00:24 — unchanged | MIT | Yes, local palace mode | Hooks for both Claude Code and Codex + MCP (per baseline); not independently re-verified this wave | Local palace store; optional ghcr container | DV/vendor-scored: recall_all@5 0.857 raw / 0.887 hybrid v4 (BENCHMARKS.md 2026-04); third-party arXiv 2604.21284 puts palace modes at 89.4%/84.2% recall_any. Not independently reproduced by us. | trial | None — main moved (pushed 09-25), no new tag. | +| **Hindsight** (vectorize-io/hindsight) — trial | v0.10.1 @ `f8950b0c07d9`, 2026-09-21T15:24 — unchanged | MIT | Yes, but LLM-extraction-on-every-retain is heavy (~86h/470q at gpt-oss-20b Mac rate) | Shared capture for Claude and Codex via official coding-agents package (per baseline); **newer integration tags are not visible through `releases/latest`**: `integrations/coding-agents/v0.7.0` @ `0c0869b7`, 2026-09-24, plus separate paperclip/obsidian/hermes integration tags | Postgres + worker + local LLM | DV: LongMemEval-S QA 91.4% (Gemini-3) / 83.6% (gpt-oss-20b), arXiv 2512.12818 — reproduced only by co-authors, not independent. BEAM-10M 64.1%. | trial | **Pin `integrations/coding-agents/v0.7.0`** for the capture path (not surfaced by `releases/latest`). **Mac trial blocker recorded**: `pg0-embedded` 0.15.2's macOS Postgres 18.1.0 links `/opt/homebrew/opt/openssl@3` dylibs; `initdb` runs `postgres -V` through `popen`/`/bin/sh`, so macOS strips `DYLD_*` and the embedded DB cannot start on a Mac without Homebrew. | +| **Attemory** (AttemorySystem/attemory) — defer | v0.1.3 @ `603c03afa9a0`, 2026-08-13T06:54 — unchanged | MIT (repo); prebuilt core SDK unverified | Unverified — core is a closed prebuilt SDK | MCP support "only planned" (baseline) | Unverified (closed core) | DV: 96.38% recall_all@5 on cleaned LME-S (local Qwen3.5-9B, undated) | defer | None found; dormant since release (pushed_at 2026-08-13, no activity since). | +| **GBrain** (garrytan/gbrain) — defer | **v0.59.0.0** @ `e78f1c38b947`, 2026-09-26T23:00 — **changed** (was v0.56.2.0 @ `db56c778`; 3 minor bumps in ~48h) | MIT | Local-provider docs (Ollama / llama.cpp llama-server / LM Studio embeddings + a fully-local llama-server Qwen3-Reranker recipe) already existed at the baseline pin v0.56.2.0 — not new this wave. New-install default, and embedding/reranking generally, still default to hosted Voyage `voyage-4`. | Not confirmed as a native Claude Code/Codex plugin this wave (generic CLI/API + `docs/integrations/`) — unknown | PGLite (embedded) | DV: v0.59.0.0 changelog reports an "evidence-checking" QA reader moving 308→324/361 judged-correct (n=361, **not** recall_all@5/470 — different metric, not comparable). Prior: 95.53% "sessions within 5 chunks" (chunks ≠ sessions), 86.6% QA, hosted models. | defer | "Relies on hosted models" stands: embedding and reranking still default to hosted Voyage, and the local-provider docs cited above already existed at the baseline pin, not a new capability. reindex/embed hangs remain **open**: #5284 (PGLite COMMIT wedge, updated 09-23) and #5449 (checkpoint self-deadlock, opened 09-24) — #5449 has a fix series but it has not shipped in v0.59.0.0. Defer stands. | +| **Mem0 OSS** (mem0ai/mem0) — defer | Python OSS **v2.2.1** @ `94c3fe9f238f`, 2026-09-25T17:35 — **changed** (was v2.2.0 @ `47a69e1e72dc`; +15 commits/47 files). Repo-wide "latest release" is `ts-v3.3.1` (TS SDK, same commit, published 75s later) — different package line, not the baseline's Python pin. | Apache-2.0 | Needs glue code; v2.2.1 fix #7350 stops ConfigManager injecting OpenAI's baseURL/model into other providers — small win for local/non-OpenAI use | Claude Code plugin needs a Platform key (not passwordless) | Pluggable vector store, Python/TS SDKs | DV: managed platform 94.4% LongMemEval / 92.5% LoCoMo / 64.1% BEAM-1M / 48.6% BEAM-10M (mem0.ai/blog, updated 2026-09-22, self-reported); OSS 91.0%/88.6%; independent-ish re-run 73.8% (Maximem, 2026-05-27) | defer | None material; patch available (v2.2.1), includes a local-provider-relevant fix. | +| **Graphiti/Zep** (getzep/graphiti) — defer | v0.30.2 @ `eaa4128681bc` (now resolved; baseline had it unresolved), 2026-09-08T20:38 — unchanged | Apache-2.0 | No — MCP server needs Docker + Neo4j | Generic MCP server only; no Claude-Code- or Codex-specific plugin found in README | Neo4j + Docker | Zep 71.2% LongMemEval (GPT-4o judge) per mem0.ai/blog (2026-09-22), "consistent across Zep's own reporting and an independent academic comparison" per that source, not independently confirmed by us; LoCoMo 94.7% claimed by Zep vs. 75.1% in ByteRover's third-party re-test (disputed) | defer | None; Neo4j weight confirmed. | +| **Supermemory** (supermemoryai/supermemory) — defer | server-v0.0.8 @ `5d2b5855fe49` (now resolved), 2026-08-17T18:39 — unchanged | MIT (repo); server ships as binaries | **No** for the documented/default path — primary MCP endpoint is hosted (`mcp.supermemory.ai`); self-hosting the binary server is possible but not the advertised path | Has an explicit Claude Code plugin (github.com/supermemoryai/claude-supermemory) + hosted MCP for many clients | Hosted service by default; binary self-host alternative | No LongMemEval/LoCoMo/BEAM score found this wave — unknown | defer | None; flag hosted-by-default MCP as a local-on-Mac concern. | +| **Cognee** (topoteretes/cognee) — defer | v1.6.1 @ `eb90d0374075` (now resolved), 2026-09-24T17:50 — unchanged | Apache-2.0 | Needs Docker for its MCP/UI service; supports local LLM/embedding providers via `.env` (not confirmed no-hosted-key by default) | **Corrects the baseline's group disqualifier**: README shows **both** a Claude Code plugin **and** a Codex plugin with hooks (`~/.codex/config.toml` → `hooks = true`, `codex plugin add cognee@cognee`), both in topoteretes/cognee-integrations @ `4042bcf8` (unpinned); both need a running Cognee server | Docker; graph store + vector store | No LongMemEval/LoCoMo/BEAM score found this wave — unknown | defer | **Disqualifier correction**: the "no Claude/Codex plugin" disqualifier no longer applies — cognee has native Claude Code + Codex plugins with hooks (topoteretes/cognee-integrations @ `4042bcf8`, unpinned, needs a running Cognee server). It **stays deferred for other reasons**: Docker/graph-store weight, no published retrieval benchmark. | +| **claude-mem** (thedotmack/claude-mem) — defer | **v13.28.0** @ `57c383037344`, 2026-09-26T18:37 — **changed** (was v13.25.3; 5 releases in ~24h: 13.26.0 cloud-sync migration off Cloudflare → new `sync.cmem.ai` hub, 13.26.1 sync fixes, 13.27.0 agent-cost-report skill pricing Claude Code + Codex transcripts via OpenRouter, 13.27.1 fixes, 13.28.0 non-interactive installer defaults to Anthropic provider) | Apache-2.0 | Core plugin is local (SQLite + hook scripts). The general interactive installer (`npx`) **pre-selects hosted "CMEM Pro"** — a hosted *observer* (inference via `cmem.ai`), not a sync tier as an earlier draft called it; the **non-interactive installer default is the Anthropic plan (local memory)**. Telemetry is **on by default (opt-out)**. | Native Claude Code plugin: 5 lifecycle hooks (SessionStart, UserPromptSubmit, PostToolUse, Stop, SessionEnd) + MCP search tools. No evidence of a native Codex lifecycle hook — it only reads Codex transcripts for the cost-report skill. | Local SQLite + worker; CMEM Pro is a hosted observer (inference via cmem.ai), pre-selected by the interactive installer, opt-out via `--provider host` or the non-interactive install path | No LongMemEval/LoCoMo/BEAM score found this wave — likely the actual disqualifier vs. the group's generic text | defer | None to the decision; the CMEM Pro hosted-observer surface should be checked for "local-on-Mac, no hosted key" before any trial, since the general interactive installer pre-selects it. | +| **MemOS** (MemTensor/MemOS) — defer | v2.0.34 @ `41bf5c7fa89e` (now resolved), 2026-09-23T05:24 — unchanged | Apache-2.0 | Not re-verified this wave | Not re-verified this wave | Not re-verified this wave | DV only, or none (baseline) | defer | None found; not deeply re-checked beyond version/license this wave. | +| **Honcho** (plastic-labs/honcho) — reject | v3.2.1 @ `79cb31645f4e`, 2026-09-24T19:43 — unchanged, byte-identical to baseline | AGPL-3.0 | Needs Postgres + Redis + worker; embedder can silently fall back to OpenAI (#915) | SDK-based, not hook/plugin native | Postgres + Redis + worker | DV 90.4% QA | reject | None. | +| **OpenViking** (volcengine/OpenViking) — reject | v0.4.21 @ `3fca2577520f`, 2026-09-20T08:45 — unchanged, byte-identical to baseline | AGPL-3.0 | Not re-verified this wave | Not re-verified this wave | Not re-verified this wave | DV LoCoMo 80.32% | reject | None. | +| **Basic Memory** (basicmachines-co/basic-memory) — reject | v0.23.2 @ `c0bd87c6d5a4`, 2026-08-25T20:44 — unchanged, byte-identical to baseline | AGPL-3.0 | Not re-verified this wave | Not re-verified this wave | Not re-verified this wave | DV LoCoMo R@5 76.4% | reject | None. | +| **Letta server** (letta-ai/letta) — reject (retired) | 0.16.8 @ `1131535716e8`, 2026-05-14T17:14 — unchanged (last server release; still no server release in >4 months, confirms "retired") | Apache-2.0 | n/a | n/a | n/a | none published | reject | None. | + +## New / newly-relevant entrants (released or pushed since 2026-08-01, not in baseline) + +| System | Latest release @commit, date | License | Local-on-Mac | Claude/Codex integration | Best evidence | 2026-09-25 decision | Suggested | +|---|---|---|---|---|---|---|---| +| **Memorix** (AVIDS2/memorix) | v1.9.6 @ `e44f2a296b5f`, 2026-09-23T07:57 | Apache-2.0 | Unverified — "memory living under the Git project" suggests local files; embedding/LLM requirements not confirmed | Explicitly markets MCP compatibility with **Claude Code, Codex**, Cursor, Windsurf, Gemini CLI, Antigravity, OpenClaw, Hermes, Pi, Copilot, Kiro, OpenCode, Trae | None found this wave — no LongMemEval/LoCoMo/BEAM score | n/a (new) | Candidate for **defer**: gap = no published retrieval benchmark. Re-check next wave. | +| **mcp-memory-service** (doobidoo/mcp-memory-service) | v11.14.0 @ `bed24ef72141`, 2026-09-25T09:44 | Apache-2.0 | Plausibly local (REST API + knowledge graph + "autonomous consolidation"; typical of this class) but not independently confirmed this wave | Markets itself for Claude + LangGraph/CrewAI/AutoGen pipelines; **lists Codex CLI among its supported MCP clients** (a compatible-clients listing — client support, not a native Codex lifecycle-hook integration) | None found this wave — no LongMemEval/LoCoMo/BEAM score | n/a (new) | Candidate for **defer**: same benchmark gap. Re-check next wave. | + +## Considered, not given a row + +- **ByteRover / campfirein/byterover-cli** (formerly Cipher, 4,966 stars): last release v3.16.1 (2026-05-27), last push 2026-06-25 — **outside** the since-2026-08-01 window despite appearing at 92.2–96.1% LoCoMo / 92.8% LongMemEval-S in third-party leaderboards (self-reported; mem0.ai's own blog flags internal inconsistency — ByteRover's own comparison table scores Zep at 75.1% and Mem0 at 66.9%, both far below those vendors' self-reports, and ByteRover's 96.1% LoCoMo figure is flagged as identical to a separate "ZeroMemory" claim). License shows `NOASSERTION` via GitHub API. Has a Claude plugin (campfirein/brv-claude-plugin), no Codex plugin found. +- **JordanMcCann/agentmemory**: **name collision** with the baseline trial candidate (rohitg00/agentmemory) — a different, unrelated repo. MIT license, claims "#1 on LongMemEval — 96.2% (481/500)," but created and pushed exactly once on 2026-03-26, zero GitHub Releases, 47 stars — dormant, self-reported, unreproduced. Excluded as neither released nor updated since 2026-08-01. +- **Mastra** (mastra-ai/mastra): very active TS agent framework (pushed 2026-09-27), not a Claude-Code/Codex memory plugin itself. Its "Observational Memory" research claims 94.87% LongMemEval (gpt-5-mini judge; benchmark code at `github.com/mastra-ai/mastra/tree/main/explorations/longmemeval`) per mastra.ai/research. License shows `NOASSERTION` via GitHub API — unresolved, not manually checked this wave. +- **Agent Zero Memory**: arXiv 2608.29606 claims 95.60% LongMemEval / 93.60% LoCoMo (ahead of Mastra 94.87%, Mem0 92.50%, ByteRover 2.0 92.20%) per mem0.ai's blog summary. No discoverable public GitHub repository (searched; only unrelated small hobby projects matched). Paper-only. +- **DolphinBench**: a new Mem0-authored memory benchmark ("Mapping the Pareto Frontier of Agent Memory," blog dated 2026-09-22) — not one of the four benchmarks named in this task (LongMemEval-S/M, LoCoMo, BEAM, MemoryAgentBench). Flagged as an emerging benchmark to watch, not scored here. + +## MemoryAgentBench + +No system checked in this sweep publishes a MemoryAgentBench score as of 2026-09-27 (baseline or new entrants). Gap, not resolved. + +
diff --git a/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/rerank.md b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/rerank.md new file mode 100644 index 000000000..8b7b153d4 --- /dev/null +++ b/blueprints/memory-layer-s3/inputs/sweep-20260927-mac/rerank.md @@ -0,0 +1,117 @@ + + +**S3 input from mac-coordinator-64gb-20260925, 2026-09-27: `rerank` sweep table** (suggested path: `inputs/sweep-20260927-mac/rerank.md`). Sonnet sweep, independently verified by an Opus pass against primary sources; its "Verification" section lists every correction applied. The published and self-reported numbers here are discovery evidence, not S3 evidence. + +
Table + +# Reranker sweep, 2026-09-27 — Mac memory-lane trial candidates + +## Verification (2026-09-27, independent Opus pass) +- KaLM-Reranker-V1-Small Mac path reworded: not a "patched llama.cpp fork" but upstream llama.cpp pinned at 277a105 plus 7 patches bundled in the HF repo (`llama.cpp/patches/0001`-`0007`); stock llama.cpp does not recognize t5gemma2; only CUDA and CPU builds are documented, no Mac path. Source: verify/rerank/r_kalmgguf_README.md; verify/rerank/m_kalmgguf.json (patch file listing); verify/rerank/g_commit_277a105.json (https://github.com/ggml-org/llama.cpp/commit/277a105dc8f8643dab54331926a9830860a03292). +- Qwen3-Reranker-0.6B: "verified... b11057-compatible" softened to maintainer-demonstrated on an October 2025 build (b11057 itself untested); footprint corrected to 0.64GB actual (639,153,184 bytes). Source: verify/rerank/g_issue16407.json + g_issue16407_comments.json (https://github.com/ggml-org/llama.cpp/issues/16407, ggerganov comment 2025-10-03T10:25:20Z); verify/rerank/m_ggml06.json (`totalFileSize`). +- BGE-v2-m3 "beats every Qwen3-Reranker size" limited to NDCG@10/MAP (margins +0.0053/+0.0114 over Qwen3-0.6B, no CIs reported); Qwen3-Reranker sizes lead on R@5/R@20 in the same Table 6, so treated as an effective tie; added the one unresolved report of a bge GGUF ranking differently on M1 llama.cpp than on vLLM/CUDA. Source: verify/rerank/a_html_2605.06132v2.html (Table 6, full row incl. R@5/R@20); verify/rerank/g_issue16407_comments.json (heibaidaolx123 comment, 2026-02-02T09:33:48Z). +- Table 6: GPT-4o-mini was evaluated on 481 of 500 LongMemEval queries (19 skipped on API failures). Source: verify/rerank/a_html_2605.06132v2.html ("GPT-4o-mini evaluated 481 out of 500 queries on LongMemEval"). +- Labeled the "Ranked top 3" list explicitly as a deliberately varied trial lineup (best stock artifact / best memory evidence / lowest-friction ONNX), not the Full table's sort-key order. +- Added two omitted repos: ggml-org/jina-reranker-v1-turbo-en-GGUF (sha 607d8664, 2024-12-12, apache-2.0, no memory evidence) and Qwen/Qwen3-VL-Reranker-2B/8B (created 2026-01-07, multimodal). Source: verify/rerank/m_ggml_jina_v1t.json; verify/rerank/s_qwen_rerank.json. + +Method: HF API (`api/models`, `pipeline_tag=text-ranking`, per-org `search=rerank`, sorted `createdAt`), +per-model full metadata (`api/models/{id}`, `.sha`=40-hex, `.lastModified`, `.tags`, `.safetensors.total`), +raw READMEs via `huggingface.co/{id}/raw/main/README.md`, arXiv HTML for 2606.22807v3 (KaLM) and +2605.06132v2 (MemReranker), GitHub API for ggml-org/llama.cpp#16407 (state + comments), WebSearch for +MTEB/LightOn context. No ctx_fetch_and_index/ctx_search tool was available in this environment (ToolSearch +found none); curl substituted for it against public read-only HF/GitHub/arXiv endpoints, consistent with +this repo's own `pin_method.huggingface` convention. No weights downloaded. + +Memory footprint: BF16 = params × 2 bytes; Q8_0 ≈ params × 1.06; Q4_K_M ≈ params × 0.6 — "computed" unless +marked "actual" (from a published file size). Mac-suitability sort key, applied in order: (1) runs +unmodified on stock llama.cpp b11057 `--reranking`, or ships an official MLX build, or an official ONNX +arm64 build; (2) commercial-usable licence; (3) no custom loader / no `trust_remote_code`; (4) +memory-retrieval-specific evidence (LongMemEval/LMEB), independent weighted over same-team self-reported. + +Baseline file: catalogs/foundation/memory-stack-20260925.json — non_repository_decisions rows 1310-1600 +(production ai-memory-llm-reranking L1310; MemReranker-4B L1338; KaLM-Reranker-V1-Small L1381; +ettin-reranker-400m-v1 L1425; Qwen3-Reranker-4B L1469; Jina Reranker v3.5 L1511; ms-marco-MiniLM-L6-v2 L1553). + +**Correction (post first draft):** the MemReranker paper's Table 6 has 10 numeric columns +(MAP MRR NDCG@1 NDCG@3 **NDCG@10** NDCG R@3 R@5 R@20 F1); the first draft of this file misread column 6 +("NDCG", i.e. a different, non-@10 metric) as NDCG@10 for every row except MemReranker-4B. Corrected below. + +## Production (context, not ranked) +ai-memory LLM listwise reranking = Ollama `qwen3.5-9b-64k` (`qwen3.5:9b-mlx`, tag digest `5efc0dd55d4a`), +reasoning low, 20s hard timeout. No harness result yet (C4 has not run on the Mac). Footprint: unspecified +exact GB in the catalog (9B nominal, MLX-quantized; not resolved here) — unknown. + +## Shared evidence table (arXiv 2605.06132 Table 6, LongMemEval n=500, BGE-M3 top-50 candidates, self-reported by MemReranker's authors) +| Model | MAP | NDCG@10 | +|---|---|---| +| GPT-4o-mini (pointwise LLM judge, hosted; 481/500 queries scored — 19 skipped on API failures, all other rows cover the full 500) | 0.5684 | 0.6265 | +| Qwen3-Reranker-4B | 0.6501 | 0.7185 | +| BGE-v2-m3 | 0.7069 | 0.7573 | +| Qwen3-Reranker-8B | 0.6966 | 0.7551 | +| Qwen3-Reranker-0.6B | 0.6955 | 0.7520 | +| Gemini-3-Flash (pointwise LLM judge, hosted) | 0.7259 | 0.7733 | +| MemReranker-0.6B (*no published HF repo found — see unknowns*) | 0.7538 | 0.7987 | +| **MemReranker-4B** | **0.8043** | **0.8354** | + +## Full table, sorted by Mac suitability (numbered = ranked contenders; unnumbered = out of scope/hosted/rejected) + +| # | Model (HF id) | Rev (40-hex) | Date (created→lastMod) | Params | License | Mac artifact | Custom loader | Footprint | Strongest evidence | +|---|---|---|---|---|---|---|---|---|---| +| 1 | Qwen/Qwen3-Reranker-0.6B | e61197ed45024b0ed8a2d74b80b4d909f1255473 | 2025-05-29→2026-04-16 (not in baseline; pre-window) | 596M | Apache-2.0 | **ggml-org official GGUF** `ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF` rev a02f48bb4f057028298c21fa033da2b30d7742d5 (2025-10-03). Verified, not inferred: llama.cpp maintainer **ggerganov** posted this exact repo in issue #16407 on 2025-10-03 ("I just uploaded a correct one"), ran `llama-server -hf ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF ... --rerank` and showed well-separated scores (0.999 vs 0.0001-class); issue closed 2025-10-09 | No (`custom_code:false`) | 0.64GB actual (639,153,184 bytes) | LongMemEval-500: NDCG@10 0.7520/MAP 0.6955 — ties Qwen3-Reranker-8B, beats the 4B, **effectively ties BGE-v2-m3 (0.7573/0.7069): BGE leads NDCG@10/MAP by +0.0053/+0.0114 with no CIs reported, but this model leads BGE on R@5/R@20 in the same Table 6** — differs from it mainly in artifact provenance | +| 2 | cross-encoder/ettin-reranker-400m-v1 | 5dca36282a5d85f368d2544002513a29159b4c9e | 2026-05-15→2026-05-19 | 395M | Apache-2.0 | Official ARM64 ONNX int8 `onnx/model_qint8_arm64.onnx`, 397.5MB actual | No | 397.5MB actual; BF16 0.79GB computed | MTEB(eng,v2) retrieval-after-rerank 0.6091 vs ms-marco-MiniLM 0.5082; NanoBEIR 0.7193 (DV, no memory-specific number) | +| 3 | cross-encoder/ettin-reranker-1b-v1 | 7d20e9baad17016fdf5549c08f69a2d7ca3e60c3 | 2026-05-15→2026-05-19 | 1.03B | Apache-2.0 | Same family/day as #2; ONNX arm64 sibling assumed, **not independently confirmed** (siblings fetch returned empty, likely rate-limited) | No | BF16 2.06GB computed | Same family as #2, no size-specific score found | +| 4 | cross-encoder/ms-marco-MiniLM-L6-v2 (+ Xenova ONNX) | 233902d25c440f23af6f7d6e94d2946bac0bee0a / a09144355adeed5f58c8ed011d209bf8ee5a1fec | 2026-08-09 (ce) / 2025-06-30 (Xenova) | ~23M (documented) | Apache-2.0 (ce copy); Xenova ONNX copy carries **no license tag** | Bundled ONNX by **Xenova (community conversion account, not the original publisher)**, ships inside agentmemory 0.9.29, off by default — flagged community | No | <100MB | 0.5082 (DV, ettin's table) — the floor every candidate above beats | +| 5 | BAAI/bge-reranker-v2-m3 | 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e | 2024-03-15→2024-06-24 | 568M | Apache-2.0 | None official found (community ONNX/GGUF conversions exist); it is the model llama.cpp's own server README names as its reference reranking example | No | BF16 1.14GB computed | LongMemEval-500: NDCG@10 **0.7573**/MAP **0.7069** — **leads on NDCG@10 and MAP only** (+0.0053/+0.0114 over Qwen3-Reranker-0.6B, no confidence intervals reported); Qwen3-Reranker sizes lead on R@5/R@20 in the same Table 6, so this is effectively a tie rather than beating every Qwen3-Reranker size outright. One unresolved report (llama.cpp issue #16407, comment 2026-02-02) found a bge-reranker-v2-m3 GGUF ranking differently on M1 llama.cpp (Metal) than on vLLM/CUDA — still the reigning "current leader" baseline by reputation | +| 6 | IAAR-Shanghai/MemReranker-4B | 22890bb4adff85b83b2181ed91e940ffb5a4ec45 | 2026-04-27→2026-05-12 | 4.41B | Apache-2.0 | Community MLX only; no official Mac artifact | No (`custom_code:false`) | BF16 8.82GB computed | **Best in sweep**: LongMemEval-500 NDCG@10 0.8354/MAP 0.8043, beats BGE-v2-m3, all Qwen3-Reranker sizes, GPT-4o-mini and Gemini-3-Flash (0.7733) in its own Table 6. Same-team/self-reported (IAAR-Shanghai, GitHub linked to MemTensor/MemOS, correspondence @memtensor.cn) | +| 7 | Qwen/Qwen3-Reranker-8B | 77d193c791ed757ca307ee72715aa132723da912 | 2025-05-29→2026-04-16 | 8.19B | Apache-2.0 | No official GGUF (community only) | No | BF16 16.4GB computed | LongMemEval-500: NDCG@10 0.7551/MAP 0.6966 — barely ahead of the 0.6B at 13× the size, both behind BGE-v2-m3 | +| 8 | Qwen/Qwen3-Reranker-4B | 22e683669bc0f0bd69640a1354a6d0aebcfeede5 | 2025-06-03→2026-04-16, unchanged since baseline | 4.02B | Apache-2.0 | No ggml-org/publisher GGUF (only the 0.6B has one); community conversions only, mixed quality | No | BF16 8.04GB computed | MTEB-R 69.76 (DV); LMEB LongMemEval 53.72 (KaLM paper, D); LongMemEval-500 NDCG@10 0.7185/MAP 0.6501 — **worst of its own family** | +| 9 | ibm-granite/granite-embedding-reranker-english-r2 | d09d3d6971b689bf9c23839e45a470874d46e13a | 2025-08-04→2025-11-06 | 149.6M | Apache-2.0 | None official (community ONNX-fp16 fork only, `TahaRauf/...`) | No | BF16 0.30GB computed — **smallest text-only reranker found** | BEIR Avg 55.0-55.8, MLDR(en) 44.9-45.8, Miracl(en) 54.2-55.2 (own card; beats ms-marco-MiniLM-L12-v2, bge-reranker-base/large; ties gte-reranker-modernbert-base). No memory-specific number | +| 10 | mixedbread-ai/mxbai-rerank-large-v2 | ca7e1ee484c37c0ddd8d178a9a5c33cec575c5e6 | 2025-03-03→2026-04-08 | 1.54B | Apache-2.0 | None official found | No | BF16 3.09GB computed | "Current leader" baseline by name recognition; no memory-specific number captured in this sweep | +| 11 | zeroentropy/zerank-2-reranker | 5eae30d5ee3c6b2df2ef6d723bde45172d761c4c | 2025-11-19→2026-07-24 (in-window touch) | 4B (documented) | Apache-2.0 | None official; safetensors only; community GPU-only quant (FP8/compressed-tensors) exists, no Mac build | No (card: trust_remote_code no longer required as of May 2026) | BF16 ~8GB computed | Self-reported NDCG@10 domain avg 0.6714 > Cohere rerank-v3.5 (0.5847) and Gemini-2.5-Flash-listwise (0.5999); "Conversational" domain 0.6140 is a generic proxy, not LongMemEval itself | +| 12 | zeroentropy/zerank-1-small-reranker | a65fd51c450e9b47fdddab98e31166ecad21af8d | 2025-06-30→2026-07-24 (in-window touch) | 1.72B | Apache-2.0 | None official | Unverified (not re-checked) | BF16 3.44GB computed | zerank-1 avg 0.6456 (unclear if this specific "small" size or the full zerank-1-reranker; not disambiguated in fetched text) | +| 13 | KaLM-Embedding/KaLM-Reranker-V1-Small (+ -R2) | V1: 8c2141c8af7417cd4a58fe7b58f79f1c00ad1d49 (2026-06-02→2026-09-21); R2: 68b74f09ff2ab5df9b9a9a48c812a81aaff8815e (2026-08-14→2026-09-23) | see above | V1 2.12B total/1B activated (encoder-decoder, T5Gemma2) | Apache-2.0 | **Official GGUF exists but fails the stock-runtime criterion**: `KaLM-Embedding/KaLM-Reranker-V1-Small-Q8_0-GGUF` (91ba8005363b484c48d07eb48b2c35fa5ea0aa99, 2026-07-02) needs **upstream llama.cpp pinned at 277a105 plus 7 patches bundled in the Hugging Face repo** (`llama.cpp/patches/0001`-`0007`) + custom build target `llama-kalm-reranker`; stock llama.cpp does not recognize the t5gemma2 architecture; only CUDA and CPU builds are documented, no Mac path | **Yes** — ships `kalm_cross_encoder.py`/`kalm_reranker.py`/`kalm_reranker_utils.py` + a vLLM plugin; not flagged by the `custom_code` tag, confirmed instead by file listing | GGUF actual 1.82GB (Q8_0, text params 1.70B); BF16 4.24GB computed | LMEB dialogue: V1-Small 87.58 (baseline, DV); V1-Large 89.54 on the LongMemEval sub-task specifically, beating Qwen3-Reranker-8B's 78.93 in the same KaLM paper table (arXiv 2606.22807v3 Appendix G, re-verified in this session — same-team, runs LMEB itself). **R2 has no LMEB/LongMemEval number in v3 of the paper** — confirmed independently, not just stale from 09-25 | +| 14 | nvidia/llama-nemotron-rerank-1b-v2 | 828765652b05bd439c9789d2a6d093db1caa1443 | 2025-10-16→2026-08-26 (in-window update) | 1.24B | "other": OpenMDW-1.1 + Llama 3.2 Community Licence; card states "ready for commercial use" but attribution/acceptable-use conditions attach | None official (safetensors only) | **Yes** (`custom_code:true`, `trust_remote_code=True` in its own usage snippet) | BF16 2.47GB computed | No BEIR/MTEB number captured in the fetched README excerpt; multilingual (26 languages), 8192-token cross-encoder from Llama-3.2-1B, hosted alternative via NVIDIA NIM | + +### Out of scope / hosted-only / rejected (not numbered — not comparable contenders for this role) +| Model | Rev (40-hex) | Date | License | Why excluded | +|---|---|---|---|---| +| jinaai/jina-reranker-v3.5 (+ -mlx, -GGUF) | v3.5: e8a93f33f0b22108f8c2364f8484ce3422552fbc; mlx: 3dd4ac901ccdcac85abe3815df0a0aaaf44e4a21; GGUF: 884f7c67aa3ac24edb89064da8c7bfd03f4a90f5 | 2026-07-14→2026-07-30 | **CC-BY-NC-4.0 — rejected, non-commercial** (baseline, unchanged) | Best Mac artifact coverage of the entire sweep (official MLX **and** official GGUF) but disqualified twice over: non-commercial licence, and newly-confirmed `custom_code:true` (baseline hadn't flagged this) | +| lightonai/LightOn-rerank-{PW,LW}-{0.8B,2B,4B} | PW-2B: c5333533d5ebea2142490d5eacd42f0c8363c808 (representative) | 2026-07-08→2026-07-22 | Apache-2.0 | Multimodal-first (`image-text-to-text`, Qwen3.5-VL LoRA): LW-2B nDCG@10 62.66 on ViDoRe V3 (document-**image** retrieval); "competitive on text-only BEIR" claimed with no exact figure found. No custom loader, would need the VL stack regardless | +| Alibaba-NLP/core-reranker-8b (+ -2b) | 8b: d95d36a34300e34b87cc1793827ac798642c4488 | 2026-08-30→2026-09-04 | CC-BY-4.0 (commercial-OK) | **N/A for text memory retrieval** — a Qwen3-VL-Reranker-based multimodal *compositional-image-reasoning* reranker (COLA/SugarCrepe++/NegBench/MCMR); no BEIR/LongMemEval text evidence at all. (-2b sibling exists, sha not separately verified) | +| Contrastive-LM/CLM-v0.1-8B | not resolved (excluded before full fetch) | 2026-09-21 | Apache-2.0 | An agent state/action **trajectory verifier** (DeepSWE 81.6%, Terminal-Bench 2.1 87.6%), tagged text-ranking/reranker but not a document reranker | +| voyageai/rerank-3 | 2d2c29517b5c3bbf130c879fda12b1678e598158 | 2026-09-23 | apache-2.0 tag, **no weights published** | Hosted API only (siblings = config/tokenizer, no safetensors/bin) | +| CohereLabs/rerank-v3.5 | d9c89bde0658d73e59e60818bcfcdf8eb388d930 | 2025-01-02→2026-03-25 | none published | Hosted API only (3 siblings, no weights); scores 0.5847 avg NDCG@10 in zerank-2's own table — below zerank-1/2 and Gemini-3-Flash | +| ggml-org/jina-reranker-v1-turbo-en-GGUF | 607d8664c787e517e5d6e339d21f680f9002c931 | 2024-12-12 (pre-window) | apache-2.0 | **Omitted from the original sweep, added on verification.** ggml-org official GGUF (jina-bert-v2 arch, 37.6M params, 77MB f16) — otherwise a genuine stock-runtime candidate, but pre-window and no memory-specific evidence found | +| Qwen/Qwen3-VL-Reranker-2B (+ -8B) | 2B: 4bd860ac4f15ad1897a214615cccc700f8f71818; 8B: b212dc8c91a8164aef1ea2de9c1a867611e75c04 | 2026-01-07 | apache-2.0 per the Qwen3-Reranker family, **not independently verified for these two VL repos** | **Omitted from the original sweep, added on verification.** Multimodal (vision-language) sibling of the Qwen3-Reranker text family — out of scope for text-only memory retrieval | + +## Negative findings (publishers/orgs swept with nothing new since 2026-07-01) +- **BAAI**: newest is bge-reranker-v2.5-gemma2-lightweight (2024) and the Matroyshka-ReRanker family (2025-05); no update in the window. +- **mixedbread-ai**: mxbai-rerank-{base,large}-v2 last touched 2026-04-08; no v3. +- **Tencent**: no `rerank`-named model under the `Tencent` HF org. +- **MemTensor**: no dedicated reranker product; ships MemOperator-{0.6B,1.7B,4B} and MemReader-4B (2025-07, different function, not verified as rerankers) — but MemReranker-4B (row 6) is affiliated with this same group per its paper's GitHub link and author email domain. +- **answerdotai / jhu-clsp**: the ettin family lives under `cross-encoder`, not these org accounts directly. +- **Qwen**: no "Qwen3.5-Reranker" exists — production's own listwise LLM family has no dedicated-reranker sibling on HF. +- The 4-6 week window (since ~2026-08-16) from named serious text-reranker publishers contains only the KaLM V1→R2 refresh (Aug 14) and Alibaba-NLP's multimodal core-reranker (Aug 30, out of scope) — nothing new from Qwen, BAAI, jina, mixedbread, zeroentropy, or ettin specifically in that narrower window; nvidia's rerank-1b-v2 got a same-repo update Aug 26. + +## Ranked top 3 for the Mac trial lineup + +This lineup is a deliberately varied trial set — best stock artifact (#1), best memory evidence (#2), lowest-friction ONNX (#3) — not the Full table's sort-key order; see that table above for the sort-key ranking. + +1. **Qwen/Qwen3-Reranker-0.6B** (not in baseline; pre-window release, current-leader pick). Wins: the only model in the entire sweep with an official, unmodified artifact that the llama.cpp maintainer himself demonstrated working — **maintainer-demonstrated on an October 2025 build** (ggerganov, llama.cpp issue #16407, 2025-10-03); compatibility with the b11057 tag specifically is untested, not verified. Apache-2.0, 0.64GB actual, no custom loader. Loses against production: its LongMemEval-500 number (0.7520 NDCG@10) is competitor-reported, not our own NCE, and it is an **effective tie with BAAI/bge-reranker-v2-m3** on that same shared table (BGE ahead +0.0053 NDCG@10/+0.0114 MAP, this model ahead on R@5/R@20) — the two differ mainly in artifact provenance, not a clean evidence-quality win either way. A single small cross-encoder pass is far cheaper per-candidate than a 9B generative listwise call, but Ollama has no rerank endpoint, so either candidate still adds a second server process (llama-server/ONNX runtime) beside Ollama. +2. **IAAR-Shanghai/MemReranker-4B** (already a baseline trial; reaffirmed with richer evidence). Wins: the single strongest concrete memory-specific result found anywhere in this sweep — it beats Gemini-3-Flash (a closer analog to an LLM-based reranker than GPT-4o-mini) on the exact same 500-query LongMemEval split, in the paper's own Table 6. Loses: entirely self-reported by its own authors (IAAR-Shanghai/MemTensor); only a community MLX build exists for Mac; baseline's own next-step note stands — it needs "an upstream reranker slot in the chosen memory system; we do not maintain forks" (catalogs/foundation/memory-stack-20260925.json L1349). +3. **cross-encoder/ettin-reranker-400m-v1** (already a baseline trial; reaffirmed as the zero-friction pick). Wins: the only candidate that is both officially packaged for Mac (ARM64 int8 ONNX, 397.5MB, ready today) and needs no patch, no custom loader, no conversion work — lowest integration risk of anything examined. Loses: zero memory-specific or LongMemEval evidence anywhere (this sweep and the baseline agree); its only strength is generic MTEB/NanoBEIR numbers, unproven on the dialogue/temporal-reasoning queries the production reranker actually has to handle. + +Close misses, explicitly not top-3: **BAAI/bge-reranker-v2-m3** now reads as arguably the strongest all-round candidate on the evidence (ties pick #1 on the shared table — ahead on NDCG@10/MAP, behind on R@5/R@20 — no custom loader, Apache-2.0) but has no confirmed official Mac artifact of any kind, only community conversions — kept at #5 in the ranked table rather than top-3 pending an artifact check. **KaLM-Reranker-V1(-R2)** has the strongest *self-reported* raw number (89.54 for V1-Large on LongMemEval specifically) but the *worst* Mac-artifact story of the sweep — its own GGUF card says stock llama.cpp/Ollama/LM Studio are not supported, and the non-GGUF path needs custom loader code either way; R2 (the checkpoint anyone would actually want) has no memory number at all yet. **zerank-2** has a strong generic self-reported benchmark and no `trust_remote_code`, but no Mac artifact of any kind and no LongMemEval-specific number. No candidate in this sweep satisfies both "official stock-runtime artifact" and "memory-benchmark evidence" at once. + +## Unknowns +- **MemReranker-0.6B** appears in the shared evidence table (NDCG@10 0.7987/MAP 0.7538, second-best in the table) but no matching HF repo was found under the IAAR-Shanghai org listing — the smallest memory-strong reranker in the evidence isn't (yet, or publicly) shipped as weights. +- ettin-reranker-1b-v1's ONNX arm64 sibling file was not independently confirmed (empty siblings response, likely a rate-limited/short JSON body from the HF API, not re-verified with a second attempt). +- Whether zerank-2's/zerank-1-small's "Yes/No"-logit classification head converts cleanly through `convert_hf_to_gguf.py`'s Qwen3-Reranker path is unverified — not attempted by anyone found in this sweep. +- nvidia/llama-nemotron-rerank-1b-v2 and lightonai/LightOn-rerank-* have no BEIR/MTEB/memory number captured from the portions of their cards fetched (README excerpts, not full cards); a further read could surface one. +- No dedicated ctx_fetch_and_index/ctx_search tool was available in this session (ToolSearch returned only WebSearch and ai-memory's memory_query); HF/GitHub/arXiv reads went through direct `curl` to public read-only endpoints instead, consistent with this repo's own `pin_method.huggingface` convention — flagged as a tooling deviation, not a silent one. +- ai-memory/memory_query was not called: no `.ai-memory.toml` was found under the working directory to source `workspace`/`project`, and none was supplied by the caller; the task's own baseline JSON was used as the source of record instead. +- MTEB reranking leaderboard (huggingface.co/spaces/mteb/leaderboard) was not scraped beyond one WebSearch pass (JS-heavy Space, no plain-HTML table); no current #1 pulled from it directly — current-leader claims here rest on the KaLM/MemReranker papers' own comparison tables instead. +- LightOn's exact text-only BEIR number (blog says "competitive," no figure) and the precise meaning of its reported "0.6469" for the 4B variant (scale/benchmark unstated in the fetched snippet) were not resolved. +- Alibaba-NLP/core-reranker-2b's full 40-hex revision was not re-fetched after the first truncated read (excluded from the role anyway; low priority to close). + +
diff --git a/blueprints/memory-stack/longmemeval/A17-SUPERSESSION.md b/blueprints/memory-stack/longmemeval/A17-SUPERSESSION.md new file mode 100644 index 000000000..83ad2d348 --- /dev/null +++ b/blueprints/memory-stack/longmemeval/A17-SUPERSESSION.md @@ -0,0 +1,31 @@ +# A17: the confirmatory rerun R1 is superseded by S3 (PROPOSED, 2026-09-27) + +**Status: proposed, not in effect.** It takes effect only when the user confirms it at the freeze of the S3 preregistration (`blueprints/memory-layer-s3/PREREGISTRATION.md`, sections 9 and 12, PR #390). Until then nothing runs under either protocol. + +This amendment continues the memory-stack protocol's own lineage: +- A1–A16.3 live in the frozen [`PREREGISTRATION.md`](PREREGISTRATION.md) (v3, #380) and [`v4/PREREGISTRATION.md`](v4/PREREGISTRATION.md) (v4, #386). +- Frozen text is never edited, so A17 is a new file. + +## What A17 was to be + +VelaNext was retired, so the confirmatory rerun R1 needed another host and a new amendment naming it before any run (`docs/decisions/2026-09-25-retire-vela-velanext.md:36-44`). The workstation `nativestack-5975wx-20260925` took R1 as A17 (`docs/decisions/2026-09-25-workstation-sota-refresh.md:542-546`; issue #274). No A17 text was ever written before this file, and no session held a draft (checked on 2026-09-27 with the sessions that had owned it). + +## The amendment + +1. **Host.** R1's host would have been the workstation `nativestack-5975wx-20260925`. +2. **R1 is not run under A1–A16.3.** The durable-memory decision on both hosts is instead made by the S3 head-to-head preregistration, which names its own reference, arms, statistics and decision rule. +3. **No look is spent.** No confirmatory arm of A1–A16.3 has run on any host. No C3′, C4′, C4 or D2h result exists (`docs/decisions/2026-09-25-retire-vela-velanext.md:43-44`; `catalogs/foundation/memory-stack-20260925.json:1318`; `docs/decisions/2026-09-27-mac-single-writer-staged.md:48`). This amendment therefore selects nothing after seeing results. +4. **Carried into S3:** + - The user's 2026-09-25 decision to use official ai-memory **v2.4.0** as the control (issue #274; `docs/decisions/2026-09-25-workstation-sota-refresh.md:228`) becomes S3's reference, as C4′ with C3′ as a diagnostic. + - This protocol's C3, C4 and D2h configurations are the starting point for S3's C3′, C4′ and agentmemory configuration (v4 `lme_harness.py:759-784,1562-1575,1618-1619`), with two stated differences: + - v2.4.0 has no embedding-prefix setting, so C3′/C4′ use Qwen3-Embedding-4B's native, unprefixed input; + - agentmemory's adapter must execute the pinned shipped hooks or prove equivalence (S3 sections 1 and 2). + - The user's 2026-09-27 decision that the memory layer is chosen on merit and the winner supersedes ai-memory (PR #383) is S3's decision. +5. **Not carried into S3:** + - this protocol's family α values (0.0333, 0.0167), its +5 pp/−2 pp rule and its one-look-per-family rule; + - the v4 = v3 reproduction gate; + - the VelaNext-only drivers and summarizer. + + S3's statistics, gates and new isolated runner replace them. +6. **Descriptive record.** The historical Mac C3 (0.570, build `19b6429`) and all earlier Mac results stay descriptive, as A15.2 already requires. +7. **Frozen files stay frozen.** The v3 and v4 files remain the provenance and adapter source S3 reads. They are never edited and never run on a shared host (v4 `README.md:21-24,86-93`). diff --git a/docs/decisions/2026-09-30-memory-s3-r7.md b/docs/decisions/2026-09-30-memory-s3-r7.md new file mode 100644 index 000000000..ce535d7d9 --- /dev/null +++ b/docs/decisions/2026-09-30-memory-s3-r7.md @@ -0,0 +1,242 @@ +# Decision: memory-layer S3 preregistration r7, a symmetric merit rule behind a maintenance gate (2026-09-30) + +**Decided by:** the coordinator session, under the user's standing requirements of 2026-09-30 as relayed verbatim by the coordinator's r7 brief. A delegated Opus 5.5 worker drafted it on branch `claude/memory-layer-s3-r7-20260930`, based on `f7e5e228` (PR #390's head `d161f691` merged with `origin/main`). + +**Status:** draft, with the pre-freeze candidate additions committed on the PR branch, not frozen. The user accepted the three decisions below on 2026-09-30; the complete execution bundle still needs freeze acceptance and a GPT-6 cross-family review ([preregistration, section 11](../../blueprints/memory-layer-s3/PREREGISTRATION.md#11-freeze-procedure)). Round 1 addressed the initial GPT-6 needs_changes verdict; round 2 addressed Opus's review of `a266a383`, GPT-6's N1 re-check and settled usefulness/fallback choices. The separately authorized round-3 brief repairs the review of `0d027e06` and applies the accepted dependency scope. The pre-freeze candidate additions below supersede round 3's landscape deferral for the named candidates. + +## Context + +r6 (DRAFT, 2026-09-27) compared seven primary memory systems against a frozen ai-memory v2.4.0 reference arm (C3′/C4′): +- a candidate replaced the reference only through a superiority route or a non-inferiority-plus-cost route; +- the Holm family held 14 hypotheses; +- LongMemEval's own scorer and judge prompt were to run unchanged. + +On 2026-09-30 the user restated the memory-layer requirements below, and the coordinator's brief asked for r7 with eight named changes. + +## Why: the user's words + +As relayed by the brief, which quotes the user's messages of 2026-09-30; this record did not hear them directly: +- the memory layer uses "the best sota repos, with highest quality performance, rag and token efficiency for full memory foundation"; +- "the best candidates only with evidence, always check the repos quality, active maintenance, NEVER STALED REPOS"; +- "No bias but evaluate on the sotaness and the quality of repos itself"; the current incumbent is not the default. + +## Decision + +The preregistration becomes DRAFT r7 ([`blueprints/memory-layer-s3/PREREGISTRATION.md`](../../blueprints/memory-layer-s3/PREREGISTRATION.md)): +1. **Maintenance gate (section 1).** + - A repository is stale if it is archived, or if it has no default-branch commit in 90 days. This is the floor of OpenSSF Scorecard's Maintained check (v5.5.0 `checks/evaluation/maintained.go:31-32,88-100`). + - The gate runs through `gh api` at freeze and before every run. + - **ACCEPTED user decision 3:** every repository selected to run (harness, memory/RAG system or tool) must pass Maintained. Transitive dependencies require a per-arm SBOM and a clean osv-scanner result; stale transitive dependencies are flagged without blocking their harness. LongMemEval `9e0b455f` appears only as a citation, and its dataset file is static data pinned by sha256. +2. **Candidates at current releases, with tag and full commit (section 3).** + - Primary candidates: agentmemory v0.9.29, Hindsight v0.10.2, OpenViking v0.4.22, cognee v1.6.2, MemPalace v3.10.0, basic-memory v0.23.2 and memsearch v0.4.21. + - ai-memory v2.4.2 competes without protected status. + - Reserves: engram, Graphiti and supermemory-local. + - Mem0, Letta and beads are out on fit, each with its locator. + - These historical sets are superseded by "Pre-freeze candidate additions (2026-09-30)" below. +3. **Harness (section 4).** + - AMB runs byte-identical at `03c1d0f1`. + - The LongMemEval adapters, the recall scorer (implemented from the paper's metric definition) and the thin provider clients live in this repository. Since the repair round every system has one, Hindsight included, and no AMB provider is reused. + - AMB's locked mem0 1.0.5, cognee 0.5.4 and hindsight-all 0.4.17 implement no candidate arm, but provider-module imports execute. The accepted dependency scope replaces round 2's executing-closure maintenance rule: include imports in every arm's SBOM/OSV scan; unknown vulnerability acceptance keeps AMB pending. Flag `rank_bm25` 0.2.2 (supplied source's last commit 2024-10-08), without making its age a harness blocker. + - Answerer and judge are OpenAI-compatible `cx/gpt-6-astra-max` through the local gateway. + - Vendor-bias controls: a judge-free primary metric, equal recall-depth policy, one parity checklist for every system, a GPT-6 adversarial audit, and no use of the vendor leaderboard. Open QA receives the shortest final-ranking prefix covering five distinct sessions, capped at 20 items, with emitted context tokens recorded per arm. +4. **Symmetric merit rule (sections 6–7), replacing the reference and the replacement routes.** + - All-pairs paired cluster-bootstrap tests use r6's seed, 20260927, with Holm adjustment through `statsmodels` `multipletests` over the frozen m = k(k−1)/2 pairs. + - The leader on `recall_all@5` wins. Several systems sharing the maximum are all leaders. A system inside the 0.02 practical-equivalence band joins them in tie-break 1 only if Holm finds it significantly worse than no leader (repair round 1). Tie-break 1 is gross cost at frozen list prices; tie-break 2 is a Scorecard-based repo-quality score with an exact formula. +5. **Token ledger (5.2):** + - OmniRoute ingest usage per system and role; + - o200k recall payloads counted with gpt-tokenizer 4.0.0; + - native per-session usage from `claude -p --output-format json` and `codex exec --json`, run in scratch `CLAUDE_CONFIG_DIR` and `CODEX_HOME` homes; + - latency p95 and resources. +6. **RAG section (8).** + - A document-RAG head-to-head: qmd, LightRAG, a llama_index hybrid pipeline, OpenViking, memsearch and cognee search, each wrapped as an MTEB 2.21.10 `SearchProtocol` model. + - This historical set is superseded by "Pre-freeze candidate additions (2026-09-30)" below. + - Embedder and reranker selection on LoCoMo (memory), CoIR (code) and a disjoint half of the `MTEB(eng, v2)` retrieval tasks (documents). Incumbents compete as equals. + - S8 regenerates SWE-bench Verified tasks through Harbor's maintained adapter and stays pending until its own dated amendment. Under the user's round-2 decision it is a short-code do-no-harm check, not the main usefulness test or a coupling that blocks host selection. +7. **Long-workflow usefulness and fallback (ACCEPTED 2026-09-30).** + - LongMemEval-V2 `2cc8c540` is the main test: the same direct native harness argv per domain for all arms/control, five paired replications, superiority to no retrieval and no significant loss to any **eligible, completely observed** best memory system, with a separately frozen Holm family. Outcome-independent shared-trajectory haystack components are the resampling unit; preflight must verify at least 20 against downloaded files, or usefulness stays pending. A baseline final-failure rate above 5% makes the entire family pending. MemoryAgentBench `538026089` is secondary only for applicable native modes (currently agentmemory normalized-fact CR, pending acceptance); a vendored copy cannot represent a selected candidate. MemoryArena `6cd9de14` is stale and citation-only. This replaces the minus-one floor and S8 usefulness coupling. + - When complete valid evidence establishes that no system qualifies, deploy the operationally eligible system with the best primary score. Eligibility requires the selected-repository gate, dependency SBOM/OSV acceptance, demonstrated native deployment/lifecycle, gateway/network/latency/observer acceptance and no failed hard gate. There is no incumbent default. Fallback is labelled usefulness-unqualified; an empty set, insufficient usefulness clusters or a pending contender able to change qualification/selection remains unresolved. +8. **Status:** DRAFT r7; the complete freeze needs the user's acceptance and a GPT-6 cross-family review. + +## Evidence (live, 2026-09-30) + +- **Gate at 2026-09-30T02:52:42Z.** + - STALE: `snap-research/locomo` (last commit 2024-08-13), `xiaowu0162/LongMemEval` (2026-05-11), `harbor-framework/harbor-datasets` (2026-05-16), and also `CoIR-team/coir` (2025-06-30). + - ACTIVE: AMB, whose default-branch head is the pin `03c1d0f1`; mteb (release 2.21.10); ossf/scorecard; every candidate, reserve and excluded system; and every RAG candidate. + - The full capture, with every tag-to-commit resolution, image digest and Hugging Face revision, is [`inputs/r7/live-verification-20260930.json`](../../blueprints/memory-layer-s3/inputs/r7/live-verification-20260930.json). +- **Mode.** AMB scores a retrieval-typed dataset only from `raw_response["documents"]` (`runner.py:222-227`), which only its LLM-free `retrieval` mode supplies (`modes/retrieval.py:56-61`). `agentic-rag` constructs `GeminiLLM()` (`modes/agentic_rag.py:8,26-27`). +- **Supermemory.** AMB's `supermemory` provider hard-codes the cloud status URL and sends the key there (`memory/supermemory.py:77-85`). It also ignores `k` (`:7,113-119`) and returns one merged document (`:127-128`). +- **Mem0.** The Claude Code plugin defaults to `https://api.mem0.ai` (`integrations/claude-code-plugin/core/memory_core.py:31` at `94c3fe9f`), with a `MEM0_API_URL` override (`:1995`). It calls Platform routes (`:1916,1996,2532`) that the self-hosted server (`server/main.py:367-547`) does not serve. Codex uses `mcp.mem0.ai` (`docs/integrations/codex.mdx:71,78`). +- **Gateway cache.** OmniRoute's semantic cache serves only requests whose temperature is exactly 0 (`src/lib/semanticCache.ts:483-493`, identical at `a58000c7` and `v3.8.51`). AMB's header-less client omits temperature, so the cache cannot serve its calls. +- **LongMemEval scorer.** Its `eval_utils.py:6` calls `np.asfarray`, removed in NumPy 2.0 (`doc/source/release/2.0.0-notes.rst:197`). +- **Scorecard aggregate.** It is the risk-weighted mean of conclusive checks (`pkg/scorecard/scorecard_result.go:83-120`), with weights Critical 10, High 7.5, Medium 5 and Low 2.5 (`README.md:605-613`). +- **Repair round 1, 2026-09-30T04:23:48Z.** The reads went through `gh api graphql`, because the account's shared REST quota was refusing requests. They are captured in [`inputs/r7/repair1-verification-20260930.json`](../../blueprints/memory-layer-s3/inputs/r7/repair1-verification-20260930.json): + - the gate for the S8 sources: harbor, SWE-bench/SWE-bench, huggingface/datasets, AnswerDotAI/fastcore and astral-sh/uv ACTIVE; harbor-datasets still STALE; + - Harbor v0.23.0 resolves to `1e5c5c6d`, and its SWE-bench adapter loads `princeton-nlp/SWE-bench_Verified` (a separate dataset at `c104f840`), not the maintained `SWE-bench/SWE-bench_Verified` at `78f471bf`; + - AMB's `hindsight-http` provider sets `enable_observations=False` on every bank, and Hindsight 0.10.2 applies it as a bank override against its documented `true` defaults; + - GitHub's REST description at `af2c1025` supplies the check-run states, pagination and the 1000-suite limit. + +## Brief items adjusted on primary evidence + +- (a) "rag mode only" is kept for the answer-generating open-QA variant. The retrieval variant has to run AMB's `retrieval` mode (evidence above). +- (b) AMB's `supermemory` provider is not reused for supermemory-local; a thin client in this repository replaces it. +- (c) Mem0's plugin URL is a default with an override, not a hard-coded value. The exclusion stands on the Platform routes it calls. +- (d) Free VRAM on the RTX 4090 measured 12,726 MiB, then 12,005 MiB at 03:13:24Z, not about 13.7 GB. Nemotron-3-Embed-8B-BF16 does not fit and is `pending`. +- (e) CoIR's repository is stale too, so CoIR enters only as MTEB task data. +- (f) AMB's per-category LongMemEval judge prompts have no abstention branch. The 30 abstention questions form their own reported stratum, and the gap is a recorded validity limit. + +## Review and repair (2026-09-30) + +**Review.** GPT-6 reviewed the complete r7 draft (the preregistration and this record at `a146c419`, unchanged at `a6ca5072`) across model families, through the gateway as `cx/gpt-6-astra` at effort max. +- Verdict: **needs_changes**, with seven findings: three high and four medium. +- The coordinator checked the review's 12 quoted source excerpts against their sources and reports that all 12 matched. This repair did not repeat that check. +- The findings are captured verbatim in [`inputs/r7/gpt6-review-20260930.md`](../../blueprints/memory-layer-s3/inputs/r7/gpt6-review-20260930.md). + +**Repair round 1 (historical).** The original bounded loop authorized one repair; the table below records that round at `a266a383`. Every upstream locator it relied on was read at its pinned commit and captured (Evidence, repair round 1). The separately issued repair-round-2 brief now authorizes the additional work below; this is not a claim that a new independent review was performed in this worker turn. + +| Finding | Severity | Resolution | Where (preregistration) | +|---|---|---|---| +| F1: stale harbor-datasets task bundles could execute code | high | Resolved. "Static data" excludes executed or interpreted files, and harbor-datasets supplies nothing. A task-level provenance manifest is enforced before every trial. S8 regenerates SWE-bench Verified tasks through Harbor's maintained adapter, requires per-task field identity with the maintained dataset revision, and takes every executable part from a gate-passing source. S8 stays pending until its amendment. | 1, 8.4 | +| F2: AMB's Hindsight provider turns observations and consolidation off; "equal candidate depth" undefined | high | Resolved. A thin local Hindsight provider speaks the documented REST API and creates banks without configuration fields. The effective bank configuration is read back and matched to frozen settings before any ingest, and ingest ends only after consolidation. A common depth policy applies (k = 20, native mapping, upstream-default recall effort). The parity audit exercises every provider's actual ingest and recall paths and injected failures. The `hindsight-all` 0.4.17 statement is corrected. | 2, 3.5, 4.3, 4.5 | +| F3: tie-break membership undefined with several maximizers | high | Resolved. The maximizer set is explicit. A non-maximizer is admitted within 0.02 only if Holm finds it significantly worse than no leader. Comparisons use exact rationals. The frozen executable examples, extracted from the committed text, pass; mutating the quantifier, the cost boundary or the arithmetic makes them fail. | 7.2, 7.3 | +| F4: gross versus control-subtracted cost | medium | Resolved. C is the gross, nonnegative cost of 5.2's frozen workload, and the difference from the no-memory control is only reported. Several equal minima, zero included, go to 7.4. C_RAG is defined. | 5.2, 7.3, 8.3 | +| F5: the fallback preflight mixed concurrency K and 1 | medium | Resolved. There are two measurements. The fallback rate under the confirmatory dispatch at K, with a stated denominator, alone controls stage disabling; warm latency is measured at concurrency 1. | 7.5 | +| F6: the CI term could score unfinished runs | medium | Resolved. K = 10 needs a complete, paginated set in which every run is completed and passing; otherwise K = 0, with the reason recorded. R and K are labelled coordinator heuristics. | 7.4 | +| F7: no minimum usefulness for deployment | medium | Round 1 added S8 non-inferiority and left a usefulness residual. **Superseded by the accepted round-2 LongMemEval-V2 rule**; S8 now serves only the separate short-code do-no-harm scope. | 5.5, 7.1, 8.4 | + +**Round-1 residuals and their current status:** +- The 0/25 example and pending-S8 host-selection coupling are superseded by the user's accepted LongMemEval-V2 rule below; the former minus-one/options text is no longer operative. +- S8's design is left to its dated amendment: its arms, task count, seeds, margin and test, and how each memory system mounts inside a Harbor task. Its task and provenance manifests, and the receipt for the relayed 36-task check, are required before freeze. +- The thin Hindsight provider, the effective-bank check, the depth mapping and the failure-injection parity tests are specified, not built or run. +- The coordinator's 12/12 quote check and the 36-task check are relayed, not repeated here. +- r7's open items outside this change's paths remain; see Limitations. + +**Repair round 2 (historical).** The brief reports an independent Opus **needs_changes** review of `a266a383` (H1–H2, M1–M7, L1–L4) and GPT-6's **accept_with_minor** re-check with N1. Pinned local source reads, clone identities and labelled synthetic checks are retained in [`inputs/r7/repair2-verification-20260930.json`](../../blueprints/memory-layer-s3/inputs/r7/repair2-verification-20260930.json). No network/API request, install, deployment or provider call was made. Round 3 supersedes the dependency gate and closes the protocol gaps below. + +| Finding | Resolution in the protocol | Preregistration section | +|---|---|---| +| H1 | Final failed question-seeds score zero; reconcile native files to the frozen denominator, retain attempts/usage and mark an arm pending above 5% final failures. | 4.3, 4.5 | +| H2 / earlier F7 | Accepted main LongMemEval-V2 paired multi-seed Holm rule; no max-of-k floor or usefulness dependence on S8. | 5.5, 6, 7.1, 8.4 | +| M1 | Accepted best-primary operationally eligible fallback, with no failed hard gate and no protected incumbent. | 7.6 | +| M2 | Round 2 gated executing dependencies. **Superseded by accepted user decision 3:** gate selected repositories; require dependency SBOM/OSV acceptance with stale flags. Denied runtime egress remains unchanged. | 1, 4.3, 4.6 | +| M3 | Every thin provider defaults both recall methods to k=20 and returns raw_response=None; audit omitted-k RAG calls and prompt bytes. | 4.3, 4.5 | +| M4 | Each question×seed×arm (and host/lane/track/attempt) has a unique output_dir; manifest names the exact native split file and sidecar. | 4.3 | +| M5 | Native count/token depth escalates by one frozen multiplier/cap rule until five distinct expanded source sessions or exhaustion/cap. | 4.5 | +| M6 | Header-limited exemption requires complete source-backed native call observations or a frozen measured route. agentmemory and MemPalace are explicitly pending without these receipts. | 2.2, 4.4 | +| M7 | Joint embedder/reranker/local-generation serving peak must fit measured free VRAM minus reserve, or a pre-approved GPU window. | 8.2, 9 | +| L1 | State gross-C boundary, omitted local inference charge and total-operating-cost sensitivity at acceptance. | 5.2, 7.3 | +| L2 | cx/GPT-6 remains unpriced; excluded from priced subtotal, never zeroed or compared as a complete C. Cost-dependent ties remain unresolved. | 5.2, 7.1 | +| L3 | Empty-context native QA zeros are unjudged and flagged; all question/user/session IDs exposed to providers are opaque, with no abstention suffix. | 4.1, 5.5 | +| L4 | Prospective main/QA cost/power/subset plans and verified pre-query snapshot rules specify which ingest can be reused and counted once. | 5.2, 5.5 | +| N1 | An accepted/fallback stage outcome is mandatory even after gateway success. Pinned native invalid-score signals and validation predicate back the synthetic successful-transport invalid-score control. An unobservable outcome leaves the arm pending. | 7.5 | + +**Corrections / anti-pattern record, verified this round:** +- "Unused provider" was treated as "unexecuted dependency". AMB's package initializer imports every provider, and those modules import BM25, Mem0, Qdrant and sentence-transformers. Verified from `vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/memory/__init__.py:1`, `bm25.py:1`, `mem0.py:5` and `hybrid_search.py:5`. Round 2 applied the candidate gate to those imports; accepted user decision 3 supersedes that policy with per-arm SBOM/OSV coverage and stale flags, while preserving the import observation. +- Gateway completion was treated as stage success. ai-memory validates scores **after** the call and logs/preserves the old ranking on invalid scores; verified at `akitaonrails/ai-memory@a0ca8d1a:crates/ai-memory-mcp/src/server.rs:2133`, `:2157`, `:2176`. The new observation rule fails closed on missing outcomes and catches invalid scores despite transport success; it also accepts valid scores that preserve order. +- Equal request k was assumed for both lanes. RAG omits k and the base async method defaults to 10 (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/modes/rag.py:49`; `src/memory_bench/memory/base.py:54`). Thin providers now explicitly default both methods to 20. Uniform raw_response=None prevents the prompt's JSON/text branch from varying by arm (`src/memory_bench/dataset/longmemeval.py:131`). +- Every abstention was described as judged. The runner skips judging on empty context (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/runner.py:228`); native unjudged zeros now have their own flag/count. +- A memory-config JSON was insufficiently distinguished from backend registration, and native scripts from direct harness flags. Verified at `xiaowu0162/LongMemEval-V2@2cc8c540:memory_modules/memory.py:170`, `README.md:236`, `evaluation/run_eval.py:301-343`, `evaluation/harness.py:190,193-196`: custom backends require registration, the wrapper omits evaluator base URL, and evaluator CLI effort accepts high rather than max. Privacy is stated at `README.md:219-223`; `run_eval.py:70` is the question-ID flag (`:69` is `--limit`). All arms/control now share the direct native entry point and still require wire acceptance. +- MemoryAgentBench's README names an absent `bash_files/eniac` path; the supplied pinned files contain `bash_files/sh/run_memagent_rag_agents.sh:18`. The protocol cites the real script/native CLI and does not inherit the author's shell/GPU/model choices. +- Deployment leads needed concrete configuration corrections: cognee auth/API telemetry/version suffix; supermemory failure/404/deadline polling; MemPalace shared init/store/backend and writable files; ai-memory external env wiring and fresh port/name/volume; Graphiti in-network Neo4j URI and pulling/pinning its image before `--pull never`. Section 3.5 names the verified upstream files rather than the private deployment-spec path. +- The first publication check rejected an upstream relative documentation path whose directory is named `home`. The equivalent memsearch native CLI configuration table (`zilliztech/memsearch@2a4652fa:docs/cli.md:303`) was read and substituted; the failed output/correction is retained in the round-2 receipt. This fixes publication compatibility without introducing a private host path or changing the provider claim. + +**Repair round 3 (this revision).** The current brief reports the final pre-freeze Opus review of `0d027e06` and supplies accepted user decision 3. Findings are resolved as protocol repairs below. Local pinned-source observations and actual synthetic check outputs are in [`inputs/r7/repair3-verification-20260930.json`](../../blueprints/memory-layer-s3/inputs/r7/repair3-verification-20260930.json); no new cross-family review or native run is claimed. + +| Item | Resolution | Preregistration section | +|---|---|---| +| 1: LME2 invocation parity | Resolved: same direct harness argv per domain for every arm/control; upstream no-retrieval config; identical reader/judge settings; explicit evaluator base URL; mandatory request-body wire check. | 5.5 | +| 2: undefined resampling unit | Resolved: full-tier connected components of shared trajectories/native memory, independent of outcomes; minimum 20 verified components. Fewer leaves usefulness pending, including the likely two-domain-haystack case. | 5.5, 6 | +| 3: M5 undoes open-QA parity | Resolved: shortest final-ranking prefix covering five sessions, capped at 20 items; context tokens recorded per arm. | 4.3, 4.5 | +| 4: MemoryAgentBench applicability | Resolved: per-system native applicability table; normalized-fact agentmemory CR is conditional secondary; unsupported systems and vendored cognee never supply candidate results. | 5.5 | +| 5: executable eligible-best mismatch | Resolved: `best` is computed over the supplied eligible set only; ineligible high-scorer example/negative control added. | 7.1 | +| 6: pending higher scorer in fallback | Resolved: pending higher/equal/unknown primary scorer or qualification uncertainty leaves fallback unresolved. | 7.6 | +| 7: baseline failures | Resolved: baseline final failures above 5% make the entire usefulness family pending; boundary examples added. | 4.5, 6, 7.1 | +| 8: line references | Resolved: judge effort `harness.py:193-196`, privacy `README.md:219-223`, question IDs `run_eval.py:70`; protocol, decision log and round-2 receipt corrected. | 5.5; round-2 receipt | +| User decision 3 | **ACCEPTED:** selected repositories pass Maintained; dependencies pass OSV and appear in per-arm SBOMs; stale flags do not block the harness. | 1, 4.3 | + +**Round-3 corrections / anti-pattern log and verification paths:** +- Wrapper/direct-harness parity was inferred from model names. The wrapper forwards Qwen-oriented sampling defaults and omits reader effort/judge URL (`xiaowu0162/LongMemEval-V2@2cc8c540:evaluation/run_eval.py:75-78,301-343`); one ID list is rejected across domains (`data/public_data.py:88-93`). The common direct argv fixes those differences; real request bodies remain required. +- "History cluster" did not specify an independent unit. The same native haystack builds shared memory once (`evaluation/harness.py:1169-1170,1180-1213`), and leaderboard points keep the same domain haystacks (`leaderboard/README.md:149-150`). The downloaded-file component check and minimum 20 now prevent question/seed counts from masquerading as independent histories; under the minimum the lane stays pending. +- An initial request of 20 was confused with a 20-item answer context. AMB formats every document returned after escalation (`vectorize-io/agent-memory-benchmark@03c1d0f1:src/memory_bench/modes/rag.py:49,52-54`). The provider now applies the shortest-prefix cap after the final ranking; synthetic tests exercise repeated sessions, multi-session items, late fifth sessions and empty results. +- A generic secondary-benchmark claim exceeded native dispatch. `HUST-AI-HYZ/MemoryAgentBench@538026089:agent.py:64-89,236-255,518-531` and `methods/agentmemory.py:13-17,57-82` establish the applicable modes and normalized-fact boundary. The table excludes unsupported routes and vendored candidate copies. +- The round-2 executable best set included ineligible scorers, although its prose excluded them. The corrected 7.1 example explicitly receives eligibility and fails the negative control that removes that filter. Likewise, the 7.6 pending guard fails controls that ignore pending higher/equal/unknown scorers. +- The earlier executing-closure/no-shim maintenance rule could exclude a maintained harness for a stale imported library. The user explicitly accepted a different scope; AMB's `memory/__init__.py:2` → `bm25.py:1` and supplied `dorianbrown/rank_bm25@47aa3ddf` commit date verify the stale flag. The repository's `.github/workflows/security-scan.yml:46-86` pins OSV-Scanner v2.6.0 and scans lockfiles. Scanner discovery returned exit 127 here; no vulnerability result or installation is invented. + +**Residual execution requirements:** adapter/bootstrap/collector implementations and native parity/request-body checks are specified, not built/run here; downloaded main-usefulness data, the component-count preflight, cost/power/seed results and per-arm SBOM/OSV acceptance are absent. A native small-tier design with about two components stays pending under the frozen minimum, with no alternate question-level test. Header-limited routes, supermemory's isolated bind/auth boundary, joint serving-model residency and actual successful-gateway invalid-score control remain pending. MemoryAgentBench has not run; agentmemory's secondary mode covers normalized-fact CR only. S8's amendment/36-task receipt and out-of-scope A17 update remain required in their stated scopes. Landscape completeness is deferred. These limits cannot be converted into host acceptance by the repository validator. + +## Pre-freeze candidate additions (2026-09-30) + +**Decision.** The coordinator adds four memory primaries to the unfrozen draft under the user's instruction of 2026-09-30 that every layer's candidates be the current best maintained repositories, evaluated on merit with no incumbent default. These are pre-freeze changes, not late-candidate admissions under the rule that "never edits this text". This section supersedes round 3's landscape deferral for the named candidates; it does not claim an exhaustive landscape or freeze acceptance. + +- **Memory family:** thedotmack/claude-mem v13.28.0 (`57c38303734482a596b3e2a052b969b30584be0d`), doobidoo/mcp-memory-service v11.14.0 (`bed24ef7214192bfe59ad46f6852165678735a77`), MemMachine/MemMachine v0.3.9 (`1d3e0ce2782b83571e1974198c4d317e601cb263`) and agentscope-ai/ReMe v0.4.1.12 (`4f7c8786e3975b6ff1b20685ed56c3ccbaa4ed9f`) passed the same official local-install and both-client source admission rule as the existing eight. Section 3.1 records their release dates and screen locators; 3.5 records deployment corrections and pending acceptance. MemMachine's later-head extraction was checked against the release, with unsupported transfers left pending. +- **Fit and layer boundaries:** SimpleMem v0.3.0 is out on fit because the screen found no official Codex setup at release `c830efa3` or main `db80b6a7`; its Claude path copies a skill (`aiming-lab/SimpleMem@c830efa3:SKILL/README.md:13`). The landscape's "Mem0 OSS core" primary recommendation is not adopted: the Codex plugin calls Platform routes (`mem0ai/mem0@94c3fe9f:integrations/codex-plugin/core/memory_core.py:1996`), and the pinned tree supplies no self-hosted OpenMemory directory. Mem0 stays at most diagnostic. HippoRAG 2 belongs to document RAG, with the dated head-pin exception in 8.3: `398bfdc388692795c105c1b903db60031975b669` (2026-09-29, package 2.0.0a5), because GitHub's v1.0.0 is legacy HippoRAG 1 and PyPI's 2.0.0a4 predates maintained fixes. Its wrapper uses native indexing and retrieval-only ranked passages (`OSU-NLP-Group/HippoRAG@398bfdc3:README.md:110,121-127`; `src/hipporag/HippoRAG.py:704,773,794`). +- **Reserves and late candidates:** preserve the existing three memory reserves and append SuperLocalMemory, memU, AgentMem, EverOS, MemOS and MemoryOS in the order and at the release identities in 3.2. AgentMem needs a pinnable release or dated head-pin decision before screening. MemMachine and MCP Memory Service are now primaries; Honcho and r6's pending names retain the late-candidate rule. RAGFlow v1.0.0-rc1 is a document-RAG reserve. GitNexus v1.6.12 and Serena stay outside S3, which has no code-retrieval tool arm. Serena's measurement as a token tool in the separate Harbor E2E and GitNexus's absence from that run are relayed facts; no receipt for that run exists in this checkout, and the Harbor receipt is required before freeze. GitNexus is a candidate for its next round. + +**Evidence.** The discovery input is [`inputs/r7/landscape-memory-20260930.json`](../../blueprints/memory-layer-s3/inputs/r7/landscape-memory-20260930.json), sha256 `8f877d85ea57421f1c50704b6a2f9e88b0f8e03e8f281e9f165e03b78cfd75b8`. The maintenance capture and screens are [`inputs/r7/landscape-foldin-gate-20260930.json`](../../blueprints/memory-layer-s3/inputs/r7/landscape-foldin-gate-20260930.json), sha256 `bed16fa91e45d116d0ee968bb86b5e9c7fcd503eff820b1c0a92bc33c5a4ac65`, checked at 2026-09-30T16:30:31Z with window start 2026-07-02T16:30:31Z: all 16 repositories ACTIVE, none archived. These are captured metadata and pinned-source reviews, not executed deployment acceptance. + +**Alternatives.** Adding all seven landscape-recommended primaries to the memory family is rejected: SimpleMem and Mem0 fail the official local/both-client admission rule, and HippoRAG supplies a document-retrieval SDK without that integration. Section 6 retains a p = 1 placeholder for every pair involving an ineligible member; adding all seven would therefore take m to 105 and weaken every Holm step for the eligible comparisons. Adding none is also rejected: four maintained systems pass the same source admission rule the current eight passed, and incumbent status supplies no reason to omit them. + +**Counts and power.** The supplied count/power calculation first reproduces the current k = 8 figures exactly. Its k = 12 figures are m = 66 retrieval pairs and m_U = 78 usefulness pairs; seven document-RAG candidates give 21 pairs. The all-system ingest planning scenario becomes 3.243B tokens per host and 6.486B across both hosts, and route (a)'s full-scenario process starts become 470 × 5 × 12 = 28,200 per host (2.2). For the illustrative 470-question paired normal approximation, a true difference of 0.05 has strictest-step power 0.54, 0.18 and 0.08 at discordance 0.10, 0.20 and 0.30 (α = 0.05/66); last-step power remains 0.93, 0.68 and 0.51 (α = 0.05). For 0.02, power is at most 0.02 at the strictest step and 0.28 at the last. These figures do not replace the frozen cluster-based power report, loosen a threshold or authorize a post-result subset. + +**Corrections and verification paths.** HippoRAG's GitHub release v1.0.0 is legacy HippoRAG 1: it cites arXiv 2405.14831 (`OSU-NLP-Group/HippoRAG@b144c46d:README.md:5`), while the selected tree declares 2.0.0a5 (`OSU-NLP-Group/HippoRAG@398bfdc3:pyproject.toml:7`). Its wrapper must read result attributes directly because `QuerySolution.to_dict()` truncates to five (`src/hipporag/utils/misc_utils.py:70`). Deployment corrections and unresolved source claims are recorded with their pinned locators in 3.5; none is promoted to a passed native check. + +- **Codex route correction / anti-pattern log.** A client-list locator did not establish MCP Memory Service's Codex route. The actual recipe (`doobidoo/mcp-memory-service@bed24ef7:docs/integration/multi-client.md:450-477`) uses third-party `mcp-proxy`; its no-native-HTTP claim is contradicted by installed Codex 0.159.2 and `openai/codex@ff6aec96:codex-rs/cli/src/mcp_cmd.rs:145-157,343-359`, with bearer auth at `openai/codex@ff6aec96:codex-rs/config/src/mcp_types.rs:640-644`. The referenced example file is absent at `bed24ef7`. Section 3.5 now leaves both the sidecar route and labelled native-HTTP adaptation pending pin/gate acceptance before freeze. MemMachine's pinned skill or gated installer (`MemMachine/MemMachine@1d3e0ce2:packages/skills/README.md:13-24`) drives `mem-cli` from `memmachine-client` (`MemMachine/MemMachine@1d3e0ce2:packages/client/pyproject.toml:38`), which must be pinned in the arm's SBOM; its Codex path likewise remains pending before freeze. +- **Harbor evidence correction / anti-pattern log.** Serena's token-tool measurement and GitNexus's absence from that run were stated as observed without a receipt in this checkout. Both are now labelled relayed in the protocol and gate screens, with the Harbor receipt required before freeze. + +**Overturn conditions.** If SimpleMem or Mem0 ships an official local path in both clients at a release, re-screen it. If HippoRAG publishes a HippoRAG 2 GitHub release, pin that release by dated revision. If a screened-in system turns out to lack a working official path, it stays in the family as ineligible with p = 1 for every affected pair, per section 6; do not shrink the family. Re-check maintenance and release/head identities at freeze. Later changes still follow the dated-amendment rule. + +## Decisions for acceptance + +The repair-round-2 and round-3 briefs settle the following three decisions; they are **ACCEPTED 2026-09-30**, not new approval questions: + +- **No-winner fallback — ACCEPTED 2026-09-30.** The user's intent is "Best-scoring eligible system": choose the best observed primary score among systems that pass the selected-repository gate and dependency SBOM/OSV policy, deploy and pass required hard gates. A failed/pending hard gate cannot enter the selectable set; a pending contender that could change the result keeps fallback unresolved. Treat the incumbent symmetrically (7.6). +- **Long-workflow usefulness — ACCEPTED 2026-09-30.** The user asked "no-memory may win at short term, but for long autonomous workflow?", then requested "full sota resolution, to make sure our architecture is latest sota aligned, upstream clean e2e commands". Adopt LongMemEval-V2 as the MAIN test, with five paired replications, superiority to no retrieval and no significant loss to the eligible best, Holm-adjusted; choose small-tier coverage/subset prospectively from cost/power. Round 3 fixes invocation parity and the outcome-independent resampling unit with a minimum of 20 components; insufficient clusters or baseline failures above 5% leave the family pending. MemoryAgentBench is SECONDARY for applicable modes only, never through vendored candidate copies; short-code S8 is resolve-rate non-inferiority plus token reporting only; stale MemoryArena is citation-only (5.5, 6, 7.1, 8.4). +- **Dependency maintenance scope — ACCEPTED 2026-09-30, user decision 3.** The user said: "with best sota practice you recommend, make sure all repos candidates are latest sota". The 90-day Maintained rule excludes any stale repository we choose to run: harnesses, memory systems, RAG systems and tools. Transitive dependencies pass osv-scanner with **no known vulnerability** and appear in an **SBOM per arm**. A stale transitive dependency is flagged in the report and does not block its harness. Flagged so far: AMB's imported `rank_bm25` 0.2.2, supplied source last commit 2024-10-08. This supersedes the round-2 executing-transitive/no-shim maintenance gate; it does not exempt a selected candidate by calling it a library (1, 4.3). + +The remaining coordinator choices are listed for the complete bundle's freeze acceptance; the three decisions above are already settled: +- the optional-stage rule, which selects an optional recall stage such as ai-memory's opt-in LLM reranker per system on the development split; since repair round 1 it also covers documented non-default recall-effort levels, such as Hindsight's `budget: "high"`; +- the fallback preflight, generalized to every system with a silent-fallback LLM stage. Repair round 1 split it into a fallback measurement under the confirmatory dispatch at K, which alone controls stage disabling, and a warm-latency measurement at concurrency 1; +- agentmemory's embedder follows the same development selection as every configurable system, and r6's coordinator-directed MiniLM exception (B:C) becomes a diagnostic unless reaffirmed; +- the concrete main-usefulness subset/window, fixed haystack-component manifest/minimum, paired schedule, context policy, native registration/bootstrap and request-body wire receipt are frozen from preflight under the accepted rule, without an outcome-count floor; +- the cost scalar is the gross, nonnegative priced cost of 5.2's frozen workload at a list-price table frozen at freeze, with r6's 0.80× cost ratio. Since repair round 1, control-subtracted differences are only reported, and equal minima, zero included, go to the repo-quality tie-break; +- the repo-quality score adds a release-cadence term and a CI-on-pinned-commit term, each at Scorecard's Low weight and each a coordinator heuristic. Since repair round 1, the CI term needs a complete, finished and passing check and status set; +- the `MTEB(eng, v2)` retrieval tasks are split by corpus size: the five smallest evaluate RAG systems and the five largest select document embedders; +- the RAG feasibility window defaults to 96 h per host; +- **S8 do-no-harm margin/test:** freeze the resolve-rate margin, paired test/client-wise multiplicity and token-overhead accounting with S8's amendment. Its pending state no longer couples to usefulness or initial host selection; +- **repair round 1, the S8 route** (8.4). The brief's option (a) is taken: regenerate SWE-bench Verified tasks through Harbor v0.23.0's maintained adapter, with per-task field identity against `SWE-bench/SWE-bench_Verified@78f471bf` and a provenance manifest. The brief's alternative, option (b), keeps S8 pending with no route; +- **common recall depth and failures (4.5):** k=20 on both provider methods, escalation by the common 1/2/4/8/16 multiplier rule and supported cap, upstream-default effort; open QA uses the shortest final-ranking prefix covering five sessions capped at 20 items; zero scores for final failures and pending above 5%, including family-wide pending for baseline failures; +- **complete cost policy (5.2):** unpriced routes/unknown usage cannot become zero or a partial-total C; the current cx/GPT-6 mapping is unpriced, and local inference's omitted charge and gross-C boundary must be stated at acceptance; +- **observable fallback preflight (7.5):** native accepted/fallback outcomes with complete per-request coverage, including successful-transport invalid-output rejection; unobservable stages remain pending. + +## Alternatives considered + +- **Keep r6's reference design, with ai-memory v2.4.2 as the reference.** Rejected: it gives the incumbent protected status, contrary to the user's third requirement. +- **A leader-versus-rest family (m = k − 1).** Rejected: the leader is chosen from the same data, so the family is not fixed before results. The all-pairs family is fixed in advance and treats every system alike. +- **Hsu's multiple comparisons with the best.** Not adopted: the brief specifies Holm-adjusted paired tests through `statsmodels` `multipletests`, and a second, unpinned procedure would widen the review surface. +- **r6's +0.05 superiority margin as the winning condition.** Rejected: the brief makes the best point estimate the winner, with a 0.02 practical-equivalence band. +- **Running LongMemEval's `eval_utils.py` unchanged, as r6 planned.** Rejected: the repository is stale, and the code cannot run under NumPy 2. +- **LongMemEval's `get_anscheck_prompt` for open QA.** Not used: the brief directs AMB's per-category prompts. The abstention gap is recorded instead. +- **Stars, downloads or awesome-list presence as quality.** Rejected by the user's rule. Scorecard checks, release cadence and CI on the pinned commit are used instead, and only to break ties. +- **Selecting document embedders on the same MTEB tasks that evaluate RAG systems.** Rejected as leakage. The tasks are partitioned before any result. + +## Overturn conditions + +- The user changes an accepted decision or rejects another coordinator choice above: revise before freeze, never after a confirmatory result. +- The GPT-6 cross-family review finds that the all-pairs family or the tie-break set misstates the estimand or fails error control: revise before freeze. +- The maintenance gate finds a selected repository stale: its code stops, and a dated amendment follows. A stale transitive dependency instead receives a report flag; a known vulnerability or missing dependency SBOM/OSV coverage blocks execution under the accepted scope. +- Any pinned system publishes a newer release before freeze: re-pin by dated revision before freeze. +- Any header-limited client's wire check shows cache/dedup/replay, missing effective effort or incomplete attribution: retain pending status until a source-backed measured exemption/native route closes the gap before scoring. +- A stage has a successful gateway response but no observable validation/fallback disposition, or the invalid-score control is missed: 7.5 remains pending, regardless of the apparent gateway completion rate. +- The Harbor lane's receipt contradicts the relayed codebase-memory `/testbed` refusal: correct 8.4. +- The receipt for the relayed 36-task check contradicts field identity, or a regenerated S8 task differs from `SWE-bench/SWE-bench_Verified@78f471bf`: exclude the task, or correct 8.4 before freeze. +- Hindsight's effective bank configuration cannot be read back, or differs from the frozen settings: the Hindsight arm is pending until it matches. +- A GPU window approved by the user, or an official quantized artifact: Nemotron-3-Embed-8B moves from `pending` into the selection run. + +## Limitations + +- No deployment, provider call or scored run was made. Rounds 2–3 use supplied pinned local sources and synthetic examples; historical live metadata is not a fresh gate or host run. Round 3 makes no network request, and no downloaded-haystack count or clean vulnerability scan is claimed. +- The user's words are relayed by the brief. +- The ai-memory v2.4.2 re-pin is not yet in `manifests/stack.json`, which still lists 2.4.1. +- `blueprints/memory-stack/longmemeval/A17-SUPERSESSION.md` (lines 16 and 19–21) still describes the r4–r6 reference arm. It lies outside this change's paths and must be revised before the freeze. +- The gateway's strict `json_schema` support and the codebase-memory `/testbed` refusal are relayed facts, pending their own receipts. The same holds for the coordinator's 12/12 quote check of the GPT-6 review, the 36-task field-identity check that 8.4 cites, and the Harbor token-tool E2E's Serena measurement and GitNexus absence; the Harbor receipt is required before freeze. +- During the repair round, GitHub's REST API refused requests on the account's shared quota. The repair's reads used `gh api graphql` at pinned commits, the Hugging Face and PyPI APIs, and one raw read of GitHub's REST description at a pinned commit. +- Illustrative power to separate differences of 0.02 is at most 0.28, so the rule decides on point estimates and uses the tests to label results and to bar significantly worse systems from the tie-break. diff --git a/manifests/evidence.json b/manifests/evidence.json index 60ba0251d..d6cbcb2b6 100644 --- a/manifests/evidence.json +++ b/manifests/evidence.json @@ -7323,6 +7323,101 @@ "sha256": "7adef5c4578f4636a1596ff3e67f5dab43c051f6015c1d0e477f071579168462", "bytes": 16076 }, + { + "path": "blueprints/memory-layer-s3/PREREGISTRATION.md", + "sha256": "249e9448f3331fcca470c3af38a6a26b8a6157e018495d87cfa558f5526e6573", + "bytes": 267215 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r6/eval-frameworks-report.md", + "sha256": "c5f25772de014b945a20b02c2e4b464d34c59f90e9f09060500600ebff0c7f55", + "bytes": 17143 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r6/gateway-ab-designs.md", + "sha256": "d38e9973eda2ae1c84fb2f635959a2068a4358267bb4df51586460959a0ee73b", + "bytes": 9207 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r6/memory-merit-synth.md", + "sha256": "89da63f6197d03384e7b716270cb04130d4b3f534dc2a88a247791d05824688f", + "bytes": 21124 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r6/r6-claude-review.md", + "sha256": "aac90a9b092c7a1ea38cb42c455bd00768cabc7a8696b6035001c458924a4824", + "bytes": 16488 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r6/s3-r6-inputs.md", + "sha256": "22389274ad57b42b4ad365a9495ab630331cc2ba4e7f76cfe9a10c3d52901e31", + "bytes": 5018 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r7/gpt6-review-20260930.md", + "sha256": "ce7275817f4664f9ca1c60b48cb3cb1f600bf96834fd9c247d104cf943239d00", + "bytes": 7868 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r7/landscape-foldin-gate-20260930.json", + "sha256": "bed16fa91e45d116d0ee968bb86b5e9c7fcd503eff820b1c0a92bc33c5a4ac65", + "bytes": 13895 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r7/landscape-memory-20260930.json", + "sha256": "8f877d85ea57421f1c50704b6a2f9e88b0f8e03e8f281e9f165e03b78cfd75b8", + "bytes": 33086 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r7/live-verification-20260930.json", + "sha256": "c02d1c506d31164891d3f1782a55993d1d5ae61f9a5c4b9c7e285aa839e24f69", + "bytes": 13757 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r7/repair1-verification-20260930.json", + "sha256": "e17484588948dc7150afba464d4806924bc08ae7858a726bda5ef1b70c9a2b35", + "bytes": 15941 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r7/repair2-verification-20260930.json", + "sha256": "f8e3514c11903a4530a74f5976a4da9dc457a2fb5267843c452428e3a24e8650", + "bytes": 17294 + }, + { + "path": "blueprints/memory-layer-s3/inputs/r7/repair3-verification-20260930.json", + "sha256": "30de6af2d63e3676e669b27cec79df0ec413105484ce6977e537b76bbe878bc7", + "bytes": 21177 + }, + { + "path": "blueprints/memory-layer-s3/inputs/sweep-20260927-mac/README.md", + "sha256": "f483ab23aaf9f2d0c414364895b6e6e7dcd5a08f863ad708baf25567d4d39be6", + "bytes": 3433 + }, + { + "path": "blueprints/memory-layer-s3/inputs/sweep-20260927-mac/embed.md", + "sha256": "a50bed303d79b3f35bb92e4a18a6f8270f8728ee1b006f83c08c332768b4014d", + "bytes": 22181 + }, + { + "path": "blueprints/memory-layer-s3/inputs/sweep-20260927-mac/gen.md", + "sha256": "bbb5c6a881305868ff3e2e40dc62f87fd3083f1fb2ee12281a61f905470ba44c", + "bytes": 20470 + }, + { + "path": "blueprints/memory-layer-s3/inputs/sweep-20260927-mac/memllm.md", + "sha256": "dfcee43461c567b3eaf467720bd1af2344aaa027bbdeecbfc2d81aaafdc9bfeb", + "bytes": 28704 + }, + { + "path": "blueprints/memory-layer-s3/inputs/sweep-20260927-mac/memsys.md", + "sha256": "556eb2c0e26cc35835ea615a0c251edc11eb80c7f12d7f6b5ce81f5e239157de", + "bytes": 19813 + }, + { + "path": "blueprints/memory-layer-s3/inputs/sweep-20260927-mac/rerank.md", + "sha256": "943368b875af6a09ddd4712da61c6cd6133835c0aa6c65522d23139d0970efd1", + "bytes": 23880 + }, { "path": "blueprints/memory-lifecycle-probe/README.md", "sha256": "c50ba6b05cba66095a357f1185429761b251462b9b91d1d23a51532b317bc042", @@ -8398,6 +8493,11 @@ "sha256": "cf45ee658753072c0e197c98fc851f610e278ab01ed5dfb61b73e2e5854eb6ae", "bytes": 85 }, + { + "path": "blueprints/memory-stack/longmemeval/A17-SUPERSESSION.md", + "sha256": "377ab0d329aa8629824cde33ec9ab7b8ffed4cbb98f588e20c5103229ce14518", + "bytes": 3185 + }, { "path": "blueprints/memory-stack/longmemeval/PREREGISTRATION.md", "sha256": "1f3ca9d554cff24dc785c35f275335027a831dd386f460890f8a3082a07c11c2", @@ -14938,6 +15038,11 @@ "sha256": "93a7459d627c02946f41b4284467801187435114082d928ee014fc7bccc3a747", "bytes": 28168 }, + { + "path": "docs/decisions/2026-09-30-memory-s3-r7.md", + "sha256": "1fe95662d20072f701466b897c67cd1e6476988a945a785a849acba2e14cf4cb", + "bytes": 45756 + }, { "path": "docs/decisions/2026-09-30-omniroute-rebuild.md", "sha256": "b2d99153c8dffb6169923e9129cfb6e4343cd9f5fc443ad8839fa5f162e20c1e",