feat(memory): kNN-nominate/LLM-decide dedup + lossless merges (memory-core-redesign slice 3)

[Aaronontheweb]

Implements Slice 3 of the memory-core-redesign OpenSpec change (#1570): write-side dedup becomes kNN-nominate → LLM-decide, and curation merges become lossless-or-append. Closes tasks 3.1–3.7.

Stage A — lossless merge machinery (landed on this branch previously)

• Merged-body response protocol (task 3.2): CurationPromptBuilder UPDATE/CONSOLIDATE responses now emit a complete merged document body after a --- separator (CurationDecision.MergedBody); candidates can be rendered full-content for the decider.
• MergeGuard (task 3.3): deterministic lossless-merge validator — load-bearing tokens (URLs, numbers, versions, dates, code identifiers) from every source body must survive at ≥95% retention, and the merged body must not length-collapse.
• Guard-validated write routing (task 3.4): every LLM-tier UPDATE/CONSOLIDATE and every deterministic CONSOLIDATE routes through ApplyGuardedMergeOrAppend — guard-pass writes the merged body (MergeDocument), guard-fail or body-absent degrades to a structural append (existing body + dated separator + proposal, AppendDocument — reachable for the first time since the enum was written). The raw markdown_body = excluded.markdown_body overwrite is unreachable from curation decisions. The overwrite-unreachable trace closed three paths: LLM UPDATE (guard or append), LLM CONSOLIDATE (guard or append), and the previously unguarded deterministic Consolidate — pre-Slice-3, a fuzzy-anchor ≥80%-overlap Consolidate reached the store with no guard at all (GuardDestructiveUpdate is a no-op for Consolidate); it now always appends, since the rules tier never synthesizes a merged body. The one deliberate exception: the deterministic exact-anchor UPDATE keeps its raw overwrite because GuardDestructiveUpdate already proves the proposal is a content superset of the target before that decision can exist.
• Config surface (task 3.5): Memory.Curation { NominatorSimilarityThreshold=0.86, NominatorK=5, LlmMaxOutputTokens=4096, LlmTimeoutSeconds=10 } + schema sync, replacing Slice-1 hardcoded constants.

Stage B — embedding kNN nominator (this PR's new commits)

Evaluation order in the shared MemoryCurationEvaluator (design D4), identical on both write pipelines:

1. Record bypass and exact-anchor deterministic fast paths — unchanged.
2. Embedder available + no exact anchor match → embed the proposal ({title}\n{content}, the exact concatenation embed-on-write uses, so proposals and stored documents share embedding space), query MemoryVectorIndex.TopK(NominatorK, NominatorSimilarityThreshold), hydrate nominees into full-content candidates tagged with their cosine (ExistingMemoryCandidate.CosineSimilarity; anchor/lexical candidates carry null).
3. Any nominee forces the LLM tier with full-content candidate previews — regardless of Jaccard bands.
4. No nominee + no anchor match → Create with zero LLM calls (median nominee count on a random write is 0 — the common case stays cheap).
5. Embedder unavailable → the pre-existing lexical content-term search runs unchanged as the degraded path (curation_nominator_degraded marker).

Invariants — cosine nominates ONLY:

• The May 2026 measurement (ratified in the OpenSpec design; corroborated at corpus scale in the July audit §5) found no cosine threshold separates true duplicates from merely-related siblings — siblings live at 0.905–0.941, inside the duplicate band. Cosine similarity therefore never auto-merges and never auto-skips, anywhere.
• Nominee present but no LLM available (the daemon checkpoint worker today) or LLM call fails → conservative Create, deliberately bypassing TryAutoResolveAmbiguous: semantic-near content is exactly the ambiguity the May data says deterministic Jaccard logic cannot adjudicate, and a duplicate document is recoverable where a wrong auto-merge is not.
• Known limitation (documented on NominateAsync): proposals evaluated within one batch cannot nominate each other — neither is committed/indexed while the other is evaluated. Cross-batch and steady-state dedup are unaffected.

Wiring: new MemoryVectorIndexHolder (defers index construction until the warmed-up embedder's model id/dimensions exist — same rationale as MemoryEmbedderHolder) registered in daemon DI and threaded into both pipelines: inline actor via MemoryCurationActor.CreateProps/SessionMemoryServices, daemon worker via MemoryCurationEngine's constructor. New store hydration query SQLiteMemoryStore.GetCandidatesByIdsAsync. New markers: curation_nominated count/topCosine, curation_nominator_degraded, curation_nominee_no_llm_decision; all existing markers intact.

Test matrix (task 3.6 — all synthetic fixtures, hand-crafted cosine geometry)

• Nomination forcing: paraphrase pair at cosine 0.93 with word-Jaccard <0.4 → LLM tier invoked (recording IChatClient) even though the lexical tier finds zero candidates and would have silently said Create.
• Sibling never-auto-merge: scripted LLM answers CREATE → two separate documents persist; no-LLM variant → conservative Create — never a merge without the LLM.
• Novel-skips-curator: no nominee + no anchor → Create with zero IChatClient invocations.
• Degraded path: unavailable embedder (and null-holder variant) → lexical candidates still produced, curation_nominator_degraded fires.
• Both-pipelines parity: MemoryCurationEvaluatorParityTests extended — actor-style vs engine-style construction with a shared embedder/index reach identical forced-LLM decisions.
• Actor e2e (Akka.TestKit): proposal through the real MemoryCurationActor with fake embedder + scripted LLM to a committed store write; AwaitAssertAsync, no sleeps.
• Stage A matrix (MergeGuard property tests, append fallback, merge routing) landed with Stage A.

Gates

• Netclaw.Actors.Tests 2614/2614, Netclaw.Daemon.Tests 832/832, Netclaw.Cli.Tests 1231/1231, Netclaw.Configuration.Tests 461/461, Netclaw.Embeddings.Tests 18/18 — all green (Release).
• Full solution Release build: 0 warnings / 0 errors.
• dotnet slopwatch analyze: 0 issues. Add-FileHeaders.ps1 -Verify: clean.
• Memory eval gate (NETCLAW_EVAL_CATEGORY=Memory, Qwen3.6-27B @ spark-acad): scoreboard below. Note: embeddings default OFF, so the nominator idles in the eval daemon — this gate protects the shared curation paths against regressions.

Category: Memory Pipeline
  [PASS] memory_recall_active           5/5 (1.00)  — recall active, not degraded
  [PASS] memory_identity_preference_routing 5/5 (1.00)  — durable user preference routed to memory, not SOUL.md
  [PASS] memory_explicit_store          4/5 (0.80)  — explicit remember request uses store_memory
  [PASS] memory_checkpoint_enqueue      5/5 (1.00)  — checkpoint enqueued for non-identity fact
  [PASS] memory_recall_filters          5/5 (1.00)  — candidate selection with score filtering
  Category: 5/5 passed (GREEN)

─────────────────────────────────────────────────
Overall: 5/5 cases passed (100.0%)

(An earlier run of the same gate scored 4/5: memory_checkpoint_enqueue 3/5 — investigated per the eval-debugging protocol, the two failing runs' stdout contained only [error] LLM provider transport error: Connection refused (spark-acad…:8000) — the remote endpoint dropped mid-suite, so those turns never executed. Endpoint recovered; the clean rerun above is the gate result.)

Skill sync

netclaw-memory SKILL.md bumped 1.8.0 → 1.9.0: documents that with embeddings enabled, near-duplicate proposals are adjudicated by the curation LLM (skip/update/consolidate/create) — similarity alone never merges — and merges are lossless-or-append.

OpenSpec change: memory-core-redesign (#1570) — tasks 3.1–3.7 checked.