opsx: memory-core-redesign — proposal, design, spec deltas, tasks

openspec/changes/memory-core-redesign/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-03

openspec/changes/memory-core-redesign/proposal.md

@@ -0,0 +1,149 @@
+# Proposal: memory-core-redesign
+
+Source PRD: PRD-007 (agent personality and local memory). Evidence base:
+`docs/research/memory-audit-2026-07.md` (July 2026 measured audit) and
+`docs/research/memory-recall-findings-2026-05.md` (May 2026 autoresearch).
+
+## Why
+
+The memory system's two intelligence layers have never functioned — the LLM
+curation tier has zero successful decisions in its production lifetime and
+recall ranks by a lexical score measured to carry no relevance signal — so the
+corpus accretes near-duplicates (14% redundant, doubling in five weeks) while
+automatic recall injects 46% pollution into live turns (19% of recall events
+actively misleading). The quick-win slice (July 2026) stopped the worst
+bleeding; this change adds the missing semantic judgment and lifecycle, and
+removes the dead structure that three redesign cycles left behind.
+
+## What Changes
+
+- **Semantic judgment (embeddings).** In-process ONNX embedding infrastructure
+  (snowflake-arctic-embed 137M int8, CPU, zero sidecars): embed-on-write, a
+  brute-force in-memory vector index, and model provisioning with a pinned
+  hash-verified allowlist, downloaded at daemon initialization.
+- **Write-side dedup becomes nominate→decide.** Embedding kNN nominates
+  near-duplicates (τ≈0.86, k=5, config); any nominee forces the LLM curation
+  tier to decide merge/enrich/keep. No cosine auto-merge tier (measured: no
+  threshold separates duplicates from siblings). One shared curation evaluator
+  replaces the two divergent pipelines.
+- **Lossless merges.** CONSOLIDATE/UPDATE produce an LLM-synthesized merged
+  body validated by a deterministic MergeGuard (load-bearing-token retention)
+  with a structural-append fallback — the raw `markdown_body` overwrite path
+  becomes unreachable. **BREAKING** for any consumer that assumed merge ==
+  replace.
+- **Read-side hybrid recall with an absolute relevance floor.** Query
+  embedding + FTS5 union, weighted fusion, and a cosine floor calibrated
+  against the real-traffic gold set — turns where nothing is relevant inject
+  nothing (measured: 65% of real queries).
+- **Taxonomy rebalance around real behaviors.** `Searchable` recall mode is
+  removed from the automatic pool (**BREAKING** semantic fix: today
+  searchable ⊂ auto); durable-fact formation defaults to searchable with auto
+  reserved for identity/preferences/environment; `Trace` (72 h short-lived
+  memory) gets a reachable producer and a recallable-while-fresh mode; an
+  expiry sweep actually deletes expired rows.
+- **Tool-use lessons.** New `tool_lesson` memory class (Document/merge/
+  searchable) anchored per tool, captured explicitly (`store_memory`) and by
+  the sidecar distillation prompt; recalled via per-tool context injection on
+  first tool use per session — outside the pre-turn recall budget.
+- **Maintenance tooling.** `netclaw memory` CLI group: `backfill-embeddings`,
+  `consolidate --dry-run` (ratification plan file) / `--apply --plan` (gated,
+  backup-first), `status`; maintenance-run ledger table.
+- **Subtraction.** Remove: the unused `memory_edges` graph, the inert
+  4-demo-facet planner inference, the dead `verified-tool-finding` +25 recall
+  bonus, and gate the checkpoint worker's turn-complete lane at enqueue time
+  (95% of its intake is dropped by design today). **BREAKING** only at the
+  schema-surface level; no functional behavior depends on any of these
+  (verified by audit).
+
+## Capabilities
+
+### New Capabilities
+
+- `memory-embeddings` — ONNX embedding runtime: model provisioning
+  (allowlist, SHA-256, atomic download), embed-on-write, vector index, and
+  loud-degradation semantics (doctor check, daemon status, structured logs;
+  lexical recall keeps serving but never silently).
+- `memory-maintenance` — operator-driven corpus lifecycle: embedding
+  backfill, consolidation dry-run/apply with human ratification and
+  backup-first apply, expiry sweep, `netclaw memory status`, maintenance
+  ledger.
+
+### Modified Capabilities
+
+- `netclaw-agent-memory` — hybrid recall + absolute cosine floor and
+  injection semantics; nominate→decide curation with lossless merge; recall-
+  mode semantics fix (searchable leaves the automatic pool); trace revival
+  (producer, fresh-recall, deletion); tool-lesson class + per-tool context
+  injection; formation-side recall-mode assignment; removal of graph-edge and
+  facet-inference requirements (the spec's flagged open decisions — "keyword
+  vs vector search, embedding strategy, injection budgets" — are resolved by
+  this change).
+
+## Impact
+
+- **Code**: `src/Netclaw.Embeddings` (new project), `Netclaw.Actors/Memory`
+  (curation evaluator, store schema + vector queries, policy gates, enums),
+  `Netclaw.Actors/Sessions` (recall coordinator, tool-execution pipeline for
+  lessons), `Netclaw.Daemon` (DI, warmup hosted service, checkpoint gating),
+  `Netclaw.Cli` (memory command group, doctor checks),
+  `Netclaw.Configuration` (Memory.Embeddings/Recall/Curation config objects +
+  schema sync), observer sidecar distillation prompt, `netclaw-memory` system
+  skill.
+- **Dependencies**: `Microsoft.ML.OnnxRuntime` (CPU; linux-x64 + linux-arm64),
+  `FastBertTokenizer`, `System.Numerics.Tensors`. Model artifact (~90–140 MB)
+  distributed at runtime, never embedded in the binary.
+- **Data**: new `memory_embeddings` and `memory_maintenance_runs` tables
+  (owned by `SQLiteMemoryStore.InitializeAsync`, not daemon migrations);
+  one-time ratified consolidation pass over the existing corpus (operator-
+  gated; out of automatic paths); expiry sweep begins deleting expired
+  records.
+- **Evals**: recall-quality gold-set regression suite (real-traffic gold from
+  the audit); eval cases for tool lessons and zero-injection behavior; the
+  scenario suite's paraphrase-gap case (P09) flips back to expected-recall.
+
+### In scope (MVP)
+
+Slices 2–6 as designed: embedding foundation; write-side nominate→decide +
+lossless merge; read-side hybrid + cosine floor; taxonomy rebalance + tool
+lessons + trace revival + expiry sweep; maintenance CLI + subtraction items.
+
+### Out of scope
+
+- Multilingual embedding models (future pass; vectors are keyed by
+  `model_id`, thresholds live in config next to `ModelId`, so a model swap is
+  `config change + backfill --force`; re-evaluate .NET SentencePiece
+  tokenizer support then).
+- ANN indexes (brute-force cosine is sub-ms at ≤50k vectors).
+- Mirroring the model artifact into the R2 feeds infra (post-PoC decision;
+  pinned HF URLs first).
+- Structural (code-level) detection of tool-use corrections (explicit +
+  sidecar capture only).
+- Applying consolidation to any live corpus as part of this change's
+  implementation (tooling ships; each apply run remains an operator decision).
+
+## Security and Operational Impact
+
+- **Model supply chain**: `ModelId` selects from a pinned in-code allowlist
+  (id → URL + size + SHA-256); arbitrary URLs are rejected; downloads are
+  atomic (temp + rename) and hash-verified before load. A failed or missing
+  model is a **loud** degraded state (doctor Error, daemon status
+  `embeddings: degraded`, rate-limited structured logs) — lexical recall
+  keeps serving; no silent fallback.
+- **Policy parity**: vector-sourced recall candidates pass the identical
+  audience/boundary/sensitivity/recall-mode gates as lexical candidates
+  (scenario-tested requirement, not an implementation detail).
+- **Destructive-operation gating**: consolidation `--apply` executes only a
+  previously written, human-editable plan file, refuses a live daemon by
+  default, and takes a `VACUUM INTO` backup first. The expiry sweep deletes
+  only rows already invisible to every recall/search path.
+- **Resource envelope**: measured on the reference box (i9-9900K) — full
+  1,216-doc backfill 4.5–8.3 min, <0.5 GB RSS; steady-state embed-on-write
+  ~13 docs/day. The recall-time query-embedding sub-budget must be
+  re-measured on the ONNX int8 path before the hybrid slice ships (Ollama
+  measurements ran 4–30× above the design assumption for full documents;
+  queries are far shorter).
+- **Operations**: new doctor checks (embedding provisioning/coverage,
+  curation-LLM health — the latter shipped with the quick-win slice);
+  `netclaw memory status` becomes the corpus-health surface; runbook
+  `docs/runbooks/memory-health-and-evals.md` gains embedding/consolidation
+  sections.

openspec/changes/memory-core-redesign/specs/memory-embeddings/spec.md

@@ -0,0 +1,100 @@
+# Spec: memory-embeddings (new capability)
+
+## ADDED Requirements
+
+### Requirement: In-process embedding runtime
+
+The system SHALL compute memory embeddings in-process with a CPU ONNX runtime
+and a managed tokenizer — no sidecar processes, no network inference hop.
+Embedding components SHALL sit behind a narrow interface owned by the memory
+subsystem so actor code carries no ONNX dependency, and the runtime SHALL
+support both linux-x64 and linux-arm64.
+
+#### Scenario: Embeddings compute without external services
+
+- **GIVEN** a healthy daemon with the embedding model provisioned
+- **WHEN** a memory document is written
+- **THEN** its embedding is computed in-process
+- **AND** no network call or child process is involved in inference
+
+### Requirement: Pinned model provisioning
+
+The embedding model SHALL be selected by id from a pinned in-code allowlist
+mapping model id to download URL, byte size, and SHA-256. Arbitrary model URLs
+SHALL be rejected. Provisioning SHALL download atomically (temporary file then
+rename), verify the hash before load, and run at daemon initialization when
+auto-download is enabled or on explicit operator command. The model artifact
+SHALL NOT be embedded in the application binary.
+
+#### Scenario: Hash mismatch refuses the model
+
+- **GIVEN** a downloaded model artifact whose SHA-256 does not match the
+  allowlist entry
+- **WHEN** provisioning verifies the artifact
+- **THEN** the artifact is discarded and not loaded
+- **AND** the failure is surfaced as a doctor-visible error
+
+#### Scenario: Unknown model id is rejected
+
+- **GIVEN** configuration naming a model id absent from the allowlist
+- **WHEN** the daemon initializes embeddings
+- **THEN** provisioning refuses with a configuration error identifying the
+  allowlisted ids
+
+### Requirement: Embed-on-write with derived backfill state
+
+Every recallable memory document SHALL receive an embedding keyed by
+`(item id, model id)` with a content hash of its normalized text. Writes SHALL
+embed after commit; a startup gap-repair sweep SHALL embed any item missing a
+current-model embedding. Re-embedding SHALL be skipped when the content hash
+is unchanged. Backfill progress SHALL be derived from the store (items lacking
+a current-model embedding), never tracked in separate mutable state. Vectors
+are derived data: loss or deletion of embeddings SHALL be recoverable by
+re-embedding without any loss of memory content.
+
+#### Scenario: Crash between write and embed self-heals
+
+- **GIVEN** a document committed whose embedding upsert was interrupted
+- **WHEN** the daemon next starts and the gap-repair sweep runs
+- **THEN** the missing embedding is computed and stored
+- **AND** the embedding doctor check reports full coverage afterward
+
+#### Scenario: Model change re-embeds without data loss
+
+- **GIVEN** a corpus embedded under model A
+- **WHEN** the operator switches configuration to allowlisted model B and runs
+  a forced backfill
+- **THEN** embeddings for model B are created alongside or replacing model A's
+- **AND** memory content is unmodified
+
+### Requirement: Loud degradation without silent fallback
+
+Memory recall and curation SHALL continue on their lexical paths when the
+embedding model is missing, corrupt, or the runtime fails, and the degraded
+state SHALL be loud: a doctor check reports the cause, the daemon runtime
+status reports embeddings as degraded, and recall/curation log structured
+degradation events. The system SHALL NOT silently revert to lexical behavior
+without these signals.
+
+#### Scenario: Missing model degrades loudly
+
+- **GIVEN** auto-download is disabled and no model artifact is present
+- **WHEN** the daemon starts and a turn triggers recall
+- **THEN** recall serves lexical-only results
+- **AND** daemon status reports embeddings degraded
+- **AND** `netclaw doctor` reports the missing model as an error with
+  remediation
+
+### Requirement: Embedding coverage diagnostics
+
+A doctor check SHALL report embedding provisioning state and corpus coverage:
+model present and hash-valid, count of items lacking current-model embeddings,
+and a warning when embeddings exist under multiple model ids (mixed-model
+corpus invalidates similarity thresholds).
+
+#### Scenario: Mixed-model corpus warns
+
+- **GIVEN** embeddings stored under two different model ids
+- **WHEN** the embedding doctor check runs
+- **THEN** it warns that similarity thresholds are calibrated per model
+- **AND** recommends a forced backfill under the active model

openspec/changes/memory-core-redesign/specs/memory-maintenance/spec.md

@@ -0,0 +1,83 @@
+# Spec: memory-maintenance (new capability)
+
+## ADDED Requirements
+
+### Requirement: Embedding backfill command
+
+The CLI SHALL provide `netclaw memory backfill-embeddings` to provision the
+model if needed and embed every item lacking a current-model embedding, with a
+`--force` mode that re-embeds everything under the active model. Backfill
+SHALL be safe against a live daemon (small batched writes under WAL) and SHALL
+report progress and a final coverage summary.
+
+#### Scenario: Backfill completes coverage
+
+- **GIVEN** a corpus with items lacking current-model embeddings
+- **WHEN** the operator runs the backfill command
+- **THEN** all recallable items receive embeddings under the active model
+- **AND** the command reports counts embedded, skipped (hash-unchanged), and
+  failed
+
+### Requirement: Ratified consolidation with dry-run plan files
+
+Corpus consolidation SHALL be two-phase and operator-gated. A dry-run SHALL
+build near-duplicate clusters by embedding similarity, synthesize a proposed
+lossless merge per cluster, and write a human-editable plan file plus a
+readable report — with no database mutation. An apply run SHALL execute a
+previously written plan file verbatim (operators veto by editing or deleting
+plan entries), SHALL refuse to run against a live daemon by default, SHALL
+take a database backup before mutating, and SHALL re-embed merged results and
+rebuild affected search rows. Every apply SHALL be recorded in a maintenance
+ledger.
+
+#### Scenario: Dry-run mutates nothing
+
+- **GIVEN** a corpus containing near-duplicate clusters
+- **WHEN** the operator runs consolidation in dry-run mode
+- **THEN** a plan file and report are produced
+- **AND** the database bytes are unchanged
+
+#### Scenario: Apply executes only the ratified plan
+
+- **GIVEN** a reviewed plan file with one cluster entry deleted by the
+  operator
+- **WHEN** the operator runs apply with that plan
+- **THEN** a backup of the database is created first
+- **AND** the deleted entry's cluster is left untouched
+- **AND** the remaining entries are applied and recorded in the maintenance
+  ledger
+
+#### Scenario: Apply refuses a live daemon
+
+- **GIVEN** the daemon is running
+- **WHEN** the operator runs consolidation apply without the explicit
+  live-override flag
+- **THEN** the command refuses and names the running daemon
+
+### Requirement: Expiry sweep deletes expired rows
+
+The system SHALL periodically delete memory rows whose expiry has passed
+beyond a grace window. Expired rows are already excluded from every recall and
+search surface, so deletion SHALL be behavior-neutral for reads; each sweep
+SHALL log the number of rows removed per class.
+
+#### Scenario: Expired evidence is physically removed
+
+- **GIVEN** evidence records whose expiry passed beyond the grace window
+- **WHEN** the maintenance sweep runs
+- **THEN** those rows are deleted from the store
+- **AND** the sweep logs the per-class deletion counts
+
+### Requirement: Memory status surface
+
+The CLI SHALL provide `netclaw memory status` reporting corpus composition
+(counts by class and recall mode), embedding coverage for the active model,
+pending checkpoints, expired-row counts awaiting sweep, and the most recent
+maintenance-ledger entries.
+
+#### Scenario: Operator inspects corpus health
+
+- **GIVEN** a daemon with a populated memory store
+- **WHEN** the operator runs the status command
+- **THEN** it reports class/recall-mode counts, embedding coverage, pending
+  checkpoints, and recent maintenance runs