opsx: memory-relevance-gate — cross-encoder relevance gate for memory recall (design)

openspec/changes/memory-relevance-gate/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-06

openspec/changes/memory-relevance-gate/proposal.md

@@ -0,0 +1,167 @@
+# Proposal: memory-relevance-gate
+
+Source PRD: PRD-007 (agent personality and local memory), continuing
+`memory-core-redesign` (`openspec/changes/memory-core-redesign/`, PR #1570;
+Slice 4 shipped the hybrid recall + calibrated cosine floor this change builds
+on). Evidence base: `~/recall-research-local/2026-07/gate-shootout/` (4-design
+gate shoot-out, 2026-07-06) and `~/recall-research-local/2026-07/gold-expansion/`
+(450-query out-of-sample gold expansion + gate re-validation, 2026-07-06) —
+operator-local research stores holding real (PII) traffic data, never
+committed, per the same convention documented in
+`docs/research/memory-audit-2026-07.md`.
+
+## Why
+
+Even with hybrid recall and the calibrated per-model cosine floor
+(memory-core-redesign Slice 4), most nothing-relevant queries still cause an
+injection: floor-only zero-injection accuracy measured **16.7%** on the July
+gold set (`gold-prod-2026-07`, 93 queries) and **7.3%** on the 450-query
+out-of-sample expanded gold set. Cosine similarity measures topical
+"aboutness," not usefulness-for-answering — a candidate can clear the floor
+and still be the wrong thing to inject. This is the dominant remaining
+recall-quality defect because **60–65% of real queries have nothing relevant**
+to recall at all (replicated across 543 labeled real-traffic queries: 93 July
++ 450 expansion), so the floor's residual miss rate lands on the majority
+case, not the tail.
+
+## What Changes
+
+- **New relevance-gate stage after the cosine floor.** A tiny cross-encoder
+  scores `(query, candidate)` jointly for each of the (≤`AutoRecallMaxItems`
+  = 3) floor-surviving candidates; anything below a calibrated threshold S* is
+  dropped. Zero survivors after the gate ⇒ inject nothing, same as zero
+  survivors at the floor today.
+- **Winner of a 4-design measured shoot-out, out-of-sample validated**:
+  `Xenova/ms-marco-MiniLM-L-6-v2`, `model_quantized.onnx` (int8, 22.07 MB,
+  SHA-256 `e9d8ebf845c413e981c175bfe49a3bfa9b3dcce2a3ba54875ee5df5a58639fbe`).
+  Out-of-sample (450-query expanded gold set, disjoint from the calibration
+  set) at S*=0.02: zero-injection accuracy **86.8%** (95% CI 82.3–90.3) vs
+  7.3% floor-only, recall retention **98.3%**, F0.5 **0.130** vs 0.100
+  floor-only, mean injected **0.251** vs 2.538 floor-only.
+- **Reuses memory-core-redesign's infrastructure wholesale** — this is the
+  change's selling point, not an afterthought: the same consumer-defined-seam
+  pattern (`IMemoryEmbedder` → `IRelevanceScorer`), the same
+  allowlist-manifest provisioning pattern (`EmbeddingModelProvisioner` gains a
+  relevance-model manifest entry kind carrying pinned URL/SHA-256/size *and*
+  the calibrated operating threshold), the same warmup hosted service, and the
+  same loud-degradation contract (rate-limited log marker + doctor
+  visibility) — no new machinery class, only a new manifest entry and a new
+  scoring step in an existing pipeline.
+- **One mental switch.** Gate activation is tied to
+  `Memory.Embeddings.Enabled` — there is no separate "turn semantic recall
+  quality on" knob. `Memory.Recall.RelevanceGate { Enabled (nullable, follows
+  Embeddings), Threshold (nullable, follows the manifest's calibrated S*) }`
+  exists only for an explicit operator override.
+- **Logging.** `memory_retrieval_final` gains `gateScores` and `droppedByGate`
+  fields. A new eval case asserts the zero-injection behavior end-to-end:
+  seeded corpus, off-topic question, assert no `[memory-recall]` block and a
+  gate marker in the logs.
+- **Rejected alternatives** (recorded for provenance; not shipped):
+  - *Distribution-shape statistical gate* (`z_top50 ≥ 2.80`): looked viable
+    in-sample (70% zero-injection) but failed out-of-sample — 65.2%
+    zero-injection accuracy, **86.5% recall retention (below the ≥90%
+    constraint)**, F0.5 0.089, *worse* than the 0.100 floor-only baseline.
+  - *Learned feature gate* (candidate-/query-level logistic regression and
+    GBM over cosine/margin/z-score/length/age features): query-level variant
+    measured out-of-fold AUC 0.545 (chance = 0.500, i.e. no signal);
+    candidate-level variant's positive-class support grew only 8→39 across
+    the gold expansion — still not enough to certify signal over
+    small-sample luck, and it showed an 80%-relative recall collapse on a
+    differently-composed transfer set.
+  - *Per-memory offender priors* (`pollution_count`/`injection_count` per
+    `docId`): structurally cold-start-bound — only 1.1% of top-3 recall
+    candidates carry 3+ injection observations to build a prior from, 5.3%
+    even at a relaxed 2+ threshold; 80.6% of top-3 candidates are cold-start
+    with no addressable history at all.
+
+## Capabilities
+
+### New Capabilities
+
+- `memory-relevance-gate`: the `IRelevanceScorer` seam and
+  `OnnxCrossEncoderScorer` implementation, the relevance-model provisioning
+  manifest kind (pinned URL/SHA-256/size + calibrated threshold), and the
+  post-floor gate stage wired into automatic recall.
+
+### Modified Capabilities
+
+- `netclaw-agent-memory`: the automatic pre-turn recall requirement gains a
+  post-floor relevance-gate stage — floor-surviving candidates are scored and
+  filtered before injection; zero survivors after the gate is a "nothing
+  injected" outcome exactly like zero survivors at the floor; gate
+  unavailability or sub-budget timeout degrades to floor-only behavior with a
+  loud marker.
+- `memory-embeddings`: the pinned-allowlist provisioning requirement is
+  generalized to a manifest entry *kind* so it can provision relevance
+  (cross-encoder) models alongside embedding models, and the warmup hosted
+  service provisions/warms both.
+
+## Impact
+
+- **Code**: new `IRelevanceScorer` seam (`Netclaw.Actors/Memory`), new
+  `OnnxCrossEncoderScorer` (`Netclaw.Embeddings`, pair encoding `[CLS] q [SEP]
+  d [SEP]` with `token_type_ids`, sigmoid over the single-logit head, dynamic
+  sequence length bucket-of-8 matching the embedder's convention);
+  `EmbeddingModelProvisioner`'s allowlist gains a relevance-model manifest
+  kind; `SQLiteMemoryRecallCoordinator` gains the post-floor gate stage under
+  a CE sub-budget; `Netclaw.Configuration` gains
+  `Memory.Recall.RelevanceGate`; doctor/status surfaces extend to cover the
+  relevance model; `netclaw-memory` skill update.
+- **Dependencies**: none new — reuses the `Microsoft.ML.OnnxRuntime` +
+  managed-tokenizer stack memory-core-redesign Slice 2 already adopted. One
+  new pinned model artifact (~22 MB int8), never embedded in the binary,
+  downloaded and hash-verified at provisioning time exactly like the
+  embedding model is today.
+- **Data/config**: `netclaw-config.v1.schema.json` gains the new nodes, all
+  nullable with manifest-derived defaults — additive, non-breaking.
+- **Evals**: new zero-injection gate eval case; `memory_retrieval_final`'s
+  log schema gains two additive fields (`gateScores`, `droppedByGate`).
+- **Target branch**: implementation lands on `feature/memory-embeddings` (the
+  in-flight branch carrying memory-core-redesign's embedding and recall
+  slices), not directly on `dev` — this change's tasks assume that branch's
+  `IMemoryEmbedder`/`MemoryEmbedderHolder`/`SQLiteMemoryRecallCoordinator`
+  hybrid-recall code as their starting point.
+
+### In scope (MVP)
+
+- The cross-encoder scorer, its provisioning manifest entry, the coordinator
+  wiring (score → threshold → drop), the config surface, degradation
+  semantics, logging fields, and the zero-injection eval case.
+- Recording the shoot-out's rejected alternatives and residual failure modes
+  in `design.md` for provenance.
+
+### Out of scope
+
+- Domain-calibrated or class-conditional thresholds for the measured MS
+  MARCO under-scoring of procedural/command-style memories (residual, ~1.7%
+  of retained recall at S*=0.02) — future work, not this change.
+- Consolidating `MemoryEmbedderHolder` and a prospective relevance-scorer
+  holder into one combined embedding-runtime holder — noted as an optional
+  simplification in `design.md`, not required for this change to ship.
+- Any change to the cosine floor itself, the embedding model, or the fusion
+  weights (memory-core-redesign Slice 4 territory; this change only adds a
+  stage after that pipeline's existing output).
+- Re-running or expanding the judged gold sets further; this change consumes
+  the existing gate-shootout and gold-expansion results as already-ratified
+  inputs.
+
+## Security and Operational Impact
+
+- **Model supply chain**: the relevance model is provisioned through the
+  same pinned-allowlist mechanism as the embedding model — id → URL + byte
+  size + SHA-256, arbitrary URLs rejected, atomic download (temp + rename),
+  hash-verified before load. No new supply-chain surface, only a new
+  manifest entry kind on the existing one.
+- **Resource envelope**: measured on the reference CPU — ~11 ms p50 / ~35 ms
+  p95 to score 3 pairs (quantized int8), ~103 MB incremental RSS. Combined
+  with int8 embeddings (263 MB) and daemon peak (397 MB), the operator's
+  measured total is ≈763 MB — inside the 1 GB K8s pod limit, with headroom
+  noted rather than assumed.
+- **Degradation**: gate unavailability (model not provisioned) or exceeding
+  its CE sub-budget (~60 ms, linked CTS) degrades to floor-only behavior — the
+  pre-existing, already-shipped recall path — plus a rate-limited
+  `memory_recall_gate_degraded` log marker and doctor visibility. Never a
+  silent fallback, matching memory-core-redesign's degradation contract.
+- **Operations**: no new operator action required — gate activation follows
+  `Memory.Embeddings.Enabled`; the existing warmup hosted service and doctor
+  checks extend to cover the new model without a new CLI verb.

openspec/changes/memory-relevance-gate/specs/memory-embeddings/spec.md

@@ -0,0 +1,43 @@
+# Delta: memory-embeddings (memory-relevance-gate)
+
+## MODIFIED Requirements
+
+### Requirement: Pinned model provisioning
+
+Memory-subsystem models SHALL be selected by id from a pinned in-code
+allowlist mapping model id to download URL, byte size, and SHA-256, covering
+more than one kind of model artifact (embedding models and relevance-scoring
+models share the same allowlist mechanism). A relevance-model manifest entry
+SHALL additionally carry a calibrated similarity threshold alongside its
+download and verification fields, so a model's operating point travels with
+its id rather than living as a disconnected configuration default. Arbitrary
+model URLs SHALL be rejected for every manifest kind. Provisioning SHALL
+download atomically (temporary file then rename), verify the hash before
+load, and run at daemon initialization when auto-download is enabled or on
+explicit operator command. No model artifact SHALL be embedded in the
+application binary.
+
+#### Scenario: Hash mismatch refuses the model
+
+- **GIVEN** a downloaded model artifact whose SHA-256 does not match the
+  allowlist entry
+- **WHEN** provisioning verifies the artifact
+- **THEN** the artifact is discarded and not loaded
+- **AND** the failure is surfaced as a doctor-visible error
+
+#### Scenario: Unknown model id is rejected
+
+- **GIVEN** configuration naming a model id absent from the allowlist
+- **WHEN** the daemon initializes embeddings
+- **THEN** provisioning refuses with a configuration error identifying the
+  allowlisted ids
+
+#### Scenario: Relevance manifest entry's threshold travels with its model id
+
+- **GIVEN** an allowlisted relevance-model manifest entry carrying a
+  calibrated threshold
+- **WHEN** that model id is provisioned and becomes active
+- **THEN** the calibrated threshold from that same manifest entry is what
+  governs gating, not a threshold associated with any other model id
+- **AND** switching to a different allowlisted relevance-model id switches
+  the effective threshold to that id's own calibrated value

openspec/changes/memory-relevance-gate/specs/memory-relevance-gate/spec.md

@@ -0,0 +1,167 @@
+# Spec: memory-relevance-gate (new capability)
+
+## ADDED Requirements
+
+### Requirement: In-process cross-encoder relevance scoring
+
+The system SHALL score floor-surviving recall candidates against the query
+with an in-process CPU ONNX cross-encoder — no sidecar processes, no network
+inference hop, mirroring the embedding runtime's execution model. The scorer
+SHALL sit behind a narrow interface owned by the memory subsystem so actor
+code carries no ONNX dependency, SHALL preserve candidate order across a
+batch call, and SHALL encode each `(query, candidate)` pair jointly (not as
+two independently embedded vectors) so the score reflects usefulness for
+answering the query rather than topical similarity alone.
+
+#### Scenario: Candidates score without external services
+
+- **GIVEN** a healthy daemon with the relevance model provisioned
+- **WHEN** automatic recall has floor-surviving candidates to gate
+- **THEN** each candidate is scored in-process against the query
+- **AND** no network call or child process is involved in scoring
+
+#### Scenario: Query text is never truncated to fit a candidate
+
+- **GIVEN** a floor-surviving candidate whose combined length with the query
+  exceeds the model's maximum sequence length
+- **WHEN** the pair is encoded for scoring
+- **THEN** the candidate side is truncated to fit
+- **AND** the query side is preserved in full
+
+### Requirement: Relevance model provisioning carries a calibrated operating point
+
+The relevance model SHALL be provisioned through the same pinned-allowlist
+mechanism as other memory-subsystem models (id → download URL, byte size,
+SHA-256, arbitrary URLs rejected), and its manifest entry SHALL additionally
+carry a calibrated similarity threshold that travels with the model id. A
+relevance model SHALL NOT be usable with a threshold calibrated for a
+different model id.
+
+#### Scenario: Calibrated threshold ships with the model id
+
+- **GIVEN** the relevance model manifest entry for the active model id
+- **WHEN** the recall coordinator applies the gate
+- **THEN** it uses the threshold carried by that manifest entry unless the
+  operator has configured an explicit override
+- **AND** no separate operator calibration step is required to get a
+  working default
+
+#### Scenario: Hash mismatch refuses the relevance model
+
+- **GIVEN** a downloaded relevance model artifact whose SHA-256 does not
+  match the allowlist entry
+- **WHEN** provisioning verifies the artifact
+- **THEN** the artifact is discarded and not loaded
+- **AND** the gate reports unavailable rather than scoring with an unverified
+  artifact
+
+### Requirement: Post-floor relevance gate on automatic recall
+
+After the existing absolute cosine floor admits candidates, the system SHALL
+score each surviving candidate (bounded to the automatic recall item limit)
+against the query and SHALL drop any candidate whose score falls below the
+active threshold. When every floor-surviving candidate is dropped by the
+gate, the turn SHALL inject nothing, identical in kind to the existing
+zero-survivors-at-the-floor outcome. The gate SHALL run under its own latency
+sub-budget nested inside the overall recall timeout.
+
+#### Scenario: Topically-adjacent but unhelpful candidate is rejected
+
+- **GIVEN** a floor-surviving candidate whose cosine similarity to the query
+  clears the absolute floor but whose content does not help answer the query
+- **WHEN** the relevance gate scores the candidate
+- **THEN** the candidate scores below the active threshold
+- **AND** the candidate is dropped before injection
+
+#### Scenario: Genuinely relevant candidate survives the gate
+
+- **GIVEN** a floor-surviving candidate that directly answers the query
+- **WHEN** the relevance gate scores the candidate
+- **THEN** the candidate scores above the active threshold
+- **AND** the candidate remains eligible for injection
+
+#### Scenario: All candidates gated out means nothing injected
+
+- **GIVEN** every floor-surviving candidate for a turn scores below the
+  active threshold
+- **WHEN** automatic recall completes for that turn
+- **THEN** no memory items are injected
+- **AND** the recall context block is omitted entirely from the prompt
+
+### Requirement: Loud degradation without silent fallback
+
+Automatic recall SHALL degrade to the floor-only result, unfiltered by the
+relevance gate, when the relevance model is unavailable (not provisioned,
+hash verification failed, runtime load error) or the gate exceeds its
+per-turn sub-budget. The degraded state SHALL be loud: a doctor check
+reports the cause, and a rate-limited structured log event records the
+degradation reason. The system SHALL NOT silently apply or silently skip
+gating without one of these signals.
+
+#### Scenario: Missing relevance model degrades to floor-only, loudly
+
+- **GIVEN** the relevance model is not provisioned
+- **WHEN** a turn triggers automatic recall with floor-surviving candidates
+- **THEN** recall injects the floor's own result unfiltered by any gate
+- **AND** a rate-limited degradation event is logged
+- **AND** `netclaw doctor` reports the missing relevance model with
+  remediation
+
+#### Scenario: Gate sub-budget timeout degrades to floor-only
+
+- **GIVEN** the relevance model is available but scoring exceeds its
+  configured sub-budget for a turn
+- **WHEN** the sub-budget elapses
+- **THEN** the gate stops waiting and recall injects the floor's own result
+  unfiltered for that turn
+- **AND** the degradation is logged at a rate-limited interval, not on every
+  occurrence
+
+### Requirement: Gate activation follows embedding enablement
+
+The relevance gate SHALL be active whenever automatic embeddings are
+enabled, without requiring a separate operator decision, while still
+allowing an explicit override in either direction. The active similarity
+threshold SHALL default to the value carried by the provisioned model's
+manifest entry, while allowing an explicit operator override.
+
+#### Scenario: Enabling embeddings enables the gate with no extra configuration
+
+- **GIVEN** an operator enables automatic memory embeddings with no gate
+  configuration present
+- **WHEN** the daemon starts
+- **THEN** the relevance gate is active using the manifest-provided
+  threshold for the provisioned relevance model
+
+#### Scenario: Explicit override disables the gate independent of embeddings
+
+- **GIVEN** automatic memory embeddings are enabled
+- **AND** the operator has explicitly disabled the relevance gate
+- **WHEN** automatic recall runs
+- **THEN** hybrid recall with the absolute cosine floor still applies
+- **AND** no candidate is scored or dropped by the relevance gate
+
+### Requirement: Gate decisions are observable in retrieval logging and evals
+
+The final retrieval log record for a turn SHALL include the relevance score
+computed for each gated candidate and the count of candidates dropped by the
+gate. The eval suite SHALL include a case that seeds a memory corpus, poses
+an off-topic query, and asserts both that no recall context block is added
+to the prompt and that a gate-degradation-or-decision marker is present in
+the logs for that turn.
+
+#### Scenario: Retrieval log records gate scores and drop count
+
+- **GIVEN** a turn where the relevance gate scored and dropped at least one
+  floor-surviving candidate
+- **WHEN** the final retrieval log line is written
+- **THEN** it includes the score computed for each gated candidate
+- **AND** it includes the count of candidates the gate dropped
+
+#### Scenario: Zero-injection eval case passes on an off-topic query
+
+- **GIVEN** a seeded memory corpus with no content relevant to a specific
+  off-topic question
+- **WHEN** the eval case asks that question
+- **THEN** the assembled prompt contains no `[memory-recall]` block
+- **AND** the turn's logs contain a relevance-gate marker for the decision