feat: local embedding offline benchmark runner (#128)

docs/evaluation/2026-07-29-embedding-benchmark.json

@@ -0,0 +1,8 @@
+{
+  "models": {
+    "baseline": "pending_corpus",
+    "primary": "pending_corpus"
+  },
+  "result": "pending_corpus",
+  "schemaVersion": "context-engine-embedding-benchmark-frozen-result-v1"
+}

eval/README.md

@@ -42,3 +42,75 @@ authorized Package, and case pass rate. Package Evidence has no relevance-rank
 contract, so this version does not report recall@k. Quality is `measured`;
 Reliability and Budget remain `not-evaluated`. This report does not modify or
 feed the release-gate report, and there is no CI threshold.
+
+## Offline embedding comparison
+
+`context-engine-embedding-benchmark` is an offline evaluation CLI. It compares
+the exact 384-dimensional primary and baseline model identities required by
+issue #128 and never composes an `EmbeddingProvider` into Supply or Runtime.
+The main environment deliberately contains no model framework. The closed
+`sentence-transformers` backend is installed explicitly with
+`uv sync --extra benchmark`. Each local pinned model directory is the exact
+complete `SentenceTransformer` snapshot declared by the tracked registry: its
+weight, model/config, tokenizer, modules, and pooling files must match the
+registry's closed path-and-SHA-256 artifact list, with no missing or extra
+files. The tracked
+`embedding-benchmark/model-registry.json`, not run input or a local manifest,
+owns the expected repository id, immutable revision, artifact digest,
+dimension, complete transformation pipeline, pooling, prompt prefixes,
+precision, and batching. See that registry for the exact ordered pipelines;
+both emit unit-norm 384-dimensional vectors under the same geometry. Unknown
+models and local identity overrides are refused, and the runner hashes every
+on-disk artifact against the tracked expected digests before and after the
+backend loads the model.
+
+`SentenceTransformer` requires a directory path and reopens its artifacts, so
+the two checks do not create a tamper-proof snapshot. A concurrent writer could
+swap files during backend construction and restore the registered bytes before
+the post-load check. That adversary is outside M1's single trusted local operator
+threat model: the runner provides provenance under that trusted operator, not
+protection against adversarial concurrent writes. If an untrusted party gains
+concurrent write access to model storage, stronger snapshot or locking semantics
+require their own ADR.
+
+The input follows `embedding-benchmark/input.schema.json`; the output follows
+`embedding-benchmark/report.schema.json` and must be written below the ignored
+`.context-engine/` directory. The input includes the typed RFC 8785/SHA-256
+`sha256-rfc8785-accidental-edit-detection-v1` lock. It detects accidental edits,
+truncation, and re-serialization performed without relocking, but it is not a
+defense against deliberate forgery: the M1 threat model is one trusted local
+operator who can recompute a colocated digest. If the boundary changes to an
+untrusted caller or remote runner, signing returns as its own ADR. All benchmark
+JSON entry points reject out-of-range or non-finite numbers and bound document
+size, nesting, strings, and containers with a typed refusal. Retrieval scores
+are not implemented by this runner: it imports the one fixed judge
+factory owned by issue #129, which is solely responsible for case hit,
+macro/micro Evidence recall, and slice breakdowns. Until #129 lands, the real
+CLI deliberately fails closed with `retrieval judge is unavailable` rather
+than degrading to a second metric implementation.
+
+The maintainer corpus is private and pending delivery to a durable,
+maintainer-controlled location outside disposable Git worktrees. Once available,
+an operator may run:
+
+```bash
+uv run context-engine-embedding-benchmark run \
+  --dataset "$DURABLE_GOLDEN_ROOT/embedding-benchmark-v1.json" \
+  --backend sentence-transformers \
+  --primary-model-dir "$PINNED_MODEL_ROOT/qwen3-embedding-0.6b" \
+  --baseline-model-dir "$PINNED_MODEL_ROOT/multilingual-e5-small" \
+  --output .context-engine/eval/embedding-benchmark-v1.json
+```
+
+The tracked frozen result contains metrics only. It currently records
+`pending_corpus`; it must never contain queries, note titles, paths, excerpts,
+or model weights. The model verdict uses Pareto dominance across case hit,
+macro Evidence recall, and micro Evidence recall: one model wins only when it is
+no worse on all three and strictly better on at least one. Exact equality is a
+tie; mixed wins are `inconclusive`. Per-slice results remain diagnostic and no
+weighted composite or tiebreak manufactures a winner. A losing or inconclusive
+primary is a valid benchmark outcome, not a runner error.
+
+The issue remains open while the corpus is pending. A real-model run, frozen
+numeric result, actual model verdict, and measured comparison to the standing
+3.8% twin baseline are not complete until that durable corpus arrives.

eval/__init__.py

@@ -0,0 +1 @@
+"""Offline evaluation tools and deterministic judge contracts."""

eval/embedding-benchmark/input.schema.json

@@ -0,0 +1,60 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://context-engine.invalid/eval/embedding-benchmark/input.schema.json",
+  "title": "ContextEngine offline embedding benchmark input v1",
+  "description": "M1 assumes one trusted local operator. The colocated digest detects accidental edits, truncation, or re-serialization performed without relocking; it is not a defense against deliberate forgery because an operator can recompute the digest. If callers or runners become remote or untrusted, signing requires its own ADR.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schemaVersion", "lock", "documents", "cases"],
+  "properties": {
+    "schemaVersion": {"const": "context-engine-embedding-benchmark-input-v1"},
+    "lock": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["profile", "contentDigest"],
+      "properties": {
+        "profile": {
+          "description": "Typed accidental-edit detector for one trusted local operator; not a tamper-proof or anti-forgery signature.",
+          "const": "sha256-rfc8785-accidental-edit-detection-v1"
+        },
+        "contentDigest": {"type": "string", "pattern": "^[0-9a-f]{64}$"}
+      }
+    },
+    "documents": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["documentRef", "text"],
+        "properties": {
+          "documentRef": {"$ref": "#/$defs/nonblank"},
+          "text": {"$ref": "#/$defs/nonblank"}
+        }
+      }
+    },
+    "cases": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["caseRef", "query", "expectedDocumentRefs", "slice"],
+        "properties": {
+          "caseRef": {"type": "string", "pattern": "^[a-z][a-z0-9_-]{0,127}$"},
+          "query": {"$ref": "#/$defs/nonblank"},
+          "expectedDocumentRefs": {
+            "type": "array",
+            "minItems": 1,
+            "uniqueItems": true,
+            "items": {"$ref": "#/$defs/nonblank"}
+          },
+          "slice": {"$ref": "#/$defs/nonblank"}
+        }
+      }
+    }
+  },
+  "$defs": {
+    "nonblank": {"type": "string", "minLength": 1, "pattern": ".*\\S.*"}
+  }
+}

eval/embedding-benchmark/model-registry.json

@@ -0,0 +1,58 @@
+{
+  "schemaVersion": "context-engine-embedding-model-registry-v1",
+  "models": {
+    "primary": {
+      "backend": "sentence-transformers",
+      "artifacts": [
+        {"path": "1_Pooling/config.json", "sha256": "37bf193fa101f19101bfad9c31d3eb0f786e247b7b1e5cb7f007d730eed1ddbd"},
+        {"path": "config.json", "sha256": "b5bf1f51fc45be473a54718cef92448d90a1be001bf9b9a44b8c7f10a19feaa9"},
+        {"path": "config_sentence_transformers.json", "sha256": "10667c72ddb772627bf1780cb7f86af8e2ae0032b8c243c731172064105c6961"},
+        {"path": "generation_config.json", "sha256": "28396d421a2108acce96383f6a7de78008f7f1b17f807958f3c14c51dbfb65fb"},
+        {"path": "merges.txt", "sha256": "8831e4f1a044471340f7c0a83d7bd71306a5b867e95fd870f74d0c5308a904d5"},
+        {"path": "model.safetensors", "sha256": "0437e45c94563b09e13cb7a64478fc406947a93cb34a7e05870fc8dcd48e23fd"},
+        {"path": "modules.json", "sha256": "84e40c8e006c9b1d6c122e02cba9b02458120b5fb0c87b746c41e0207cf642cf"},
+        {"path": "tokenizer.json", "sha256": "def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a"},
+        {"path": "tokenizer_config.json", "sha256": "253153d0738ceb4c668d2eff957714dd2bea0b56de772a9fdccd96cbf517e6a0"},
+        {"path": "vocab.json", "sha256": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"}
+      ],
+      "identity": {
+        "modelId": "Qwen/Qwen3-Embedding-0.6B",
+        "revision": "97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3",
+        "artifactDigest": "8cb25677d5be69ce6ac88ebbdfb5dad30980fee39c35c6324a583e325917eddc",
+        "dimension": 384,
+        "pooling": "last_token",
+        "queryPrefix": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:",
+        "documentPrefix": "",
+        "transformationPipeline": "l2 -> truncate 1024->384 -> l2",
+        "precision": "float32",
+        "batchSize": 8
+      }
+    },
+    "baseline": {
+      "backend": "sentence-transformers",
+      "artifacts": [
+        {"path": "1_Pooling/config.json", "sha256": "987f7a67a38fa564c849bb5d277c52ab9088a84368fc0be31a354125aebb12a0"},
+        {"path": "config.json", "sha256": "69137736cab8b8903a07fe8afaafdda25aac55415a12a55d1bffa9f581abf959"},
+        {"path": "model.safetensors", "sha256": "1a55775f53449dac10a2bcbc312469fac40b96d53198c407081a831f81c98477"},
+        {"path": "modules.json", "sha256": "c6e29747481e8b5dd2b58401966aeac910de39092f90cda9a704b1545f902b04"},
+        {"path": "sentence_bert_config.json", "sha256": "948201d8329907aae938fa62f9ceeed53f5694dacc2b87b9f3b78b37ee986529"},
+        {"path": "sentencepiece.bpe.model", "sha256": "cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865"},
+        {"path": "special_tokens_map.json", "sha256": "d05497f1da52c5e09554c0cd874037a083e1dc1b9cfd48034d1c717f1afc07a7"},
+        {"path": "tokenizer.json", "sha256": "0b44a9d7b51c3c62626640cda0e2c2f70fdacdc25bbbd68038369d14ebdf4c39"},
+        {"path": "tokenizer_config.json", "sha256": "a1d6bc8734a6f635dc158508bef000f8e2e5a759c7d92f984b2c86e5ff53425b"}
+      ],
+      "identity": {
+        "modelId": "intfloat/multilingual-e5-small",
+        "revision": "614241f622f53c4eeff9890bdc4f31cfecc418b3",
+        "artifactDigest": "961e0953328daf4df7916c1406751c5c63fe19e14a5c7e977d9f8b38964c5cf3",
+        "dimension": 384,
+        "pooling": "mean",
+        "queryPrefix": "query: ",
+        "documentPrefix": "passage: ",
+        "transformationPipeline": "l2 -> keep native 384 -> l2",
+        "precision": "float32",
+        "batchSize": 8
+      }
+    }
+  }
+}

eval/embedding-benchmark/model-registry.schema.json

@@ -0,0 +1,66 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://context-engine.invalid/eval/embedding-benchmark/model-registry.schema.json",
+  "title": "ContextEngine offline embedding tracked model registry v1",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schemaVersion", "models"],
+  "properties": {
+    "schemaVersion": {"const": "context-engine-embedding-model-registry-v1"},
+    "models": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["primary", "baseline"],
+      "properties": {
+        "primary": {"$ref": "#/$defs/model"},
+        "baseline": {"$ref": "#/$defs/model"}
+      }
+    }
+  },
+  "$defs": {
+    "digest": {"type": "string", "pattern": "^[0-9a-f]{64}$"},
+    "identity": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["modelId", "revision", "artifactDigest", "dimension", "transformationPipeline", "pooling", "queryPrefix", "documentPrefix", "precision", "batchSize"],
+      "properties": {
+        "modelId": {"type": "string", "minLength": 1},
+        "revision": {"type": "string", "pattern": "^(?:[0-9a-f]{40}|[0-9a-f]{64})$"},
+        "artifactDigest": {"$ref": "#/$defs/digest"},
+        "dimension": {"const": 384},
+        "transformationPipeline": {
+          "description": "Complete ordered vector transformation, including upstream normalization, dimensional reduction, and final normalization. Both supported pipelines emit unit-norm 384-dimensional vectors.",
+          "enum": ["l2 -> truncate 1024->384 -> l2", "l2 -> keep native 384 -> l2"]
+        },
+        "pooling": {"type": "string", "minLength": 1},
+        "queryPrefix": {"type": "string"},
+        "documentPrefix": {"type": "string"},
+        "precision": {"type": "string", "minLength": 1},
+        "batchSize": {"type": "integer", "minimum": 1}
+      }
+    },
+    "model": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["backend", "artifacts", "identity"],
+      "properties": {
+        "backend": {"const": "sentence-transformers"},
+        "artifacts": {
+          "description": "The exact complete local snapshot consumed by SentenceTransformer. Every required config, tokenizer, pooling, and weight file is present and digest-bound; extra or missing files are refused.",
+          "type": "array",
+          "minItems": 1,
+          "items": {
+            "type": "object",
+            "additionalProperties": false,
+            "required": ["path", "sha256"],
+            "properties": {
+              "path": {"type": "string", "minLength": 1},
+              "sha256": {"$ref": "#/$defs/digest"}
+            }
+          }
+        },
+        "identity": {"$ref": "#/$defs/identity"}
+      }
+    }
+  }
+}