Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions bench/BASELINE-LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,17 @@ architect):
| #286 teams-slice-1 AccessContextV1 contract | 2026-08-21 | additive, eval-unreachable (all-new paths; verdict 5356722388) | no re-baseline owed | RECORDED |
| #297 auto-focus keeps newest request (summary-steering) | 2026-08-21 | compaction/assembly focus string for over-cap user turns (ingest-side summarization included) | next full-500 V1-M run re-baselines (deterministic); next main-tree LoCoMo row records post-#297 | OPEN |
| #300 teams-slice-2 policy/catalog/scope-storage | 2026-08-21 | additive, eval-unreachable (all-new paths, zero call sites; verdict 5358203472) | no re-baseline owed | RECORDED |

Rows appended by the v0.23.2 security/privacy train (architect, 2026-09-05; merge shas derived with
`gh pr view <n> --json mergeCommit`; the single discharge event for every OPEN row below is the F53
re-bank registration, #367/#380, which runs on the shipped posture — durable lossless + provider-copy
transform ON — and registers as a NEW row, never as a delta against F53's raw-embed config):

| boundary (merge) | date | surface | required re-baseline | status |
|---|---|---|---|---|
| #364 read-only current-schema clones (9e43907f; closes #363) | 2026-08-26 | compaction path under concurrent SQLite writers: a Hermes per-agent host clone whose current-schema startup hit a busy WAL fell back silently to the lossy compressor; startup is now read-only at the two idempotent migration/repair boundaries | bench-inert by construction: `benchmarking/longmemeval.py` builds `MessageStore` / `SummaryDAG` / `VectorStore` directly and never runs `LCMEngine` compaction or clone startup, so no bench row can observe this boundary and none is owed a re-baseline; the observable gate is the multi-clone live soak (release-gauntlet Phase C), which already ran post-#364 on the rc3/GA tree (59bba769 ⊇ 9e43907f) with the pre-existing #247 conflicts as its only finding | RECORDED — bench-inert by construction; live-soak gate discharged at rc3/GA |
| #366 private-key redaction ordering (bc033109; SECURITY #365) | 2026-08-26 | **provider-copy content shaping** (`protect_embedding_text`): private-key patterns apply before assignment patterns (e.g. `password_assignment`) on both durable and provider paths, and a transform-independent residual validator fails closed (raise) rather than dispatching a leaked key body; transform identity privacy:v1→v3 (`ingest_protection.py:396` on main), so every cloud vector identity re-embeds. Durable path moves only when `sensitive_patterns_enabled=True` (default False, `config.py:702`) | F53 re-bank registers privacy:v3 + the provider-copy posture; a corpus without credential-shaped material is expected-inert, but the re-bank must report the TRANSFORM-CHANGE COUNT — documents for which `protect_embedding_text` returned `changed=True` or raised, measured by a full prewarm pass over the prepared corpus (input-hash parity; the harness has no blocked counter and a redaction changes text without blocking) — 0 means inert for that corpus, >0 means the boundary is live and the row moves | OPEN — discharged by the F53 re-bank |
| #370 privacy policy errors fail loud on the recall path (f21fc84b; closes #367 ask 3 — ask 1, the trio ledger row, landed via #368 b2142ed8) | 2026-08-26 | INSTRUMENT + recall path: `lcm_recall` no longer swallows `EmbeddingPrivacyPolicyError` into a silent FTS degrade (four handler sites; proactive-recall counter + warn-once); `benchmarking/longmemeval.py` now applies the production provider-copy transform on the run path (`evaluate_question` builds an `embedding_privacy_enabled=None` config and calls `protect_embedding_text` directly, ~:2405-2743) and via `_embedding_privacy_context` (:478) in the prewarm/determinism paths (:1588/:1655) | the loud-fail half moves no baseline by construction (a run that would have silently degraded now stops loud; already reflected in the privacy-trio row's HARNESS GAP CLOSED clause); the harness-transform half is NOT inert by construction — `protect_embedding_text` can rewrite a document or query (`changed=True`) without raising, so it is the same provider-copy boundary as #366/#374 and is discharged by the F53 re-bank's transform-change count (see the #366 row) | OPEN — loud-fail half RECORDED; harness-transform half discharged by the F53 re-bank |
| #374 lossless by default (fbe24cce; closes #371 #346-amendment) | 2026-08-26 | **recall-path content shaping in both directions**: (a) the durable store is lossless unless `sensitive_patterns_enabled=True` — v0.23.1 configs that forced durable redaction under a cloud provider now retain raw text, so FTS hits and rerank snippets see raw; (b) provider copies — embed documents, embed query (`tools.py:3762`), rerank query + snippets (`tools.py:5457/5461`) — pass through the transform by default for cloud providers via `embedding_privacy_enabled` (`config.py:707`, None = auto-on for cloud), and the explicit opt-out is a distinct vector identity (`privacy:off`) | F53 re-bank on the shipped posture; the F53 declared config predates the flag and embedded raw text with no transform, so the re-bank is a new row registration | OPEN — discharged by the F53 re-bank |
| #384 + #391 truncated-PEM scanner (c0402f31 + 59bba769; rc1/rc2 gauntlet catches, #383 #389) | 2026-08-26 / 2026-08-27 | provider-copy content shaping: marker-independent backstops (orphan full-width base64 runs; raw-text proximity-to-placeholder backstop restored with git-SHA hex exclusion) make a chunk carrying PEM-shaped material near a redacted key fail closed (embed refused) instead of dispatching; #391 removes the key-plus-git-SHA over-block #384 introduced | same discharge event as #366/#374; expected-inert on PEM-free corpora, verified by the re-bank's transform-change count == 0 (see the #366 row) | OPEN — expected-inert; verify at re-bank |
| #155 session-end prefix matching extracted to a mixin (a84ca050) | 2026-09-04 | compaction path refactor only; maintainer AST-identity check: 18 moved methods, 18 identical, 0 different, no shadowing (acceptance-155 receipt) | ledger entry only, no re-baseline owed | RECORDED |
Loading