feat(lock): inject reference-tier locks as a bounded manifest (#1016-B)

CHANGELOG/v3.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Layered locks, injection: reference-tier locks render as a bounded manifest ([#1016](https://github.com/robotrocketscience/aelfrice/issues/1016)).** Second slice of #1016-B, building on the `lock_tier` field. At the hook injection paths (per-prompt `UserPromptSubmit` and `SessionStart`), a `reference`-tier lock is now surfaced as a **one-line manifest entry** — `ref <id>: "<topic>"` inside an `<aelfrice-locks-manifest>` block — instead of its full text, with the agent reading full text on demand via `aelf locked` / `aelf search`. `retrieve()` gains `manifest_reference_locks` (the hook paths pass it on): reference locks then cost only their manifest line in the budget, **freeing relevance budget** that fat locks otherwise consume (#1014/#1015). The topic is a deterministic string transform (first sentence or an 80-char cap — no ML). `frozen` locks are unchanged, and the whole feature is **byte-identical until a lock is demoted to `reference`** (frozen cost == full content; the default flag is off), so existing stores, the rebuilder, `aelf search`, and the benchmarks are unaffected. The rebuild-block and search-tool formatters still render reference locks verbatim — folding them into the manifest is a follow-up.
 - **Layered locks, foundation: a `lock_tier` field on beliefs ([#1016](https://github.com/robotrocketscience/aelfrice/issues/1016)).** First slice of the #1016-B bounded-lock work — the data model, **no injection change yet**. Locks gain a `lock_tier` orthogonal to `lock_level`: `frozen` (always injected verbatim — identity + always-relevant invariants) vs `reference` (bounded — destined for manifest-only injection in the next slice). The migration adds the column defaulting **every existing lock to `frozen`**, so behaviour is unchanged until the user opts a lock down (#379 no-silent-loss). `aelf lock --reference` / `--frozen` set or change the tier (re-locking demotes/promotes an existing lock); `aelf locked` annotates `[reference]` locks. Python-side `LOCK_TIERS` validates writes (insert + update raise on a bad tier), matching the `retention_class`/`scope` precedent. The manifest injection + disjoint lock-channel budget land in the follow-up slice.
 - **Lock-dedup hygiene: near-duplicate detection at lock time + a lock-scoped audit ([#1016](https://github.com/robotrocketscience/aelfrice/issues/1016)).** Locks are injected unbounded and never trimmed (#379), so re-locking slightly-reworded ground truth quietly accumulates near-duplicate locks that inflate the injection — on a real 24-lock store, 3 locks were one fact re-locked with κ/kappa wording drift (12.5%, matching the #1016 ~14% estimate). This is the #1016-C sub-task, implemented as **dedup** (the R&D-validated lever) rather than the issue's original distillation idea (distillation tested weak — locks are already dense). Two surfaces reuse the existing `dedup` engine (Jaccard ≥ 0.8 **and** Levenshtein ≥ 0.85): (1) `aelf lock` now prints a hygiene **warning** when the new lock is a near-duplicate of an existing lock, naming it and suggesting `aelf unlock`/`aelf delete` — warning only, the lock still writes (it is user ground truth); (2) `aelf doctor --dedup --dedup-locks` scopes the dedup audit to the user-locked set so the existing backlog cluster is findable without wading through the full-store report. New `dedup.find_near_duplicate_locks()` and a `locked_only` flag on `dedup_audit()`; the default full-store audit is byte-identical. No belief is auto-deleted — locks are user-asserted ground truth, so cleanup stays user-confirmed.
 - **`aelf doctor` ingest-gap detector ([#1011](https://github.com/robotrocketscience/aelfrice/issues/1011)).** `turns.jsonl` captures every turn in real time, but conversational *ingestion* folds them into beliefs only on certain triggers — so a session can log turns that never become retrievable (the #1011 logged-but-not-ingested signal). The auditor's new `ingest_gap` check compares the newest `turns.jsonl` timestamp against the newest conversation-derived belief (`agent_inferred` / `user_transcript`) and **warns** (`severity='warn'`, exit stays 0) when turns are newer — i.e. there is an un-ingested backlog. Mixed ISO formats (belief `…Z` vs turn `…+00:00`) are parsed to datetimes, not string-compared. The CLI owns the `turns.jsonl` read (raw last line, so a noise-filtered newest turn can't hide a gap) and the auditor stays pure-store. New store query `latest_belief_created_at(origins)`. On a real store it surfaces `~6d` of un-ingested turns.

src/aelfrice/hook.py

@@ -150,6 +150,14 @@
 CLOSE_TAG: Final[str] = "</aelfrice-memory>"
 SESSION_START_OPEN_TAG: Final[str] = "<aelfrice-baseline>"
 SESSION_START_CLOSE_TAG: Final[str] = "</aelfrice-baseline>"
+# #1016-B: reference-tier locks are injected as a one-line manifest
+# inside the memory/baseline block instead of verbatim, so lock injection
+# stays bounded; the agent reads full text on demand.
+LOCKS_MANIFEST_OPEN_TAG: Final[str] = (
+    '<aelfrice-locks-manifest note="bounded reference locks (#1016); read '
+    'full text on demand via `aelf locked` / `aelf search`">'
+)
+LOCKS_MANIFEST_CLOSE_TAG: Final[str] = "</aelfrice-locks-manifest>"
 
 # Sub-block tags injected on the first UserPromptSubmit of a session (#578).
 # Placed INSIDE <aelfrice-memory> before per-turn retrieval hits.
@@ -1731,14 +1739,55 @@ def _filter_session_exclusions(
         return hits
 
 
-def _format_hits(hits: list[Belief]) -> str:
-    lines: list[str] = [OPEN_TAG, _FRAMING_HEADER]
+def _split_belief_lines(
+    hits: list[Belief],
+) -> tuple[list[str], list[str]]:
+    """Render hits into verbatim `<belief>` lines + reference manifest lines.
+
+    #1016-B: a reference-tier lock is emitted as a single manifest entry
+    instead of full content (bounded injection); everything else — frozen
+    locks and non-locked hits — renders verbatim as before. Returns
+    `(belief_lines, manifest_lines)`; an empty `manifest_lines` means no
+    reference locks were present (byte-identical to the pre-#1016 output).
+    """
+    # Local import: keep the heavy retrieval module off hook.py's
+    # module-load path (these formatters run only after a retrieve()).
+    from aelfrice.retrieval import (  # noqa: PLC0415
+        is_reference_lock,
+        lock_manifest_line,
+    )
+    belief_lines: list[str] = []
+    manifest_lines: list[str] = []
     for h in hits:
+        if is_reference_lock(h):
+            # Escape framing tags in the manifest line exactly as belief
+            # content is escaped, so a reference lock cannot spoof the
+            # envelope (#1037 review). The belief id is a hex hash; only
+            # the topic could carry a tag.
+            manifest_lines.append(
+                "  " + _escape_for_hook_block(lock_manifest_line(h))
+            )
+            continue
         lock_attr = "user" if h.lock_level == LOCK_USER else "none"
         content = _escape_for_hook_block(h.content)
-        lines.append(
+        belief_lines.append(
             f'<belief id="{h.id}" lock="{lock_attr}">{content}</belief>'
         )
+    return belief_lines, manifest_lines
+
+
+def _manifest_block_lines(manifest_lines: list[str]) -> list[str]:
+    """Wrap reference-lock manifest lines in their block, or [] if none."""
+    if not manifest_lines:
+        return []
+    return [LOCKS_MANIFEST_OPEN_TAG, *manifest_lines, LOCKS_MANIFEST_CLOSE_TAG]
+
+
+def _format_hits(hits: list[Belief]) -> str:
+    belief_lines, manifest_lines = _split_belief_lines(hits)
+    lines: list[str] = [OPEN_TAG, _FRAMING_HEADER]
+    lines.extend(belief_lines)
+    lines.extend(_manifest_block_lines(manifest_lines))
     lines.append(CLOSE_TAG)
     lines.append("")
     return "\n".join(lines)
@@ -2345,12 +2394,9 @@ def _format_hits_with_session_start(
     lines: list[str] = [OPEN_TAG, _FRAMING_HEADER]
     if session_start_block:
         lines.append(session_start_block)
-    for h in hits:
-        lock_attr = "user" if h.lock_level == LOCK_USER else "none"
-        content = _escape_for_hook_block(h.content)
-        lines.append(
-            f'<belief id="{h.id}" lock="{lock_attr}">{content}</belief>'
-        )
+    belief_lines, manifest_lines = _split_belief_lines(hits)
+    lines.extend(belief_lines)
+    lines.extend(_manifest_block_lines(manifest_lines))
     lines.append(CLOSE_TAG)
     lines.append("")
     return "\n".join(lines)
@@ -2812,7 +2858,12 @@ def _retrieve_baseline_with_block(
     """
     store = _open_store()
     try:
-        hits = retrieve(store, "", token_budget=token_budget)
+        # #1016-B: SessionStart renders reference-tier locks as a manifest,
+        # so budget them at manifest size (byte-identical until demotion).
+        hits = retrieve(
+            store, "", token_budget=token_budget,
+            manifest_reference_locks=True,
+        )
     finally:
         store.close()
     if not hits:
@@ -2828,13 +2879,10 @@ def _format_baseline_hits(hits: list[Belief]) -> str:
     the model can tell which channel a belief arrived through. Lock
     state is carried as a `lock` attribute on the inner <belief>.
     """
+    belief_lines, manifest_lines = _split_belief_lines(hits)
     lines: list[str] = [SESSION_START_OPEN_TAG, _FRAMING_HEADER]
-    for h in hits:
-        lock_attr = "user" if h.lock_level == LOCK_USER else "none"
-        content = _escape_for_hook_block(h.content)
-        lines.append(
-            f'<belief id="{h.id}" lock="{lock_attr}">{content}</belief>'
-        )
+    lines.extend(belief_lines)
+    lines.extend(_manifest_block_lines(manifest_lines))
     lines.append(SESSION_START_CLOSE_TAG)
     lines.append("")
     return "\n".join(lines)

src/aelfrice/hook_search.py

@@ -68,8 +68,13 @@ def search_for_prompt(
     read, calls `record_retrieval` to write one audit row per returned
     belief.
     """
+    # #1016-B: this is a hook injection path whose formatter renders
+    # reference-tier locks as a one-line manifest, so budget them at
+    # manifest size (frees relevance budget; byte-identical until a lock
+    # is demoted to reference).
     hits: list[Belief] = retrieve(
         store, prompt, token_budget=token_budget,
+        manifest_reference_locks=True,
     )
     record_retrieval(store, hits, stderr=stderr)
     return hits

src/aelfrice/retrieval.py

@@ -95,7 +95,12 @@
     heat_kernel_score,
     seeds_from_bm25,
 )
-from aelfrice.models import LOCK_NONE, LOCK_USER, Belief
+from aelfrice.models import (
+    LOCK_NONE,
+    LOCK_TIER_REFERENCE,
+    LOCK_USER,
+    Belief,
+)
 from aelfrice.scoring import (
     DEFAULT_POSTERIOR_WEIGHT,
     ZETA_ALPHA_DEFAULT,
@@ -546,6 +551,57 @@ def _belief_tokens(b: Belief) -> int:
     return _estimate_tokens(b.content)
 
 
+# --- #1016-B layered locks: reference-tier manifest -------------------
+
+# Cap on the manifest topic. A reference lock is surfaced as one line —
+# `ref <id>: "<topic>"` — so injection stays ~constant regardless of the
+# lock's full length; the agent reads full text on demand.
+_LOCK_TOPIC_MAX: Final[int] = 80
+
+
+def _lock_topic(content: str) -> str:
+    """Deterministic one-line topic for a reference lock's manifest entry.
+
+    Whitespace-collapsed; the first sentence if it ends within the cap,
+    else a hard char-cap with an ellipsis. No ML — a pure string
+    transform so the manifest is reproducible. Internal double-quotes are
+    flattened to single so the `"<topic>"` wrapper stays unambiguous.
+    """
+    collapsed = " ".join(content.split()).replace('"', "'")
+    if not collapsed:
+        return ""
+    for sep in (". ", "? ", "! "):
+        idx = collapsed.find(sep)
+        if 0 <= idx < _LOCK_TOPIC_MAX:
+            return collapsed[: idx + 1].strip()
+    if len(collapsed) <= _LOCK_TOPIC_MAX:
+        return collapsed
+    return collapsed[:_LOCK_TOPIC_MAX].rstrip() + "…"
+
+
+def lock_manifest_line(b: Belief) -> str:
+    """One-line manifest entry for a reference-tier lock (#1016-B)."""
+    return f'ref {b.id}: "{_lock_topic(b.content)}"'
+
+
+def is_reference_lock(b: Belief) -> bool:
+    """True iff `b` is a user lock demoted to the bounded reference tier."""
+    return b.lock_level == LOCK_USER and b.lock_tier == LOCK_TIER_REFERENCE
+
+
+def lock_injection_tokens(b: Belief, *, manifest_reference_locks: bool) -> int:
+    """Token cost of injecting a locked belief.
+
+    When `manifest_reference_locks` is on, a reference lock costs only its
+    one-line manifest entry (the #1016-B bound); otherwise — and for every
+    frozen lock — it costs full content, identical to `_belief_tokens`. So
+    the default (off) is byte-identical to pre-#1016 budgeting.
+    """
+    if manifest_reference_locks and is_reference_lock(b):
+        return _estimate_tokens(lock_manifest_line(b))
+    return _belief_tokens(b)
+
+
 # --- Config flag resolution ----------------------------------------------
 
 
@@ -2524,6 +2580,7 @@ def retrieve(
     heat_kernel_enabled: bool | None = None,
     eigenbasis_cache: GraphEigenbasisCache | None = None,
     use_type_aware_compression: bool | None = None,
+    manifest_reference_locks: bool = False,
 ) -> list[Belief]:
     """Return L0 locked + L2.5 entity + L1 BM25 + L3 BFS expansions.
 
@@ -2662,7 +2719,17 @@ def _cost(b: Belief) -> int:
     # that callers passing a tight `token_budget` never get more
     # tokens back than they asked for, while still letting the
     # default 2400-budget caller see the full 400-token L2.5 slice.
-    locked_used: int = sum(_belief_tokens(b) for b in locked)
+    #
+    # #1016-B: when manifest_reference_locks is on (the hook injection
+    # paths), reference-tier locks cost only their one-line manifest entry,
+    # freeing relevance budget. Frozen locks (every lock by default) still
+    # cost full content, so this is byte-identical until a lock is demoted.
+    locked_used: int = sum(
+        lock_injection_tokens(
+            b, manifest_reference_locks=manifest_reference_locks
+        )
+        for b in locked
+    )
     # #379 locks are uncapped + never trimmed; reserve a relevance floor so
     # they can't starve L2.5/L1 to zero. No-op (byte-identical) unless locks
     # leave less than the floor — see RELEVANCE_BUDGET_FLOOR_FRACTION.