feat(consolidate): audit-only near-duplicate cluster report — aelf doctor --consolidate (#1312)

CHANGELOG/v4.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Consolidation is auditable without being buildable ([#1312](https://github.com/robotrocketscience/aelfrice/issues/1312), [#1176](https://github.com/robotrocketscience/aelfrice/issues/1176) proposal 4).** New `aelf doctor --consolidate` clusters active beliefs at the shipped dedup predicate (Jaccard >= 0.8 **and** Levenshtein ratio >= 0.85), picks each component's **medoid** — the member minimising summed Levenshtein distance to the others, tie-broken id-ASC — and reports how many beliefs a contraction would retire. Read-only: no edge is inserted, no belief is retired, no log row is written, and `dedup.py` stays audit-only. The contraction write path is **deliberately not built**, because the same measurement that funded the report priced it: on the development store it would retire **996 of 44,594 active beliefs (2.23%)**, which does not carry a high-risk write path or the `SUPERSEDES` direction convention it would establish with nothing gating it (there are still **0** such edges). Proposal 4 did clear its pre-registered kill gate — polarity-only member pairs, tested against the vocabularies `relationship_detector` already ships, are **0.17-0.24%** of examined pairs and appear in only **1-2 of the 50 largest clusters** — so the thresholds are sound and it is the reach, not the predicate, that fails. Candidate pairs come from order-4 shingle blocking with a `df <= 32` cap rather than dedup's O(n^2) prefilter, which is ~991M pairs at this scale; tightening that cap from 400 to 32 left the components bit-identical, and skipped high-df shingles are **counted and reported**, never dropped silently. Output is aggregate counts only — no belief content, no ids, no paths — so the report is safe to paste into an issue, and the published 2.23% is reproducible from the shipped command rather than from a scratch script.
 - **Two-tier headline rendering measured and not built ([#1286](https://github.com/robotrocketscience/aelfrice/issues/1286), [#1177](https://github.com/robotrocketscience/aelfrice/issues/1177) proposal 15).** `benchmarks/two_tier_render_bound.py` runs the proposal's own kill experiment before any build code exists: replay the blocks the hook recorded, re-render them under `render_mode(b, rank, cost)` — verbatim for a user lock, for `rank < 5`, or for `cost <= 60`, headline otherwise — and report the delta in injected tokens against a **~10% ship bar**. Over **353 real `user_prompt_submit` blocks** after the 2026-06-30 regime break the pooled saving is **1.39%**, per-block median **0.42%** (q1/q3 0.00%/2.52%, max 12.47%), and the rule changes nothing at all on **34.0%** of blocks while making **16** of them larger. **It fails the bar, and it is not the constants.** A sweep over the two knobs moves the pooled saving from 1.39% to at most **1.48%**, that ceiling being the most aggressive form of the rule — headline every non-locked belief regardless of rank or length. The reason is compositional and generalises past this proposal: of the injected tokens, **61.0% are user-locked belief lines the rule may not touch** and **32.9% is framing, `<core>`, manifest and markup**, leaving **6.1% eligible**. So **6.1% is the absolute ceiling for *any* lock-exempt render rule**, below the bar before a single belief is headlined — the parent's own prediction that the saving would be "dominated by the lock and `<core>` redundancy" is confirmed rather than inferred. Two secondary findings: the `rank < K_verbatim` clause is **dead on live data** (0 of 19,474 belief lines), because locks lead every block and the lock exemption fires first, and the `rank_in_pack` reading used here counts every belief line rather than only the non-locked tier, which headlines strictly *more* beliefs — so the result is an upper bound, not a point estimate. The measurement reuses the shipped `compression._headline` and `retrieval._estimate_tokens` rather than restating them, and its parse is pinned by mutation-verified tests against `hook._split_belief_lines`, because a regex that drifts from the renderer parses zero beliefs and reports a 0.00% saving indistinguishable from a real null. Aggregates only; no belief text is emitted. **Recommendation on record: do not fund the build.** The accounting lie the proposal also names — budget computed on a compressed size that is never rendered — is real and survives this verdict as separate, smaller work.
 - **Injection-block ordering is an explicit, recorded policy ([#1274](https://github.com/robotrocketscience/aelfrice/issues/1274)).** Position in the injected block was a side effect of lane concatenation (`locked + l25 + l1 + hrr + spine + bfs`), not a policy anyone chose, so it could not be measured without a rewrite. `retrieval.order_for_injection()` names three — `lane` (identity, the default), `locks_last`, `score_desc` — behind `resolve_order_policy()` on the usual env -> argument -> TOML precedence, applied at the render boundary in `hook._split_belief_lines` and recorded as `order_policy` on the hook-audit row. Every policy is a stable, total permutation (ties break on the original index), so the rendered order is a pure function of (hits, policy, scores) and replay reproduces it. **Default unchanged: under `lane` the block is byte-identical.** `score_desc` needs rerank scores, which `Belief` does not carry; absent them it degrades to `lane` and traces to stderr rather than silently substituting the posterior, and an unscored hit sorts last rather than first because scores are log-domain negative. The pre-flight that sized this (`benchmarks/order_policy_movable_bound.py`, 341 real blocks after the 2026-06-30 regime break) found the movable set is real — **71.6%** of blocks carry >=2 non-locked hits (`score_desc`'s threshold) and **72.9%** carry both tiers, which is `locks_last`'s, since relocating the whole locked tier reorders a block holding just one of each — but also that **position 1 is a user lock in 100%** of them, a median of **50** locks deep, so any lock-relocating arm spends the locked tier's position and must be scored on lock-following rather than accuracy alone. The proposed `u_shaped` default is deliberately not implemented for that reason. Note for anyone running the A/B: the bench adapters retrieve with `include_locked=False` and ingest at `LOCK_NONE`, so all three policies are the same permutation there and a null would be an inert instrument, not a result.
 - **Lock-consistency annotation, library half ([#1175](https://github.com/robotrocketscience/aelfrice/issues/1175)).** The L0 locked tier is injected ahead of everything else and no consistency check runs between it and the L1/L2.5 tiers, so the model can read the user's explicit ground truth and then read a contradiction of it in the same prompt. `src/aelfrice/lock_consistency.py` computes which retrieved beliefs slot-conflict with a lock. **ANNOTATE, never DROP** — the slot keys come from a preceding-alphabetic-token heuristic that emits junk, so a filter would delete beliefs on a bad key, and handing the model the defeat rather than the deletion is what the #605 posture requires. **The suppression is the design, and it was measured rather than asserted — including two rejected attempts.** Unsuppressed, 6.12% of retrieved unlocked beliefs conflict with a lock across 189 real prompts replayed against a live 44,584-belief store, and **63.7% of those trace to one version-and-date lock**. Capping the share of a single pack one lock may flag was refuted by the data: that lock flags a p50 of just 2.9% of any one pack, so its noise is spread across queries rather than concentrated within one and a per-retrieval threshold cannot see it. Dropping keys that are multi-valued inside one belief — the dominant lock carries key `v` with four distinct values in a single sentence, which cannot support a functional dependency — helps to 4.21% and evicts that lock, but concentration got **worse**, with the next version-and-date lock taking over at 72.4%; it treated a symptom. Inspecting what the extractor actually emits (`v=3.6`, `shipped=2026.0`, a SHA fragment as a key in `b88fd4=9.0`, a CI run id) gave the third rule. All three together — key multi-valued in the belief, key in `{v, version, rev, release}`, value a 4-digit year — take the rate to **1.38% with the top lock at 27.2%**, spread across five locks rather than dominated by one. That is the difference between an annotation a user reads and one they learn to ignore. Suppression is symmetric and slot-scoped, both pinned by tests: an ambiguous key is as meaningless on the candidate side as on the lock side (passing raw candidate slots through passed all 23 other tests, because every other case suppresses on the lock side first), and a belief carrying both a version literal and a genuine disagreement is still annotated on the genuine one. The filter deliberately lives in this module rather than in `value_compare.extract_values`: narrowing the shipped extractor would change what counts as a numeric slot for every consumer, which is the wider blast radius [#1228](https://github.com/robotrocketscience/aelfrice/issues/1228) explicitly declined. Pure and deterministic — local predicates over a belief's own slots, no store read, no corpus scan, no per-store threshold. Library only; the retrieval wiring and the `hook.py` render land separately, matching how `clustering.py` shipped.

src/aelfrice/cli.py

@@ -6090,6 +6090,8 @@ def _cmd_doctor(args: argparse.Namespace, out: object) -> int:
         return _cmd_doctor_replay(args, out)
     if getattr(args, "dedup", False):
         return _cmd_doctor_dedup(args, out)
+    if getattr(args, "consolidate", False):
+        return _cmd_doctor_consolidate(args, out)
     if getattr(args, "relationships", False):
         return _cmd_doctor_relationships(args, out)
     if getattr(args, "detect_stale", False):
@@ -6404,6 +6406,63 @@ def _cmd_doctor_dedup(args: argparse.Namespace, out: object) -> int:
     return 0
 
 
+def _cmd_doctor_consolidate(args: argparse.Namespace, out: object) -> int:
+    """Run the audit-only consolidation report (#1312, #1176 proposal 4).
+
+    Clusters near-duplicate active beliefs at the shipped dedup
+    thresholds, picks each component's medoid, and reports how many
+    beliefs a contraction would retire. Read-only: no edges are
+    inserted, no beliefs are retired, no log rows are written.
+
+    Contraction itself is deliberately not built — the operator funded
+    the report and not the write path, because the measured reach is
+    ~2% of the store.
+
+    Exit 0 regardless of cluster count: clusters are diagnostic, not
+    failure conditions. Exit 1 only on malformed thresholds.
+    """
+    from aelfrice.consolidate import (
+        DEFAULT_MAX_SHINGLE_DF,
+        consolidation_audit,
+        format_consolidation_report,
+    )
+    from aelfrice.dedup import load_dedup_config
+
+    config = load_dedup_config()
+    j_override = getattr(args, "consolidate_jaccard", None)
+    l_override = getattr(args, "consolidate_levenshtein", None)
+    df_override = getattr(args, "consolidate_max_shingle_df", None)
+
+    store = _open_store()
+    try:
+        report = consolidation_audit(
+            store,
+            jaccard_min=(
+                float(j_override)
+                if j_override is not None
+                else config.jaccard_min
+            ),
+            levenshtein_min=(
+                float(l_override)
+                if l_override is not None
+                else config.levenshtein_min
+            ),
+            max_shingle_df=(
+                int(df_override)
+                if df_override is not None
+                else DEFAULT_MAX_SHINGLE_DF
+            ),
+        )
+    except ValueError as exc:
+        print(f"aelf doctor consolidate: {exc}", file=sys.stderr)
+        return 1
+    finally:
+        store.close()
+
+    print(format_consolidation_report(report), file=out)  # type: ignore[arg-type]
+    return 0
+
+
 def _cmd_doctor_relationships(args: argparse.Namespace, out: object) -> int:
     """Run the v2.0 semantic-relationship audit (#201).
 
@@ -8820,6 +8879,60 @@ def _positive_int(s: str) -> int:
             "or [dedup] in .aelfrice.toml."
         ),
     )
+    p_doctor.add_argument(
+        "--consolidate",
+        dest="consolidate",
+        action="store_true",
+        default=False,
+        help=(
+            "cluster near-duplicate beliefs and report what a "
+            "contraction would retire (#1312). Read-only: no edges are "
+            "inserted and no belief is retired. Bypasses the "
+            "hooks/graph checks. Runs silently for a minute or more on a "
+            "large store — the medoid phase dominates and costs "
+            "O(k^2 * L^2) in the largest cluster's member count k and "
+            "belief length L, so a store with one big cluster of long "
+            "near-duplicates is far slower than its belief count "
+            "suggests. Tune via --consolidate-jaccard / "
+            "--consolidate-levenshtein / --consolidate-max-shingle-df."
+        ),
+    )
+    p_doctor.add_argument(
+        "--consolidate-jaccard",
+        dest="consolidate_jaccard",
+        type=float,
+        default=None,
+        metavar="F",
+        help=(
+            "with --consolidate: override the Jaccard threshold "
+            "(0.0-1.0). Default: [dedup] jaccard_min in .aelfrice.toml "
+            "> 0.8."
+        ),
+    )
+    p_doctor.add_argument(
+        "--consolidate-levenshtein",
+        dest="consolidate_levenshtein",
+        type=float,
+        default=None,
+        metavar="F",
+        help=(
+            "with --consolidate: override the Levenshtein-ratio "
+            "threshold (0.0-1.0). Default: [dedup] levenshtein_min in "
+            ".aelfrice.toml > 0.85."
+        ),
+    )
+    p_doctor.add_argument(
+        "--consolidate-max-shingle-df",
+        dest="consolidate_max_shingle_df",
+        type=int,
+        default=None,
+        metavar="N",
+        help=(
+            "with --consolidate: skip 4-grams appearing in more than N "
+            "beliefs when blocking candidate pairs (default 32). "
+            "Skipped shingles are reported, never dropped silently."
+        ),
+    )
     p_doctor.add_argument(
         "--dedup-jaccard",
         dest="dedup_jaccard",