feat(dedup): audit-only aelf doctor dedup (#197 R1)

docs/LIMITATIONS.md

@@ -41,6 +41,7 @@ The default retrieval mode (recall, not audit) is correctly served by latest-ser
 - **Confidence drops below 0.5 do not auto-flag.** A belief whose posterior drifts under the prior is not surfaced as a warning at v1.x. The only automatic state change driven by negative evidence is locked-belief demotion-pressure (≥5 contradictions → auto-demote). To find drifting beliefs, query `aelf stats` directly.
 - **Jeffreys prior reads as 0.5.** A belief with no feedback reports posterior mean exactly `0.5`. That means "no evidence yet," not "coin-flip true."
 - **`aelf onboard` is non-incremental on duplicates.** Re-runs are idempotent; existing beliefs are not re-scored or refreshed.
+- **Near-duplicates from different ingest paths persist.** `INSERT OR IGNORE` on `(source, sentence)` content_hash dedupes exact matches but not paraphrases (e.g. "don't push to main" locked + "never push directly to main" scanned both surface in retrieval). v2.0 ships `aelf doctor --dedup` (#197) — read-only audit that lists Jaccard + Levenshtein-confirmed duplicate clusters; the write-path SUPERSEDES hook is bench-gated and deferred behind the corpus benchmark.
 - **No bulk operations.** No batch lock, no `delete <pattern>`, no merge.
 - **No edit.** A wrong belief is corrected by inserting a new one with a `SUPERSEDES` edge; the original stays.
 - **No graph viz.** Inspect with `sqlite3 "$(python -c 'from aelfrice.cli import db_path; print(db_path())')"`.

docs/dedup.md

@@ -0,0 +1,84 @@
+# Dedup — `aelf doctor --dedup`
+
+Audit-only near-duplicate detection over the belief store, shipped at v2.0 per [#197](https://github.com/robotrocketscience/aelfrice/issues/197).
+
+The detector pairs two cheap deterministic signals:
+
+- **Jaccard** over lowercase Unicode-word tokens — fast prefilter, no allocation per character pair.
+- **Levenshtein ratio** (`1 - edit_distance / max(len_a, len_b)`) — second-stage confirmation; guards against shared-vocabulary false positives that Jaccard alone would accept.
+
+Both thresholds must clear for a pair to count as a near-duplicate.
+
+## Usage
+
+```bash
+aelf doctor --dedup
+```
+
+Read-only: walks every belief pair, runs the prefilter, emits a clustered report. No edges are inserted, no beliefs are mutated.
+
+### Per-run flags
+
+```bash
+aelf doctor --dedup \
+  --dedup-jaccard 0.7 \
+  --dedup-levenshtein 0.9 \
+  --dedup-max-pairs 1000
+```
+
+Each `--dedup-*` flag overrides one knob for the current run only.
+
+### Project defaults via `.aelfrice.toml`
+
+```toml
+[dedup]
+jaccard_min = 0.8
+levenshtein_min = 0.85
+max_candidate_pairs = 5000
+```
+
+Walk-up resolution: the loader walks from cwd up through ancestor directories looking for `.aelfrice.toml`. Malformed values fall back to the module defaults with a stderr trace; the loader never raises.
+
+## Defaults
+
+| knob | default | source |
+| --- | --- | --- |
+| `jaccard_min` | 0.8 | research-line ratification |
+| `levenshtein_min` | 0.85 | research-line ratification |
+| `max_candidate_pairs` | 5000 | research-line ratification |
+
+The 0.8 Jaccard floor is intentionally strict — token-set divergence between e.g. "don't" / "do not" pushes that pair below the floor even though Levenshtein would clear. Lower the threshold via `--dedup-jaccard` for paraphrase-style detection at the cost of more false positives.
+
+## Output shape
+
+```
+aelf doctor dedup
+========================================
+Beliefs scanned         : 1483
+Candidate pairs visited : 1099303
+Duplicate pairs         : 12
+Duplicate clusters      : 4
+
+Clusters:
+  belief-abc-123  (3 members)
+    * belief-abc-123
+      belief-abc-456
+      belief-abc-789
+  ...
+
+Top duplicate pairs (jaccard, levenshtein):
+  belief-abc-123  ~  belief-abc-456  (j=0.923, l=0.971)
+  ...
+```
+
+`Candidate pairs visited` is the raw O(n²) count. `Duplicate pairs` is the post-Jaccard, post-Levenshtein survivor count. Clusters are the union-find collapse of those pairs into connected components; the `*` marks each cluster's deterministic representative (the lexicographically smallest member id).
+
+## What's not in this command
+
+The audit is **read-only by design**. The write-path SUPERSEDES hook — collapsing duplicates by inserting `SUPERSEDES` edges from older to newer at every `ingest_turn` / `onboard` / `apply_feedback` write — is the bench-gated R2 deferred behind the v2.0 corpus benchmark per #197 ratification. Until that lands, use this command to inspect candidate clusters and review them by hand.
+
+## Related
+
+- Spec memo: [`v2_dedup.md`](v2_dedup.md).
+- Issue: [#197](https://github.com/robotrocketscience/aelfrice/issues/197).
+- Scope cut: dedup is one of six bench-gated v2.0 modules; corpus contract at [#307](https://github.com/robotrocketscience/aelfrice/issues/307), bench-gate harness at [#319](https://github.com/robotrocketscience/aelfrice/issues/319).

src/aelfrice/cli.py

@@ -2135,6 +2135,8 @@ def _cmd_doctor(args: argparse.Namespace, out: object) -> int:
         return _cmd_doctor_promote_retention(args, out)
     if getattr(args, "replay", False):
         return _cmd_doctor_replay(args, out)
+    if getattr(args, "dedup", False):
+        return _cmd_doctor_dedup(args, out)
     scope = getattr(args, "scope", None)
     exit_code = 0
     if scope in (None, "hooks"):
@@ -2165,6 +2167,64 @@ def _cmd_doctor(args: argparse.Namespace, out: object) -> int:
     return exit_code
 
 
+def _cmd_doctor_dedup(args: argparse.Namespace, out: object) -> int:
+    """Run the v2.0 dedup audit (#197 R1).
+
+    Walks the store, finds near-duplicate belief pairs with Jaccard >=
+    `--dedup-jaccard` AND Levenshtein ratio >= `--dedup-levenshtein`,
+    and prints a clustered report. Read-only: no edges are inserted,
+    no beliefs are mutated. The write-path SUPERSEDES hook is the
+    bench-gated R2 deferred behind the corpus benchmark.
+
+    Exit 0 on success regardless of cluster count — clusters are
+    diagnostic, not failure conditions. Exit 1 only on store-open
+    errors.
+    """
+    from aelfrice.dedup import (
+        DedupConfig,
+        dedup_audit,
+        format_audit_report,
+        load_dedup_config,
+    )
+
+    config = load_dedup_config()
+    j_override = getattr(args, "dedup_jaccard", None)
+    l_override = getattr(args, "dedup_levenshtein", None)
+    mp_override = getattr(args, "dedup_max_pairs", None)
+    config = DedupConfig(
+        jaccard_min=(
+            float(j_override) if j_override is not None else config.jaccard_min
+        ),
+        levenshtein_min=(
+            float(l_override)
+            if l_override is not None
+            else config.levenshtein_min
+        ),
+        max_candidate_pairs=(
+            int(mp_override)
+            if mp_override is not None
+            else config.max_candidate_pairs
+        ),
+    )
+
+    store = _open_store()
+    try:
+        report = dedup_audit(
+            store,
+            jaccard_min=config.jaccard_min,
+            levenshtein_min=config.levenshtein_min,
+            max_candidate_pairs=config.max_candidate_pairs,
+        )
+    except ValueError as exc:
+        print(f"aelf doctor dedup: {exc}", file=sys.stderr)
+        return 1
+    finally:
+        store.close()
+
+    print(format_audit_report(report), file=out)  # type: ignore[arg-type]
+    return 0
+
+
 def _cmd_doctor_replay(args: argparse.Namespace, out: object) -> int:
     """Run the v2.x full-equality replay probe (#262).
 
@@ -2963,6 +3023,55 @@ def build_parser(*, show_advanced: bool = False) -> argparse.ArgumentParser:
             "(exists for forward compatibility)."
         ),
     )
+    p_doctor.add_argument(
+        "--dedup",
+        dest="dedup",
+        action="store_true",
+        default=False,
+        help=(
+            "find near-duplicate beliefs (Jaccard + Levenshtein gate) "
+            "and print clustered candidates (#197). Read-only: no edges "
+            "are inserted. Bypasses the hooks/graph checks. Tune via "
+            "--dedup-jaccard / --dedup-levenshtein / --dedup-max-pairs "
+            "or [dedup] in .aelfrice.toml."
+        ),
+    )
+    p_doctor.add_argument(
+        "--dedup-jaccard",
+        dest="dedup_jaccard",
+        type=float,
+        default=None,
+        metavar="F",
+        help=(
+            "with --dedup: override the Jaccard prefilter threshold "
+            "(0.0-1.0). Default: [dedup] jaccard_min in .aelfrice.toml > "
+            "0.8."
+        ),
+    )
+    p_doctor.add_argument(
+        "--dedup-levenshtein",
+        dest="dedup_levenshtein",
+        type=float,
+        default=None,
+        metavar="F",
+        help=(
+            "with --dedup: override the Levenshtein-ratio confirmation "
+            "threshold (0.0-1.0). Default: [dedup] levenshtein_min in "
+            ".aelfrice.toml > 0.85."
+        ),
+    )
+    p_doctor.add_argument(
+        "--dedup-max-pairs",
+        dest="dedup_max_pairs",
+        type=int,
+        default=None,
+        metavar="N",
+        help=(
+            "with --dedup: cap reported duplicate pairs after Jaccard "
+            "prefilter; deterministic truncation by (id_a, id_b). "
+            "Default: [dedup] max_candidate_pairs in .aelfrice.toml > 5000."
+        ),
+    )
     p_doctor.set_defaults(func=_cmd_doctor)
 
     p_sweep_feedback = sub.add_parser(