diff --git a/docs/feature-intentional-clustering.md b/docs/feature-intentional-clustering.md new file mode 100644 index 000000000..9724ac781 --- /dev/null +++ b/docs/feature-intentional-clustering.md @@ -0,0 +1,301 @@ +# Feature spec: Intentional clustering (#436) + +**Status:** spec, no implementation +**Issue:** #436 +**Recovery-inventory line:** [`docs/ROADMAP.md`](ROADMAP.md) — *"Intentional clustering | v2.0.0"* +**Substrate prereqs:** edge graph (foundation), `dedup.DuplicateCluster` union-find pattern (`src/aelfrice/dedup.py:155-185`, shipped #197), heat kernel authority (#150, shipped v1.7.0), BFS multi-hop (#143, shipped v1.3.0) + +--- + +## Purpose + +When a query needs more than one belief to answer (e.g. *"how do we deploy + what are the prerequisites?"*), the existing rank+pack pipeline can return K beliefs that all sit in the same graph neighbourhood — the highest BM25F + heat-kernel scores cluster around one topic, and a complementary cluster covering the second half of the query never makes the cut. Intentional clustering is a **retrieval-time pass** that biases the top-K output toward **distinct graph-connected clusters**, paying a small per-belief score cost in exchange for multi-fact coverage. + +This is **not** the same mechanism as the heat kernel (#150). Heat kernel propagates authority *along* the graph from BM25 seeds — beliefs near a hot belief get a score boost. Clustering operates on the **already-ranked** candidate set: it changes which beliefs survive the pack, not their individual scores. + +--- + +## Contract + +```python +from aelfrice.clustering import RetrievalCluster, cluster_candidates, pack_with_clusters + +@dataclass(frozen=True) +class RetrievalCluster: + cluster_id: int # zero-indexed, dense + member_ids: tuple[str, ...] # belief ids, ordered by descending rank score + representative_id: str # member_ids[0] — the highest-ranked member + seed_score: float # representative's pre-clustering rank score + +# Step 1: identify clusters in the ranked candidate pool. +def cluster_candidates( + candidates: list[Belief], + candidate_scores: dict[str, float], + *, + edges: Iterable[Edge], + edge_weight_floor: float = DEFAULT_CLUSTER_EDGE_FLOOR, +) -> list[RetrievalCluster]: ... + +# Step 2: cluster-diverse pack at fixed token_budget. +def pack_with_clusters( + clusters: list[RetrievalCluster], + *, + token_budget: int, + cluster_diversity_target: int = DEFAULT_CLUSTER_DIVERSITY_TARGET, + fallback_to_score: bool = True, +) -> list[Belief]: ... +``` + +The two halves are split so the union-find pass is testable in isolation. Production retrieval calls them in sequence: + +``` +candidates = lane_fan_out + rank # existing behaviour +clusters = cluster_candidates(candidates, candidate_scores, edges=...) +out = pack_with_clusters(clusters, token_budget=token_budget, ...) +``` + +`out` replaces the existing pack-loop output (`retrieval.py:1048-1085, :1197-1232`) when the flag is on. + +--- + +## Cluster definition + +**Graph-connected components within the post-rank candidate pool, scoped to the edge subgraph induced by the candidates.** + +Concretely: take the top-N ranked candidates (where N is the existing pack-loop's seed pool size). Build a subgraph where vertices are the candidate belief IDs and edges are the elements of `store.edges_for_beliefs(candidate_ids)` whose `weight ≥ edge_weight_floor`. Run union-find over the edge list (the same primitive `dedup.py:248` calls path-compressed union-find on). Each connected component is one `RetrievalCluster`; isolated candidates are singletons. + +### Why graph-connected, not topic-coherent or co-occurrence-derived + +Three options were on the table per the issue acceptance: + +| Option | Reasoning | Decision | +|---|---|---| +| **graph-connected** | Reuses the existing edge graph (the union of `SUPPORTS`, `CONTRADICTS`, `RELATES_TO`, `CITES`, `DERIVED_FROM` edges plus their tuned weights at `models.py:55-66`). No new substrate. Deterministic. | **chosen** | +| topic-coherent | Requires an embedding model or a topic classifier. The project's posture is no embeddings in retrieval — feedback memory `feedback_avoid_embeddings_nondeterminism`. Out. | rejected | +| co-occurrence-derived | "Belief A and belief B were both ranked in the top-K for query Q in the past" needs a query-event log indexed by belief co-rank. The `ingest_log` doesn't index that, and the `rebuild_logs` (#288) are not yet a queryable derived table. Could ship in v2.x once #288 grows a queryable view. | deferred | + +The graph-connected definition is the cheapest, most deterministic, and uses substrate already on `main`. + +### Edge-weight floor + +`DEFAULT_CLUSTER_EDGE_FLOOR = 0.4` — picked to include `CITES` (`EDGE_VALENCE[EDGE_CITES] = 0.5` per `models.py:66`) but exclude `RELATES_TO` (`0.3` per `bfs_multihop.md` § Edge weights). Beliefs that only "relate to" each other are too weak a signal to be considered the same cluster; beliefs that cite each other are. + +The floor is a tunable. Per-store override via `[retrieval] cluster_edge_weight_floor = 0.4` in `.aelfrice.toml`. Bench-evidence is the gate for changing the default. + +--- + +## Pack algorithm + +`pack_with_clusters` replaces the existing tail-trim with a **diversity-aware greedy fill**: + +``` +out: list[Belief] = [] +used_tokens: int = 0 +covered_clusters: set[int] = set() + +# Stage 1: cluster representatives, in descending seed_score. +for cluster in sorted(clusters, key=lambda c: -c.seed_score): + if len(covered_clusters) >= cluster_diversity_target: + break + rep = cluster.member_ids[0] # representative + cost = _belief_tokens(rep) + if used_tokens + cost > token_budget: + if fallback_to_score: + break + continue + out.append(rep) + used_tokens += cost + covered_clusters.add(cluster.cluster_id) + +# Stage 2: remaining budget — fill from the score-ranked tail. +score_ranked_remaining = [b for b in flattened_score_order(clusters) + if b.id not in {x.id for x in out}] +for b in score_ranked_remaining: + cost = _belief_tokens(b) + if used_tokens + cost > token_budget: + break + out.append(b) + used_tokens += cost + +return out +``` + +- **Stage 1** guarantees the top-K hits at least `cluster_diversity_target` distinct clusters (or as many as exist), one representative per cluster, paid for at the budget cost of the representatives. +- **Stage 2** fills the remaining budget by score, skipping already-included beliefs. This recovers single-fact behaviour on queries where the candidate pool has fewer clusters than the diversity target — single-fact recall is preserved by acceptance #3. +- **`fallback_to_score=True`** (default) — when a cluster representative does not fit in the remaining budget, abandon Stage 1 and let Stage 2 finish the pack from the score-ranked tail. This keeps the pack non-degenerate on tight budgets. Setting `False` causes the pack to skip-but-continue Stage 1, which is the strict-diversity mode for benchmarks; default keeps backward-compatible recall behaviour on single-cluster queries. + +`DEFAULT_CLUSTER_DIVERSITY_TARGET = 3` — ships as the default. Three clusters covers most multi-fact queries without crowding out the score-ranked tail. Tunable via `[retrieval] cluster_diversity_target` in `.aelfrice.toml`. + +--- + +## Where clustering sits + +**Post-rank, pre-pack.** After lane fan-out + score composition, before the budget pack: + +``` +query + -> lane fan-out (BM25F + heat kernel + HRR-structural + BFS) + -> score composition + rank + -> [cluster_candidates → pack_with_clusters] if use_intentional_clustering + -> compose +``` + +When the flag is OFF, the existing pack loop runs unchanged. When ON, both `cluster_candidates` and `pack_with_clusters` run; `cluster_candidates` is a single union-find pass over the candidate edge subgraph, `pack_with_clusters` is the diversity-aware greedy fill above. + +### Configuration + +`use_intentional_clustering` follows the convention at `retrieval.py:118-131`: + +1. `retrieve(..., use_intentional_clustering=True)` kwarg. +2. `AELFRICE_INTENTIONAL_CLUSTERING=1` env var. +3. `[retrieval] use_intentional_clustering = true` in `.aelfrice.toml`. +4. Default OFF at v2.0.0 until bench-gate clears. + +Two adjacent knobs (also TOML-only, no env var): + +- `[retrieval] cluster_edge_weight_floor` — default `0.4`. +- `[retrieval] cluster_diversity_target` — default `3`. + +--- + +## Latency budget + +Issue acceptance #4: *"Cluster pass runs inside the retrieve() budget; no separate query."* The cost decomposes into: + +- **One `store.edges_for_beliefs(candidate_ids)` call.** Indexed lookup; existing pattern from BFS multi-hop. Cost is dominated by the `IN (...)` clause; at typical candidate-pool size (≤200) this is sub-ms in SQLite-backed numpy. +- **One union-find pass.** Path-compressed union-by-size, the same primitive `dedup.py:248` uses. O(α(N)·E) where α is the inverse Ackermann. At N=200 candidates and E ≤ 1000 inter-candidate edges, this is microseconds. +- **One score-ranked tail iteration.** Same as the existing pack loop, no algorithmic change. + +Total expected overhead at N=200, E=500: **<1 ms**, well below the existing per-call budget (~7-8 ms heat kernel at N=50k per `graph_spectral.py:227`). The latency-budget acceptance is met without a benchmark — the algorithmic profile is bounded. + +--- + +## Reconciliation + +### vs. #150 (heat kernel) + +Heat kernel **changes scores** by propagating authority along the graph from BM25 seeds. Beliefs near hot beliefs get score-boosted. Clustering **leaves scores untouched** but biases the **selection** of which K of the score-ranked candidates survive the pack. + +The two compose. Heat kernel's input is the BM25 lane; its output is a per-belief authority score that participates in the rank. Clustering's input is the post-rank candidate pool; its output is the packed top-K. Different stages, different inputs, different outputs. + +### vs. BFS multi-hop (#143) + +BFS multi-hop **expands** the candidate pool: starting from a seed, walk N hops to surface beliefs that BM25 didn't find. Clustering **prunes** the candidate pool to a diverse subset. BFS adds candidates; clustering chooses among them. + +When both are on (`bfs_enabled=True` AND `use_intentional_clustering=True`), BFS-discovered beliefs become candidates that the clusterer then groups. The two compose with no shared code path. + +### vs. #197 (dedup) `DuplicateCluster` + +`dedup.DuplicateCluster` (`dedup.py:155-185`) and this spec's `RetrievalCluster` are **different relations**: + +| | `DuplicateCluster` (#197) | `RetrievalCluster` (#436) | +|---|---|---| +| input | Jaccard-prefiltered duplicate **pairs** | post-rank candidate edges | +| edge type | implicit (duplicate-pair predicate) | explicit (`SUPPORTS`/`CITES`/etc., weight ≥ floor) | +| consumer | dedup audit / SUPERSEDES write-path | retrieval pack | +| timing | offline audit | retrieval-time | + +Both use union-find. The implementation PR may share the path-compressed union-find primitive — that is a refactor opportunity, not a requirement. + +### vs. type-aware compression (#434) + +Both are pre-pack transforms in the same stage. The order is: + +``` +rank → cluster_candidates → compress → pack_with_clusters +``` + +Compression reduces per-belief token cost; clustering biases selection toward cluster-diverse top-K. They compose: a `transient`-class belief in cluster X gets stub-compressed before being considered a representative; if its stub fits the budget, the cluster gets covered cheaply. + +### vs. doc linker (#435) + +Doc linker projects metadata onto returned beliefs (anchors). It does not change which beliefs are returned; clustering does. They sit at orthogonal stages: clustering at pack-time, linker at output-projection time. They compose with no interaction. + +--- + +## Acceptance + +### A1 — multi-fact corpus (issue acceptance #2) + +A new labeled `multi_fact` corpus lives under `tests/corpus/v2_0/multi_fact/`. Each row encodes: + +```jsonl +{"query": "...", + "expected_belief_ids": ["b/...", "b/..."], + "expected_clusters": [["b/..."], ["b/..."]], + "n_clusters_required": 2, + "tag": "complementary|conjunctive|sequential"} +``` + +`expected_clusters` partitions `expected_belief_ids` into the cluster groupings the labeller saw — this lets the bench distinguish "found both clusters" from "found two beliefs from the same cluster." Public CI runs with `AELFRICE_CORPUS_ROOT` unset and skips via the autouse `bench_gated` marker; labeled content lives in the lab repo only, per the published corpus policy. + +### A2 — multi-fact recall uplift (issue acceptance #3a) + +On the `multi_fact` fixture: + +``` +recall@k(use_intentional_clustering=ON) > recall@k(use_intentional_clustering=OFF) +cluster_coverage@k(ON) > cluster_coverage@k(OFF) +``` + +`cluster_coverage@k` is a new metric: number of `expected_clusters` represented in the top-K, divided by `n_clusters_required`. Both inequalities must hold for ship; the second is the multi-fact-coherence claim, the first is the recall-doesn't-degrade backstop. + +Threshold: **strictly positive uplift** on both. Magnitudes are bench-evidence-derived, not pre-set. + +### A3 — single-fact non-regression (issue acceptance #3b) + +On the existing `retrieve_uplift` fixture (the v0.1 single-fact fixture #154 used): + +``` +NDCG@k(use_intentional_clustering=ON, BM25F=ON) ≥ NDCG@k(use_intentional_clustering=OFF, BM25F=ON) - 0.005 +``` + +Tolerance band matches the BM25F bench-gate band at #154 — half-point of NDCG noise floor. + +### A4 — latency (issue acceptance #4) + +A microbench under `tests/bench_gate/` measures `cluster_candidates + pack_with_clusters` wall-time at N ∈ {50, 100, 200} candidates. Assertion: **<5 ms p99** at N=200. The latency-budget claim under §"Latency budget" is the analytical bound; the microbench is the empirical confirmation. + +### A5 — composition tracker + +The #154 composition tracker doc gains a row for `use_intentional_clustering`: input shape, output shape, where it sits, bench verdict. Like the doc linker (#435) and type-aware compression (#434), this is **not a lane** — it is a pack-stage selection transform. + +--- + +## Bench-gate / ship-or-defer policy + +`needs-spec` → `bench-gated` once this spec lands. Implementation is the next gate. **The implementation PR ships only on positive bench evidence per A2 and non-regression per A3.** A mechanically-correct implementation that fails either ships merged-but-default-OFF or gets reverted; it does not ship default-ON without a benchmark cut. + +A2 is the ship gate; A3 is the don't-break-existing-things backstop; A4 is a latency assertion that should pass by construction. + +--- + +## Out of scope at v2.0.0 + +- **Topic-coherent clustering** (embedding-derived). No embeddings in retrieval; rejected. +- **Co-occurrence-derived clustering** (history-derived). Requires a queryable rebuild-logs view (#288 follow-up). Deferred to v2.x. +- **Cluster-aware ranking** (per-cluster score adjustment). Out of scope; this spec changes selection, not ranking. Per-cluster ranking is a different spec. +- **Streaming / incremental clustering** (online union-find as candidates arrive). The pack runs over a finished candidate list. Streaming is a future concern, not a v2.0 ship gate. +- **Cluster surface in `RetrievalResult`.** Returning the cluster assignment to consumers is not in this spec — only the packed beliefs are returned. If a consumer needs to know "which cluster did this belief come from," that is a follow-up surface (additional `cluster_assignments` parallel list, the way doc-anchors lands at #435). + +--- + +## Implementation prereqs + +- `src/aelfrice/dedup.py:248` — path-compressed union-find primitive to reuse or duplicate. +- `src/aelfrice/models.py:55-66` — `EDGE_VALENCE` weights informing the default `cluster_edge_weight_floor = 0.4`. +- `src/aelfrice/store.py` — needs `edges_for_beliefs(candidate_ids)` (or equivalent batched edge fetch). If the method does not yet exist by name, the impl PR adds it; the existing `edges_from(belief_id)` is the unbatched precedent. +- `src/aelfrice/retrieval.py:118-131` — flag-resolution convention. +- `src/aelfrice/retrieval.py:1048-1085, :1197-1232` — pack loops to replace. +- `tests/corpus/v2_0/`, `tests/bench_gate/` — corpus + harness. + +All substrate is on `main` as of `e646383`. No new dependencies. No schema changes. + +--- + +## Open questions for review + +1. **Edge subgraph scope.** `cluster_candidates` builds the subgraph from the candidate pool's outgoing edges. Should it also include incoming edges from non-candidate beliefs (e.g. a candidate's citer, even if the citer isn't a candidate)? Spec says no — keep the subgraph candidate-induced. The candidate pool is the universe; non-candidates are out of consideration. Confirm at impl-PR review. +2. **Singleton handling.** A candidate with no in-pool edges is a singleton cluster of size 1. The Stage 1 loop visits it like any other cluster — it gets a representative (itself), it consumes one diversity slot. Is that correct, or should singletons be Stage-2-only? Spec keeps singletons in Stage 1 because demoting them would penalise queries whose hits are intentionally graph-isolated (e.g. lock-asserted policy beliefs). Confirm at impl-PR review. +3. **Diversity-target tuning.** Default `3`. If a typical multi-fact query in the wild needs 4-5 clusters, the default under-shoots. Tune from bench evidence, not pre-set. +4. **Locked beliefs and clustering.** L0 (locked) beliefs are never trimmed (`retrieval.py:950`). The pack-with-clusters output should pre-include locked beliefs ahead of Stage 1, exactly like the existing pack does (`retrieval.py:1015`). The spec says so implicitly via the phrase "replaces the existing pack-loop output"; the impl PR makes the L0-skip explicit at the top of the algorithm.