Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 122 additions & 0 deletions docs/design/feature-edge-type-expansion.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Edge-type expansion: SUPPORTS + SUPERSEDES at ingest (#999)

Status: spec (needs-spec → spec-ready)
Issue: #999
Depends on: #988 (CONTRADICTS substrate), #201 (detector ratification), #197 (dedup bench gate)
Blocks: #1001 (`+HRR-expand` ablation arm)

## Problem

The deterministic HRR vocabulary-bridge expansion lane (`use_hrr_expand`,
#981/PR #997) finds semantic neighbors by traversing typed edges. The ingest
substrate built in #988 emits **CONTRADICTS only**
(`relationship_detector.py:905`), so the lane traverses a graph missing every
other edge class. The #981 ceiling measurement (2,098 edges on LoCoMo10, all
CONTRADICTS) is the binding constraint, not the lane logic. Until the substrate
carries more edge types, the `#1001` ablation measures a starved graph and a
flat F1 delta would be **misread** as "the lane does not help."

This spec adds the two highest-value semantic edge types for a conversational
corpus — **SUPPORTS** and **SUPERSEDES** — to the ingest substrate.
Code-edge types (CALLS / CITES / TESTS / IMPLEMENTS) are out of scope: they
are source-graph relations with no deterministic lexical signal in
conversational belief text.

## Key finding: both writers already exist, deferred

Neither edge type needs a new detector. The detection already exists; only the
write-path is gated off.

- **SUPPORTS** — `relationship_detector.analyze()` already returns a `REFINES`
verdict for pairs with residual-content overlap above the relatedness floor
**and** agreeing modality (`relationship_detector.py:335-340`). REFINES is
counted in audits (`n_refines`) but never written: `write_semantic_edges`
is `CONTRADICTS only` by design (#988). `EDGE_SUPPORTS` is a defined model
type (`models.py:32`, valence `+1.0`).
- **SUPERSEDES** — `dedup.dedup_audit()` already clusters near-duplicate
beliefs (union-find) and documents the write-path: "will use the *oldest*
member as the SUPERSEDES target" (`dedup.py:160`), "deferred behind the #197
bench gate" (`dedup.py:329`). `EDGE_SUPERSEDES` is a defined model type
(`models.py:35`, valence `0.0`).

So #999 = **activate two already-designed, deferred write-paths**, gated
default-off under the existing `AELFRICE_AUTO_RELATIONSHIPS` flag. This mirrors
exactly how #988 activated the #201-ratified CONTRADICTS writer default-off,
and it keeps the #605/#897 determinism-and-narrow-surface decision intact: no
LLM, no embedding, no new default-on behavior.

## Design

### SUPPORTS (relationship_detector.py)

Add `write_supports_edges(store, *, jaccard_min, residual_overlap_min,
max_candidate_pairs, max_edges_per_belief) -> SemanticEdgeWriteReport`,
structurally identical to `write_semantic_edges` but acting on the `REFINES`
slice of the audit instead of the `CONTRADICTS` slice.

- Candidate set: `report.pairs` where `label == LABEL_REFINES`.
- REFINES carries no confidence gradient (score is fixed `0.0`,
`relationship_detector.py:337`); the residual-overlap floor already cleared
in `analyze()` is the relatedness gate. So **all** REFINES pairs are
eligible — no `confidence_min` partition (unlike CONTRADICTS, which splits
high/sub-confidence between `write_semantic_edges` and
`write_potentially_stale_edges`).
- Edge is **symmetric** (mutual support): canonicalize `src = min(id)`,
`dst = max(id)` — same as the CONTRADICTS writer.
- **Idempotent**: skip if `(src, dst, SUPPORTS)` exists.
- **Write-gated**: reuse `max_edges_per_belief` (Exp-48 coverage-dilution
guard); pre-existing edges count toward the per-belief budget across re-runs.
- **Deterministic**: pairs processed in the audit's `(belief_a_id,
belief_b_id)` order, so which edges survive the cap is deterministic.

### SUPERSEDES (dedup.py)

Add `write_supersedes_edges(store, *, jaccard_min, levenshtein_min,
max_candidate_pairs, max_edges_per_belief) -> SupersedesWriteReport`, acting on
`dedup_audit()` clusters.

- For each cluster, the **oldest** member (min `created_at`, tie-break min
`id`) is the SUPERSEDES *target*. Every other member emits one edge
`src = member` → `dst = oldest` (directional: the newer belief supersedes the
older).
- **Idempotent**: skip if `(member, oldest, SUPERSEDES)` exists.
- **Write-gated**: `max_edges_per_belief`, same semantics.
- **Deterministic**: members are already lexicographically sorted in
`DuplicateCluster.member_ids`; clusters sorted by `representative_id`.
- Direction rationale: SUPERSEDES is the one asymmetric type here. Newer→older
matches the lane's intent (a query hitting the current belief should reach
the ones it replaced, and vice-versa, via the directed edge the lane probes).

### Ingest wiring (ingest.py)

The CONTRADICTS writer is already invoked behind
`is_auto_relationship_detection_enabled()` (ingest.py:307-311). Add the two new
writers in the same guarded block, same order every run:
`write_semantic_edges` → `write_supports_edges` → `write_supersedes_edges`.
When the flag is off, ingest stays byte-identical to today.

## Determinism & test plan

Mirror the #988 CONTRADICTS tests for each new writer:

1. Flag-off no-op: ingest writes zero SUPPORTS/SUPERSEDES edges.
2. Byte-equal edge table across two runs on the same store snapshot
(tie-break by `(src, dst)` ASC).
3. Idempotency: second `write_*` run writes 0 new edges, skips N existing.
4. Write-gate: a belief at `max_edges_per_belief` drops further pairs
deterministically.
5. SUPPORTS: a REFINES pair produces exactly one symmetric SUPPORTS edge; a
CONTRADICTS pair produces none.
6. SUPERSEDES: a 3-member duplicate cluster produces 2 edges both pointing at
the oldest member; direction is newer→oldest.
7. No regression: existing CONTRADICTS / POTENTIALLY_STALE writers unchanged.

## Out of scope

- CALLS / CITES / TESTS / IMPLEMENTS edge types (no deterministic conversational
signal).
- Flipping `AELFRICE_AUTO_RELATIONSHIPS` to default-on — that reverses #605 and
is explicitly routed elsewhere.
- The incremental per-turn detection rewrite — tracked separately (#1000); this
spec keeps the existing full-audit candidate-pair path.
- Running the `+HRR-expand` ablation — #1001, blocked on this landing.
120 changes: 120 additions & 0 deletions src/aelfrice/dedup.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,11 @@
DEFAULT_JACCARD_MIN: Final[float] = 0.8
DEFAULT_LEVENSHTEIN_MIN: Final[float] = 0.85
DEFAULT_MAX_CANDIDATE_PAIRS: Final[int] = 5000
# Write-gate for the #999 SUPERSEDES edge writer: per-belief cap on
# auto-written edges (the Exp-48 coverage-dilution guard, mirroring the
# relationship_detector default). Defined locally — dedup must not import
# relationship_detector (that module imports this one at load time).
DEFAULT_MAX_EDGES_PER_BELIEF: Final[int] = 8

# --- Similarity primitives ---------------------------------------------

Expand Down Expand Up @@ -388,6 +393,121 @@ def dedup_audit(
)


# --- SUPERSEDES edge writer (#999 — the #197-deferred cluster hook) ------


@dataclass(frozen=True)
class SupersedesWriteReport:
"""Summary of one ``write_supersedes_edges`` run.

``n_clusters``
Near-duplicate clusters returned by ``dedup_audit`` — the
candidate set this writer acts on.

``n_edges_written``
Member→oldest edges newly written this run.

``n_edges_skipped_existing``
Edges whose triple already existed (idempotency guard fired).

``n_edges_skipped_gated``
Edges dropped because an endpoint hit ``max_edges_per_belief``.
"""

n_clusters: int
n_edges_written: int
n_edges_skipped_existing: int
n_edges_skipped_gated: int


def write_supersedes_edges(
store: MemoryStore,
*,
jaccard_min: float = DEFAULT_JACCARD_MIN,
levenshtein_min: float = DEFAULT_LEVENSHTEIN_MIN,
max_candidate_pairs: int = DEFAULT_MAX_CANDIDATE_PAIRS,
max_edges_per_belief: int = DEFAULT_MAX_EDGES_PER_BELIEF,
) -> SupersedesWriteReport:
"""Emit SUPERSEDES edges within near-duplicate clusters (#999).

Activates the write-path ``dedup_audit`` documented but deferred
behind the #197 bench gate: within each duplicate cluster, the
**oldest** member (min ``created_at``, tie-break min ``id``) is the
SUPERSEDES *target*; every other member emits one directional edge
``src = member`` → ``dst = oldest`` (the newer belief supersedes the
older one it duplicates).

Wired into ingest only behind the default-off ``auto_detect`` flag
(``relationship_detector.is_auto_relationship_detection_enabled``); a
fresh install writes no edges and the #605/#897 scope decision stands.

**Idempotent.** Each ``(member, oldest, SUPERSEDES)`` triple is checked
before insert and skipped if present.

**Write-gated.** ``max_edges_per_belief`` caps how many SUPERSEDES
edges any one belief accretes; pre-existing edges count toward the
budget for a hard per-belief bound across re-runs.

**Deterministic.** Clusters are sorted by ``representative_id`` and
members are lexicographically sorted, so the oldest-member resolution
and the write-gate survivors are stable across runs.

Stdlib-only: no LLM, no embedding. Delegates clustering to
``dedup_audit``.
"""
from aelfrice.models import Edge, EDGE_SUPERSEDES

report = dedup_audit(
store,
jaccard_min=jaccard_min,
levenshtein_min=levenshtein_min,
max_candidate_pairs=max_candidate_pairs,
)

edges_per_belief: dict[str, int] = {}
n_written = 0
n_skipped_existing = 0
n_gated = 0
for cluster in report.clusters:
members = [
b
for mid in cluster.member_ids
if (b := store.get_belief(mid)) is not None
]
if len(members) < 2:
continue
# Oldest member is the SUPERSEDES target (dedup.py cluster note).
oldest = min(members, key=lambda b: (b.created_at, b.id))
for b in members:
if b.id == oldest.id:
continue
src_id, dst_id = b.id, oldest.id
if (
edges_per_belief.get(src_id, 0) >= max_edges_per_belief
or edges_per_belief.get(dst_id, 0) >= max_edges_per_belief
):
n_gated += 1
continue
if store.get_edge(src_id, dst_id, EDGE_SUPERSEDES) is not None:
n_skipped_existing += 1
edges_per_belief[src_id] = edges_per_belief.get(src_id, 0) + 1
edges_per_belief[dst_id] = edges_per_belief.get(dst_id, 0) + 1
continue
store.insert_edge(
Edge(src=src_id, dst=dst_id, type=EDGE_SUPERSEDES, weight=1.0)
)
n_written += 1
edges_per_belief[src_id] = edges_per_belief.get(src_id, 0) + 1
edges_per_belief[dst_id] = edges_per_belief.get(dst_id, 0) + 1

return SupersedesWriteReport(
n_clusters=len(report.clusters),
n_edges_written=n_written,
n_edges_skipped_existing=n_skipped_existing,
n_edges_skipped_gated=n_gated,
)


@dataclass(frozen=True)
class DedupConfig:
"""Resolved `[dedup]` section of `.aelfrice.toml`.
Expand Down
17 changes: 11 additions & 6 deletions src/aelfrice/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -296,20 +296,25 @@ def _ingest_turn_ids(
anchor_text=anchor,
))

# #988: optionally build the semantic-edge substrate at ingest. Writes
# CONTRADICTS edges across the store for this turn's new beliefs so the
# graph lanes (HRR-expand #981, BFS) are no longer inert on benchmark
# corpora. Default-OFF (is_auto_relationship_detection_enabled): when
# off, this branch is never entered and ingest is byte-identical to
# today. Gated on `inserted` so corroboration-only turns skip the audit.
# #988/#999: optionally build the semantic-edge substrate at ingest.
# Writes CONTRADICTS (#988), SUPPORTS (REFINES verdicts, #999), and
# SUPERSEDES (dedup clusters, #999) edges across the store so the graph
# lanes (HRR-expand #981, BFS) are no longer starved on a CONTRADICTS-only
# graph. Default-OFF (is_auto_relationship_detection_enabled): when off,
# this branch is never entered and ingest is byte-identical to today.
# Gated on `inserted` so corroboration-only turns skip the audit.
if inserted:
from aelfrice.dedup import write_supersedes_edges
from aelfrice.relationship_detector import (
is_auto_relationship_detection_enabled,
write_semantic_edges,
write_supports_edges,
)

if is_auto_relationship_detection_enabled():
write_semantic_edges(store)
write_supports_edges(store)
write_supersedes_edges(store)

return inserted

Expand Down
Loading
Loading