Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG/v4.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **`[relationship_detector]` was half-honoured at ingest: the flag took effect, the thresholds did not ([#1299](https://github.com/robotrocketscience/aelfrice/issues/1299)).** `ingest.py` resolved `auto_detect` from `.aelfrice.toml` and then called `write_semantic_edges(store, new_belief_ids=...)` with no threshold arguments, so `jaccard_min` / `confidence_min` / `max_candidate_pairs` reached the read-only `aelf doctor --relationships` audits and were silently ignored on the one path that actually mutates the graph. Adjacent keys in the same section, opposite reach, no trace on stderr — and the asymmetry ran the risky direction, with the audit tunable and the writer pinned at the module defaults. Ingest now threads the resolved config through. The three keys are parsed from the **same** `.aelfrice.toml` read that resolves `auto_detect` (new `resolve_ingest_relationship_config`), so the per-turn config-probe count is unchanged — measured 11 probes with no config file and 4 with one four directories up, before and after — rather than adding a second filesystem walk to a hot path ([#1289](https://github.com/robotrocketscience/aelfrice/issues/1289)/[#1298](https://github.com/robotrocketscience/aelfrice/issues/1298)). That measurement is env-unset; the env var is the other half of the precedence and it moves the count, so it is stated separately. `AELFRICE_AUTO_RELATIONSHIPS=0` decides the question without needing any threshold, and the flag-only resolver short-circuited on env before touching the filesystem — so that install paid **0** probes per turn and still does, the resolver returning before the walk rather than reading a config whose only consumer will not run. With the var set truthy the walk is real and new (0 -> 11 on a deep tree): the thresholds are then actually used, which is the point of the fix, so that one is a cost and not a regression. Precedence for `auto_detect` is unchanged (env > TOML > default-off), and default-off means a fresh install is byte-identical. `residual_overlap_min` and `max_edges_per_belief` still have no TOML key at all; `docs/user/CONFIG.md` now documents the section with per-key reach so which is which is readable.
- **PHILOSOPHY and the write-log memo claimed `edges` are a materialized projection of the log; they never have been ([#1283](https://github.com/robotrocketscience/aelfrice/issues/1283)).** All six `derive()` return paths emit `edges=[]`, so `ingest_log.derived_edge_ids` is NULL on **every** row (0 of 139,592 on the development store), and every real edge is written outside the log by `ingest.py`, `temporal_spine.py` and the relationship / contradiction detectors. The substrate claim therefore overreached for the graph: a replay from empty cannot reconstruct the L3 BFS graph, the temporal spine or the `CONTRADICTS` substrate, and `replay.py` reports edge divergence only in the bucket documented as *"never promoted into `has_drift`"*. Both documents now state the ratified contract (edges are **log-derived**, recompute keyed on `(created_at, ingest_log ULID)`, operator ruling 2026-08-01) separately from the shipped state, and name what is still missing. The key is the log's ULID rather than anything read off the belief table because the writer actually orders by `(created_at, rowid)` and `rowid` is implicit here — VACUUM may renumber it; measured, the ULID key reproduces **93.7%** of the live `TEMPORAL_NEXT` set against **7.4%** for a belief-table key (`benchmarks/spine_order_provenance.py`). Docs only — no code, no defaults, and the recompute itself is not built.
- **LIMITATIONS claimed a residual exposure-as-evidence path that has not existed since #1162 ([#1267](https://github.com/robotrocketscience/aelfrice/issues/1267)).** The sharp-edges entry said retrieval "still enqueues each surfaced belief" and that `aelf sweep-feedback` "applies a small alpha bump (default +0.05)", citing [#1091](https://github.com/robotrocketscience/aelfrice/issues/1091) as having only *flagged* the sweep for audit-only treatment. All three were stale: the sweeper has written nothing since [#1162](https://github.com/robotrocketscience/aelfrice/issues/1162) (it classifies what it *would* have applied and returns `mutated=False`), and the enqueue inside `retrieve()` is gated on `AELFRICE_IMPLICIT_FEEDBACK_ENQUEUE`, default off. A reader auditing where their posteriors come from was pointed at a mutation path that no longer fires. New `benchmarks/posterior_channel_audit.py` drives all three `apply_feedback` routes against a fresh store and fails non-zero if any default moves, so the entry cannot go stale silently again.
- **An explicit `token_budget=2400` was silently downgraded to 2000 when the entity index was off ([#1271](https://github.com/robotrocketscience/aelfrice/issues/1271)).** `retrieve_with_tiers` declared the budget as a `None` sentinel meaning "caller did not ask", resolved it to a concrete int, then reconstructed the intent downstream as `token_budget != DEFAULT_TOKEN_BUDGET` — a different predicate that agrees most of the time and is wrong for exactly the caller who passes the default on purpose. `2399` and `2401` were honoured; `2400` was not, so the behaviour was discontinuous at precisely the default. It invalidated a measurement on [#1269](https://github.com/robotrocketscience/aelfrice/issues/1269) before it was found, where a 16.7% budget cut on one arm was read as a lane-precedence effect. `resolve_token_budget_with_provenance` now returns `(budget, defaulted)` and `resolve_token_budget` delegates to it so the precedence cannot drift; env and TOML both count as expressing a preference, and the legacy path stays for callers who expressed none. `retrieve()` resolves the budget before delegating, so it passes the provenance down explicitly — without that thread every no-argument call would have been promoted from 2000 to 2400, breaking the v1.2 parity the branch exists for.
Expand Down
53 changes: 53 additions & 0 deletions docs/user/CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,24 @@ temporal_spine_budget = 32
# is byte-identical. AELFRICE_TEMPORAL_SPINE_WRITE env var overrides.
write_temporal_spine = true

[relationship_detector]
# #988 / #1299. Deterministic contradiction detector. `auto_detect` is
# default-OFF: when false, ingest writes no CONTRADICTS edges and the
# section only affects the read-only `aelf doctor --relationships` /
# `--detect-stale` audits. AELFRICE_AUTO_RELATIONSHIPS env var overrides
# `auto_detect`; the three thresholds have no env override.
auto_detect = false
# Minimum token Jaccard for a belief pair to enter the classifier.
# Default 0.4. Unrelated to `dedup`'s 0.8 — different module, different
# consumer.
jaccard_min = 0.4
# Minimum verdict score for a `contradicts` pair to be auto-emitted as
# an edge. Default 0.5. Sub-confidence pairs are the POTENTIALLY_STALE
# writer's domain.
confidence_min = 0.5
# Cap on candidate pairs scored per audit run. Default 5000.
max_candidate_pairs = 5000

[rebuilder]
# v3.0+ / #718 (PR #719). Selects the query-rewriting stack used by
# the context rebuilder. Default `"stack-r1-r3"` since v3.0; runs
Expand Down Expand Up @@ -708,6 +726,41 @@ chains (idempotent, `--dry-run` supported), and `aelf doctor` reports
spine presence + edge count. `AELFRICE_TEMPORAL_SPINE_WRITE` env var
overrides.

## `[relationship_detector]` (v4.x+)

The deterministic contradiction detector (#201 / #988). Two consumers read
this section and they are **not** equivalent, so each key is documented with
its reach:

* the **ingest write path** — runs on every ingested turn, inserts
`CONTRADICTS` edges, gated entirely on `auto_detect`;
* the **audit commands** — `aelf doctor --relationships` (read-only report)
and `aelf doctor --detect-stale` (writes `POTENTIALLY_STALE` edges), which
run on demand and ignore `auto_detect`.

| Key | Type | Default | Ingest write path | `aelf doctor` audits |
|---|---|---|---|---|
| `auto_detect` | bool | `false` | **honoured** — the on/off switch | not read (the flag gates ingest only) |
| `jaccard_min` | float `[0.0, 1.0]` | `0.4` | **honoured** (since [#1299](https://github.com/robotrocketscience/aelfrice/issues/1299); silently ignored before) | **honoured**, `--relationships-jaccard` overrides |
| `confidence_min` | float `[0.0, 1.0]` | `0.5` | **honoured** (since #1299; silently ignored before) | **honoured**, `--relationships-confidence` overrides |
| `max_candidate_pairs` | int `>= 1` | `5000` | **honoured** (since #1299; silently ignored before) | **honoured**, `--relationships-max-pairs` overrides |
| `residual_overlap_min` | — | `0.4` | **no TOML key** — not parsed anywhere | **no TOML key** |
| `max_edges_per_belief` | — | `8` | **no TOML key** by design (Exp-48 write-gate, caller kwarg only) | n/a (audits do not write `CONTRADICTS`) |

So the section still does not mean exactly one thing for every key: the last
two rows are module constants with no configuration surface at all. What
changed in #1299 is that the three keys that *do* parse now reach the path
that mutates the graph, instead of only the read-only audit.

`auto_detect` resolves env > TOML > default-off: `AELFRICE_AUTO_RELATIONSHIPS`
(`1`/`true`/`yes`/`on` vs `0`/`false`/`no`/`off`) wins over the file. The three
thresholds have no env override.

Default-off is load-bearing. A fresh install writes no semantic edges; turning
`auto_detect` on makes every ingested turn run an incremental contradiction
audit over the beliefs inserted that turn. Wrong-typed values fall back to the
default with a stderr trace and never raise.

## `[implicit_feedback]` (v1.6.0+)

Retrieval-exposure feedback: the queue that records which beliefs a
Expand Down
20 changes: 17 additions & 3 deletions src/aelfrice/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -325,14 +325,28 @@ def _ingest_turn_ids(
# touching at least one belief from `inserted` are evaluated, which is
# provably equivalent to a full-store audit for discovering new edges
# while avoiding the O(n²) re-scan of the whole store every turn.
# #1299: the thresholds come from the same `[relationship_detector]`
# section as the flag. Resolving both in one walk keeps the per-turn
# `.aelfrice.toml` probe count identical to the flag-only version
# (#1289/#1298 — this is a hot path; a second walk is not free).
if inserted:
from aelfrice.relationship_detector import (
is_auto_relationship_detection_enabled,
resolve_ingest_relationship_config,
write_semantic_edges,
)

if is_auto_relationship_detection_enabled():
write_semantic_edges(store, new_belief_ids=inserted)
auto_relationships, rel_config = (
resolve_ingest_relationship_config()
)
if auto_relationships:
write_semantic_edges(
store,
jaccard_min=rel_config.jaccard_min,
residual_overlap_min=rel_config.residual_overlap_min,
confidence_min=rel_config.confidence_min,
max_candidate_pairs=rel_config.max_candidate_pairs,
new_belief_ids=inserted,
)

# #1064: optionally chain this turn's new beliefs into the
# per-session temporal spine (TEMPORAL_NEXT, src = successor,
Expand Down
68 changes: 67 additions & 1 deletion src/aelfrice/relationship_detector.py
Original file line number Diff line number Diff line change
Expand Up @@ -497,11 +497,19 @@ def relationships_audit(

@dataclass(frozen=True)
class RelationshipDetectorConfig:
"""Resolved ``[relationship_detector]`` section of `.aelfrice.toml`."""
"""Resolved ``[relationship_detector]`` section of `.aelfrice.toml`.

``residual_overlap_min`` has **no TOML key** — it is a caller-supplied
knob that carries the module default, kept on the dataclass so every
consumer passes a uniform quadruple to the detector.
"""
jaccard_min: float = DEFAULT_JACCARD_MIN
residual_overlap_min: float = DEFAULT_RESIDUAL_OVERLAP_MIN
confidence_min: float = DEFAULT_CONFIDENCE_MIN
max_candidate_pairs: int = DEFAULT_MAX_CANDIDATE_PAIRS
# #1299: parsed here so the ingest path can resolve the flag and the
# thresholds in ONE `.aelfrice.toml` walk instead of two.
auto_detect: bool = False


def _load_unit_float(
Expand Down Expand Up @@ -594,10 +602,21 @@ def load_relationship_detector_config(
mp_resolved = DEFAULT_MAX_CANDIDATE_PAIRS
else:
mp_resolved = mp_obj
ad_obj: Any = section.get(AUTO_DETECT_KEY, False)
if isinstance(ad_obj, bool):
ad_resolved = ad_obj
else:
print(
f"aelfrice relationship_detector: ignoring [{SECTION}] "
f"{AUTO_DETECT_KEY} in {candidate} (expected bool)",
file=serr,
)
ad_resolved = False
return RelationshipDetectorConfig(
jaccard_min=jm,
confidence_min=cm,
max_candidate_pairs=mp_resolved,
auto_detect=ad_resolved,
)
if current.parent == current:
break
Expand Down Expand Up @@ -685,6 +704,52 @@ def is_auto_relationship_detection_enabled(
return False


def resolve_ingest_relationship_config(
start: Path | None = None,
) -> tuple[bool, RelationshipDetectorConfig]:
"""Resolve the ingest-time flag **and** thresholds in one config walk.

Returns ``(enabled, config)``. Before #1299 the ingest path read only
``auto_detect`` and then called ``write_semantic_edges`` with no
threshold arguments, so ``[relationship_detector] jaccard_min`` /
``confidence_min`` / ``max_candidate_pairs`` were honoured by the
``aelf doctor`` audit commands and silently ignored by the one path
that actually mutates the graph.

Threading the config must not cost a second filesystem walk on a hot
path (#1289/#1298): ``load_relationship_detector_config`` already
walks to the nearest `.aelfrice.toml`, so ``auto_detect`` is parsed
from that same read and handed to
``is_auto_relationship_detection_enabled`` as its ``explicit``
argument. A non-``None`` ``explicit`` is decisive before the
resolver's own TOML step, so exactly one walk happens per call and
the documented precedence (env > TOML > default-off) is unchanged.

One walk is still one walk too many when the env var has already
settled the question in the *off* direction: the thresholds are read
only to be passed to a writer that will not run. The flag-only
resolver this replaced short-circuited on env before touching the
filesystem, so an ``AELFRICE_AUTO_RELATIONSHIPS=0`` install paid
**zero** probes per ingested turn; loading the config first would
have made that 11 on a deep tree. Env-off therefore returns before
the walk. Env-*on* still walks, because the thresholds are then
actually needed — that is the intended cost of #1299, not a
regression.

The returned config is meaningful only when ``enabled`` is True. On
the env-off short-circuit it is the module defaults rather than the
file's values; no caller reads it, and reading it would reintroduce
the walk this branch exists to avoid.
"""
if _env_auto_relationships_override() is False:
return False, RelationshipDetectorConfig()
config = load_relationship_detector_config(start)
enabled = is_auto_relationship_detection_enabled(
config.auto_detect, start=start,
)
return enabled, config


# --- Report formatter --------------------------------------------------


Expand Down Expand Up @@ -1068,6 +1133,7 @@ def write_semantic_edges(
"is_auto_relationship_detection_enabled",
"load_relationship_detector_config",
"relationships_audit",
"resolve_ingest_relationship_config",
"write_potentially_stale_edges",
"write_semantic_edges",
]
Loading
Loading