Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 133 additions & 0 deletions docs/value_compare.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Value-comparison contradiction gate (#422)

`aelfrice.value_compare` is the typed-slot relatedness gate that
replaces the residual-overlap floor in `relationship_detector` for
contradiction detection. Stdlib-only, deterministic, no embeddings.

## Why it exists

The R2 detector from #201 used residual-Jaccard token overlap to
decide if two beliefs were "about the same subject" before checking
modality / quantifier disagreement. On the labeled adversarial
corpus this gate caught 2 of 60 (`recall = 0.033`) — the failure
mode is paraphrase: real natural-language contradictions almost
never share enough surface tokens to clear the overlap floor.

The v3 gate sidesteps this by extracting **typed slots** from each
belief and firing `contradicts` when the two beliefs disagree on a
slot value, regardless of token overlap. Slot match is the
relatedness signal.

## What gets extracted

`extract_values(text)` returns a `ValueSlots` with two tuples:

**Numeric slots** — `NumericSlot(key, value)`. The `key` is the
alphabetic token immediately preceding a number (with optional
``=`` / ``:`` / ``is`` / ``of`` / ``to`` / ``equals`` separator).
``value`` is parsed as float. Examples that match:

| input | extracted |
|---|---|
| ``alpha = 0.5`` | ``(alpha, 0.5)`` |
| ``timeout: 30`` | ``(timeout, 30.0)`` |
| ``set retries to 3`` | ``(retries, 3.0)`` |
| ``max_depth=4 depth=2`` | ``(max_depth, 4.0)`` and ``(depth, 2.0)`` |

Filler keys (``is``, ``of``, ``the``, ``a``, …) are dropped — see
`_NUMERIC_KEY_DROP`. Unit-aware comparison is **out of scope** for
v3: the regex's greedy capture of trailing tokens introduced too
many false negatives (e.g. ``alpha = 0.5 prior`` vs ``alpha = 1.0
in config`` produced different "units" — ``prior`` vs ``in`` — and
silently skipped the conflict). If unit-aware comparison becomes
needed, file a separate issue with a curated unit vocabulary.

**Enum slots** — `EnumSlot(category, group_id, member)`. Members
come from `ENUM_VOCAB`, a curated taxonomy of 9 categories grouped
by mutual exclusion:

| category | groups |
|---|---|
| `execution_mode` | `{sync, synchronous}`, `{async, asynchronous}` |
| `default_state` | `{default-on, enabled}`, `{default-off, disabled}` |
| `storage_mode` | `{indexed}`, `{scan, full-scan, table-scan}` |
| `completeness` | `{full}`, `{incremental}`, `{partial}` |
| `strictness` | `{strict}`, `{lax, permissive}` |
| `necessity` | `{required}`, `{optional}` |
| `visibility` | `{public}`, `{private}` |
| `access_mode` | `{readonly, read-only}`, `{writable, read-write}` |
| `determinism` | `{deterministic}`, `{non-deterministic, nondeterministic, stochastic}` |

Members within a group are aliases — they do **not** conflict with
each other. `group_id` is the alphabetically-first member of the
group, used as a stable cross-belief identifier. Adding a category
extends the contradiction surface; the dict in
``src/aelfrice/value_compare.py`` is the single source of truth.

## When the comparator fires

`find_conflicts(slots_a, slots_b)` returns a tuple of `SlotConflict`:

- **Numeric conflict**: same key, values outside relative tolerance
(default 1%). The 0/0 case is silent (degenerate denominator).
- **Enum conflict**: same category, disjoint group_id sets. Aliases
collapse to one group, so `sync` vs `synchronous` is silent.

When the comparator returns multiple distinct conflicts on the same
pair, all of them are surfaced — the integration layer decides how
many it takes to fire.

## Integration

`relationship_detector.analyze(a, b, use_value_comparison=True)`
runs the gate **before** the residual-overlap floor. If any
conflict is found:

```
RelationshipVerdict(
label="contradicts",
score=1.0,
residual_overlap=0.0,
rationale="value_comparison:numeric=alpha",
)
```

Score is pinned at 1.0 so the auto-emit policy (#422 acceptance #3)
can use a single threshold for "slot fired" vs "modality-only
fired" verdicts.

`use_value_comparison=False` (the default) preserves v1 behaviour
byte-for-byte — the v3 path adds zero overhead until flipped.

## Determinism

Same `(text_a, text_b)` produces byte-identical slots and identical
conflicts across runs. No embeddings, no learned classifiers, no
random seeds. This is load-bearing for replay-equality (#262 / #403)
and bench-gate reproducibility (per the v2.0 README rationale).

## Bench gate

`tests/bench_gate/test_contradiction_v3.py` runs the v3 path against
the labeled adversarial `contradiction` corpus and asserts:

- recall on `contradicts` ≥ 0.5
- precision on `contradicts` ≥ 0.7

Skip-on-no-corpus on public CI. Below-floor blocks the v3 default-on
flip; the operator decides whether to widen `ENUM_VOCAB`, tune
`DEFAULT_NUMERIC_REL_TOL`, or accept the precision/recall trade-off
under audit-only surface (`aelf resolve` style human-in-loop).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (typo): Consider changing "human-in-loop" to the more standard "human-in-the-loop".

"Human-in-loop" reads like a typo and is uncommon; using the standard "human-in-the-loop" would improve clarity without changing the meaning.

Suggested change
under audit-only surface (`aelf resolve` style human-in-loop).
under audit-only surface (`aelf resolve` style human-in-the-loop).


## Maintenance

Adding a category:
1. Append the entry to `ENUM_VOCAB` with mutually-exclusive groups.
2. Pin a unit test in `tests/test_value_compare.py` covering the
new contradiction shape.
3. Re-run the bench gate. If recall/precision drop, the new
category is too noisy — narrow it or revert.

Tuning numeric tolerance: `DEFAULT_NUMERIC_REL_TOL` is pinned at 1%.
Drift on this value silently changes the gate; a unit test in
`test_value_compare.py` pins it explicitly.
54 changes: 51 additions & 3 deletions src/aelfrice/relationship_detector.py
Original file line number Diff line number Diff line change
Expand Up @@ -234,12 +234,47 @@ def analyze(
text_b: str,
*,
residual_overlap_min: float = DEFAULT_RESIDUAL_OVERLAP_MIN,
use_value_comparison: bool = False,
) -> RelationshipVerdict:
"""Classify the relationship between two belief texts.

Pure function over the two strings. Read by the audit pass and by
the bench gate at ``relationship_detector.classify``.

When ``use_value_comparison=True``, the typed-slot value-comparison
gate (#422) runs **before** the residual-overlap floor. If the
pair has at least one mutual-exclusion slot conflict, the verdict
is ``contradicts`` with a high score regardless of token overlap —
natural-language paraphrase is the failure mode that #201's R2
gate hit (recall 0.033) and the value-comparison gate
deliberately bypasses it. When the flag is ``False`` (the default)
behaviour is unchanged from the v1 detector.
"""
if use_value_comparison:
from aelfrice.value_compare import extract_values, find_conflicts
slots_a = extract_values(text_a)
slots_b = extract_values(text_b)
conflicts = find_conflicts(slots_a, slots_b)
if conflicts:
# Slot-conflict verdict bypasses the residual-overlap
# floor — the slot match itself is the relatedness
# signal. Score is fixed at 1.0 (highest confidence
# this gate emits) so the auto-emit policy in #422
# acceptance #3 can route on (conflicts present AND
# score >= floor).
kinds = sorted({c.kind for c in conflicts})
keys = sorted({c.key for c in conflicts})
rationale = "value_comparison:" + ",".join(
Comment on lines +265 to +267

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Rationale formatting can mis-associate conflict kinds and keys.

kinds and keys are independently deduped, sorted, and then zipped. This breaks the original (kind, key) associations, so a given kind may be paired with the wrong key, especially when mixing numeric and enum conflicts, producing a misleading rationale. Instead, build the rationale from the original conflicts sequence, preserving (c.kind, c.key) pairs (e.g., by iterating conflicts directly or sorting a collection of (kind, key) tuples).

f"{kind}={key}" for kind, key in zip(kinds, keys)
) if len(kinds) == len(keys) else (
"value_comparison:" + "+".join(kinds)
)
return RelationshipVerdict(
label=LABEL_CONTRADICTS,
score=1.0,
residual_overlap=0.0,
rationale=rationale,
)
sa = extract_signals(text_a)
sb = extract_signals(text_b)
overlap = _residual_jaccard(sa.residual_content, sb.residual_content)
Expand Down Expand Up @@ -281,9 +316,22 @@ def analyze(
)


def classify(text_a: str, text_b: str) -> str:
"""Bench-gate entry point: return the verdict label only."""
return analyze(text_a, text_b).label
def classify(
text_a: str,
text_b: str,
*,
use_value_comparison: bool = False,
) -> str:
"""Bench-gate entry point: return the verdict label only.

When ``use_value_comparison=True``, the v3 typed-slot gate (#422)
runs ahead of the residual-overlap floor.
"""
return analyze(
text_a,
text_b,
use_value_comparison=use_value_comparison,
).label


# --- Audit report types ------------------------------------------------
Expand Down
Loading
Loading