diff --git a/CHANGELOG.md b/CHANGELOG.md index f2859f241..a0f12975d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,8 @@ installable release; see the roadmap in [README.md](README.md). - **Rebuilder pack accounting now honours `use_type_aware_compression`** ([#798](https://github.com/robotrocketscience/aelfrice/issues/798)). `rebuild_v14` was re-packing `retrieve()`'s candidate set with verbatim token cost regardless of the flag, so any ON-arm extras `retrieve()` admitted at compressed cost got trimmed back to the OFF-arm count. The downstream A4 continuation-fidelity bench gate ([#775](https://github.com/robotrocketscience/aelfrice/issues/775) / [PR #776](https://github.com/robotrocketscience/aelfrice/pull/776)) was therefore structurally vacuous — per-row fidelity delta = 0 by construction, regardless of corpus. Fix resolves the flag once at `rebuild_v14` entry (`resolve_use_type_aware_compression(use_type_aware_compression)`), threads it into the `retrieve()` call and into `_estimate_belief_tokens(b, *, compress_on=...)` at all three pack sites (L0 init, session tier, L1 / L2.5 tier). The rebuild block content itself stays verbatim — the change is in *how many* beliefs survive the budget, not what each surviving belief renders as. Default-OFF and the legacy `_retrieve_for_rebuild` (v1.2.0a0 alpha contract) are byte-identical. Unblocks the A4 axis of the [#769](https://github.com/robotrocketscience/aelfrice/issues/769) flip-default decision. Operator-decision history: Option A per [#798 thread](https://github.com/robotrocketscience/aelfrice/issues/798); Options B (rebuilder emits `compressed_beliefs[i].rendered`) and C (drop A4 from #769 acceptance) declined. Two new tests in `tests/test_context_rebuilder.py` (`test_rebuild_v14_pack_size_matches_compression_flag`, `test_rebuild_v14_compression_off_byte_identical_default`). +- **Subfloor noise-pattern filter at sentence-level ingest** ([#809](https://github.com/robotrocketscience/aelfrice/issues/809), continuation of [#785](https://github.com/robotrocketscience/aelfrice/issues/785) § 3). `_ingest_turn_ids` (the transcript / commit-ingest sentence path) now runs each candidate sentence through `_looks_like_subfloor_noise` before belief creation. Matched sentences — code-fence boundaries (` ```bash`, `` ``` ``), header stubs ending with `:` (`"Acceptance criteria:"`, `"Pipeline composition, in order of evidence:"`), and markdown bullet stubs (`"- run tests"`, `"* foo"`, `"+ baz"`) — do not become freestanding belief rows. When a matched sentence sits between two full-length-belief sentences within the same turn, it attaches as `anchor_text` on a new intra-turn `DERIVED_FROM` edge between the surrounding beliefs (src=later, dst=earlier, matching the inter-turn convention in `ingest_jsonl`); unanchored matches are silently dropped. Multiple sub-floor clauses between the same pair join with `" | "` and truncate to `ANCHOR_TEXT_MAX_LEN`. Closes 19% of the short-reinforced-bloat leak documented in the `retrieval-corpus-bloat` lab campaign (companion to PR #795's §1 speaker-attribution gate, which closed 51%). The pattern check is scoped to short content by a length cap (`_SUBFLOOR_MAX_LEN = 80`, the spec literal): long-form content that happens to start with `- ` or end in `:` (real prose statements, multi-sentence list items) is preserved. A pattern-only check over-applied and dropped long-form prose ending in `:`; a standalone length-floor over-applied and dropped short legit claims like `"The default port is 8080."`. The combination (pattern AND < 80) catches the load-bearing noise class while preserving both short legit claims and long-form prose. Code-fence and bullet patterns are already largely handled upstream by `extract_sentences` (paired-fence wholesale strip, line-leading list-marker strip); the gate is a backstop for malformed / mid-line cases that survive. The header-ending-in-`:` pattern is NOT handled upstream and is the load-bearing pattern in the normal pipeline. Acknowledged false-positive risk on `"He said:"` and similar — the lab campaign named the pattern explicitly; trade-off accepted at empirical scope. Architectural deviation from spec letter (which describes the gate on "triple subject/object slots") is documented on the helper's docstring: the noun-phrase-based `triple_extractor` produces slots typically far below any length floor, so the gate has to live on the sentence-level path where the observable leak originates. 20 new tests in `tests/test_ingest_subfloor_noise.py` (four added for the length-floor boundary at 80). + ## [3.1.0] - 2026-05-14 ### Added diff --git a/src/aelfrice/ingest.py b/src/aelfrice/ingest.py index 91d858dbf..eeef0ecf9 100644 --- a/src/aelfrice/ingest.py +++ b/src/aelfrice/ingest.py @@ -17,10 +17,11 @@ from __future__ import annotations import json +import re from dataclasses import dataclass from datetime import datetime, timezone from pathlib import Path -from typing import cast +from typing import Final, cast from aelfrice.derivation_worker import run_worker from aelfrice.extraction import extract_sentences @@ -35,6 +36,69 @@ ) from aelfrice.store import MemoryStore +# #809 / #785 § 3: pattern-based subfloor-noise detector with +# length-floor scope. +# +# Three short-fragment classes are filtered at the sentence-level +# ingest boundary: code-fence boundaries (` ```bash`, `` ``` ``), +# header stubs ending in `:` ("Acceptance criteria:", "Open +# questions:"), and markdown bullet stubs (`- run tests`, `* foo`). +# A matched sentence does not become a freestanding belief row; when +# it sits between two full-length-belief sentences in the same turn +# it attaches as `anchor_text` on an intra-turn DERIVED_FROM edge, +# preserving relational meaning without inflating the belief-row +# count. Unanchored matches are silently dropped. +# +# The pattern check is **scoped to short content** by a length cap +# (`_SUBFLOOR_MAX_LEN`). The spec's intent is to filter short +# fragments that carry no semantic claim; long-form content that +# happens to start with `- ` or end in `:` (real prose statements, +# multi-sentence list items that survived `extract_sentences`) is +# load-bearing and must not be dropped. A pattern-only check +# (without the length cap) over-applies and drops long-form prose +# ending in `:` — e.g., +# "If you look at the way the rebuilder picks beliefs, the order is +# always the same:". +# +# Why 80 chars: matches the spec literal in +# docs/feature-ingest-speaker-gate.md §3. Empirically, sentences of +# this kind that exceed ~80 chars are dominantly real prose; below +# ~80 chars they are dominantly header stubs / fragment markers. +# +# A standalone 80-char length floor (without the pattern check) is +# also wrong — it would drop short legit claims like "The default +# port is 8080." that the test suite encodes as ingest-eligible. +# The combination (pattern AND < 80) catches the load-bearing noise +# class while preserving both short legit claims and long-form +# prose. +# +# Acknowledged residual false-positive risk: a short complete +# sentence ending with `:` ("He said:", "Note:") still drops. +# These are ambiguous in isolation and the cost of preserving them +# is a more elaborate rule (verb-detection, list-following-context) +# that gets fragile fast. Trade-off accepted at empirical scope; +# re-measure if production data surfaces a non-trivial miss rate. +_SUBFLOOR_MAX_LEN: Final[int] = 80 +_SUBFLOOR_BULLET_PREFIX = re.compile(r"^[-*+]\s") + + +def _looks_like_subfloor_noise(sentence: str) -> bool: + """True when `sentence` matches one of the three short-fragment + noise patterns (code-fence boundary, `:`-suffix header stub, + markdown bullet stub) AND its stripped length is below + `_SUBFLOOR_MAX_LEN`. Long-form content is never noise regardless + of leading or trailing markers — see module docstring.""" + stripped = sentence.strip() + if not stripped or len(stripped) >= _SUBFLOOR_MAX_LEN: + return False + if stripped.startswith("```"): + return True + if stripped.endswith(":"): + return True + if _SUBFLOOR_BULLET_PREFIX.match(stripped): + return True + return False + def _now_utc_iso() -> str: return datetime.now(timezone.utc).isoformat() @@ -116,12 +180,41 @@ def _ingest_turn_ids( list is the per-sentence derived belief id (in input order, with duplicates dropped) — `ingest_jsonl` uses the last entry to wire DERIVED_FROM edges between consecutive turns within a session. + + #809 adds a pattern-based subfloor filter: sentences matching + `_looks_like_subfloor_noise` (code-fence prefix, header ending in + `:`, bullet stub) do not become belief rows. When a sub-floor + sentence sits between two full-length-belief sentences in the + same turn, it attaches as `anchor_text` on an intra-turn + DERIVED_FROM edge between the surrounding beliefs; unanchored + sub-floor sentences are silently dropped. """ sentences = extract_sentences(text) sentences = [s for s in sentences if not is_transcript_noise(s)] if not sentences: return [] + # #809: partition sentences into full-length belief candidates and + # sub-floor clauses pending demotion to edge anchor_text. The + # `subfloor_between[i]` list holds sub-floor clauses (in original + # order) that preceded `full_sentences[i]`; entries before the + # first full sentence and after the last full sentence are + # unanchored and silently dropped. + full_sentences: list[str] = [] + subfloor_between: list[list[str]] = [] + pending_subfloor: list[str] = [] + for sentence in sentences: + if _looks_like_subfloor_noise(sentence): + pending_subfloor.append(sentence) + else: + full_sentences.append(sentence) + subfloor_between.append(pending_subfloor) + pending_subfloor = [] + # Anything left in pending_subfloor has no following full sentence — + # unanchored, silently dropped. + if not full_sentences: + return [] + ts = created_at or _now_utc_iso() # Snapshot the canonical belief set so we can identify which derived # ids in this turn correspond to brand-new inserts (vs corroborations @@ -130,7 +223,7 @@ def _ingest_turn_ids( ids_before: set[str] = set(store.list_belief_ids()) log_ids: list[str] = [] - for sentence in sentences: + for sentence in full_sentences: log_id = store.record_ingest( source_kind=INGEST_SOURCE_FILESYSTEM, source_path=source, @@ -145,20 +238,51 @@ def _ingest_turn_ids( # at end-of-turn is the per-batch invocation pattern from the spec. run_worker(store) + # Resolve each log_id to its canonical belief id once, in input + # order. Used twice: (a) for the public return value (newly + # inserted beliefs, deduped), (b) for the #809 intra-turn edge + # wiring below (per-sentence belief id, position-preserving). + log_belief_ids: list[str | None] = [] inserted: list[str] = [] seen: set[str] = set() for log_id in log_ids: entry = store.get_ingest_log_entry(log_id) - if entry is None: + bid: str | None = None + if entry is not None: + ids = entry.get("derived_belief_ids") or [] + if isinstance(ids, list) and ids: + head = ids[0] + if isinstance(head, str): + bid = head + log_belief_ids.append(bid) + if bid is not None and bid not in ids_before and bid not in seen: + seen.add(bid) + inserted.append(bid) + + # #809: wire intra-turn DERIVED_FROM edges between consecutive + # full-length beliefs whose original-prose ordering was separated + # by one or more sub-floor clauses. Edge direction matches the + # inter-turn DERIVED_FROM convention in `ingest_jsonl` (src is the + # later belief, dst is the earlier one — "this is derived from + # that earlier one"). Anchor_text is the joined sub-floor clauses, + # truncated to ANCHOR_TEXT_MAX_LEN. + for i in range(1, len(log_belief_ids)): + between = subfloor_between[i] + if not between: continue - ids = entry.get("derived_belief_ids") or [] - if not isinstance(ids, list): + prior_bid = log_belief_ids[i - 1] + curr_bid = log_belief_ids[i] + if prior_bid is None or curr_bid is None or prior_bid == curr_bid: continue - for bid in ids: - if (isinstance(bid, str) and bid not in ids_before - and bid not in seen): - seen.add(bid) - inserted.append(bid) + anchor = " | ".join(between)[:ANCHOR_TEXT_MAX_LEN] + if store.get_edge(curr_bid, prior_bid, EDGE_DERIVED_FROM) is not None: + continue + store.insert_edge(Edge( + src=curr_bid, dst=prior_bid, + type=EDGE_DERIVED_FROM, weight=1.0, + anchor_text=anchor, + )) + return inserted diff --git a/tests/test_ingest_subfloor_noise.py b/tests/test_ingest_subfloor_noise.py new file mode 100644 index 000000000..1779fc111 --- /dev/null +++ b/tests/test_ingest_subfloor_noise.py @@ -0,0 +1,315 @@ +"""Regression suite for the ingest subfloor-noise filter (#809). + +`_ingest_turn_ids` (the sentence-level transcript / commit ingest path) +filters out sentences matching `_looks_like_subfloor_noise`. The +pattern set is defense-in-depth across the three noise classes the +lab campaign named: + +1. **Code-fence boundaries** (` ```bash`, `` ``` ``). Already handled + upstream by `extract_sentences` (it strips paired triple-backtick + regions wholesale); the gate here is a backstop for malformed / + unpaired fence text that survives the strip. +2. **Bullet stubs** (`- run tests`, `* foo`). Already handled upstream + by `extract_sentences` (it strips line-leading list markers); the + gate here is a backstop for bullets that appear mid-line and + survive. +3. **Header stubs ending in `:`** ("Acceptance criteria:", "Pipeline + composition, in order of evidence:"). NOT handled by + `extract_sentences` — this is the load-bearing pattern in the + normal pipeline. + +Below the gate: + +- matched sentences do not become freestanding belief rows; +- if a matched sentence sits *between* two full-length beliefs in the + same turn, it attaches as `anchor_text` on an intra-turn + DERIVED_FROM edge between those two beliefs; +- if unanchored (no surrounding full-length belief in the same turn), + it is silently dropped. + +Empirical basis: `retrieval-corpus-bloat` R0/R2 (lab campaign, +2026-05-11) attributed 19% of short-reinforced beliefs in the +alpha+beta >= 10 stratum to these three pattern classes. Operator- +ratified scope (#809) is pattern-based rather than length-based so +legit short factual claims survive the gate. +""" +from __future__ import annotations + +from collections.abc import Iterator +from pathlib import Path + +import pytest + +from aelfrice.ingest import _ingest_turn_ids, _looks_like_subfloor_noise +from aelfrice.models import EDGE_DERIVED_FROM +from aelfrice.store import MemoryStore + + +# Full-length fixtures (real claims, ingest-eligible under +# `is_transcript_noise`). Each sentence terminates with `.` so +# `extract_sentences` splits on it cleanly. +FULL_A = "The configuration file lives at /etc/aelfrice/conf." +FULL_B = "Astronomers process supernova imagery nightly using clusters." +FULL_C = "The pipeline orders feedback events ahead of corroboration." + +# Sub-floor (header-ending-in-`:`) fixtures. These must be placed on +# their own newline-separated lines in fixture text — `:` is not a +# sentence boundary in `extract_sentences`, so without the newline +# they merge with surrounding prose and don't reach the gate. +SUB_HEADER = "Acceptance criteria:" +SUB_HEADER_LONG = "Pipeline composition, in order of evidence:" + + +@pytest.fixture +def store(tmp_path: Path) -> Iterator[MemoryStore]: + s = MemoryStore(str(tmp_path / "subfloor.db")) + yield s + s.close() + + +# --- _looks_like_subfloor_noise unit checks (all three patterns) --------- + + +def test_helper_flags_header_ending_with_colon() -> None: + assert _looks_like_subfloor_noise("Acceptance criteria:") is True + assert _looks_like_subfloor_noise( + "Pipeline composition, in order of evidence:" + ) is True + + +def test_helper_flags_codefence_prefix() -> None: + """Backstop: `extract_sentences` already strips paired code-fence + regions, but the helper catches unpaired / malformed fence text + that survives.""" + assert _looks_like_subfloor_noise("```bash") is True + assert _looks_like_subfloor_noise("```") is True + assert _looks_like_subfloor_noise("```python") is True + + +def test_helper_flags_bullet_stubs() -> None: + """Backstop: `extract_sentences` already strips line-leading list + markers, but the helper catches bullets that survive (e.g., + mid-line `- foo` after sentence-split).""" + assert _looks_like_subfloor_noise("- run tests") is True + assert _looks_like_subfloor_noise("* foo bar") is True + assert _looks_like_subfloor_noise("+ another bullet") is True + + +def test_helper_preserves_legit_short_claims() -> None: + """Pattern-gate (per #809 operator-ratified scope) does NOT flag + short factual claims that a strict length floor would have + dropped.""" + assert _looks_like_subfloor_noise( + "The configuration file lives at /etc/aelfrice/conf." + ) is False + assert _looks_like_subfloor_noise( + "The default port is 8080 for the dashboard." + ) is False + assert _looks_like_subfloor_noise( + "Hubble observes galaxies nightly." + ) is False + + +def test_helper_strips_before_checking() -> None: + """Whitespace around the noise marker doesn't bypass the gate.""" + assert _looks_like_subfloor_noise(" ```bash ") is True + assert _looks_like_subfloor_noise("\t- bullet\t") is True + assert _looks_like_subfloor_noise(" Acceptance criteria: ") is True + + +def test_helper_empty_sentence_not_flagged() -> None: + """Empty / whitespace-only sentences are not noise-marked here — + they're filtered upstream by `extract_sentences` (< 10 chars).""" + assert _looks_like_subfloor_noise("") is False + assert _looks_like_subfloor_noise(" ") is False + + +# --- Filter: header stubs do not become belief rows ---------------------- + + +def test_header_stub_alone_in_turn_dropped(store: MemoryStore) -> None: + """A header-ending-in-`:` sentence alone produces zero beliefs.""" + ids = _ingest_turn_ids(store, SUB_HEADER, source="user") + assert ids == [] + assert store.count_beliefs() == 0 + + +def test_all_subfloor_turn_drops_all_silently( + store: MemoryStore, +) -> None: + """A turn composed entirely of subfloor sentences produces zero + beliefs AND zero edges — no surrounding full-length belief to + anchor to.""" + text = f"{SUB_HEADER}\n\n{SUB_HEADER_LONG}" + ids = _ingest_turn_ids(store, text, source="user") + assert ids == [] + assert store.count_beliefs() == 0 + assert store.count_edges() == 0 + + +def test_subfloor_at_start_dropped_when_no_preceding_full( + store: MemoryStore, +) -> None: + """Sub-floor leading a turn has no preceding full-length belief to + pair with — unanchored, silently dropped.""" + text = f"{SUB_HEADER}\n\n{FULL_A}" + ids = _ingest_turn_ids(store, text, source="user") + assert len(ids) == 1 + assert store.count_edges() == 0 + + +def test_subfloor_at_end_dropped_when_no_following_full( + store: MemoryStore, +) -> None: + """Sub-floor trailing the last full-length sentence has no + surrounding belief on its right side — silently dropped.""" + text = f"{FULL_A}\n\n{SUB_HEADER}" + ids = _ingest_turn_ids(store, text, source="user") + assert len(ids) == 1 + assert store.count_edges() == 0 + + +# --- Demotion: subfloor between two full sentences -> edge anchor -------- + + +def test_header_stub_between_full_demotes_to_edge_anchor( + store: MemoryStore, +) -> None: + """Spec § 3 primary test (codebase-adapted): a paragraph with + (full_A, header_stub, full_B) creates two beliefs and one + intra-turn DERIVED_FROM edge full_B -> full_A whose anchor_text + carries the sub-floor header.""" + text = f"{FULL_A}\n\n{SUB_HEADER}\n\n{FULL_B}" + ids = _ingest_turn_ids(store, text, source="user") + assert len(ids) == 2 + assert store.count_beliefs() == 2 + + later_id, earlier_id = ids[1], ids[0] + edge = store.get_edge(later_id, earlier_id, EDGE_DERIVED_FROM) + assert edge is not None + assert SUB_HEADER in edge.anchor_text + + +def test_multiple_header_stubs_between_two_full_concatenated( + store: MemoryStore, +) -> None: + """Two header stubs between the same pair of full-length sentences + both land in the edge's anchor_text (joined by ' | ').""" + text = f"{FULL_A}\n\n{SUB_HEADER}\n\n{SUB_HEADER_LONG}\n\n{FULL_B}" + ids = _ingest_turn_ids(store, text, source="user") + assert len(ids) == 2 + + edge = store.get_edge(ids[1], ids[0], EDGE_DERIVED_FROM) + assert edge is not None + assert SUB_HEADER in edge.anchor_text + assert SUB_HEADER_LONG in edge.anchor_text + + +def test_subfloor_between_each_pair_of_three_full_sentences( + store: MemoryStore, +) -> None: + """Three full sentences with sub-floor between each consecutive + pair create three beliefs and two demotion edges, each carrying + the appropriate sub-floor clause.""" + text = ( + f"{FULL_A}\n\n{SUB_HEADER}\n\n{FULL_B}\n\n{SUB_HEADER_LONG}\n\n" + f"{FULL_C}" + ) + ids = _ingest_turn_ids(store, text, source="user") + assert len(ids) == 3 + + edge_b_to_a = store.get_edge(ids[1], ids[0], EDGE_DERIVED_FROM) + assert edge_b_to_a is not None + assert SUB_HEADER in edge_b_to_a.anchor_text + + edge_c_to_b = store.get_edge(ids[2], ids[1], EDGE_DERIVED_FROM) + assert edge_c_to_b is not None + assert SUB_HEADER_LONG in edge_c_to_b.anchor_text + + +# --- Full-length sentences: existing behavior unchanged ------------------ + + +def test_full_length_consecutive_sentences_no_intra_turn_edge( + store: MemoryStore, +) -> None: + """Two full-length sentences adjacent with no sub-floor between + them produce no intra-turn DERIVED_FROM edge — the demotion path + only fires when sub-floor sits between full-length beliefs.""" + text = f"{FULL_A} {FULL_B}" + ids = _ingest_turn_ids(store, text, source="user") + assert len(ids) == 2 + assert store.count_edges() == 0 + + +def test_legit_short_factual_claim_still_ingested( + store: MemoryStore, +) -> None: + """The pattern-gate (vs length-floor) trade-off: short factual + claims that a length floor would have dropped survive here.""" + text = "The default port is 8080." # 25 chars, no noise markers + ids = _ingest_turn_ids(store, text, source="user") + assert len(ids) == 1 + assert store.count_beliefs() == 1 + + +# --- Length-floor boundary on the three-pattern check -------------------- + + +def test_helper_does_not_flag_long_form_prose_ending_with_colon() -> None: + """A long sentence ending with `:` is a real prose statement, not + a header stub. The gate's pattern check is scoped to short + content (< 80 chars stripped); long content survives regardless + of the trailing `:`. Without this boundary, prose like the + sentence below — 81 chars — would be incorrectly dropped from + ingest.""" + s = ("If you look at the way the rebuilder picks beliefs, " + "the order is always the same:") + assert len(s) == 81 + assert _looks_like_subfloor_noise(s) is False + + +def test_helper_does_not_flag_long_bullet_paragraph() -> None: + """A long-form belief that happens to start with `- ` (e.g. a + list item that survived `extract_sentences`'s leading-marker + strip via an edge case) is load-bearing content, not a stub. + The length-floor preserves it.""" + s = ("- Keeps each belief's confidence as a Beta-Bernoulli " + "posterior so that confidence updates remain interpretable") + assert len(s) >= 80 + assert _looks_like_subfloor_noise(s) is False + + +def test_helper_flags_short_header_below_floor() -> None: + """The pattern check still fires on canonical short header stubs. + Sanity-check the boundary in the other direction.""" + assert _looks_like_subfloor_noise("Acceptance criteria:") is True + assert _looks_like_subfloor_noise("```bash") is True + assert _looks_like_subfloor_noise("- run tests") is True + + +def test_helper_drops_at_boundary_just_below_floor() -> None: + """A 79-char `:`-suffix string drops (< 80). An 80-char string + survives (>= 80). The boundary is half-open at 80.""" + s79 = "x" * 78 + ":" + s80 = "x" * 79 + ":" + assert len(s79) == 79 + assert len(s80) == 80 + assert _looks_like_subfloor_noise(s79) is True + assert _looks_like_subfloor_noise(s80) is False + + +def test_ingest_idempotent_on_repeat_under_demotion( + store: MemoryStore, +) -> None: + """Re-ingesting the same turn does not create duplicate beliefs or + duplicate intra-turn DERIVED_FROM edges (deduped via `get_edge` + check).""" + text = f"{FULL_A}\n\n{SUB_HEADER}\n\n{FULL_B}" + ids_first = _ingest_turn_ids(store, text, source="user") + n_beliefs = store.count_beliefs() + n_edges = store.count_edges() + _ = _ingest_turn_ids(store, text, source="user") + assert store.count_beliefs() == n_beliefs + assert store.count_edges() == n_edges + assert len(ids_first) == 2