feat(ingest): subfloor noise-pattern filter + intra-turn edge-anchor demotion (#809)

CHANGELOG.md

@@ -14,6 +14,8 @@ installable release; see the roadmap in [README.md](README.md).
 
 - **Rebuilder pack accounting now honours `use_type_aware_compression`** ([#798](https://github.com/robotrocketscience/aelfrice/issues/798)). `rebuild_v14` was re-packing `retrieve()`'s candidate set with verbatim token cost regardless of the flag, so any ON-arm extras `retrieve()` admitted at compressed cost got trimmed back to the OFF-arm count. The downstream A4 continuation-fidelity bench gate ([#775](https://github.com/robotrocketscience/aelfrice/issues/775) / [PR #776](https://github.com/robotrocketscience/aelfrice/pull/776)) was therefore structurally vacuous — per-row fidelity delta = 0 by construction, regardless of corpus. Fix resolves the flag once at `rebuild_v14` entry (`resolve_use_type_aware_compression(use_type_aware_compression)`), threads it into the `retrieve()` call and into `_estimate_belief_tokens(b, *, compress_on=...)` at all three pack sites (L0 init, session tier, L1 / L2.5 tier). The rebuild block content itself stays verbatim — the change is in *how many* beliefs survive the budget, not what each surviving belief renders as. Default-OFF and the legacy `_retrieve_for_rebuild` (v1.2.0a0 alpha contract) are byte-identical. Unblocks the A4 axis of the [#769](https://github.com/robotrocketscience/aelfrice/issues/769) flip-default decision. Operator-decision history: Option A per [#798 thread](https://github.com/robotrocketscience/aelfrice/issues/798); Options B (rebuilder emits `compressed_beliefs[i].rendered`) and C (drop A4 from #769 acceptance) declined. Two new tests in `tests/test_context_rebuilder.py` (`test_rebuild_v14_pack_size_matches_compression_flag`, `test_rebuild_v14_compression_off_byte_identical_default`).
 
+- **Subfloor noise-pattern filter at sentence-level ingest** ([#809](https://github.com/robotrocketscience/aelfrice/issues/809), continuation of [#785](https://github.com/robotrocketscience/aelfrice/issues/785) § 3). `_ingest_turn_ids` (the transcript / commit-ingest sentence path) now runs each candidate sentence through `_looks_like_subfloor_noise` before belief creation. Matched sentences — code-fence boundaries (` ```bash`, `` ``` ``), header stubs ending with `:` (`"Acceptance criteria:"`, `"Pipeline composition, in order of evidence:"`), and markdown bullet stubs (`"- run tests"`, `"* foo"`, `"+ baz"`) — do not become freestanding belief rows. When a matched sentence sits between two full-length-belief sentences within the same turn, it attaches as `anchor_text` on a new intra-turn `DERIVED_FROM` edge between the surrounding beliefs (src=later, dst=earlier, matching the inter-turn convention in `ingest_jsonl`); unanchored matches are silently dropped. Multiple sub-floor clauses between the same pair join with `" | "` and truncate to `ANCHOR_TEXT_MAX_LEN`. Closes 19% of the short-reinforced-bloat leak documented in the `retrieval-corpus-bloat` lab campaign (companion to PR #795's §1 speaker-attribution gate, which closed 51%). The pattern check is scoped to short content by a length cap (`_SUBFLOOR_MAX_LEN = 80`, the spec literal): long-form content that happens to start with `- ` or end in `:` (real prose statements, multi-sentence list items) is preserved. A pattern-only check over-applied and dropped long-form prose ending in `:`; a standalone length-floor over-applied and dropped short legit claims like `"The default port is 8080."`. The combination (pattern AND < 80) catches the load-bearing noise class while preserving both short legit claims and long-form prose. Code-fence and bullet patterns are already largely handled upstream by `extract_sentences` (paired-fence wholesale strip, line-leading list-marker strip); the gate is a backstop for malformed / mid-line cases that survive. The header-ending-in-`:` pattern is NOT handled upstream and is the load-bearing pattern in the normal pipeline. Acknowledged false-positive risk on `"He said:"` and similar — the lab campaign named the pattern explicitly; trade-off accepted at empirical scope. Architectural deviation from spec letter (which describes the gate on "triple subject/object slots") is documented on the helper's docstring: the noun-phrase-based `triple_extractor` produces slots typically far below any length floor, so the gate has to live on the sentence-level path where the observable leak originates. 20 new tests in `tests/test_ingest_subfloor_noise.py` (four added for the length-floor boundary at 80).
+
 ## [3.1.0] - 2026-05-14
 
 ### Added

src/aelfrice/ingest.py

@@ -17,10 +17,11 @@
 from __future__ import annotations
 
 import json
+import re
 from dataclasses import dataclass
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import cast
+from typing import Final, cast
 
 from aelfrice.derivation_worker import run_worker
 from aelfrice.extraction import extract_sentences
@@ -35,6 +36,69 @@
 )
 from aelfrice.store import MemoryStore
 
+# #809 / #785 § 3: pattern-based subfloor-noise detector with
+# length-floor scope.
+#
+# Three short-fragment classes are filtered at the sentence-level
+# ingest boundary: code-fence boundaries (` ```bash`, `` ``` ``),
+# header stubs ending in `:` ("Acceptance criteria:", "Open
+# questions:"), and markdown bullet stubs (`- run tests`, `* foo`).
+# A matched sentence does not become a freestanding belief row; when
+# it sits between two full-length-belief sentences in the same turn
+# it attaches as `anchor_text` on an intra-turn DERIVED_FROM edge,
+# preserving relational meaning without inflating the belief-row
+# count. Unanchored matches are silently dropped.
+#
+# The pattern check is **scoped to short content** by a length cap
+# (`_SUBFLOOR_MAX_LEN`). The spec's intent is to filter short
+# fragments that carry no semantic claim; long-form content that
+# happens to start with `- ` or end in `:` (real prose statements,
+# multi-sentence list items that survived `extract_sentences`) is
+# load-bearing and must not be dropped. A pattern-only check
+# (without the length cap) over-applies and drops long-form prose
+# ending in `:` — e.g.,
+# "If you look at the way the rebuilder picks beliefs, the order is
+# always the same:".
+#
+# Why 80 chars: matches the spec literal in
+# docs/feature-ingest-speaker-gate.md §3. Empirically, sentences of
+# this kind that exceed ~80 chars are dominantly real prose; below
+# ~80 chars they are dominantly header stubs / fragment markers.
+#
+# A standalone 80-char length floor (without the pattern check) is
+# also wrong — it would drop short legit claims like "The default
+# port is 8080." that the test suite encodes as ingest-eligible.
+# The combination (pattern AND < 80) catches the load-bearing noise
+# class while preserving both short legit claims and long-form
+# prose.
+#
+# Acknowledged residual false-positive risk: a short complete
+# sentence ending with `:` ("He said:", "Note:") still drops.
+# These are ambiguous in isolation and the cost of preserving them
+# is a more elaborate rule (verb-detection, list-following-context)
+# that gets fragile fast. Trade-off accepted at empirical scope;
+# re-measure if production data surfaces a non-trivial miss rate.
+_SUBFLOOR_MAX_LEN: Final[int] = 80
+_SUBFLOOR_BULLET_PREFIX = re.compile(r"^[-*+]\s")
+
+
+def _looks_like_subfloor_noise(sentence: str) -> bool:
+    """True when `sentence` matches one of the three short-fragment
+    noise patterns (code-fence boundary, `:`-suffix header stub,
+    markdown bullet stub) AND its stripped length is below
+    `_SUBFLOOR_MAX_LEN`. Long-form content is never noise regardless
+    of leading or trailing markers — see module docstring."""
+    stripped = sentence.strip()
+    if not stripped or len(stripped) >= _SUBFLOOR_MAX_LEN:
+        return False
+    if stripped.startswith("```"):
+        return True
+    if stripped.endswith(":"):
+        return True
+    if _SUBFLOOR_BULLET_PREFIX.match(stripped):
+        return True
+    return False
+
 
 def _now_utc_iso() -> str:
     return datetime.now(timezone.utc).isoformat()
@@ -116,12 +180,41 @@ def _ingest_turn_ids(
     list is the per-sentence derived belief id (in input order, with
     duplicates dropped) — `ingest_jsonl` uses the last entry to wire
     DERIVED_FROM edges between consecutive turns within a session.
+
+    #809 adds a pattern-based subfloor filter: sentences matching
+    `_looks_like_subfloor_noise` (code-fence prefix, header ending in
+    `:`, bullet stub) do not become belief rows. When a sub-floor
+    sentence sits between two full-length-belief sentences in the
+    same turn, it attaches as `anchor_text` on an intra-turn
+    DERIVED_FROM edge between the surrounding beliefs; unanchored
+    sub-floor sentences are silently dropped.
     """
     sentences = extract_sentences(text)
     sentences = [s for s in sentences if not is_transcript_noise(s)]
     if not sentences:
         return []
 
+    # #809: partition sentences into full-length belief candidates and
+    # sub-floor clauses pending demotion to edge anchor_text. The
+    # `subfloor_between[i]` list holds sub-floor clauses (in original
+    # order) that preceded `full_sentences[i]`; entries before the
+    # first full sentence and after the last full sentence are
+    # unanchored and silently dropped.
+    full_sentences: list[str] = []
+    subfloor_between: list[list[str]] = []
+    pending_subfloor: list[str] = []
+    for sentence in sentences:
+        if _looks_like_subfloor_noise(sentence):
+            pending_subfloor.append(sentence)
+        else:
+            full_sentences.append(sentence)
+            subfloor_between.append(pending_subfloor)
+            pending_subfloor = []
+    # Anything left in pending_subfloor has no following full sentence —
+    # unanchored, silently dropped.
+    if not full_sentences:
+        return []
+
     ts = created_at or _now_utc_iso()
     # Snapshot the canonical belief set so we can identify which derived
     # ids in this turn correspond to brand-new inserts (vs corroborations
@@ -130,7 +223,7 @@ def _ingest_turn_ids(
     ids_before: set[str] = set(store.list_belief_ids())
 
     log_ids: list[str] = []
-    for sentence in sentences:
+    for sentence in full_sentences:
         log_id = store.record_ingest(
             source_kind=INGEST_SOURCE_FILESYSTEM,
             source_path=source,
@@ -145,20 +238,51 @@ def _ingest_turn_ids(
     # at end-of-turn is the per-batch invocation pattern from the spec.
     run_worker(store)
 
+    # Resolve each log_id to its canonical belief id once, in input
+    # order. Used twice: (a) for the public return value (newly
+    # inserted beliefs, deduped), (b) for the #809 intra-turn edge
+    # wiring below (per-sentence belief id, position-preserving).
+    log_belief_ids: list[str | None] = []
     inserted: list[str] = []
     seen: set[str] = set()
     for log_id in log_ids:
         entry = store.get_ingest_log_entry(log_id)
-        if entry is None:
+        bid: str | None = None
+        if entry is not None:
+            ids = entry.get("derived_belief_ids") or []
+            if isinstance(ids, list) and ids:
+                head = ids[0]
+                if isinstance(head, str):
+                    bid = head
+        log_belief_ids.append(bid)
+        if bid is not None and bid not in ids_before and bid not in seen:
+            seen.add(bid)
+            inserted.append(bid)
+
+    # #809: wire intra-turn DERIVED_FROM edges between consecutive
+    # full-length beliefs whose original-prose ordering was separated
+    # by one or more sub-floor clauses. Edge direction matches the
+    # inter-turn DERIVED_FROM convention in `ingest_jsonl` (src is the
+    # later belief, dst is the earlier one — "this is derived from
+    # that earlier one"). Anchor_text is the joined sub-floor clauses,
+    # truncated to ANCHOR_TEXT_MAX_LEN.
+    for i in range(1, len(log_belief_ids)):
+        between = subfloor_between[i]
+        if not between:
             continue
-        ids = entry.get("derived_belief_ids") or []
-        if not isinstance(ids, list):
+        prior_bid = log_belief_ids[i - 1]
+        curr_bid = log_belief_ids[i]
+        if prior_bid is None or curr_bid is None or prior_bid == curr_bid:
             continue
-        for bid in ids:
-            if (isinstance(bid, str) and bid not in ids_before
-                    and bid not in seen):
-                seen.add(bid)
-                inserted.append(bid)
+        anchor = " | ".join(between)[:ANCHOR_TEXT_MAX_LEN]
+        if store.get_edge(curr_bid, prior_bid, EDGE_DERIVED_FROM) is not None:
+            continue
+        store.insert_edge(Edge(
+            src=curr_bid, dst=prior_bid,
+            type=EDGE_DERIVED_FROM, weight=1.0,
+            anchor_text=anchor,
+        ))
+
     return inserted