fix(rebuilder): honour use_type_aware_compression in pack accounting (#798)

CHANGELOG.md

@@ -10,6 +10,10 @@ installable release; see the roadmap in [README.md](README.md).
 
 ## [Unreleased]
 
+### Fixed
+
+- **Rebuilder pack accounting now honours `use_type_aware_compression`** ([#798](https://github.com/robotrocketscience/aelfrice/issues/798)). `rebuild_v14` was re-packing `retrieve()`'s candidate set with verbatim token cost regardless of the flag, so any ON-arm extras `retrieve()` admitted at compressed cost got trimmed back to the OFF-arm count. The downstream A4 continuation-fidelity bench gate ([#775](https://github.com/robotrocketscience/aelfrice/issues/775) / [PR #776](https://github.com/robotrocketscience/aelfrice/pull/776)) was therefore structurally vacuous — per-row fidelity delta = 0 by construction, regardless of corpus. Fix resolves the flag once at `rebuild_v14` entry (`resolve_use_type_aware_compression(use_type_aware_compression)`), threads it into the `retrieve()` call and into `_estimate_belief_tokens(b, *, compress_on=...)` at all three pack sites (L0 init, session tier, L1 / L2.5 tier). The rebuild block content itself stays verbatim — the change is in *how many* beliefs survive the budget, not what each surviving belief renders as. Default-OFF and the legacy `_retrieve_for_rebuild` (v1.2.0a0 alpha contract) are byte-identical. Unblocks the A4 axis of the [#769](https://github.com/robotrocketscience/aelfrice/issues/769) flip-default decision. Operator-decision history: Option A per [#798 thread](https://github.com/robotrocketscience/aelfrice/issues/798); Options B (rebuilder emits `compressed_beliefs[i].rendered`) and C (drop A4 from #769 acceptance) declined. Two new tests in `tests/test_context_rebuilder.py` (`test_rebuild_v14_pack_size_matches_compression_flag`, `test_rebuild_v14_compression_off_byte_identical_default`).
+
 ## [3.1.0] - 2026-05-14
 
 ### Added

docs/feature-type-aware-compression.md

@@ -138,7 +138,7 @@ Per `docs/belief_retention_class.md` § "Recommendation summary": *"Soft down-we
 
 The rebuilder consumes retrieval output and writes a continuation-fidelity-scored block at `<git-common-dir>/aelfrice/rebuild_logs/`. With compression on, the rebuilder gets more total beliefs in the same `[rebuilder] token_budget` (default in `context_rebuilder.py:111`). Acceptance #4 makes this measurable: continuation-fidelity uplift at fixed budget.
 
-The rebuilder does not need a separate config knob; it inherits the retrieval-side flag. If `use_type_aware_compression` is on at retrieval call-time, the rebuilder reads compressed output transparently via the `RetrievalResult` shape change above.
+The rebuilder does not need a separate config knob; it inherits the retrieval-side flag. `rebuild_v14` resolves `use_type_aware_compression` once at function entry via `resolve_use_type_aware_compression(use_type_aware_compression)`, threads the resolved boolean into the `retrieve()` call so retrieve's L1/L2.5 pack accounting is compression-aware, and threads it into `_estimate_belief_tokens(b, *, compress_on=...)` at all three pack sites in rebuild_v14 (L0 init, session tier, L1 / L2.5 tier). The rebuild-block content itself remains verbatim — compression changes *how many* beliefs fit under the budget, not what each surviving belief renders as. This is the #798 fix; before it landed, rebuild_v14 used verbatim cost for its own re-pack and silently trimmed retrieve()'s ON-arm extras back to the OFF-arm count, leaving A4 structurally vacuous (per-row fidelity delta = 0 regardless of corpus).
 
 ### vs. type / source-tier axes
 
@@ -182,6 +182,8 @@ A second test compares against a fixture of `(belief_id, expected_rendered)` pai
 
 The continuation-fidelity scorer (#141 v1.4 deliverable) is run on the rebuild_logs corpus with `use_type_aware_compression={OFF, ON}`. Bench-gate: ON ≥ OFF on continuation-fidelity score at the same `[rebuilder] token_budget`. Tolerance band: `≥ baseline − 0.005` (a half-point of the fidelity-score noise floor, mirroring the BM25F bench-gate band at #154).
 
+Prerequisite: rebuild_v14's pack accounting must honour the compression flag — otherwise per-row fidelity delta is 0 by construction, as documented in #798. That prerequisite landed alongside this doc revision (see `tests/test_context_rebuilder.py::test_rebuild_v14_pack_size_matches_compression_flag` — asserts ON packs strictly more beliefs than OFF at a budget that forces trim).
+
 ### A5 — composition tracker
 
 The #154 composition tracker doc gains a row for `use_type_aware_compression`: input shape, output shape, where it sits, bench verdict. This is **not a lane** — it is a packing-stage transform. The tracker row is present for operator clarity.

src/aelfrice/context_rebuilder.py

@@ -80,7 +80,8 @@
     VALID_STRATEGIES,
     transform_query,
 )
-from aelfrice.retrieval import retrieve
+from aelfrice.compression import compress_for_retrieval
+from aelfrice.retrieval import resolve_use_type_aware_compression, retrieve
 from aelfrice.scoring import posterior_mean
 from aelfrice.store import MemoryStore
 from aelfrice.triple_extractor import extract_triples
@@ -316,6 +317,7 @@ def rebuild_v14(
     floor_l1: float = 0.0,
     query_strategy: str = DEFAULT_QUERY_STRATEGY,
     working_state: "WorkingState | None" = None,
+    use_type_aware_compression: bool | None = None,
 ) -> str:
     """v1.4 rebuild: L0 + session-scoped + L2.5/L1 via `retrieve()`.
 
@@ -358,6 +360,12 @@ def rebuild_v14(
     `aelfrice.query_understanding`. `legacy-bm25` is byte-identical
     to the v1.4 path and remains opt-in until PR-4 removes it.
     """
+    # #798: resolve compression flag once, thread through pack accounting
+    # so rebuild_v14's trim matches retrieve()'s trim under the same flag.
+    compress_on: bool = resolve_use_type_aware_compression(
+        use_type_aware_compression,
+    )
+
     locked: list[Belief] = store.list_locked_beliefs()
     locked_ids: set[str] = {b.id for b in locked}
 
@@ -371,7 +379,10 @@ def rebuild_v14(
     # composite ourselves so we can interleave session-scoped
     # beliefs in the right slot.
     retrieved: list[Belief] = retrieve(
-        store, query, token_budget=token_budget,
+        store,
+        query,
+        token_budget=token_budget,
+        use_type_aware_compression=compress_on,
     )
     # Drop L0 from retrieved (we'll prepend our own copy).
     non_locked_hits: list[Belief] = [
@@ -402,7 +413,9 @@ def _score_for(b: Belief) -> tuple[float, float | None]:
     # or any future dedup gap). Without this, the rebuild block can
     # surface 10+ identical lines. Locked wins (it's prepended whole);
     # subsequent tiers skip any hash already packed.
-    used: int = sum(_estimate_belief_tokens(b) for b in locked)
+    used: int = sum(
+        _estimate_belief_tokens(b, compress_on=compress_on) for b in locked
+    )
     out: list[Belief] = list(locked)
     seen_hashes: set[str] = {b.content_hash for b in locked}
     hash_to_packed_id: dict[str, str] = {b.content_hash: b.id for b in locked}
@@ -450,7 +463,7 @@ def _score_for(b: Belief) -> tuple[float, float | None]:
             reason = "budget_exceeded"
             n_dropped_by_budget += 1
         else:
-            cost = _estimate_belief_tokens(b)
+            cost = _estimate_belief_tokens(b, compress_on=compress_on)
             if used + cost > token_budget:
                 budget_exceeded_session = True
                 decision = "dropped"
@@ -500,7 +513,7 @@ def _score_for(b: Belief) -> tuple[float, float | None]:
             reason2 = "budget_exceeded"
             n_dropped_by_budget += 1
         else:
-            cost = _estimate_belief_tokens(b)
+            cost = _estimate_belief_tokens(b, compress_on=compress_on)
             if used + cost > token_budget:
                 budget_exceeded_non_locked = True
                 decision2 = "dropped"
@@ -1321,9 +1334,16 @@ def record_user_prompt_submit_log(
 # --- Format helpers --------------------------------------------------------
 
 
-def _estimate_belief_tokens(b: Belief) -> int:
+def _estimate_belief_tokens(b: Belief, *, compress_on: bool = False) -> int:
+    # Verbatim cost by default; compressed render cost when compress_on=True.
+    # Mirrors retrieval._cost so rebuild_v14's pack accounting matches
+    # retrieve()'s pack accounting under the same flag — closes #798.
+    # Locks always render verbatim (compress_for_retrieval honors locked=True).
     if not b.content:
         return 0
+    if compress_on:
+        cb = compress_for_retrieval(b, locked=(b.lock_level == LOCK_USER))
+        return cb.rendered_tokens
     return int((len(b.content) + _CHARS_PER_TOKEN - 1) // _CHARS_PER_TOKEN)
 
 

tests/test_context_rebuilder.py

@@ -454,3 +454,103 @@ def test_rebuild_v14_xml_escapes_working_state_text(tmp_path: Path) -> None:
         store.close()
     assert "feat/<unsafe>" in out
     assert "query: a < b & c" in out
+
+
+# ---- #798: rebuild_v14 pack accounting honors use_type_aware_compression ----
+
+
+def _mk_snapshot(bid: str, content: str) -> Belief:
+    # Snapshot-class belief whose verbatim cost > headline cost.
+    from aelfrice.models import RETENTION_SNAPSHOT
+    return Belief(
+        id=bid,
+        content=content,
+        content_hash=f"h_{bid}",
+        alpha=1.0,
+        beta=1.0,
+        type=BELIEF_FACTUAL,
+        retention_class=RETENTION_SNAPSHOT,
+        lock_level=LOCK_NONE,
+        locked_at=None,
+        demotion_pressure=0,
+        created_at="2026-04-26T00:00:00Z",
+        last_retrieved_at=None,
+    )
+
+
+def test_rebuild_v14_pack_size_matches_compression_flag(tmp_path: Path) -> None:
+    """#798: rebuild_v14 pack accounting honors `use_type_aware_compression`.
+
+    With a tight budget that forces trim under OFF, the ON arm must pack
+    strictly more beliefs than the OFF arm — otherwise the rebuilder is
+    using verbatim cost regardless of flag (the bug PR #776's A4 gate hit).
+    """
+    from aelfrice.context_rebuilder import rebuild_v14
+
+    # Headline is the first sentence ending in ". ". Padding pushes
+    # verbatim cost well above compressed (headline-only) cost.
+    pad = " padding bytes " * 8
+    beliefs = [
+        _mk_snapshot(
+            f"S{i}",
+            f"banana headline {i}. {pad}second sentence keeps verbatim large.",
+        )
+        for i in range(1, 7)
+    ]
+    store = _seed(tmp_path / "m.db", beliefs)
+    turns = [RecentTurn(role="user", text="banana")]
+    try:
+        # Tight budget: long verbatim trims, short headlines fit more.
+        off_block = rebuild_v14(
+            turns, store, token_budget=120, use_type_aware_compression=False,
+        )
+        on_block = rebuild_v14(
+            turns, store, token_budget=120, use_type_aware_compression=True,
+        )
+    finally:
+        store.close()
+
+    off_packed = off_block.count('<belief id="')
+    on_packed = on_block.count('<belief id="')
+    assert on_packed > off_packed, (
+        "use_type_aware_compression=True must let rebuild_v14 pack more "
+        f"beliefs than =False at a tight budget (off={off_packed}, "
+        f"on={on_packed}). If equal, _estimate_belief_tokens is ignoring "
+        "the flag — #798's verbatim-cost re-pack bug has regressed."
+    )
+
+
+def test_rebuild_v14_compression_off_byte_identical_default(
+    tmp_path: Path,
+) -> None:
+    """#798: default-OFF leaves the byte-identical contract intact.
+
+    `rebuild_v14(...)` with no kwarg and no env override must produce the
+    same block as `rebuild_v14(..., use_type_aware_compression=False)` —
+    the #139 / #288 regression contract.
+    """
+    import os
+
+    from aelfrice.context_rebuilder import rebuild_v14
+
+    pad = " padding bytes " * 8
+    beliefs = [
+        _mk_snapshot(
+            f"S{i}",
+            f"banana headline {i}. {pad}second sentence keeps verbatim large.",
+        )
+        for i in range(1, 5)
+    ]
+    store = _seed(tmp_path / "m.db", beliefs)
+    turns = [RecentTurn(role="user", text="banana")]
+    prior_env = os.environ.pop("AELFRICE_TYPE_AWARE_COMPRESSION", None)
+    try:
+        default_block = rebuild_v14(turns, store, token_budget=120)
+        off_block = rebuild_v14(
+            turns, store, token_budget=120, use_type_aware_compression=False,
+        )
+    finally:
+        store.close()
+        if prior_env is not None:
+            os.environ["AELFRICE_TYPE_AWARE_COMPRESSION"] = prior_env
+    assert default_block == off_block