docs(feature-type-aware-compression): spec memo for #434

docs/feature-type-aware-compression.md

@@ -0,0 +1,227 @@
+# Feature spec: Type-aware compression (#434)
+
+**Status:** spec, no implementation
+**Issue:** #434
+**Recovery-inventory line:** [`docs/ROADMAP.md`](ROADMAP.md) — *"Type-aware compression | v2.0.0"*
+**Substrate prereqs:** #290 (retention class column + per-source defaults, shipped v1.6.0), #141 (context rebuilder, shipped v1.4.0)
+
+---
+
+## Purpose
+
+At a fixed retrieval `token_budget`, today's path either includes a belief verbatim or trims it from the tail. Per [`docs/belief_retention_class.md`](belief_retention_class.md), beliefs vary in expected lifetime: `fact` rows describe stable codebase state; `snapshot` rows are research-thought; `transient` rows are PR-window scratch. The retrieval pack treats them identically. Type-aware compression spends fewer tokens per `snapshot` and `transient` belief so more total beliefs fit, without sacrificing fidelity on the `fact` rows that carry load.
+
+The compressor is **structural**: deterministic, byte-stable, no LLM. The contract is "same belief input → same compressed output, byte-for-byte" — issue #434 acceptance #3 makes this explicit, and is the single line that distinguishes this from a summariser.
+
+---
+
+## Contract
+
+```python
+from aelfrice.compression import compress_for_retrieval, CompressedBelief
+
+cb: CompressedBelief = compress_for_retrieval(belief, *, locked: bool)
+```
+
+`CompressedBelief` is a small wrapper around the original `Belief` plus its compressed form:
+
+```python
+@dataclass(frozen=True)
+class CompressedBelief:
+    belief: Belief
+    rendered: str          # the post-compression text the pack emits
+    rendered_tokens: int   # _estimate_tokens(rendered)
+    strategy: str          # "verbatim" | "headline" | "stub"
+```
+
+Inputs:
+
+- `belief: Belief` — the belief as returned by the lane fan-out.
+- `locked: bool` — `True` when `belief.lock_state == LOCK_USER` or the caller has otherwise classified the belief as L0. Locks always render `"verbatim"` regardless of `retention_class`.
+
+Output: a single `CompressedBelief` whose `rendered_tokens ≤ _belief_tokens(belief)` (i.e. compression is monotone-non-increasing in cost). The compressor is total: every `(belief, locked)` pair returns a `CompressedBelief`.
+
+The function is pure and deterministic. No store reads, no clock reads, no env reads, no random.
+
+---
+
+## Strategy table
+
+| Retention class | `locked=True` | `locked=False` | Notes |
+|---|---|---|---|
+| `fact` | verbatim | verbatim | Always full-text. The class is "stable codebase state"; truncation here is a fidelity loss. |
+| `snapshot` | verbatim | **headline** | Headline = first sentence (split on `. ` / `.\n`), with a trailing `…` when truncated. |
+| `transient` | verbatim | **stub** | Stub = belief id + retention class only, no content. Compressed-out content is recoverable via `store.get_belief(id)` if the consumer needs it. |
+| `unknown` | verbatim | verbatim | Migration-safety. Don't compress beliefs whose retention class hasn't been classified yet — that is a separate audit (#290 §6) before type-aware compression can fire on them. |
+
+The `locked → verbatim` rule mirrors the existing rule that L0 beliefs are never trimmed (`retrieval.py:950`: *"L0 beliefs are never trimmed."*) — locks override every retention-class decision the same way they override hibernation (#196) and the relevance floor (#289).
+
+### Headline strategy details
+
+- Split on the first occurrence of `. ` or `.\n` outside a balanced code-fence span (`` ``` ``). Code fences are preserved as a unit so we don't truncate mid-fence and produce un-renderable Markdown.
+- If no sentence boundary is found within the first `MAX_HEADLINE_CHARS = 240` characters, hard-truncate at the last whitespace boundary ≤ 240 and append `…`.
+- A belief whose `len(content) ≤ MAX_HEADLINE_CHARS` and contains no internal sentence boundary renders verbatim — the headline strategy never makes the rendered text *longer* than the source.
+
+### Stub strategy details
+
+- `rendered = "[stub: belief={id} class=transient]"` — fixed format, no payload.
+- Tokens: `_estimate_tokens` over the literal string above. At `_CHARS_PER_TOKEN = 4` and a typical `belief.id` length (~12 chars), that is ~10 tokens regardless of belief content size.
+- The stub is **deliberately recognizable** so a downstream consumer that wants the full content can detect the marker and call `store.get_belief(id)` against the original. This is the "compressed-out but not deleted" semantic.
+
+---
+
+## Where compression sits
+
+The compressor runs **after** lane fan-out and ranking, **before** the budget pack. The current pack loop at `retrieval.py:1048-1085` (and the parallel block at `:1197-1232`) reads:
+
+```python
+used: int = locked_used + sum(_belief_tokens(b) for b in l25)
+for b in candidates:
+    cost: int = _belief_tokens(b)
+    if used + cost > token_budget:
+        break
+    out.append(b)
+    used += cost
+```
+
+The compressed-pack rewrites this to:
+
+```python
+used: int = locked_used + sum(cb.rendered_tokens for cb in compressed_l25)
+for cb in compressed_candidates:
+    if used + cb.rendered_tokens > token_budget:
+        break
+    out.append(cb)              # CompressedBelief, not raw Belief
+    used += cb.rendered_tokens
+```
+
+`compressed_candidates` is the post-compression sequence produced by mapping `compress_for_retrieval(b, locked=...)` over the ranked candidate list. The same applies to `compressed_l25`.
+
+### Configuration
+
+A new `use_type_aware_compression` flag follows the established convention at `retrieval.py:118-131`:
+
+1. `retrieve(..., use_type_aware_compression=True)` kwarg (highest precedence).
+2. `AELFRICE_TYPE_AWARE_COMPRESSION=1` env var.
+3. `[retrieval] use_type_aware_compression = true` in `.aelfrice.toml`.
+4. Default OFF at v2.0.0 until bench-gate clears (acceptance #2).
+
+When OFF, the compressor short-circuits to `verbatim` for every belief, making `rendered_tokens == _belief_tokens(belief)` and `out` element-equivalent to today's `Belief` list (a `CompressedBelief` with `strategy="verbatim"` is rendered identically to its `belief.content`). This preserves the v1.x retrieval contract under the default.
+
+### `RetrievalResult` shape
+
+`RetrievalResult.beliefs` today is `list[Belief]`. The minimum viable change is to add a parallel `list[CompressedBelief]` field (e.g. `compressed_beliefs`) and leave `beliefs` populated for back-compat. A consumer that wants the compressed render reads `compressed_beliefs[i].rendered`; a consumer that wants the raw `Belief` keeps reading `beliefs[i]`. Both lists are the same length and ordered the same way.
+
+The implementation PR may instead replace `Belief` with `CompressedBelief` everywhere downstream and add a `cb.belief` accessor; that is a wider rename and is left to the impl PR's decision based on call-site count.
+
+---
+
+## Storage
+
+None. The compressor is a pure function over `(Belief, locked)`. No new schema, no new table, no persistence, no cache.
+
+If a future revision needs to memoise compressed renders (e.g. for very large beliefs), the cache is a per-process `dict[belief_id, CompressedBelief]` keyed off `(belief.id, belief.content_hash)` — invalidated naturally by content-hash change. That is **out of scope at v2.0.0**.
+
+---
+
+## Reconciliation
+
+### vs. the existing tail-trim
+
+The current pack drops beliefs from the tail when `token_budget` is hit (`retrieval.py:744`, `:1052`, `:1202`). Type-aware compression does not replace this; it **runs ahead of it**. Compression reduces per-belief cost; tail-trim still fires on the post-compression cost list when it must. The two compose: a `transient` belief that costs ~10 tokens after stub-compression rarely gets tail-trimmed; a verbatim `fact` belief still gets trimmed if the budget is pathologically tight.
+
+### vs. retention-class soft-downweight
+
+Per `docs/belief_retention_class.md` § "Recommendation summary": *"Soft down-weight in ranking, not hard expiry. Beliefs don't expire; their `final_score` gets a class-and-age multiplier before the floor (#289) is applied."* That is a **ranking-stage** mechanism — it changes what enters the pack. Type-aware compression is a **packing-stage** mechanism — it changes how many of the entered beliefs fit. They compose: ranking decides who is in, compression decides how many get the full-text treatment.
+
+### vs. context rebuilder (#141, v1.4)
+
+The rebuilder consumes retrieval output and writes a continuation-fidelity-scored block at `<git-common-dir>/aelfrice/rebuild_logs/`. With compression on, the rebuilder gets more total beliefs in the same `[rebuilder] token_budget` (default in `context_rebuilder.py:111`). Acceptance #4 makes this measurable: continuation-fidelity uplift at fixed budget.
+
+The rebuilder does not need a separate config knob; it inherits the retrieval-side flag. If `use_type_aware_compression` is on at retrieval call-time, the rebuilder reads compressed output transparently via the `RetrievalResult` shape change above.
+
+### vs. type / source-tier axes
+
+Belief carries three orthogonal axes: `belief.belief_type` (factual / correction / preference / requirement, `models.py:13-22`), `belief.source_kind` (the ingest path), and `belief.retention_class` (this spec's input). The compressor reads only `retention_class`. Other axes are out of scope:
+
+- `belief.belief_type` does not predict expected-lifetime — a `factual` belief can be `transient` (#281 example in `docs/belief_retention_class.md`), and a `preference` belief can be `fact`-class.
+- `belief.source_kind` is upstream of `retention_class` (the per-source defaults in `models.py:_RETENTION_DEFAULT_BY_SOURCE`); the compressor consumes the materialised retention class, not the source.
+
+---
+
+## Acceptance
+
+### A1 — corpus
+
+A labeled `compression_uplift` corpus lives under `tests/corpus/v2_0/compression_uplift/`, mirroring the v2.0 corpus scaffold. Each row encodes a (query, ground-truth-set, mixed-class belief population) triple. The `bench_gated` autouse marker keeps public CI green when `AELFRICE_CORPUS_ROOT` is unset; labeled content lives in the lab repo only, per the published corpus policy.
+
+### A2 — token-budget recovery
+
+At a fixed `token_budget` (default `2400` per `retrieval.py:97 DEFAULT_TOKEN_BUDGET`):
+
+```
+recall@k(use_type_aware_compression=ON)  >  recall@k(use_type_aware_compression=OFF)
+```
+
+on the `compression_uplift` fixture. Threshold: **strictly positive uplift** (issue acceptance #2 phrases this as "measurably more recovered facts"). The threshold is positive, not a fixed magnitude — the bench gates ship-or-defer, not headline-number-matches.
+
+### A3 — determinism
+
+`compress_for_retrieval(b, locked=L)` is byte-stable across processes. A property test asserts:
+
+```python
+@hypothesis.given(belief_strategy(), st.booleans())
+def test_compress_deterministic(b, locked):
+    assert compress_for_retrieval(b, locked=locked).rendered \
+        == compress_for_retrieval(b, locked=locked).rendered
+```
+
+A second test compares against a fixture of `(belief_id, expected_rendered)` pairs checked into the public test suite, so a refactor that perturbs the rendering breaks the test.
+
+### A4 — rebuilder fidelity
+
+The continuation-fidelity scorer (#141 v1.4 deliverable) is run on the rebuild_logs corpus with `use_type_aware_compression={OFF, ON}`. Bench-gate: ON ≥ OFF on continuation-fidelity score at the same `[rebuilder] token_budget`. Tolerance band: `≥ baseline − 0.005` (a half-point of the fidelity-score noise floor, mirroring the BM25F bench-gate band at #154).
+
+### A5 — composition tracker
+
+The #154 composition tracker doc gains a row for `use_type_aware_compression`: input shape, output shape, where it sits, bench verdict. Like the vocab bridge (#433), this is **not a lane** — it is a packing-stage transform. The tracker row is present for operator clarity.
+
+---
+
+## Bench-gate / ship-or-defer policy
+
+`needs-spec` → `bench-gated` once this spec lands. Implementation is the next gate. **The implementation PR ships only on positive bench evidence per A2 and A4.** A mechanically-correct implementation that fails either ships merged-but-default-OFF or gets reverted; it does not ship default-ON without a benchmark cut.
+
+---
+
+## Out of scope at v2.0.0
+
+- **LLM-based compression / summarisation.** Determinism (A3) is an explicit acceptance criterion. Any path that calls a model is excluded.
+- **Compression of `fact`-class beliefs.** They render verbatim. If `fact` is the dominant retention class in a real-world store, type-aware compression has nothing to do — that is the correct behaviour for a fact-heavy corpus, not a bug.
+- **Compression of locked beliefs.** Locks override retention class. This mirrors the L0-never-trimmed rule.
+- **Per-belief-type strategies.** `belief.belief_type` is not consulted. If a future revision wants type-axis compression (e.g. always-verbatim-for-`requirement`), that is a separate spec; this one closes the retention-class axis only.
+- **Online / streaming compression.** The compressor is called over the full ranked candidate list. Streaming compress-as-rank is out of scope.
+- **`MAX_HEADLINE_CHARS` tuning.** 240 is the spec default. Per-store tuning via `[retrieval] headline_max_chars` is implementable but **deferred** — ship one default first; tune on bench evidence.
+
+---
+
+## Implementation prereqs
+
+- `src/aelfrice/models.py` — `RETENTION_*` constants, `Belief.retention_class` field. Shipped v1.6.0 (#290).
+- `src/aelfrice/retrieval.py:215` — `_belief_tokens(b)` and `_estimate_tokens()`. The compressor reuses the same estimator on `rendered`.
+- `src/aelfrice/retrieval.py:118-131` — flag-resolution convention.
+- `src/aelfrice/retrieval.py:1048-1085, :1197-1232` — pack loops to rewrite.
+- `src/aelfrice/context_rebuilder.py` — consumer of compressed output for A4.
+- `tests/corpus/v2_0/` — corpus scaffold + autouse `bench_gated` marker. Shipped v1.6.0 (#307 / #311).
+- `tests/bench_gate/` — harness. Shipped v1.6.0 (#319 / #320).
+
+All substrate is on `main` as of `68dafc0`. No new dependencies. No schema changes.
+
+---
+
+## Open questions for review
+
+1. **Stub format.** `"[stub: belief={id} class=transient]"` is one option. Alternatives: empty string (zero-cost but indistinguishable from a deleted belief); JSON-shaped marker; emoji prefix. The spec defaults to the bracketed form because it is grep-friendly and survives prose mixing. Pick at impl-PR review.
+2. **`CompressedBelief` vs. parallel-list shape.** Two options under "RetrievalResult shape" above. The wider rename is correct long-term; the parallel field is faster to land. The impl PR should pick based on consumer count (a grep for `RetrievalResult.beliefs` will tell).
+3. **Headline-strategy on code-fence-only content.** A belief whose entire content is one ``` ```python ... ``` `` block has no sentence boundaries by the §"Headline strategy details" rule. The fallback (hard-truncate at 240 chars on whitespace) preserves Markdown well-formedness only by accident. Consider a "content-is-code-only → verbatim" override; settle in impl PR.
+4. **Promotion path interaction.** `aelf doctor --promote-retention` (`doctor.py:1138`) flips a belief from `snapshot` to `fact`. A belief mid-promotion shifts compression strategy from `headline` to `verbatim`. This is correct behaviour (the belief just earned full-text rendering), but it makes deterministic regression tests across promotion-state-changes brittle. The A3 fixture should pin retention class explicitly per row, not rely on whatever `retention_class_for_source` chose at fixture-build time.