diff --git a/docs/feature-type-aware-compression.md b/docs/feature-type-aware-compression.md new file mode 100644 index 00000000..49ac0470 --- /dev/null +++ b/docs/feature-type-aware-compression.md @@ -0,0 +1,227 @@ +# Feature spec: Type-aware compression (#434) + +**Status:** spec, no implementation +**Issue:** #434 +**Recovery-inventory line:** [`docs/ROADMAP.md`](ROADMAP.md) — *"Type-aware compression | v2.0.0"* +**Substrate prereqs:** #290 (retention class column + per-source defaults, shipped v1.6.0), #141 (context rebuilder, shipped v1.4.0) + +--- + +## Purpose + +At a fixed retrieval `token_budget`, today's path either includes a belief verbatim or trims it from the tail. Per [`docs/belief_retention_class.md`](belief_retention_class.md), beliefs vary in expected lifetime: `fact` rows describe stable codebase state; `snapshot` rows are research-thought; `transient` rows are PR-window scratch. The retrieval pack treats them identically. Type-aware compression spends fewer tokens per `snapshot` and `transient` belief so more total beliefs fit, without sacrificing fidelity on the `fact` rows that carry load. + +The compressor is **structural**: deterministic, byte-stable, no LLM. The contract is "same belief input → same compressed output, byte-for-byte" — issue #434 acceptance #3 makes this explicit, and is the single line that distinguishes this from a summariser. + +--- + +## Contract + +```python +from aelfrice.compression import compress_for_retrieval, CompressedBelief + +cb: CompressedBelief = compress_for_retrieval(belief, *, locked: bool) +``` + +`CompressedBelief` is a small wrapper around the original `Belief` plus its compressed form: + +```python +@dataclass(frozen=True) +class CompressedBelief: + belief: Belief + rendered: str # the post-compression text the pack emits + rendered_tokens: int # _estimate_tokens(rendered) + strategy: str # "verbatim" | "headline" | "stub" +``` + +Inputs: + +- `belief: Belief` — the belief as returned by the lane fan-out. +- `locked: bool` — `True` when `belief.lock_state == LOCK_USER` or the caller has otherwise classified the belief as L0. Locks always render `"verbatim"` regardless of `retention_class`. + +Output: a single `CompressedBelief` whose `rendered_tokens ≤ _belief_tokens(belief)` (i.e. compression is monotone-non-increasing in cost). The compressor is total: every `(belief, locked)` pair returns a `CompressedBelief`. + +The function is pure and deterministic. No store reads, no clock reads, no env reads, no random. + +--- + +## Strategy table + +| Retention class | `locked=True` | `locked=False` | Notes | +|---|---|---|---| +| `fact` | verbatim | verbatim | Always full-text. The class is "stable codebase state"; truncation here is a fidelity loss. | +| `snapshot` | verbatim | **headline** | Headline = first sentence (split on `. ` / `.\n`), with a trailing `…` when truncated. | +| `transient` | verbatim | **stub** | Stub = belief id + retention class only, no content. Compressed-out content is recoverable via `store.get_belief(id)` if the consumer needs it. | +| `unknown` | verbatim | verbatim | Migration-safety. Don't compress beliefs whose retention class hasn't been classified yet — that is a separate audit (#290 §6) before type-aware compression can fire on them. | + +The `locked → verbatim` rule mirrors the existing rule that L0 beliefs are never trimmed (`retrieval.py:950`: *"L0 beliefs are never trimmed."*) — locks override every retention-class decision the same way they override hibernation (#196) and the relevance floor (#289). + +### Headline strategy details + +- Split on the first occurrence of `. ` or `.\n` outside a balanced code-fence span (`` ``` ``). Code fences are preserved as a unit so we don't truncate mid-fence and produce un-renderable Markdown. +- If no sentence boundary is found within the first `MAX_HEADLINE_CHARS = 240` characters, hard-truncate at the last whitespace boundary ≤ 240 and append `…`. +- A belief whose `len(content) ≤ MAX_HEADLINE_CHARS` and contains no internal sentence boundary renders verbatim — the headline strategy never makes the rendered text *longer* than the source. + +### Stub strategy details + +- `rendered = "[stub: belief={id} class=transient]"` — fixed format, no payload. +- Tokens: `_estimate_tokens` over the literal string above. At `_CHARS_PER_TOKEN = 4` and a typical `belief.id` length (~12 chars), that is ~10 tokens regardless of belief content size. +- The stub is **deliberately recognizable** so a downstream consumer that wants the full content can detect the marker and call `store.get_belief(id)` against the original. This is the "compressed-out but not deleted" semantic. + +--- + +## Where compression sits + +The compressor runs **after** lane fan-out and ranking, **before** the budget pack. The current pack loop at `retrieval.py:1048-1085` (and the parallel block at `:1197-1232`) reads: + +```python +used: int = locked_used + sum(_belief_tokens(b) for b in l25) +for b in candidates: + cost: int = _belief_tokens(b) + if used + cost > token_budget: + break + out.append(b) + used += cost +``` + +The compressed-pack rewrites this to: + +```python +used: int = locked_used + sum(cb.rendered_tokens for cb in compressed_l25) +for cb in compressed_candidates: + if used + cb.rendered_tokens > token_budget: + break + out.append(cb) # CompressedBelief, not raw Belief + used += cb.rendered_tokens +``` + +`compressed_candidates` is the post-compression sequence produced by mapping `compress_for_retrieval(b, locked=...)` over the ranked candidate list. The same applies to `compressed_l25`. + +### Configuration + +A new `use_type_aware_compression` flag follows the established convention at `retrieval.py:118-131`: + +1. `retrieve(..., use_type_aware_compression=True)` kwarg (highest precedence). +2. `AELFRICE_TYPE_AWARE_COMPRESSION=1` env var. +3. `[retrieval] use_type_aware_compression = true` in `.aelfrice.toml`. +4. Default OFF at v2.0.0 until bench-gate clears (acceptance #2). + +When OFF, the compressor short-circuits to `verbatim` for every belief, making `rendered_tokens == _belief_tokens(belief)` and `out` element-equivalent to today's `Belief` list (a `CompressedBelief` with `strategy="verbatim"` is rendered identically to its `belief.content`). This preserves the v1.x retrieval contract under the default. + +### `RetrievalResult` shape + +`RetrievalResult.beliefs` today is `list[Belief]`. The minimum viable change is to add a parallel `list[CompressedBelief]` field (e.g. `compressed_beliefs`) and leave `beliefs` populated for back-compat. A consumer that wants the compressed render reads `compressed_beliefs[i].rendered`; a consumer that wants the raw `Belief` keeps reading `beliefs[i]`. Both lists are the same length and ordered the same way. + +The implementation PR may instead replace `Belief` with `CompressedBelief` everywhere downstream and add a `cb.belief` accessor; that is a wider rename and is left to the impl PR's decision based on call-site count. + +--- + +## Storage + +None. The compressor is a pure function over `(Belief, locked)`. No new schema, no new table, no persistence, no cache. + +If a future revision needs to memoise compressed renders (e.g. for very large beliefs), the cache is a per-process `dict[belief_id, CompressedBelief]` keyed off `(belief.id, belief.content_hash)` — invalidated naturally by content-hash change. That is **out of scope at v2.0.0**. + +--- + +## Reconciliation + +### vs. the existing tail-trim + +The current pack drops beliefs from the tail when `token_budget` is hit (`retrieval.py:744`, `:1052`, `:1202`). Type-aware compression does not replace this; it **runs ahead of it**. Compression reduces per-belief cost; tail-trim still fires on the post-compression cost list when it must. The two compose: a `transient` belief that costs ~10 tokens after stub-compression rarely gets tail-trimmed; a verbatim `fact` belief still gets trimmed if the budget is pathologically tight. + +### vs. retention-class soft-downweight + +Per `docs/belief_retention_class.md` § "Recommendation summary": *"Soft down-weight in ranking, not hard expiry. Beliefs don't expire; their `final_score` gets a class-and-age multiplier before the floor (#289) is applied."* That is a **ranking-stage** mechanism — it changes what enters the pack. Type-aware compression is a **packing-stage** mechanism — it changes how many of the entered beliefs fit. They compose: ranking decides who is in, compression decides how many get the full-text treatment. + +### vs. context rebuilder (#141, v1.4) + +The rebuilder consumes retrieval output and writes a continuation-fidelity-scored block at `/aelfrice/rebuild_logs/`. With compression on, the rebuilder gets more total beliefs in the same `[rebuilder] token_budget` (default in `context_rebuilder.py:111`). Acceptance #4 makes this measurable: continuation-fidelity uplift at fixed budget. + +The rebuilder does not need a separate config knob; it inherits the retrieval-side flag. If `use_type_aware_compression` is on at retrieval call-time, the rebuilder reads compressed output transparently via the `RetrievalResult` shape change above. + +### vs. type / source-tier axes + +Belief carries three orthogonal axes: `belief.belief_type` (factual / correction / preference / requirement, `models.py:13-22`), `belief.source_kind` (the ingest path), and `belief.retention_class` (this spec's input). The compressor reads only `retention_class`. Other axes are out of scope: + +- `belief.belief_type` does not predict expected-lifetime — a `factual` belief can be `transient` (#281 example in `docs/belief_retention_class.md`), and a `preference` belief can be `fact`-class. +- `belief.source_kind` is upstream of `retention_class` (the per-source defaults in `models.py:_RETENTION_DEFAULT_BY_SOURCE`); the compressor consumes the materialised retention class, not the source. + +--- + +## Acceptance + +### A1 — corpus + +A labeled `compression_uplift` corpus lives under `tests/corpus/v2_0/compression_uplift/`, mirroring the v2.0 corpus scaffold. Each row encodes a (query, ground-truth-set, mixed-class belief population) triple. The `bench_gated` autouse marker keeps public CI green when `AELFRICE_CORPUS_ROOT` is unset; labeled content lives in the lab repo only, per the published corpus policy. + +### A2 — token-budget recovery + +At a fixed `token_budget` (default `2400` per `retrieval.py:97 DEFAULT_TOKEN_BUDGET`): + +``` +recall@k(use_type_aware_compression=ON) > recall@k(use_type_aware_compression=OFF) +``` + +on the `compression_uplift` fixture. Threshold: **strictly positive uplift** (issue acceptance #2 phrases this as "measurably more recovered facts"). The threshold is positive, not a fixed magnitude — the bench gates ship-or-defer, not headline-number-matches. + +### A3 — determinism + +`compress_for_retrieval(b, locked=L)` is byte-stable across processes. A property test asserts: + +```python +@hypothesis.given(belief_strategy(), st.booleans()) +def test_compress_deterministic(b, locked): + assert compress_for_retrieval(b, locked=locked).rendered \ + == compress_for_retrieval(b, locked=locked).rendered +``` + +A second test compares against a fixture of `(belief_id, expected_rendered)` pairs checked into the public test suite, so a refactor that perturbs the rendering breaks the test. + +### A4 — rebuilder fidelity + +The continuation-fidelity scorer (#141 v1.4 deliverable) is run on the rebuild_logs corpus with `use_type_aware_compression={OFF, ON}`. Bench-gate: ON ≥ OFF on continuation-fidelity score at the same `[rebuilder] token_budget`. Tolerance band: `≥ baseline − 0.005` (a half-point of the fidelity-score noise floor, mirroring the BM25F bench-gate band at #154). + +### A5 — composition tracker + +The #154 composition tracker doc gains a row for `use_type_aware_compression`: input shape, output shape, where it sits, bench verdict. Like the vocab bridge (#433), this is **not a lane** — it is a packing-stage transform. The tracker row is present for operator clarity. + +--- + +## Bench-gate / ship-or-defer policy + +`needs-spec` → `bench-gated` once this spec lands. Implementation is the next gate. **The implementation PR ships only on positive bench evidence per A2 and A4.** A mechanically-correct implementation that fails either ships merged-but-default-OFF or gets reverted; it does not ship default-ON without a benchmark cut. + +--- + +## Out of scope at v2.0.0 + +- **LLM-based compression / summarisation.** Determinism (A3) is an explicit acceptance criterion. Any path that calls a model is excluded. +- **Compression of `fact`-class beliefs.** They render verbatim. If `fact` is the dominant retention class in a real-world store, type-aware compression has nothing to do — that is the correct behaviour for a fact-heavy corpus, not a bug. +- **Compression of locked beliefs.** Locks override retention class. This mirrors the L0-never-trimmed rule. +- **Per-belief-type strategies.** `belief.belief_type` is not consulted. If a future revision wants type-axis compression (e.g. always-verbatim-for-`requirement`), that is a separate spec; this one closes the retention-class axis only. +- **Online / streaming compression.** The compressor is called over the full ranked candidate list. Streaming compress-as-rank is out of scope. +- **`MAX_HEADLINE_CHARS` tuning.** 240 is the spec default. Per-store tuning via `[retrieval] headline_max_chars` is implementable but **deferred** — ship one default first; tune on bench evidence. + +--- + +## Implementation prereqs + +- `src/aelfrice/models.py` — `RETENTION_*` constants, `Belief.retention_class` field. Shipped v1.6.0 (#290). +- `src/aelfrice/retrieval.py:215` — `_belief_tokens(b)` and `_estimate_tokens()`. The compressor reuses the same estimator on `rendered`. +- `src/aelfrice/retrieval.py:118-131` — flag-resolution convention. +- `src/aelfrice/retrieval.py:1048-1085, :1197-1232` — pack loops to rewrite. +- `src/aelfrice/context_rebuilder.py` — consumer of compressed output for A4. +- `tests/corpus/v2_0/` — corpus scaffold + autouse `bench_gated` marker. Shipped v1.6.0 (#307 / #311). +- `tests/bench_gate/` — harness. Shipped v1.6.0 (#319 / #320). + +All substrate is on `main` as of `68dafc0`. No new dependencies. No schema changes. + +--- + +## Open questions for review + +1. **Stub format.** `"[stub: belief={id} class=transient]"` is one option. Alternatives: empty string (zero-cost but indistinguishable from a deleted belief); JSON-shaped marker; emoji prefix. The spec defaults to the bracketed form because it is grep-friendly and survives prose mixing. Pick at impl-PR review. +2. **`CompressedBelief` vs. parallel-list shape.** Two options under "RetrievalResult shape" above. The wider rename is correct long-term; the parallel field is faster to land. The impl PR should pick based on consumer count (a grep for `RetrievalResult.beliefs` will tell). +3. **Headline-strategy on code-fence-only content.** A belief whose entire content is one ``` ```python ... ``` `` block has no sentence boundaries by the §"Headline strategy details" rule. The fallback (hard-truncate at 240 chars on whitespace) preserves Markdown well-formedness only by accident. Consider a "content-is-code-only → verbatim" override; settle in impl PR. +4. **Promotion path interaction.** `aelf doctor --promote-retention` (`doctor.py:1138`) flips a belief from `snapshot` to `fact`. A belief mid-promotion shifts compression strategy from `headline` to `verbatim`. This is correct behaviour (the belief just earned full-text rendering), but it makes deterministic regression tests across promotion-state-changes brittle. The A3 fixture should pin retention class explicitly per row, not rely on whatever `retention_class_for_source` chose at fixture-build time.