feat(retrieval): use_type_aware_compression flips to default-on (#769) [stacked on #879]

CHANGELOG/v3.md

@@ -8,6 +8,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- **`use_type_aware_compression` flips to default-on** ([#769](https://github.com/robotrocketscience/aelfrice/issues/769)). The A2 + A4 bench gates (`docs/design/feature-type-aware-compression.md` §"Bench-gate / ship-or-defer policy") cleared on the lab-side `compression_a*` corpora — A2 mean recall@k uplift +0.3267 on n=25 with zero per-row regressions; A4 mean fidelity delta +0.0085 on n=15 with zero per-row regressions. Per-class non-regression rider (`drop` / `summarise` / `verbatim` each non-regressing) held. Per the `resolve_use_type_aware_compression()` docstring contract, the default is unblocked. Compose-compatibility with the (already default-on) `use_intentional_clustering` flag was the structural prerequisite; that shipped in [#878](https://github.com/robotrocketscience/aelfrice/issues/878) via the `pack_with_clusters` `cost_fn` seam. Precedence (env > kwarg > TOML > default) is unchanged; only the default value flips `False → True`. Reversible for v2.x parity via `[retrieval] use_type_aware_compression = false` in `.aelfrice.toml` or `AELFRICE_TYPE_AWARE_COMPRESSION=0`. Tests updated, mirroring the [#436](https://github.com/robotrocketscience/aelfrice/pull/436) default-flip precedent: `test_default_is_off → test_default_is_on`; `test_pack_byte_identical_when_flag_off → test_pack_byte_identical_when_flag_on` (ON-byte-identity invariant supersedes the prior OFF-byte-identity invariant); `test_retrieve_env_var_enables_compression → test_retrieve_env_var_disables_compression` (the env-var path now tests rollback to OFF rather than activation, matching the post-flip precedence semantics); `test_default_call_leaves_compressed_empty → test_default_call_populates_compressed`. Doc surfaces (`feature-type-aware-compression.md` status banner, `docs/user/CONFIG.md` TOML example + flag-doc section, `docs/concepts/ROADMAP.md` v3.0 row) corrected in a docs commit.
+
 ### Added
 
 - **`use_intentional_clustering` × `use_type_aware_compression` compose-reconciliation** ([#878](https://github.com/robotrocketscience/aelfrice/issues/878), unblocks [#769](https://github.com/robotrocketscience/aelfrice/issues/769) flip-default). `retrieve_v2` used to raise `ValueError` when both flags resolved True — the cluster pack accounted in raw `_belief_tokens` while the surrounding pack loop accounted in the compressed render's `rendered_tokens`, and composing two different cost models would have drifted the byte budget. `pack_with_clusters` (`src/aelfrice/clustering.py`) now accepts a `cost_fn: Callable[[Belief], int] | None = None` parameter; `retrieve_v2` passes its existing `_cost` closure through that seam so both arms account in the same currency (raw token estimate or compressed `rendered_tokens` depending on `compress_on`). The `cluster_on and compress_on` mutex raise is dropped. File-level comment and `retrieve_v2` docstring updated to reflect compose-compatibility. Test surface: `test_mutual_exclusion_with_compression_raises` replaced with `test_composes_with_compression` (composed path produces non-empty `beliefs` and parallel `compressed_beliefs`); `test_compressed_beliefs_populated_when_flag_on` drops its `use_intentional_clustering=False` workaround so the default-on clustering path is the one under test. Determinism ([#605](https://github.com/robotrocketscience/aelfrice/issues/605)) preserved: `cost_fn` defaults to the raw estimator when unset, so callers that don't opt into compression see byte-identical pack behaviour from this commit. Named "v2.x follow-up" by [#436](https://github.com/robotrocketscience/aelfrice/pull/436)'s default-on flip body — landing it here unblocks [#769](https://github.com/robotrocketscience/aelfrice/issues/769) (compression default-on flip) which was gated empirically (bench evidence A2 +0.3267 / A4 +0.0085) but structurally blocked on this reconciliation.

docs/concepts/ROADMAP.md

@@ -150,7 +150,7 @@ Substrate completion (all shipped):
 
 - **HRR persistence default-ON + split-format save/load** ([#553](https://github.com/robotrocketscience/aelfrice/issues/553)). `HRRStructIndex.save()` writes a per-store directory with `struct.npy` + `meta.npz`; legacy `.npz` bundles still load with a deprecation warning. Persistence defaults on; opt out via `[retrieval] hrr_persist = false` or `AELFRICE_HRR_PERSIST=0`. Ephemeral paths auto-disable.
 - **Wonder lifecycle completion** ([#542](https://github.com/robotrocketscience/aelfrice/issues/542) umbrella). Phantom promotion Surface A + Surface B ([#550](https://github.com/robotrocketscience/aelfrice/issues/550) + [#616](https://github.com/robotrocketscience/aelfrice/issues/616)) per the 2026-05-11 ratification (no count-trigger). Skill-layer subagent dispatch → `wonder_ingest` ([#552](https://github.com/robotrocketscience/aelfrice/issues/552)). Wonder/reason parity ([#645](https://github.com/robotrocketscience/aelfrice/issues/645)): Verdict/ImpasseKind classifiers, ConsequencePath fork-on-CONTRADICTS, VERDICT-driven dispatch + suggested-updates close-the-loop. `aelf wonder QUERY` positional defaults to the axes flow.
-- **Type-aware compression A2 bench gate** ([#434](https://github.com/robotrocketscience/aelfrice/issues/434)). `run_compression_a2_uplift` driver landed; strict positive `mean_recall@k(use_type_aware_compression=ON) > OFF` gate in place. Rebuilder continuation-fidelity (A4) remains the flip-default gate for the next minor.
+- **Type-aware compression A2 bench gate** ([#434](https://github.com/robotrocketscience/aelfrice/issues/434)). `run_compression_a2_uplift` driver landed; strict positive `mean_recall@k(use_type_aware_compression=ON) > OFF` gate in place. Rebuilder continuation-fidelity (A4) cleared in the same campaign; default-on flip landed in [#769](https://github.com/robotrocketscience/aelfrice/issues/769) after the [#878](https://github.com/robotrocketscience/aelfrice/issues/878) compose-reconciliation with `use_intentional_clustering`.
 - **Eval-harness completion** ([#592](https://github.com/robotrocketscience/aelfrice/issues/592), [#600](https://github.com/robotrocketscience/aelfrice/issues/600), [#687](https://github.com/robotrocketscience/aelfrice/issues/687)). Host-agent replay path writes/joins per-run JSONL; opt-in LLM-judge stage scores open-ended turns at the operator's anchor tier; Cohen's-κ runner gates inter-judge agreement ≥ 0.70 plus hot-start fidelity ≥ 0.80. Synthetic `hot_start` fixture covers post-compact "where were we?" prompts.
 - **Read-only federation** ([#650](https://github.com/robotrocketscience/aelfrice/issues/650), [#655](https://github.com/robotrocketscience/aelfrice/issues/655), [#688](https://github.com/robotrocketscience/aelfrice/issues/688), [#689](https://github.com/robotrocketscience/aelfrice/issues/689), [#690](https://github.com/robotrocketscience/aelfrice/issues/690), [#713](https://github.com/robotrocketscience/aelfrice/issues/713)). `scope` field on beliefs (`project` / `global` / `shared:<name>`); peer DB FTS5 + BFS visible through `knowledge_deps.json`; `aelf promote --to-scope` flips visibility; mutations against foreign belief IDs raise `ForeignBeliefError`. `aelf reason` annotates peer hops with `[scope:<name>]`.
 - **`query_strategy` default flip** ([#718](https://github.com/robotrocketscience/aelfrice/issues/718)). `DEFAULT_STRATEGY` flipped `legacy-bm25` → `stack-r1-r3` on bench evidence (+0.2851 absolute NDCG@k, +94.8%, p99 latency 13% of the 5 ms budget). `legacy-bm25` remains callable via explicit kwarg until PR-4 removes the code path one minor release out.

docs/design/feature-type-aware-compression.md

@@ -1,6 +1,6 @@
 # Feature spec: Type-aware compression (#434)
 
-**Status:** implemented behind default-OFF flag; pack-loop budget rewrite landed; lab bench A2 / A4 pending
+**Status:** default-ON since #769 (A2 + A4 bench gates cleared on the lab-side `compression_a*` corpora — A2 mean recall@k uplift +0.3267 / n=25 / 0 regressions; A4 mean fidelity delta +0.0085 / n=15 / 0 regressions). Composes with `use_intentional_clustering` since #878.
 **Issue:** #434
 **Recovery-inventory line:** [`docs/concepts/ROADMAP.md`](../concepts/ROADMAP.md) — *"Type-aware compression | v2.0.0"*
 **Substrate prereqs:** #290 (retention class column + per-source defaults, shipped v1.6.0), #141 (context rebuilder, shipped v1.4.0)

docs/user/CONFIG.md

@@ -9,7 +9,7 @@ This is the reference for power users whose project has a documentation idiom or
 A single optional TOML file at the root of a project (or any ancestor). It exposes two power-user surfaces:
 
 - `[noise]` — onboard-time belief filter. Changes how `aelf onboard` ingests beliefs; nothing else.
-- `[retrieval]` (v1.3+) — retrieval-time tier toggles + ranking. Knobs: `entity_index_enabled` (L2.5), `bfs_enabled` (L3), `posterior_weight` (partial Bayesian-weighted L1 ranking), `use_bm25f_anchors` (BM25F-with-anchor-text since v1.7), `use_heat_kernel` (authority scoring lane, default-on since v2.1), `use_hrr_structural` (HRR structural-query lane, default-on since v2.1), `hrr_persist` (HRR structural-index on-disk persistence, default-on since v3.0), `use_type_aware_compression` (per-belief retention-class compression, opt-in since v2.1), `use_intentional_clustering` (co-locating related beliefs, default-on since v3.0). Two placeholder flags (`use_signed_laplacian`, `use_posterior_ranking`) are recognised but emit a deprecation warning if set — their lanes have not yet shipped.
+- `[retrieval]` (v1.3+) — retrieval-time tier toggles + ranking. Knobs: `entity_index_enabled` (L2.5), `bfs_enabled` (L3), `posterior_weight` (partial Bayesian-weighted L1 ranking), `use_bm25f_anchors` (BM25F-with-anchor-text since v1.7), `use_heat_kernel` (authority scoring lane, default-on since v2.1), `use_hrr_structural` (HRR structural-query lane, default-on since v2.1), `hrr_persist` (HRR structural-index on-disk persistence, default-on since v3.0), `use_type_aware_compression` (per-belief retention-class compression, default-on since #769), `use_intentional_clustering` (co-locating related beliefs, default-on since v3.0). Two placeholder flags (`use_signed_laplacian`, `use_posterior_ranking`) are recognised but emit a deprecation warning if set — their lanes have not yet shipped.
 - `[rebuilder]` (v1.7+) — context-rebuilder knobs. Selects the query-understanding stack (`query_strategy`) and sets token-budget floors for the session-scoped and L1 belief lanes (`[rebuild_floor] session` and `[rebuild_floor] l1`).
 - `[feedback]` (v3.0+) — feedback-lane opt-ins. `sentiment_from_prose` (default `false`) wires the sentiment-feedback detector into `UserPromptSubmit` (#606).
 - `[user_prompt_submit_hook]` (v3.0+) — UPS hook knobs. `prompt_shape_gate_enabled` (default `true`) gates trivial-prompt and system-envelope short-circuits before BM25 retrieval runs (#674).
@@ -92,15 +92,16 @@ use_hrr_structural = true
 # paths.
 hrr_persist = true
 
-# v2.1+. Default `false`, opt-in. Enables type-aware compression
-# (#434) — populates RetrievalResult.compressed_beliefs with per-
-# belief renderings keyed by retention_class (snapshot → headline,
-# transient → stub, fact + locked → verbatim). The pack-loop budget
-# rewrite that turns the parallel field into recall@k uplift is a
-# follow-up; this flag at v2.1 just exposes the mechanism behind a
-# default-OFF gate. AELFRICE_TYPE_AWARE_COMPRESSION=1 env var
-# overrides.
-use_type_aware_compression = false
+# v2.1+ (#434), default `true` since #769 (A2 + A4 bench gates
+# cleared on the lab-side compression_a* corpora). Type-aware
+# compression: populates RetrievalResult.compressed_beliefs with
+# per-belief renderings keyed by retention_class (snapshot →
+# headline, transient → stub, fact + locked → verbatim). The
+# pack-loop budget rewrite accounts in compressed rendered_tokens
+# so a tight budget admits more transient/snapshot beliefs at
+# their stub/headline cost. Composes with use_intentional_clustering
+# since #878. AELFRICE_TYPE_AWARE_COMPRESSION=0 reverts.
+use_type_aware_compression = true
 
 # v3.0+ (#436). Default `true` since the multi-store production sweep
 # cleared 60/60 PASS at p99 0.328 ms (~15-30x margin under the 5 ms A4
@@ -402,7 +403,7 @@ Precedence (first decisive wins): env var `AELFRICE_HRR_PERSIST` (truthy `"1"`/`
 
 ### `use_type_aware_compression`
 
-Boolean, default `false`, opt-in (v2.1+, #434). Populates `RetrievalResult.compressed_beliefs` with per-belief renderings dispatched by `belief.retention_class`:
+Boolean, default `true` since #769 (v2.1+, #434). Populates `RetrievalResult.compressed_beliefs` with per-belief renderings dispatched by `belief.retention_class`:
 
 | Retention class | Locked | Unlocked | Notes |
 |---|---|---|---|
@@ -413,9 +414,9 @@ Boolean, default `false`, opt-in (v2.1+, #434). Populates `RetrievalResult.compr
 
 Compression is pure and deterministic — no store, clock, env, or random reads. The `compressed_beliefs` field is parallel to `beliefs` (same length, same order); consumers that want the raw belief read `.beliefs[i]`, consumers that want the compressed render read `.compressed_beliefs[i].rendered`.
 
-When disabled (default), `compressed_beliefs` is empty and `beliefs` is byte-identical to the v1.x return shape.
+Enabled by default: `compressed_beliefs` is parallel to `beliefs` (same length, same order). To disable for v2.x parity, set the env var or TOML key to `false`; with that, `compressed_beliefs` is empty and the pack accounts in raw `_belief_tokens`.
 
-Precedence (first decisive wins): env var `AELFRICE_TYPE_AWARE_COMPRESSION=0`/`1` > explicit Python kwarg `use_type_aware_compression=<bool>` > TOML `[retrieval] use_type_aware_compression` > default `false`. The default-on flip is gated on the lab-side bench in `tests/bench_gate/test_compression_uplift.py` plus the pack-loop budget rewrite (follow-up).
+Precedence (first decisive wins): env var `AELFRICE_TYPE_AWARE_COMPRESSION=0`/`1` > explicit Python kwarg `use_type_aware_compression=<bool>` > TOML `[retrieval] use_type_aware_compression` > default `true`. The default-on flip landed in #769 after the A2 + A4 bench gates (`docs/design/feature-type-aware-compression.md` §"Bench-gate / ship-or-defer policy") cleared on the lab-side `compression_a*` corpora. Composes with `use_intentional_clustering` since #878.
 
 ### Placeholder flags
 

src/aelfrice/retrieval.py

@@ -1711,9 +1711,13 @@ def resolve_use_type_aware_compression(
       1. AELFRICE_TYPE_AWARE_COMPRESSION env var (truthy / falsy normalised).
       2. Explicit `explicit` kwarg from the caller.
       3. `[retrieval] use_type_aware_compression` in `.aelfrice.toml`.
-      4. Default: False — ships behind the flag at v2.0.0; the bench gate
-         (A2 + A4 in docs/design/feature-type-aware-compression.md) flips the
-         default after lab-side benchmark evidence clears.
+      4. Default: True — flipped from False after the A2 + A4 bench
+         gates (docs/design/feature-type-aware-compression.md) cleared
+         on the lab-side compression_a* corpora (#769: A2 mean recall@k
+         uplift +0.3267 on n=25 with zero per-row regressions; A4 mean
+         fidelity delta +0.0085 on n=15). Compose-compatibility with
+         `use_intentional_clustering` shipped in #878 (pack_with_clusters
+         cost_fn seam); both flags resolving True is supported.
     """
     env = _env_type_aware_compression_override()
     if env is not None:
@@ -1723,7 +1727,7 @@ def resolve_use_type_aware_compression(
     toml_value = _read_toml_flag_for(TYPE_AWARE_COMPRESSION_FLAG, start)
     if toml_value is not None:
         return toml_value
-    return False
+    return True
 
 
 def resolve_use_intentional_clustering(