fix(retrieval): demote or exclude superseded beliefs, both arms behind a flag (#1187)

CHANGELOG/v4.md

@@ -35,6 +35,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **PRs touching only `benchmarks/` or `scripts/` reported a passing pytest matrix without running any tests ([#1160](https://github.com/robotrocketscience/aelfrice/issues/1160)).** `ci.yml` deliberately carries no `paths-ignore`, so the workflow always runs and the required `pytest (3.12)` / `pytest (3.13)` checks report on every PR including docs-only ones (#413/#427); which PRs actually execute the suite is decided inside the job by a `dorny/paths-filter` glob list. That list covered `src/**` and `tests/**` but not `benchmarks/**` or `scripts/**`, while `pythonpath = ["."]` makes both importable from the suite — 27 test modules import `benchmarks` and 6 load a `scripts/` module through `importlib`. A PR editing only those directories therefore took the `echo "No code paths changed"` branch and turned both required checks green having run nothing. The coverage it skipped is real, not nominal: widening `compute_band`'s absolute floor 2× in `benchmarks/tolerance.py` fails two tests in `test_bench_tolerance.py`, and breaking `scripts/check_migration_policy.py` errors 14 in `test_migration_policy.py` — the gate that guards destructive schema migrations. Both would have merged green. `docs/` and `CHANGELOG/` stay out of the filter deliberately, and that exclusion was checked rather than assumed: truncating `README.md` to a quarter of its length leaves the full suite passing, so no test asserts on prose and the docs-only exemption costs no coverage. The new guard **derives** its expectation instead of hand-listing names — it scans `tests/` for top-level in-repo packages actually imported or path-resolved and asserts each is matched by a glob, so a package the suite starts using fails the test until the filter covers it. Its own vacuity is guarded too (an empty scan would satisfy the coverage assertion for free, and the filter parser refuses to run unless it finds exactly one `code:` key), because a hand-maintained list compared against the constants it was copied from is the tautology #1161 hit on the doctor side. One acceptance criterion of the #1160 umbrella; the others remain open.
 - **The `eval-calibration` gate never ran on the code it calibrates ([#1160](https://github.com/robotrocketscience/aelfrice/issues/1160)).** The job pins `aelf eval --json` byte-for-byte against `benchmarks/posterior_ranking/baseline.json`, but its PR `paths:` filter named only `eval_harness.py`, `calibration_metrics.py` and `cli.py` — while the metric is produced by `retrieve()` (`eval_harness.py:167` imports it, `:182` calls it), so the entire retrieval and scoring stack sat outside the trigger. Because the `push:` trigger carries no `paths:` key, that asymmetry converted a skipped check into a broken main: a PR editing only `scoring.py` never ran the job, merged, and the post-merge run then failed the baseline assertion with no owning PR to revert. The hidden coverage is real — negating the bm25 term in `scoring.py` moves `roc_auc` from 0.8444 to 0.7347 and `spearman_rho` from 0.5241 to 0.3572, which the byte-exact assertion catches the moment it is allowed to run. Now filters on `src/aelfrice/**`, matching what `ci.yml` already does, rather than a longer enumeration: the enumeration is what drifted, and the coupling is not stable enough to enumerate safely — the harness pins `l1_limit` and passes `entity_index_enabled=False, bfs_enabled=False`, so `DEFAULT_L1_LIMIT` and `DEFAULT_K1` are both unreachable today and which modules are live moves with the call. Over-triggering costs a 0.45–0.68 s measurement; under-triggering costs a red main. The accompanying guard walks imports from the harness (via `ast.walk`, since the `retrieve` import sits inside a function body — a top-level-only scan misses exactly the dependency at issue) and asserts all 25 reachable modules match the filter, with its own vacuity pinned so an empty walk cannot satisfy the assertion for free. **Not fixed here:** the same gate is still blind to the posterior rerank it is named for — `AELFRICE_POSTERIOR_WEIGHT=0.0`, `1.0` and `5.0` all emit byte-identical metrics, because the calibration corpus builds every belief at `alpha=0.5, beta=0.5` and a constant posterior is a constant offset that cannot reorder anything. That is a separate acceptance criterion on the umbrella and needs a corpus change plus a deliberate baseline recut.
 - **Federated peers opened with `immutable=1` ignored a live WAL and read nothing ([#1198](https://github.com/robotrocketscience/aelfrice/issues/1198)).** `open_peer_connection` opened every peer with `mode=ro&immutable=1`. `immutable=1` promises SQLite the file cannot change, and SQLite acts on that promise by skipping locking and ignoring the write-ahead log entirely. Every aelfrice store runs in WAL mode, and a store held open by a running hook — the normal deployment shape — keeps its recent commits in `memory.db-wal` until something checkpoints, so the peer handle read a database that was missing them. Measured on a live 30-belief peer at the same instant: the `immutable=1` connection raised `no such table: beliefs` (the WAL held the schema too), while a plain `mode=ro` connection saw all 30. Where the schema *had* been checkpointed but recent rows had not, the failure was quieter and worse — a smaller row count with no error, so federated retrieval silently omitted beliefs that were committed and durable. Same class as the [#1173](https://github.com/robotrocketscience/aelfrice/issues/1173) `--archive` bug: a live WAL means the main database file is not the data. Peers now open with plain `mode=ro`, which honours the WAL; the read-only guarantee is unchanged (a write still raises `attempt to write a readonly database`). `immutable=1` is retained as a **fallback**, because a WAL-mode database must create a `-shm` file even when fully checkpointed and so plain `mode=ro` fails outright on read-only media — dropping it unconditionally would have turned those peers from working into silently unreachable. The fallback is reached only after an honest read has already failed **and only on `SQLITE_READONLY`** — any other `OperationalError` (a lock, a transient filesystem error) propagates, because routing those to `immutable=1` would hand back a WAL-blind handle for the connection's whole lifetime and silently restore the defect, and `_peer_conn` caches the handle and swallows failures so it would stay restored until the process restarts. Because `sqlite3.connect` on a URI defers the `-shm` attempt — a file on read-only media errors only at the first statement, though a *missing* file still raises from `connect` itself — the open now issues a `schema_version` pragma to force the pager to resolve where the fallback can act on it. This was the last untouched acceptance criterion of [#1161](https://github.com/robotrocketscience/aelfrice/issues/1161).
+- **Superseded beliefs were never demoted or excluded at retrieval ([#1187](https://github.com/robotrocketscience/aelfrice/issues/1187)).** `grep SUPERSEDES src/aelfrice/retrieval.py` returned zero hits: the retrieval pipeline had no notion of supersession, and `uri_baki.apply_supersession_demote` — the only implementation — had no importer anywhere in `src/`. Reproduced: two contradictory beliefs, `resolve_contradiction` picks the newer and writes SUPERSEDES, then `retrieve(s, "deploy target")` returns `['old', 'new']` — the **retired** belief ranked first, both injected. The user corrects "deploy target is heroku" to "fly.io" and the next prompt still leads with heroku. Live for the paths that actually write SUPERSEDES (explicit `aelf resolve`, the triple extractor's "X supersedes Y"); the auto-relationship detector has been CONTRADICTS-only since the #1005 revert. A new `use_supersession_demote` lane ships **both** treatments behind one default-OFF flag — `demote` (adds `log(supersession_demote_factor)`, default 0.5) and `exclude` (drops the belief from the candidate set before the heat-kernel seeds are computed) — selected by `supersession_treatment`, resolved env > kwarg > TOML > default like every other lane, and fed by one batched `SELECT DISTINCT dst … WHERE type='SUPERSEDES'` over the candidate set. Per the ratified shape, nothing presumes a winner: the three-arm bench (demote vs exclusion vs control) picks the default, and unlike the #1170 BFS-direction fix this lane changes `retrieve()` output on the **default** path, which is why it stays off until that bench is read. **The demote is additive, not multiplicative,** and that is a correction to the issue's suggested wiring: the composite rerank score is a log-domain quantity from `combine_log_scores` / `partial_bayesian_score` and is routinely negative (measured -13.08 on the two-belief reproduction), so `score * 0.5` *raises* it — importing `apply_supersession_demote` as written would have promoted the superseded belief to the top of the pack, the exact inversion the lane exists to fix. Adding `log(factor)` is the log-domain equivalent of scaling a probability, so the issue's factor semantics survive and the demote is unconditional; the primitive is left unimported and a test pins the negative-score premise so nobody folds the addition back into a multiplication. Two calibration facts for the bench, both measured: at factor 0.5 the penalty is -0.69, the same order as the default-ON entity-persistence penalty and far weaker than its -6.91 floor, and on the reproduction the two compose to *cancel* (the current belief grounds to entities, S1=0.5 → -0.691; the retired one grounds to none → 0), so the bench should sweep the factor rather than test 0.5 alone.
+- **Superseded beliefs were never demoted or excluded at retrieval ([#1187](https://github.com/robotrocketscience/aelfrice/issues/1187)).** `grep SUPERSEDES src/aelfrice/retrieval.py` returned zero hits: the retrieval pipeline had no notion of supersession, and `uri_baki.apply_supersession_demote` — the only implementation — had no importer anywhere in `src/`. Reproduced: two contradictory beliefs, `resolve_contradiction` picks the newer and writes SUPERSEDES, then `retrieve(s, "deploy target")` returns `['old', 'new']` — the **retired** belief ranked first, both injected. The user corrects "deploy target is heroku" to "fly.io" and the next prompt still leads with heroku. Live for the paths that actually write SUPERSEDES (explicit `aelf resolve`, the triple extractor's "X supersedes Y"); the auto-relationship detector has been CONTRADICTS-only since the #1005 revert. A new `use_supersession_demote` lane ships **both** treatments behind one default-OFF flag — `demote` (adds `log(supersession_demote_factor)`, default 0.5) and `exclude` (drops the belief from the candidate set before the heat-kernel seeds are computed) — selected by `supersession_treatment`, resolved env > kwarg > TOML > default like every other lane, and fed by one batched `SELECT DISTINCT dst … WHERE type='SUPERSEDES'` over the candidate set. Per the ratified shape, nothing presumes a winner: the three-arm bench (demote vs exclusion vs control) picks the default, and unlike the #1170 BFS-direction fix this lane changes `retrieve()` output on the **default** path, which is why it stays off until that bench is read. **The demote is additive, not multiplicative,** and that is a correction to the issue's suggested wiring: the composite rerank score is a log-domain quantity from `combine_log_scores` / `partial_bayesian_score` and is routinely negative (measured -13.08 on the two-belief reproduction), so `score * 0.5` *raises* it — importing `apply_supersession_demote` as written would have promoted the superseded belief to the top of the pack, the exact inversion the lane exists to fix. Adding `log(factor)` is the log-domain equivalent of scaling a probability, so the issue's factor semantics survive and the demote is unconditional; the primitive is left unimported and a test pins the negative-score premise so nobody folds the addition back into a multiplication. Two calibration facts for the bench, both measured: at factor 0.5 the penalty is -0.69, the same order as the default-ON entity-persistence penalty and far weaker than its -6.91 floor, and on the reproduction the two compose to *cancel* (the current belief grounds to entities, S1=0.5 → -0.691; the retired one grounds to none → 0), so the bench should sweep the factor rather than test 0.5 alone. The exclusion arm widens the candidate fetch and retries rather than filtering the fixed-size slice in place: the limit is applied by the search, so filtering afterwards shrank the pack instead of backfilling — at `l1_limit=4` with three of the top four retired it returned one belief while three current ones sat at ranks 5-7, the same shape as the lock-budget starvation fixed in [#1014](https://github.com/robotrocketscience/aelfrice/issues/1014)/[#1015](https://github.com/robotrocketscience/aelfrice/issues/1015). Bounded at three rounds so a store where nearly everything is superseded costs a fixed number of queries. This also matters for the ratified three-arm bench: with the arms differing in pack size as well as in treatment, a loss for exclusion could not have been attributed to either.
 
 ## [4.2.0] - 2026-07-21