Skip to content

feat(doctor): --classify-orphans pass for un-typed low-prior beliefs (#206) - #253

Merged
robotrocketscience merged 3 commits into
mainfrom
feat/issue-206-classify-orphans
Apr 29, 2026
Merged

feat(doctor): --classify-orphans pass for un-typed low-prior beliefs (#206)#253
robotrocketscience merged 3 commits into
mainfrom
feat/issue-206-classify-orphans

Conversation

@robotrocketscience

Copy link
Copy Markdown
Owner

Closes #206.

Summary

Adds `aelf doctor --classify-orphans` — a targeted reclassification pass that finds beliefs whose `type` was never resolved AND that have never received any feedback, then re-classifies them through the same Haiku batch path `aelf onboard --llm-classify` already uses. Strict superset of onboard: only adds calls, doesn't replace anything.

Orphan definition (both signals required)

  • `type = 'unknown' OR type IS NULL` — never successfully typed by onboard/ingest.
  • `alpha + beta <= 2` — untouched prior is alpha=1, beta=1 (sum=2); any feedback event pushes the sum above 2.

The other orphan signals from the issue (zero non-CONTAINS edges, entity_index miss) are noted in code comments as future extensions once #143 / edge-type auditing land.

CLI

  • `aelf doctor --classify-orphans` — full pass.
  • `--dry-run` — count orphans + show before-distribution; no LLM calls, no DB writes.
  • `--max N` — cap classifications per run (recommended: 500 for large stores).
  • Reports orphans found, classified, skipped, before/after type distribution, input/output token counts, request count, fallback count, and an estimated cost line.

Implementation notes

  • Reuses `aelfrice.llm_classifier.classify_batch` — no new LLM client.
  • Updates only `type` and `origin` (set to `agent_inferred`); `alpha`, `beta`, lock state, `demotion_pressure`, and timestamps are preserved.
  • Auth failures and token-cap-exceeded errors propagate out and exit 1.
  • When the whole batch falls back to regex, all orphans are marked skipped (regex-fallback is designed for fresh inserts in scan_repo, not in-place updates).

Test plan

  • 14 new tests in `tests/test_doctor_classify_orphans.py`, all green; full suite 1598 passed / 5 skipped.
  • `test_classify_orphans_95_percent_recovery` — synthetic orphan corpus, asserts >=95% recovery to non-unknown type.
  • `test_classify_orphans_max_n_caps_llm_input` — verifies the SQL LIMIT actually bounds the LLM call.
  • `test_cli_classify_orphans_no_api_key` — exits 1 with a useful message when ANTHROPIC_API_KEY is unset.
  • `test_classify_orphans_dry_run_no_writes` — no DB writes, no network in dry-run mode.

Out of scope

…eclassification core

New CLI: aelf doctor --classify-orphans [--max N] [--dry-run]

Orphan definition (both signals required):
  - type = 'unknown' or NULL (never resolved by onboard or ingest)
  - alpha + beta < 2 (untouched Bayesian prior — no feedback applied)

Future extension signals (entity_index miss, zero non-CONTAINS edges)
noted as TODO in code comments pending #143 and edge-type auditing.

store.py additions:
  find_orphan_beliefs(max_n) — SQL query implementing the definition
  count_beliefs_by_type()    — distribution snapshot for the report

doctor.py additions:
  classify_orphans()    — drives classify_batch, updates beliefs in-place
  format_orphan_report() — human-readable cost + distribution report
  OrphanRunReport       — dataclass accumulating run telemetry

cli.py additions:
  _cmd_doctor_classify_orphans() — CLI handler; gates API key, routes
  --classify-orphans / --dry-run / --max N args on p_doctor
…-orphans

Fix orphan threshold: alpha+beta <= 2 (the untouched default prior sums
to exactly 2; any feedback event pushes the sum above 2).

Adds test_doctor_classify_orphans.py with 14 tests covering:
  - find_orphan_beliefs SQL correctness (type filter, alpha+beta gate)
  - max_n cap on both query and LLM batch
  - dry_run: no network calls, no store writes
  - 95-percent recovery acceptance criterion (N=100 synthetic orphans)
  - no-orphans noop path
  - CLI integration: --classify-orphans / --dry-run / --max N / no-key
  - format_orphan_report output (cost estimate, before/after distribution)
  - count_beliefs_by_type distribution snapshot
@robotrocketscience
robotrocketscience force-pushed the feat/issue-206-classify-orphans branch from ef68124 to cee89c2 Compare April 28, 2026 23:18
@robotrocketscience robotrocketscience added the review-Gylf PR coordination mutex label Apr 29, 2026
@robotrocketscience
robotrocketscience merged commit 06e28e9 into main Apr 29, 2026
8 checks passed
@robotrocketscience
robotrocketscience deleted the feat/issue-206-classify-orphans branch April 29, 2026 00:01
robotrocketscience added a commit that referenced this pull request Apr 29, 2026
## Summary

Spec memo for #290. Phase-3 of the #286 redesign — the foundational
typing axis that ranking, aging, and promotion all consume. **Doc-only —
no code change. Posted for ratification.**

Core moves:

1. **New orthogonal axis `retention_class`** with three live values:
`fact`, `snapshot`, `transient` (plus `unknown` for migration). Distinct
from the existing `type` enum, which describes *form* not *expected
lifetime*.
2. **Defaults per ingest path.** Filesystem/git/CLI-remember → `fact`.
Transcript/hook → `snapshot`. `transient` requires explicit operator
opt-in. Table in §2.
3. **Soft down-weight in ranking, not hard expiry.** Composite score
from #289 gains a `retention_age_multiplier(class, age_days)` term.
Per-class exponential decay with floor (fact ~300d half-life, snapshot
14d, transient 1d). Locks force multiplier = 1.0.
4. **Promotion: snapshot → fact** when `corroboration_count >= 3 AND
distinct_sessions >= 2 AND no contradiction`. Reuses #190 corroboration
recorder. Doctor-pass opt-in (`aelf doctor --promote-retention`, mirrors
#253). No demotion — contradictions handle correctness.
5. **Migration via `_maybe_classify_retention_class()`**, schema-meta
gated, heuristic by `source_kind`. `legacy_unknown` rows stay `unknown`.

## Decision asks (nine)

Bottom of the memo. Schema shape, defaults table, transient opt-in,
multiplier shape, lock override, calibration plan, promotion thresholds,
no-demotion, migration heuristic.

## What ships if ratified

Three PRs:

1. Schema + score-formula change + migration helper (~500 lines)
2. Per-ingest defaults wired across scanner / ingest / hooks (~250
lines)
3. Promotion lane via `aelf doctor --promote-retention` (~300 lines)

## Test plan

- [x] Memo cross-references #286, #196, #229, #190, #253, #283, #281
- [x] Code touchpoints listed in Provenance
- [x] No code change; nothing to test
- [ ] Ratification or override comment from maintainer

Refs #290, #286
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-Gylf PR coordination mutex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v1.6] Orphan finder + targeted Haiku reclassification (aelf doctor --classify-orphans)

2 participants