diff --git a/CHANGELOG.md b/CHANGELOG.md index 30eacb2d2..854eb90a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,8 @@ installable release; see the roadmap in [README.md](README.md). ### Added +- **`aelf doctor` flags per-project DBs on pre-v1.x schema (no `origin` column)** ([#589](https://github.com/robotrocketscience/aelfrice/issues/589)). Scans all `~/.aelfrice/projects/*/memory.db` and appends a `legacy-schema per-project DBs detected` block listing each flagged DB with belief row count and idle days. DBs on the old schema cannot participate in the v2.x lifecycle (`agent_remembered`, `user_validated`, calibrated weights, `aelf:promote`). Block is quiet when no legacy DBs are found (parity with #557 quietness). `aelf migrate` is now visible in `--help` output (previously `argparse.SUPPRESS`) so the fix line in the nag block is discoverable. + - **Context-rebuilder eval harness — three integration points wired** ([#592](https://github.com/robotrocketscience/aelfrice/issues/592)). `benchmarks/context-rebuilder/eval_harness.py` was a skeleton with five `NotImplementedError` stubs since v1.2.0; this lands the three pure-Python wirings that make `--mode threshold-sweep` and `--mode budget-sweep` runnable end-to-end. `replay_to_fork` populates a fresh in-memory `MemoryStore` via `ingest_jsonl` against a tempfile holding the first `fork_turn` lines of the case JSONL. `run_rebuilder` calls `rebuild_v14()` with `floor_session=floor_l1=trigger_threshold` (mapping the harness's threshold-sweep axis onto the v1.7 per-lane composite-score floor) and returns `(rebuilt_block, latency_ms)` measured via `time.monotonic()`. `measure_token_cost` returns `estimate_tokens(rebuilt) / estimate_tokens(pre_clear)` using `benchmarks.context_rebuilder.measure.estimate_tokens` (the 4-chars-per-token heuristic that mirrors `aelfrice.context_rebuilder._CHARS_PER_TOKEN`), so harness measurements stay aligned with the rebuilder's own budget bookkeeping. `replay_post_fork` is a non-crashing stub that returns one placeholder per `eval_turn` (`{turn_idx, expected, actual="", matched=False, reason="needs_replay_client"}`) so the harness produces valid latency + token-cost numbers before the model-invocation client lands; `score_fidelity` reads `matched=False` and returns 0.0 instead of raising. Adds `debugging_session_001.meta.json` next to the bundled 16-turn synthetic fixture (midpoint fork at turn 8, eval_turns at indices 8/10/12/14) so `--corpus benchmarks/context-rebuilder/fixtures/synthetic/` resolves out of the box. 15 new tests in `tests/test_context_rebuilder_eval_harness_wiring.py` cover each wired function plus an end-to-end threshold-sweep smoke against the bundled fixture. The model-invocation client + LLM judge stages are explicit follow-ups; this commit lands the harness's runnable contract before fidelity scoring engages. - **Session-end Stop hook prompts to lock session corrections** ([#582](https://github.com/robotrocketscience/aelfrice/issues/582)). New `aelf-stop-hook` (default-on) fires once per assistant-turn end, walks the store for unlocked correction-class beliefs created in the current `session_id`, and emits a `` block to stderr listing each candidate with a pre-filled `aelf lock --statement '<...>'` command. Candidate filter: `session_id == current AND lock_level != LOCK_USER AND (type == BELIEF_CORRECTION OR origin in {agent_inferred, agent_remembered})`. Hook is informational by default; setting `AELF_AUTOLOCK_CORRECTIONS=1` in the environment makes it auto-lock the candidates instead (logs each lock to stderr for transparency). Wired into `aelf setup` / `aelf unsetup` (`--no-stop-hook` opts out) and into `aelf doctor` as a fourth default-on auto-capture hook the v2.1 nag flags when missing. Coexists with the existing transcript-ingest Stop entry as a separate entry under the same `hooks.Stop` event key. **Note**: the issue's spec assumed a `aelf correct` CLI command and a `feedback_history.kind=correct` marker that don't exist on `main` (no historical implementation); the v0 detection signal is correction-class beliefs in the current session instead. Future work to ship `aelf correct` + `feedback_history.kind` would let this hook also surface beliefs that were *modified* (not only newly-created) in the session. diff --git a/docs/INSTALL.md b/docs/INSTALL.md index 02ce34892..66f2e7294 100644 --- a/docs/INSTALL.md +++ b/docs/INSTALL.md @@ -170,6 +170,29 @@ All hooks are non-blocking. Every failure path returns exit 0 — a hook problem > **Privacy note.** Default-on transcript-ingest means every turn you type lands in the per-project SQLite DB on `PreCompact` rotation. The DB is local-only (no network, no telemetry — see § "Your data stays yours" in the README) but the JSONL has no PII scrubber. If you paste secrets, customer data, or anything you don't want indexed in chat, opt out with `--no-transcript-ingest` and use `aelf lock` / `aelf onboard` for explicit ingestion only. +### Legacy-schema detection (`aelf doctor`, v2.1+) + +`aelf doctor` scans all per-project DBs under `~/.aelfrice/projects/*/memory.db` and flags any that use the pre-v1.x schema (no `origin` column on the `beliefs` table) and have at least one row. DBs on the old schema cannot participate in the v2.x lifecycle — `agent_remembered`, `user_validated`, calibrated weights, `aelf:promote` — because the column that tracks origin is absent. + +When legacy DBs are found the doctor report appends a block like: + +``` +legacy-schema per-project DBs detected (pre-v1.x, no `origin` column). + ~/.aelfrice/projects/2e7ed55e017a/memory.db (35,332 beliefs, idle 16d) + ~/.aelfrice/projects/18a856c7a96b/memory.db (6,283 beliefs, idle 13d) +fix: `aelf migrate --from --apply` per DB to copy beliefs + into the current project's modern-schema DB. +``` + +The block is quiet when every scanned DB already has the `origin` column. Empty DBs (zero rows) are silently skipped. + +To migrate a legacy DB: + +```bash +aelf migrate --from ~/.aelfrice/projects//memory.db # dry-run +aelf migrate --from ~/.aelfrice/projects//memory.db --apply # write +``` + --- ## Update notifier diff --git a/src/aelfrice/cli.py b/src/aelfrice/cli.py index 1612aa6fe..4a1b44c51 100644 --- a/src/aelfrice/cli.py +++ b/src/aelfrice/cli.py @@ -4140,8 +4140,10 @@ def build_parser(*, show_advanced: bool = False) -> argparse.ArgumentParser: p_regime = sub.add_parser("regime", help=argparse.SUPPRESS) p_regime.set_defaults(func=_cmd_regime) - # Hidden: one-shot v1.0 -> v1.1 migration; the era is over. - p_migrate = sub.add_parser("migrate", help=argparse.SUPPRESS) + p_migrate = sub.add_parser( + "migrate", + help="copy beliefs from a legacy per-project DB into the current one", + ) p_migrate.add_argument( "--from", dest="from_path", default=None, help="legacy DB path (default: ~/.aelfrice/memory.db)", diff --git a/src/aelfrice/doctor.py b/src/aelfrice/doctor.py index 2d222a48d..8ca5c6e6e 100644 --- a/src/aelfrice/doctor.py +++ b/src/aelfrice/doctor.py @@ -40,6 +40,8 @@ import re import shlex import shutil +import sqlite3 +import time from dataclasses import dataclass, field from pathlib import Path from typing import TYPE_CHECKING, Final, Literal, cast @@ -223,6 +225,26 @@ def _load_settings_json(path: Path) -> dict[str, object]: # How many trailing lines of the hook-failures log to surface. _HOOK_FAILURES_TAIL: Final[int] = 10 +# Root directory under which per-project aelfrice state lives. +# Each sub-directory is a project-id slug; memory.db sits directly inside. +# Override via `aelfrice_projects_dir` kwarg on `diagnose()` (tests use this). +_AELFRICE_PROJECTS_DIR: Final[Path] = ( + Path.home() / ".aelfrice" / "projects" +) + + +@dataclass(frozen=True) +class LegacySchemaDB: + """One per-project DB detected as pre-v1.x (no `origin` column). + + `path` — absolute path to the memory.db file. + `row_count` — number of rows in the `beliefs` table. + `idle_days` — whole days since the file was last modified (mtime). + """ + path: Path + row_count: int + idle_days: int + @dataclass(frozen=True) class CommandFinding: @@ -292,6 +314,14 @@ class DoctorReport: missing_auto_capture_hooks: list[str] = field( default_factory=lambda: cast(list[str], []) ) + # Per-project DBs under ~/.aelfrice/projects/*/memory.db that use + # the pre-v1.x schema (no `origin` column on the `beliefs` table) + # and have at least one row. These DBs cannot participate in the + # v2.x lifecycle (agent_remembered, user_validated, calibrated + # weights, aelf:promote) without `aelf migrate` (#589). + legacy_schema_dbs: list[LegacySchemaDB] = field( + default_factory=lambda: cast(list[LegacySchemaDB], []) + ) @property def broken(self) -> list[CommandFinding]: @@ -324,6 +354,7 @@ def diagnose( known_cli_subcommands: frozenset[str] | None = None, search_tool_telemetry_path: Path | None = None, user_prompt_submit_telemetry_path: Path | None = None, + aelfrice_projects_dir: Path | None = None, ) -> DoctorReport: """Walk user and project settings.json, return a DoctorReport. @@ -339,6 +370,8 @@ def diagnose( derivable from the project root's git-common-dir), the search_tool_hook telemetry section is populated. Similarly for `user_prompt_submit_telemetry_path` (#218 AC4). + `aelfrice_projects_dir` overrides the default scan root for + per-project DBs (`~/.aelfrice/projects`); useful in tests (#589). """ user_path = user_settings if user_settings is not None else USER_SETTINGS_PATH project_path = ( @@ -358,6 +391,13 @@ def diagnose( report.hook_failures_tail = _tail_log(log_path, _HOOK_FAILURES_TAIL) report.missing_runtime_deps = _check_runtime_deps() report.missing_auto_capture_hooks = _check_auto_capture_hooks(report.findings) + # #589: scan per-project DBs for pre-v1.x schema (no `origin` column). + _proj_dir = ( + aelfrice_projects_dir + if aelfrice_projects_dir is not None + else _AELFRICE_PROJECTS_DIR + ) + report.legacy_schema_dbs = _check_legacy_schema_dbs(projects_dir=_proj_dir) if known_cli_subcommands is not None: slash_dir = ( slash_commands_dir if slash_commands_dir is not None @@ -748,6 +788,7 @@ def format_report(report: DoctorReport) -> str: _format_user_prompt_submit_telemetry_section(report, lines) _format_missing_runtime_deps_section(report, lines) _format_missing_auto_capture_section(report, lines) + _format_legacy_schema_section(report, lines) return "\n".join(lines) @@ -811,6 +852,89 @@ def _format_missing_auto_capture_section( ) +def _check_legacy_schema_dbs( + *, + projects_dir: Path | None = None, +) -> list[LegacySchemaDB]: + """Return per-project DBs that are on the pre-v1.x schema (no `origin`). + + Scans `~/.aelfrice/projects/*/memory.db` (or `projects_dir` override). + A DB is flagged when ALL of the following hold: + 1. The `beliefs` table exists. + 2. No column named `origin` is present. + 3. `SELECT COUNT(*) FROM beliefs` returns > 0. + + Opens each DB read-only (`file:...?mode=ro`) to avoid accidental writes. + Any DB that raises a connection or query error is silently skipped — + doctor is diagnostic only. + """ + root = projects_dir if projects_dir is not None else _AELFRICE_PROJECTS_DIR + if not root.is_dir(): + return [] + + results: list[LegacySchemaDB] = [] + now = time.time() + + for db_path in sorted(root.glob("*/memory.db")): + try: + uri = f"file:{db_path}?mode=ro" + con = sqlite3.connect(uri, uri=True) + except Exception: # noqa: BLE001 + continue + try: + cur = con.execute("PRAGMA table_info(beliefs)") + columns = cur.fetchall() + if not columns: + # No beliefs table — skip (uninitialised or unrelated DB). + continue + col_names = {row[1] for row in columns} # row[1] is the column name + if "origin" in col_names: + # Modern schema — skip. + continue + # Legacy schema. Skip empty DBs — uninteresting. + (row_count,) = con.execute("SELECT COUNT(*) FROM beliefs").fetchone() + if row_count == 0: + continue + mtime = db_path.stat().st_mtime + idle_days = int((now - mtime) / 86400) + results.append( + LegacySchemaDB( + path=db_path, + row_count=int(row_count), + idle_days=idle_days, + ) + ) + except Exception: # noqa: BLE001 + pass + finally: + con.close() + + return results + + +def _format_legacy_schema_section( + report: DoctorReport, lines: list[str], +) -> None: + """Append the pre-v1.x legacy-schema nag block (#589) to `lines`. + + Quiet when no legacy DBs are found (parity with #557 quietness). + """ + if not report.legacy_schema_dbs: + return + lines.append("") + lines.append( + "legacy-schema per-project DBs detected (pre-v1.x, no `origin` column)." + ) + for entry in report.legacy_schema_dbs: + lines.append( + f" {entry.path} ({entry.row_count:,} beliefs, idle {entry.idle_days}d)" + ) + lines.append( + "fix: `aelf migrate --from --apply` per DB to copy beliefs " + "into the current project's modern-schema DB." + ) + + def _format_missing_runtime_deps_section( report: DoctorReport, lines: list[str], ) -> None: diff --git a/tests/test_doctor.py b/tests/test_doctor.py index b3b20453d..67df3384d 100644 --- a/tests/test_doctor.py +++ b/tests/test_doctor.py @@ -3,6 +3,7 @@ import io import json +import sqlite3 from pathlib import Path import pytest @@ -461,3 +462,163 @@ def test_auto_capture_basenames_match_setup() -> None: setup.SESSION_START_HOOK_SCRIPT_NAME, setup.STOP_HOOK_SCRIPT_NAME, } + + +# --------------------------------------------------------------------------- +# #589 — per-project legacy-schema detection +# --------------------------------------------------------------------------- + +def _make_legacy_db(path: Path, belief_count: int = 3) -> None: + """Create a per-project memory.db WITHOUT the `origin` column (pre-v1.x schema).""" + path.parent.mkdir(parents=True, exist_ok=True) + con = sqlite3.connect(str(path)) + con.execute( + "CREATE TABLE beliefs " + "(id INTEGER PRIMARY KEY, content TEXT, type TEXT)" + ) + for i in range(belief_count): + con.execute("INSERT INTO beliefs (content, type) VALUES (?, ?)", + (f"belief {i}", "fact")) + con.commit() + con.close() + + +def _make_modern_db(path: Path, belief_count: int = 2) -> None: + """Create a per-project memory.db WITH the `origin` column (modern schema).""" + path.parent.mkdir(parents=True, exist_ok=True) + con = sqlite3.connect(str(path)) + con.execute( + "CREATE TABLE beliefs " + "(id INTEGER PRIMARY KEY, content TEXT, type TEXT, origin TEXT)" + ) + for i in range(belief_count): + con.execute( + "INSERT INTO beliefs (content, type, origin) VALUES (?, ?, ?)", + (f"belief {i}", "fact", "agent_remembered"), + ) + con.commit() + con.close() + + +def _make_empty_legacy_db(path: Path) -> None: + """Create a per-project memory.db without `origin`, but with zero rows (skip).""" + path.parent.mkdir(parents=True, exist_ok=True) + con = sqlite3.connect(str(path)) + con.execute( + "CREATE TABLE beliefs " + "(id INTEGER PRIMARY KEY, content TEXT, type TEXT)" + ) + con.commit() + con.close() + + +def test_legacy_schema_detected(tmp_path: Path) -> None: + """Legacy DB (no `origin` column, rows present) → block present in report; + modern DB with `origin` column → absent from block. Also verifies that + an empty legacy DB (zero rows) is silently skipped (#589 AC). + """ + from aelfrice.doctor import _check_legacy_schema_dbs + + projects_dir = tmp_path / "projects" + + # Legacy DB with rows — should be flagged. + legacy_path = projects_dir / "aabbccdd1234" / "memory.db" + _make_legacy_db(legacy_path, belief_count=5) + + # Modern DB with origin column — should be skipped. + modern_path = projects_dir / "11223344aabb" / "memory.db" + _make_modern_db(modern_path, belief_count=2) + + # Empty legacy DB (no rows) — should be skipped. + empty_legacy_path = projects_dir / "deadbeef0000" / "memory.db" + _make_empty_legacy_db(empty_legacy_path) + + results = _check_legacy_schema_dbs(projects_dir=projects_dir) + + paths_found = [r.path for r in results] + assert legacy_path in paths_found, "legacy DB with rows must be flagged" + assert modern_path not in paths_found, "modern DB must not be flagged" + assert empty_legacy_path not in paths_found, "empty legacy DB must not be flagged" + + assert len(results) == 1 + entry = results[0] + assert entry.row_count == 5 + assert entry.idle_days >= 0 + + +def test_legacy_schema_report_block_present(tmp_path: Path) -> None: + """When legacy DBs are found, format_report appends the nag block.""" + projects_dir = tmp_path / "projects" + legacy_path = projects_dir / "abc123def456" / "memory.db" + _make_legacy_db(legacy_path, belief_count=7) + + user_path = tmp_path / "settings.json" + _write_settings(user_path, { + "hooks": { + "UserPromptSubmit": [{"hooks": [{"type": "command", "command": "aelf-hook"}]}], + }, + }) + report = diagnose( + user_settings=user_path, + project_root=tmp_path / "noproj", + aelfrice_projects_dir=projects_dir, + ) + assert len(report.legacy_schema_dbs) == 1 + rendered = format_report(report) + assert "legacy-schema per-project DBs detected" in rendered + assert "aelf migrate" in rendered + assert str(legacy_path) in rendered + + +def test_legacy_schema_report_quiet_when_zero(tmp_path: Path) -> None: + """When no legacy DBs exist, format_report must not emit the nag block.""" + projects_dir = tmp_path / "projects" + modern_path = projects_dir / "00112233aabb" / "memory.db" + _make_modern_db(modern_path, belief_count=3) + + user_path = tmp_path / "settings.json" + _write_settings(user_path, { + "hooks": { + "UserPromptSubmit": [{"hooks": [{"type": "command", "command": "aelf-hook"}]}], + }, + }) + report = diagnose( + user_settings=user_path, + project_root=tmp_path / "noproj", + aelfrice_projects_dir=projects_dir, + ) + assert report.legacy_schema_dbs == [] + rendered = format_report(report) + assert "legacy-schema per-project DBs detected" not in rendered + + +def test_legacy_schema_quiet_when_projects_dir_missing(tmp_path: Path) -> None: + """Non-existent projects dir → no crash, empty list.""" + from aelfrice.doctor import _check_legacy_schema_dbs + + results = _check_legacy_schema_dbs( + projects_dir=tmp_path / "no-such-projects-dir" + ) + assert results == [] + + +def test_legacy_schema_idle_days_in_report(tmp_path: Path) -> None: + """The rendered block includes 'idle Xd' for the legacy DB.""" + projects_dir = tmp_path / "projects" + legacy_path = projects_dir / "ffee12345678" / "memory.db" + _make_legacy_db(legacy_path, belief_count=4) + + user_path = tmp_path / "settings.json" + _write_settings(user_path, { + "hooks": { + "UserPromptSubmit": [{"hooks": [{"type": "command", "command": "aelf-hook"}]}], + }, + }) + report = diagnose( + user_settings=user_path, + project_root=tmp_path / "noproj", + aelfrice_projects_dir=projects_dir, + ) + rendered = format_report(report) + assert "idle" in rendered + assert "beliefs" in rendered