diff --git a/.opencode/opencode.json b/.opencode/opencode.json new file mode 100644 index 0000000000..728a895bdd --- /dev/null +++ b/.opencode/opencode.json @@ -0,0 +1,10 @@ +{ + "$schema": "https://opencode.ai/config.json", + "mcp": { + "mempalace": { + "type": "local", + "command": ["python", "-m", "mempalace.mcp_server"], + "enabled": true + } + } +} diff --git a/FORK_CHANGELOG.md b/FORK_CHANGELOG.md index e01c0b3797..11b9a08e7a 100644 --- a/FORK_CHANGELOG.md +++ b/FORK_CHANGELOG.md @@ -24,6 +24,163 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- **Bundled OpenCode live-capture plugin that bypasses option-K v1.2.1 bugs (filed upstream as #4, #5)** ([`5522623`](https://github.com/jphein/mempalace/commit/5522623)) + Adds `examples/opencode/live-capture/` — a self-contained + OpenCode plugin (JS) + Python helper that POSTs verbatim + session transcripts to the daemon's `/silent-save` endpoint on + every `session.idle`. + + Background: while documenting the integration recipe + (`opencode-integration-recipe` below), end-to-end testing + revealed that the upstream option-K + [`opencode-plugin-mempalace`](https://www.npmjs.com/package/opencode-plugin-mempalace) + v1.2.1 cannot actually push drawers in a daemon-routed setup — + two compounding bugs, both filed upstream: + + - [option-K#4](https://github.com/option-K/opencode-plugin-mempalace/issues/4): + The plugin subscribes to `chat.message`, which OpenCode never + publishes. The message counter never increments, `session.idle` + sees `hasPendingMessages() === false`, and **the plugin never + mines a drawer**. Verified by inspecting bus types in + `~/.local/share/opencode/log/*.log` against the canonical event + taxonomy at [opencode.ai/docs/plugins](https://opencode.ai/docs/plugins). + - [option-K#5](https://github.com/option-K/opencode-plugin-mempalace/issues/5): + Even with #4 patched, `mineSync` calls `mempalace mine `, + which the *remote* daemon evaluates against ITS OWN filesystem. + For multi-host setups (palace-daemon on a different machine + from OpenCode), the daemon returns 400 because the local + path doesn't exist on its filesystem. + + The bundled plugin sidesteps both by: + + 1. Subscribing to `session.idle` / `session.deleted` / + `session.status[idle]` directly (no message counter). + 2. Reading OpenCode's local SQLite session DB client-side. + 3. POSTing the extracted transcript to the daemon's + `/silent-save` endpoint (the same endpoint MemPalace's + Claude Code stop hook uses). + + The Python helper imports `_extract_session_messages` and + `_session_transcript` from `mempalace/sources/opencode.py` so + the transcript shape matches `OpenCodeSourceAdapter` exactly. + + Also splits the previously combined option-K patch into two + independently-applicable files: + + - `examples/opencode/option-k-plugin-daemon-routing.patch` — + Fix 1 (option-K#1, `isInitialized()` daemon detection). + - `examples/opencode/option-k-plugin-message-updated.patch` — + Fix 2 (option-K#4, `chat.message` → `message.updated`). + + `docs/integrations/opencode.md` now documents both deployment + options (bundled plugin for remote-daemon, option-K plugin + + patches for local palaces). + + *Files:* `examples/opencode/live-capture/mempalace-live-capture.js`, `examples/opencode/live-capture/capture-session.py`, `examples/opencode/option-k-plugin-daemon-routing.patch`, `examples/opencode/option-k-plugin-message-updated.patch`, `docs/integrations/opencode.md` + + +- **Documented OpenCode integration recipe (read-side MCP + push plugin + retrospective adapter)** ([`60dc9e6`](https://github.com/jphein/mempalace/commit/60dc9e6)) + Adds `docs/integrations/opencode.md` and an `examples/opencode/` + directory capturing the three-direction OpenCode + MemPalace + integration recipe for daemon-routed setups: + + - **Read** — `~/.config/opencode/opencode.jsonc` MCP entry pointing + at `palace-daemon/clients/mempalace-mcp-wrapper.sh` (sources + `~/.config/palace-daemon/env` so the API key never lands in + plaintext config). + - **Push (live)** — option-K's `opencode-plugin-mempalace` npm + package (project-basename wings, default 15-message threshold, + session.idle flush, SIGINT/SIGTERM rescue, pre-compaction + injection — closest match to MemPalace's Claude Code stop-hook + pattern). + - **Pull (retrospective)** — the cherry-picked + `OpenCodeSourceAdapter` from upstream PR #1484, run via + `mempalace mine --source opencode` for one-shot backfill of + historical sessions. + + Includes a re-applicable patch + (`examples/opencode/option-k-plugin-daemon-routing.patch`) for + option-K's plugin v1.2.1 issue #1, where `isInitialized()` passes + `--palace /.mempalace/palace` to `mempalace status`, + forcing a local-store lookup that bypasses `PALACE_DAEMON_URL` + routing. Without the patch the plugin re-runs `mempalace init + --yes ` on every OpenCode start (idempotent against the + daemon, just wasteful); with the patch init+isInitialized + short-circuit when daemon-routed mode is detected. + + Why the recipe lives here rather than upstream: this fork's + single-writer-via-palace-daemon shape doesn't match upstream's + assumed local-CLI install pattern (Milofax #297, geco #1524, and + Dxrk #1567 all assume a local mempalace install). Upstream PRs + will eventually subsume parts of this — when they do, the YAML + entries become removable; for now the recipe captures what + actually works on a daemon-routed box. + + *Files:* `docs/integrations/opencode.md`, `examples/opencode/opencode.jsonc.example`, `examples/opencode/option-k-plugin-daemon-routing.patch` + + +- **.opencode/opencode.json — repo-root MCP config so opencode picks up mempalace automatically** ([`ba16b82`](https://github.com/jphein/mempalace/commit/ba16b82)) + Cherry-pick of upstream PR #1567 (Dxrk777). Adds + `.opencode/opencode.json` so that running `opencode` in the + mempalace repo root automatically wires `mempalace` as a local + MCP server — useful for contributors who use OpenCode for + development on the project itself. + + Two-commit cherry-pick: + + - `013ac63` — initial config with `command: ["mempalace-mcp"]` + - `ba16b82` — gemini-code-assist review feedback: switch to + `command: ["python", "-m", "mempalace.mcp_server"]` for + portability across install methods (pip vs uv vs dev install) + + This is the **dev/contributor surface** — it lives in the repo + and only matters when running OpenCode against the repo root. + Per-user setups should use `~/.config/opencode/opencode.jsonc` + with the daemon-aware wrapper (see `docs/integrations/opencode.md`). + + *Upstream:* [PR #1567](https://github.com/MemPalace/mempalace/pull/1567) (OPEN) + *Files:* `.opencode/opencode.json` + + +- **OpenCodeSourceAdapter (RFC 002) — retrospective ingest of OpenCode SQLite sessions** ([`2ffe652`](https://github.com/jphein/mempalace/commit/2ffe652)) + Cherry-pick of upstream PR #1484. Adds + `mempalace/sources/opencode.py` — an RFC 002 `BaseSourceAdapter` + that ingests OpenCode AI-coding-CLI session transcripts from + `~/.local/share/opencode/opencode.db` into the palace, formatted + to match `convo_miner`'s exchange-pair drawer shape. + + Five-commit cherry-pick: + + - `2c368c6` — initial adapter (482-line `opencode.py`, 6 + opencode-namespaced reference transformations in + `transforms.py`, entry-point registration, 28 tests, sample + SQLite-schema-verbatim fixture) + - `3ff7043` — gemini-code-assist review fixes: missing + `opencode_session_version` in metadata (broke incremental + `is_current`), `_skip_requested` private access, `filed_at` + hoisted out of chunk loop, PEP 8 import position + - `9531532` — igorls review fixes: ruff F401/E402 in tests, + route-hint wing/drawer-stage precedence mismatch (RFC 002 §2.5), + unjustified `# noqa` cleanup + - `18ab021` + `2ffe652` — CI ruff 0.4.x format passes + + Adapter conformance via RFC 002 §7.3 declared-transformation + round-trip; one drawer per exchange-pair; `source_file` shape + `opencode://#session=`; wing routes from + `session.directory` basename (matching the live-capture plugin's + taxonomy); incremental ingest works via `opencode_session_version`. + + Originated from JakobSachs's spadework on upstream PR #23 (DB + schema reverse engineering, session/message/part traversal, + tool-input/tool-output stripping). PR #23 is still OPEN but + CONFLICTING and unresponsive since 2026-04-08; #1484 carries + `Co-authored-by: JakobSachs` per coordination on #23. + + *Tests:* 28 OpenCode adapter tests pass; full suite 2133 passed / 33 skipped (zero regressions on fork main + 60-commit upstream sync baseline) + *Upstream:* [PR #1484](https://github.com/MemPalace/mempalace/pull/1484) (OPEN) + *Files:* `mempalace/sources/opencode.py`, `mempalace/sources/transforms.py`, `mempalace/sources/context.py`, `pyproject.toml`, `tests/test_sources_opencode.py`, `tests/fixtures/opencode/sample_session_2026_05_12/README.md`, `tests/fixtures/opencode/sample_session_2026_05_12/build_fixture.py`, `tests/test_corpus_origin_integration.py` + + - **Pending-writes journal + replay so daemon outages stop being silent** ([`0c34464`](https://github.com/jphein/mempalace/commit/0c34464)) Closes a silent-data-loss gap exposed by the 2026-05-17 power event: when ``PALACE_DAEMON_URL`` is set and the daemon's diff --git a/docs/fork-changes.yaml b/docs/fork-changes.yaml index a2cb28b6ca..776a4453d9 100644 --- a/docs/fork-changes.yaml +++ b/docs/fork-changes.yaml @@ -24,6 +24,198 @@ entries: + - id: opencode-live-capture-plugin + date: 2026-05-21 + bucket: Added + commit: 5522623 + area: CLI + summary: "Bundled OpenCode live-capture plugin that bypasses option-K v1.2.1 bugs (filed upstream as #4, #5)" + body: | + Adds `examples/opencode/live-capture/` — a self-contained + OpenCode plugin (JS) + Python helper that POSTs verbatim + session transcripts to the daemon's `/silent-save` endpoint on + every `session.idle`. + + Background: while documenting the integration recipe + (`opencode-integration-recipe` below), end-to-end testing + revealed that the upstream option-K + [`opencode-plugin-mempalace`](https://www.npmjs.com/package/opencode-plugin-mempalace) + v1.2.1 cannot actually push drawers in a daemon-routed setup — + two compounding bugs, both filed upstream: + + - [option-K#4](https://github.com/option-K/opencode-plugin-mempalace/issues/4): + The plugin subscribes to `chat.message`, which OpenCode never + publishes. The message counter never increments, `session.idle` + sees `hasPendingMessages() === false`, and **the plugin never + mines a drawer**. Verified by inspecting bus types in + `~/.local/share/opencode/log/*.log` against the canonical event + taxonomy at [opencode.ai/docs/plugins](https://opencode.ai/docs/plugins). + - [option-K#5](https://github.com/option-K/opencode-plugin-mempalace/issues/5): + Even with #4 patched, `mineSync` calls `mempalace mine `, + which the *remote* daemon evaluates against ITS OWN filesystem. + For multi-host setups (palace-daemon on a different machine + from OpenCode), the daemon returns 400 because the local + path doesn't exist on its filesystem. + + The bundled plugin sidesteps both by: + + 1. Subscribing to `session.idle` / `session.deleted` / + `session.status[idle]` directly (no message counter). + 2. Reading OpenCode's local SQLite session DB client-side. + 3. POSTing the extracted transcript to the daemon's + `/silent-save` endpoint (the same endpoint MemPalace's + Claude Code stop hook uses). + + The Python helper imports `_extract_session_messages` and + `_session_transcript` from `mempalace/sources/opencode.py` so + the transcript shape matches `OpenCodeSourceAdapter` exactly. + + Also splits the previously combined option-K patch into two + independently-applicable files: + + - `examples/opencode/option-k-plugin-daemon-routing.patch` — + Fix 1 (option-K#1, `isInitialized()` daemon detection). + - `examples/opencode/option-k-plugin-message-updated.patch` — + Fix 2 (option-K#4, `chat.message` → `message.updated`). + + `docs/integrations/opencode.md` now documents both deployment + options (bundled plugin for remote-daemon, option-K plugin + + patches for local palaces). + files: + - examples/opencode/live-capture/mempalace-live-capture.js + - examples/opencode/live-capture/capture-session.py + - examples/opencode/option-k-plugin-daemon-routing.patch + - examples/opencode/option-k-plugin-message-updated.patch + - docs/integrations/opencode.md + + - id: opencode-integration-recipe + date: 2026-05-21 + bucket: Added + commit: 60dc9e6 + area: Docs + summary: "Documented OpenCode integration recipe (read-side MCP + push plugin + retrospective adapter)" + body: | + Adds `docs/integrations/opencode.md` and an `examples/opencode/` + directory capturing the three-direction OpenCode + MemPalace + integration recipe for daemon-routed setups: + + - **Read** — `~/.config/opencode/opencode.jsonc` MCP entry pointing + at `palace-daemon/clients/mempalace-mcp-wrapper.sh` (sources + `~/.config/palace-daemon/env` so the API key never lands in + plaintext config). + - **Push (live)** — option-K's `opencode-plugin-mempalace` npm + package (project-basename wings, default 15-message threshold, + session.idle flush, SIGINT/SIGTERM rescue, pre-compaction + injection — closest match to MemPalace's Claude Code stop-hook + pattern). + - **Pull (retrospective)** — the cherry-picked + `OpenCodeSourceAdapter` from upstream PR #1484, run via + `mempalace mine --source opencode` for one-shot backfill of + historical sessions. + + Includes a re-applicable patch + (`examples/opencode/option-k-plugin-daemon-routing.patch`) for + option-K's plugin v1.2.1 issue #1, where `isInitialized()` passes + `--palace /.mempalace/palace` to `mempalace status`, + forcing a local-store lookup that bypasses `PALACE_DAEMON_URL` + routing. Without the patch the plugin re-runs `mempalace init + --yes ` on every OpenCode start (idempotent against the + daemon, just wasteful); with the patch init+isInitialized + short-circuit when daemon-routed mode is detected. + + Why the recipe lives here rather than upstream: this fork's + single-writer-via-palace-daemon shape doesn't match upstream's + assumed local-CLI install pattern (Milofax #297, geco #1524, and + Dxrk #1567 all assume a local mempalace install). Upstream PRs + will eventually subsume parts of this — when they do, the YAML + entries become removable; for now the recipe captures what + actually works on a daemon-routed box. + files: + - docs/integrations/opencode.md + - examples/opencode/opencode.jsonc.example + - examples/opencode/option-k-plugin-daemon-routing.patch + + - id: opencode-mcp-config-cherry-pick-1567 + date: 2026-05-21 + bucket: Added + commit: ba16b82 + area: CLI + summary: ".opencode/opencode.json — repo-root MCP config so opencode picks up mempalace automatically" + body: | + Cherry-pick of upstream PR #1567 (Dxrk777). Adds + `.opencode/opencode.json` so that running `opencode` in the + mempalace repo root automatically wires `mempalace` as a local + MCP server — useful for contributors who use OpenCode for + development on the project itself. + + Two-commit cherry-pick: + + - `013ac63` — initial config with `command: ["mempalace-mcp"]` + - `ba16b82` — gemini-code-assist review feedback: switch to + `command: ["python", "-m", "mempalace.mcp_server"]` for + portability across install methods (pip vs uv vs dev install) + + This is the **dev/contributor surface** — it lives in the repo + and only matters when running OpenCode against the repo root. + Per-user setups should use `~/.config/opencode/opencode.jsonc` + with the daemon-aware wrapper (see `docs/integrations/opencode.md`). + pr: 1567 + pr_state: OPEN + files: + - .opencode/opencode.json + + - id: opencode-source-adapter-cherry-pick-1484 + date: 2026-05-21 + bucket: Added + commit: 2ffe652 + area: CLI + summary: "OpenCodeSourceAdapter (RFC 002) — retrospective ingest of OpenCode SQLite sessions" + body: | + Cherry-pick of upstream PR #1484. Adds + `mempalace/sources/opencode.py` — an RFC 002 `BaseSourceAdapter` + that ingests OpenCode AI-coding-CLI session transcripts from + `~/.local/share/opencode/opencode.db` into the palace, formatted + to match `convo_miner`'s exchange-pair drawer shape. + + Five-commit cherry-pick: + + - `2c368c6` — initial adapter (482-line `opencode.py`, 6 + opencode-namespaced reference transformations in + `transforms.py`, entry-point registration, 28 tests, sample + SQLite-schema-verbatim fixture) + - `3ff7043` — gemini-code-assist review fixes: missing + `opencode_session_version` in metadata (broke incremental + `is_current`), `_skip_requested` private access, `filed_at` + hoisted out of chunk loop, PEP 8 import position + - `9531532` — igorls review fixes: ruff F401/E402 in tests, + route-hint wing/drawer-stage precedence mismatch (RFC 002 §2.5), + unjustified `# noqa` cleanup + - `18ab021` + `2ffe652` — CI ruff 0.4.x format passes + + Adapter conformance via RFC 002 §7.3 declared-transformation + round-trip; one drawer per exchange-pair; `source_file` shape + `opencode://#session=`; wing routes from + `session.directory` basename (matching the live-capture plugin's + taxonomy); incremental ingest works via `opencode_session_version`. + + Originated from JakobSachs's spadework on upstream PR #23 (DB + schema reverse engineering, session/message/part traversal, + tool-input/tool-output stripping). PR #23 is still OPEN but + CONFLICTING and unresponsive since 2026-04-08; #1484 carries + `Co-authored-by: JakobSachs` per coordination on #23. + tests: "28 OpenCode adapter tests pass; full suite 2133 passed / 33 skipped (zero regressions on fork main + 60-commit upstream sync baseline)" + pr: 1484 + pr_state: OPEN + files: + - mempalace/sources/opencode.py + - mempalace/sources/transforms.py + - mempalace/sources/context.py + - pyproject.toml + - tests/test_sources_opencode.py + - tests/fixtures/opencode/sample_session_2026_05_12/README.md + - tests/fixtures/opencode/sample_session_2026_05_12/build_fixture.py + - tests/test_corpus_origin_integration.py + - id: mempalace-walk-palace-mcp-tool date: 2026-05-17 bucket: Added diff --git a/docs/integrations/opencode.md b/docs/integrations/opencode.md new file mode 100644 index 0000000000..50c9f70edb --- /dev/null +++ b/docs/integrations/opencode.md @@ -0,0 +1,227 @@ +# OpenCode + MemPalace integration (fork-routed via palace-daemon) + +Two-direction integration between [OpenCode](https://opencode.ai) and a MemPalace running behind palace-daemon: + +| Direction | Mechanism | What you get | +|---|---|---| +| **Read** (agent → palace) | MCP server entry in `~/.config/opencode/opencode.jsonc` pointing at the daemon-aware stdio wrapper | OpenCode agents can call `mempalace_search`, `mempalace_kg_query`, `mempalace_diary_read`, etc. | +| **Push** (live capture, conversation → palace) | This fork's `examples/opencode/live-capture/` plugin — small JS shim + Python helper that POSTs to the daemon's `/silent-save`. For local palaces, [`opencode-plugin-mempalace`](https://www.npmjs.com/package/opencode-plugin-mempalace) (option-K) works after two patches in `examples/opencode/`. | On every `session.idle`, the current session's transcript is extracted from OpenCode's SQLite DB and POSTed to the palace as a diary entry | +| **Pull** (retrospective backfill, OpenCode SQLite → palace) | This fork's `OpenCodeSourceAdapter` (cherry-picked from upstream PR #1484) | One-shot ingest of historical OpenCode sessions from `~/.local/share/opencode/opencode.db` | + +Together these match the same shape as MemPalace's Claude Code stop-hook pattern: live capture during use, retrospective fill-in when needed, and read-side MCP for agent recall. + +## Why fork-ahead + +This fork carries the integration surface ahead of upstream because the canonical merge points are still open: + +| Upstream PR | What | Status (last checked 2026-05-21) | +|---|---|---| +| [#1484](https://github.com/MemPalace/mempalace/pull/1484) | `OpenCodeSourceAdapter` (RFC 002) | OPEN — CI green except a transient test-windows runner failure | +| [#1567](https://github.com/MemPalace/mempalace/pull/1567) | `.opencode/opencode.json` MCP config in repo root | OPEN | +| [#23](https://github.com/MemPalace/mempalace/pull/23) | Earlier OpenCode SQLite spadework (JakobSachs) | OPEN, CONFLICTING — superseded by #1484; my comment on #23 offered three coordination paths | +| [#297](https://github.com/MemPalace/mempalace/pull/297) | Milofax's auto-plugin | OPEN — codebase-mining + protocol injection design; not what this fork uses | +| [#1524](https://github.com/MemPalace/mempalace/pull/1524) | geco's npm-plugin integration guide | OPEN — uses opinionated 5-wing taxonomy that doesn't fit project-keyed palaces | + +The cherry-picks land #1484 + #1567 onto this fork's `main` so the fork ships with OpenCode support immediately. When upstream merges happen, the cherry-picked commits become no-ops and the fork-changes.yaml entries can be retired. + +## Setup recipe + +### 1. Install MemPalace CLI + +```bash +pipx install "mempalace>=3.3.5" +``` + +`mempalace` and `mempalace-mcp` end up on PATH. The CLI auto-routes through palace-daemon when `PALACE_DAEMON_URL` is in the env. + +### 2. Daemon env file + +Put the daemon URL + API key in `~/.config/palace-daemon/env` (mode 600): + +``` +PALACE_API_KEY= +PALACE_DAEMON_URL=http://your-daemon-host:8085 +``` + +### 3. MCP wrapper + +The mempalace-mcp stdio bridge needs `PALACE_API_KEY` in its environment, but MCP clients (OpenCode, Claude Code) spawn server subprocesses without inheriting shell rc. The wrapper at `palace-daemon/clients/mempalace-mcp-wrapper.sh` (see [palace-daemon PR #26](https://github.com/techempower-org/palace-daemon/pull/26)) sources the env file before exec'ing the bridge: + +```jsonc +{ + "$schema": "https://opencode.ai/config.json", + "mcp": { + "mempalace": { + "type": "local", + "command": ["/home//Projects/palace-daemon/clients/mempalace-mcp-wrapper.sh"], + "enabled": true + } + } +} +``` + +This goes in `~/.config/opencode/opencode.jsonc`. + +### 4. Live-capture plugin + +Two options, depending on whether your palace-daemon is local or remote: + +#### Option A (recommended for daemon-routed setups): this fork's `examples/opencode/live-capture/` + +Drop the bundled plugin + helper into OpenCode's global plugin directory: + +```bash +mkdir -p ~/.config/opencode/plugins +cp $REPO/examples/opencode/live-capture/mempalace-live-capture.js ~/.config/opencode/plugins/ +``` + +The plugin subscribes to `session.idle` / `session.deleted` / `session.status[idle]` +and, on each idle, spawns the companion `capture-session.py` helper which: + +1. Reads OpenCode's local SQLite session DB + (`~/.local/share/opencode/opencode.db`). +2. Extracts the role-pair transcript using this fork's + `OpenCodeSourceAdapter` (RFC 002 contract). +3. POSTs the transcript to the daemon's `/silent-save` endpoint. + +Why this fork ships its own plugin (instead of just using +`opencode-plugin-mempalace`): the option-K plugin is broken for +daemon-routed setups in two compounding ways. See +[Compatibility notes — option-K plugin](#compatibility-notes--option-k-plugin) below. + +The bundled plugin needs the helper script on disk to import the adapter +helpers. By default it looks at +`~/Projects/memorypalace/examples/opencode/live-capture/capture-session.py`; +override with the env var `MEMPALACE_LIVE_CAPTURE_SCRIPT` if your checkout +lives elsewhere. Failure output is logged to +`~/.local/share/opencode/mempalace-live-capture.log` (append-only). Set +`MEMPALACE_LIVE_CAPTURE_DEBUG=1` to also surface plugin-side notes via +OpenCode's normal log. + +#### Option B (local palaces only): the option-K npm plugin + +If your palace-daemon runs on the same host as OpenCode (or you don't use +the daemon at all and have a local palace under `~/.mempalace/palace`), +the upstream option-K plugin will work after applying the two patches in +[Compatibility notes — option-K plugin](#compatibility-notes--option-k-plugin): + +```bash +npm install -g opencode-plugin-mempalace +patch -d ~/.npm-global/lib/node_modules/opencode-plugin-mempalace/dist \ + < $REPO/examples/opencode/option-k-plugin-daemon-routing.patch +patch -d ~/.npm-global/lib/node_modules/opencode-plugin-mempalace/dist \ + < $REPO/examples/opencode/option-k-plugin-message-updated.patch +``` + +Add to `opencode.jsonc`: + +```jsonc +{ + "plugin": ["opencode-plugin-mempalace"] +} +``` + +The plugin uses project-basename wings (`wing_`), +default 15-message threshold, session.idle flush, SIGINT/SIGTERM rescue, +pre-compaction injection. Closest semantics to MemPalace's Claude Code +stop-hook — but only works against a *local* palace because of the +"path-context" issue described in the compatibility notes. + +### Compatibility notes — option-K plugin + +`opencode-plugin-mempalace` v1.2.1 has three known issues filed upstream: + +| Issue | What | Patch in this fork | +|---|---|---| +| [option-K#1](https://github.com/option-K/opencode-plugin-mempalace/issues/1) | `isInitialized()` passes `--palace` as a positional arg, forcing local-only behavior on daemon setups | `examples/opencode/option-k-plugin-daemon-routing.patch` | +| [option-K#4](https://github.com/option-K/opencode-plugin-mempalace/issues/4) | Plugin subscribes to `chat.message`, which OpenCode never publishes — counter never increments, plugin never mines | `examples/opencode/option-k-plugin-message-updated.patch` | +| [option-K#5](https://github.com/option-K/opencode-plugin-mempalace/issues/5) | Calls `mempalace mine `; remote daemon evaluates `` against its own filesystem, returns 400 (architectural) | No patch — Option A bypasses this entirely | + +#4 is the load-bearing bug for any setup: without it the plugin appears to +work (MCP connects, LLM responds) but **writes zero drawers**. #5 means +even with #4 patched, remote daemon setups still can't mine via the option-K +plugin — which is why this fork ships its own plugin (Option A). + +Re-apply both patches after `npm update opencode-plugin-mempalace` (they +are idempotent — `patch --dry-run` first to confirm). + +### 5. Retrospective backfill + +After install, ingest historical OpenCode sessions in one shot using the +bundled helper directly (the equivalent `mempalace mine --source opencode` +CLI flag will land with the upstream PR #1484 merge): + +```bash +# Sweep the last 100 sessions (idempotent — daemon dedupes via entry hash): +python $REPO/examples/opencode/live-capture/capture-session.py --recent 100 +``` + +The helper script reads `~/.local/share/opencode/opencode.db` (or the macOS +`~/Library/Application Support/opencode/...` path), yields one drawer per +session-exchange-pair, and POSTs through the daemon's `/silent-save` +endpoint. Wing routes from `session.directory` basename, matching the +live-capture plugin's taxonomy. + +## What gets stored + +A typical OpenCode turn produces a drawer like: + +| Field | Value | +|---|---| +| `wing` | `wing_` (matches both adapter and live-capture plugin) | +| `room` | content-detected via `convo_miner.detect_convo_room` (`technical` / `decisions` / `problems` / etc.) | +| `source_file` | `opencode://#session=` (adapter) or session export path (plugin) | +| `content` | verbatim user + assistant text, no summarization | +| `extract_mode` | `exchange` | +| Adapter-specific metadata | `session_id`, `session_title`, `project_dir`, `session_created_at`, `message_count`, `opencode_session_version`, `opencode_db_path` | + +## Read-side recall + +OpenCode agents can call any of the 30 MCP tools the daemon exposes: + +- `mempalace_search` — semantic search across all drawers +- `mempalace_list_wings` / `mempalace_list_rooms` / `mempalace_get_taxonomy` — palace navigation +- `mempalace_kg_query` / `mempalace_kg_timeline` — knowledge graph +- `mempalace_diary_read` / `mempalace_diary_write` — agent diaries +- `mempalace_traverse` / `mempalace_find_tunnels` — cross-wing connections + +The daemon serializes all writes through a single chokepoint, so multiple OpenCode windows + Claude Code + the live-capture plugin all coexist without HNSW corruption. + +### A note on automatic context injection + +The option-K plugin ships `experimental.session.compacting` and +`experimental.chat.system.transform` hooks that would, in principle, +inject palace context into the LLM's system prompt before every turn. +As of OpenCode 1.15.7: + +- `experimental.session.compacting` is the only one that exists in the + plugin API. It fires when the conversation is about to be compacted — + rare in short sessions and never in `opencode run` mode. +- `experimental.chat.system.transform` is **not** in the documented plugin + hook list ([opencode.ai/docs/plugins](https://opencode.ai/docs/plugins)) + and does not fire. + +So today, agents recall memories by explicitly invoking the MCP tools +(`mempalace_search` etc.) — not via implicit per-turn system-prompt +injection. The instructions in +[`MEMPALACE.md`](https://platform.claude.com/docs/en/claude-code/memory)-style +project memory help nudge the agent to consult `mempalace_search` +proactively. + +## Verifying the integration + +After setup, in any OpenCode session: + +``` +> Use mempalace_status to confirm we're connected. +``` + +Expected: agent calls the tool and reports drawer count + wing list. If you see "no palace found" or auth errors, check that the wrapper script can read `~/.config/palace-daemon/env`. + +To confirm live-capture is firing, watch `mempalace status` over a few minutes of OpenCode use — drawer count should increment. + +## Coordination notes + +- Upstream PR #1484 carries co-authored credit to JakobSachs for the original DB-schema spadework on PR #23. +- This fork-ahead carries 5 commits from #1484 + 2 commits from #1567 (see `docs/fork-changes.yaml` `opencode-adapter-cherry-pick` and `opencode-mcp-config-cherry-pick` entries). +- option-K's plugin v1.2.1 has three open issues filed by JP — [#1](https://github.com/option-K/opencode-plugin-mempalace/issues/1) (daemon-routing), [#4](https://github.com/option-K/opencode-plugin-mempalace/issues/4) (`chat.message` vs `message.updated`), and [#5](https://github.com/option-K/opencode-plugin-mempalace/issues/5) (remote-daemon path mismatch). #1 and #4 have patches in `examples/opencode/`; #5 is architectural — the bundled `examples/opencode/live-capture/` plugin sidesteps it by reading OpenCode's SQLite DB client-side and POSTing drawers via the daemon's `/silent-save` endpoint. diff --git a/examples/opencode/live-capture/capture-session.py b/examples/opencode/live-capture/capture-session.py new file mode 100755 index 0000000000..ee70aaaaf9 --- /dev/null +++ b/examples/opencode/live-capture/capture-session.py @@ -0,0 +1,261 @@ +#!/usr/bin/env python3 +"""OpenCode → palace-daemon live capture (POST one session to /silent-save). + +Reads OpenCode's local SQLite session DB, extracts the role-pair transcript +for one session using the in-tree ``OpenCodeSourceAdapter`` (RFC 002 contract), +and POSTs the result to the daemon's ``/silent-save`` endpoint. + +Why this exists +--------------- + +``opencode-plugin-mempalace`` v1.2.1's mining path is broken in two ways +for daemon-routed setups: + +1. It subscribes to ``chat.message``, which OpenCode never publishes. The + message counter never increments and the plugin never mines. (Filed + upstream as `option-K#4 `_.) +2. Even with #1 patched, it shells out to ``mempalace mine ``. With a + remote palace-daemon, the daemon evaluates ```` against ITS OWN + filesystem — not the client's — and returns 400 because the local path + doesn't exist there. (Filed upstream as `option-K#5 + `_.) + +This script sidesteps both bugs by reading opencode.db client-side and +POSTing drawer content directly. It's intentionally minimal: + +* No CLI deps beyond Python stdlib + the daemon URL/API key from env. +* Idempotent: the daemon uses the ``session_id + entry hash`` as the + silent-save key, so re-runs don't duplicate drawers (this is also how + Claude Code's stop hook behaves). +* Wing routing: takes ``--wing`` or derives ``wing_`` + from ``--cwd``, matching the live-capture plugin's taxonomy. + +Usage +----- + +:: + + capture-session.py --session-id ses_abc123 --cwd /home/jp/Projects/foo + + # Or by latest N sessions: + capture-session.py --recent 5 + +Env +--- + +* ``PALACE_DAEMON_URL`` — required (e.g. ``http://localhost:8085``). +* ``PALACE_API_KEY`` — required (passed as ``X-API-Key`` header). + +Exit codes +---------- + +* 0 — success (drawer POST returned 200) +* 1 — daemon error / network / missing env +* 2 — session not found in opencode.db +* 3 — empty transcript (nothing to save) +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sqlite3 +import sys +import urllib.error +import urllib.request +from pathlib import Path +from typing import List, Optional, Tuple + +# We rely on the in-tree adapter helpers (the same code the +# OpenCodeSourceAdapter calls). Importing keeps the transcript exactly +# what `mempalace mine --source opencode` would emit when that CLI lands. +THIS_DIR = Path(__file__).resolve().parent +REPO_ROOT = THIS_DIR.parents[2] # examples/opencode/live-capture -> repo root +sys.path.insert(0, str(REPO_ROOT)) + +from mempalace.sources.opencode import ( # noqa: E402 + _resolve_db, + _extract_session_messages, + _session_transcript, +) + + +def _sanitize_wing(name: str) -> str: + """Normalize a string into a wing name (mirrors plugin getWingFromPath).""" + cleaned = re.sub(r"[^a-zA-Z0-9_]+", "_", name).strip("_").lower() + return f"wing_{cleaned}" if cleaned else "wing_opencode" + + +def _wing_from_cwd(cwd: Optional[str]) -> str: + if not cwd: + return "wing_opencode" + return _sanitize_wing(Path(cwd).name) + + +def _recent_session_ids(conn: sqlite3.Connection, limit: int) -> List[str]: + rows = conn.execute( + "SELECT id FROM session ORDER BY time_created DESC LIMIT ?", + (limit,), + ).fetchall() + return [r[0] for r in rows] + + +def _session_meta(conn: sqlite3.Connection, session_id: str) -> Tuple[Optional[str], int]: + """Return (directory, message_count) for the session, or (None, 0). + + The OpenCode session schema stores ``directory`` as a top-level column + (verified on opencode-ai 1.15.7). Older versions kept it inside a + ``data`` JSON column; fall back gracefully if the column isn't present. + """ + directory: Optional[str] = None + try: + row = conn.execute("SELECT directory FROM session WHERE id=?", (session_id,)).fetchone() + if row: + directory = row[0] + except sqlite3.OperationalError: + # Older schema with `data` JSON column + row = conn.execute("SELECT data FROM session WHERE id=?", (session_id,)).fetchone() + if row: + try: + sj = json.loads(row[0]) + directory = sj.get("directory") + except (json.JSONDecodeError, TypeError): + pass + n = conn.execute("SELECT COUNT(*) FROM message WHERE session_id=?", (session_id,)).fetchone()[0] + return directory, int(n) + + +def _post_silent_save( + daemon_url: str, + api_key: str, + session_id: str, + wing: str, + entry: str, + topic: Optional[str], + message_count: int, +) -> dict: + body = json.dumps( + { + "session_id": session_id, + "wing": wing, + "entry": entry, + "topic": topic or "opencode session", + "agent_name": "opencode-live-capture", + "message_count": message_count, + } + ).encode("utf-8") + req = urllib.request.Request( + f"{daemon_url.rstrip('/')}/silent-save", + data=body, + headers={ + "Content-Type": "application/json", + "X-API-Key": api_key, + }, + method="POST", + ) + with urllib.request.urlopen(req, timeout=30) as r: + return json.loads(r.read().decode("utf-8")) + + +def _capture_one( + conn: sqlite3.Connection, + daemon_url: str, + api_key: str, + session_id: str, + wing_override: Optional[str], + cwd_override: Optional[str], + dry_run: bool, +) -> int: + pairs = _extract_session_messages(conn, session_id) + if not pairs: + print(f"[capture] {session_id}: empty transcript — skipped", file=sys.stderr) + return 3 + transcript = _session_transcript(pairs) + directory, total_messages = _session_meta(conn, session_id) + wing = wing_override or _wing_from_cwd(cwd_override or directory) + # Daemon's silent_save rejects topics with path separators (validates as + # a path-safe slug). Use a flat session id form. + topic = f"opencode_session_{session_id}" + if dry_run: + print( + f"[capture] DRY-RUN session={session_id} wing={wing} " + f"messages={total_messages} entry_chars={len(transcript)}" + ) + return 0 + try: + result = _post_silent_save( + daemon_url, api_key, session_id, wing, transcript, topic, total_messages + ) + except urllib.error.HTTPError as e: + print( + f"[capture] {session_id}: HTTP {e.code} — {e.read().decode('utf-8', 'replace')[:300]}", + file=sys.stderr, + ) + return 1 + except urllib.error.URLError as e: + print(f"[capture] {session_id}: network error — {e}", file=sys.stderr) + return 1 + entry_id = result.get("entry_id", "?") + count = result.get("count", "?") + print(f"[capture] {session_id}: saved entry_id={entry_id} count={count} wing={wing}") + return 0 + + +def main(argv: Optional[List[str]] = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + parser.add_argument("--session-id", help="OpenCode session id (e.g. ses_abc123)") + parser.add_argument( + "--recent", + type=int, + default=0, + help="Capture the N most recent sessions (skip --session-id when set)", + ) + parser.add_argument( + "--cwd", + help="Project working directory (used to derive wing if no --wing given)", + ) + parser.add_argument("--wing", help="Explicit wing override") + parser.add_argument("--db", help="Path to opencode.db (default: auto-detect)") + parser.add_argument( + "--dry-run", action="store_true", help="Print what would be POSTed; don't write" + ) + args = parser.parse_args(argv) + + daemon_url = os.environ.get("PALACE_DAEMON_URL") + api_key = os.environ.get("PALACE_API_KEY") + if not daemon_url or not api_key: + print( + "PALACE_DAEMON_URL and PALACE_API_KEY must be set (try `source ~/.config/palace-daemon/env`).", + file=sys.stderr, + ) + return 1 + + try: + db_path = _resolve_db(args.db) + except Exception as e: + print(f"could not locate opencode.db: {e}", file=sys.stderr) + return 1 + conn = sqlite3.connect(db_path) + try: + if args.recent > 0: + sids = _recent_session_ids(conn, args.recent) + elif args.session_id: + sids = [args.session_id] + else: + parser.error("--session-id or --recent N is required") + return 1 + rc = 0 + for sid in sids: + rc = ( + _capture_one(conn, daemon_url, api_key, sid, args.wing, args.cwd, args.dry_run) + or rc + ) + return rc + finally: + conn.close() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/opencode/live-capture/mempalace-live-capture.js b/examples/opencode/live-capture/mempalace-live-capture.js new file mode 100644 index 0000000000..ad588172bd --- /dev/null +++ b/examples/opencode/live-capture/mempalace-live-capture.js @@ -0,0 +1,159 @@ +// MemPalace live-capture plugin for OpenCode (daemon-routed setups). +// +// Place this file at one of: +// ~/.config/opencode/plugins/mempalace-live-capture.js (global, all projects) +// /.opencode/plugins/mempalace-live-capture.js (per-project) +// +// What it does +// ------------ +// On every `session.idle` event, it spawns the companion +// `capture-session.py` script (Python stdlib only) which: +// 1. Reads OpenCode's local SQLite session DB. +// 2. Extracts the role-pair transcript using the in-tree +// OpenCodeSourceAdapter (RFC 002 contract). +// 3. POSTs the transcript to the daemon's /silent-save endpoint. +// +// Why not use option-K's opencode-plugin-mempalace? +// ------------------------------------------------ +// It's broken for daemon-routed setups in two compounding ways +// (filed upstream as option-K#4 and option-K#5): +// - Subscribes to `chat.message`, which OpenCode never publishes. +// - Even when patched, calls `mempalace mine `, which the +// remote daemon evaluates against its own filesystem (404). +// +// This plugin bypasses both bugs by doing the extraction client-side and +// POSTing drawer content directly. It is intentionally minimal: +// - No state machine, no threshold counter — every `session.idle` POSTs. +// - The daemon's silent-save endpoint deduplicates by entry hash, so +// repeated POSTs of the same transcript do not create duplicate drawers. +// - No dependency on the option-K plugin or its npm install. +// +// Requirements +// ------------ +// * MemPalace repo cloned somewhere (the script imports adapter helpers). +// * Python 3.9+ on PATH (no extra pip deps). +// * Env: PALACE_DAEMON_URL and PALACE_API_KEY set in the shell that +// launches opencode (the plugin inherits process.env). +// +// Configure CAPTURE_SCRIPT below to point at your local checkout's +// capture-session.py, e.g. ~/Projects/memorypalace/examples/opencode/live-capture/capture-session.py +import { spawn } from 'child_process'; +import { existsSync, mkdirSync, openSync } from 'fs'; +import path from 'path'; + +const CAPTURE_SCRIPT = + process.env.MEMPALACE_LIVE_CAPTURE_SCRIPT || + path.join( + process.env.HOME || '', + 'Projects/memorypalace/examples/opencode/live-capture/capture-session.py' + ); + +function _firstExistingPython() { + const candidates = [ + process.env.MEMPALACE_PYTHON, + path.join(process.env.HOME || '', 'Projects/memorypalace/.venv/bin/python3'), + path.join(process.env.HOME || '', 'Projects/memorypalace/venv/bin/python3'), + '/usr/bin/python3', + '/usr/local/bin/python3', + 'python3', + ]; + for (const c of candidates) { + if (!c) continue; + if (c === 'python3') return c; // PATH resolution + try { + if (existsSync(c)) return c; + } catch (e) { /* ignore */ } + } + return 'python3'; +} +const PYTHON = _firstExistingPython(); + +function log(...args) { + // Plugin stdout/stderr surfaces in ~/.local/share/opencode/log/*.log as + // `service=plugin path= ...`. Keep noise quiet by default. + if (process.env.MEMPALACE_LIVE_CAPTURE_DEBUG) { + console.warn('[mempalace-live-capture]', ...args); + } +} + +function captureSession(sessionID, cwd) { + if (!sessionID) return; + if (!existsSync(CAPTURE_SCRIPT)) { + log(`capture script not found: ${CAPTURE_SCRIPT}`); + return; + } + if (!process.env.PALACE_DAEMON_URL || !process.env.PALACE_API_KEY) { + log('PALACE_DAEMON_URL / PALACE_API_KEY not set — skipping'); + return; + } + const args = ['--session-id', sessionID]; + if (cwd) { + args.push('--cwd', cwd); + } + // Route the capture's stdout+stderr to a rotated log so failures aren't + // invisible. Default location: ~/.local/share/opencode/mempalace-live-capture.log. + const logFile = + process.env.MEMPALACE_LIVE_CAPTURE_LOG || + path.join( + process.env.HOME || '', + '.local/share/opencode/mempalace-live-capture.log' + ); + let outStream; + try { + mkdirSync(path.dirname(logFile), { recursive: true }); + outStream = openSync(logFile, 'a'); + } catch (e) { + // If we can't open the log file, fall back to ignoring (capture + // failures stay invisible but the session itself isn't blocked). + outStream = 'ignore'; + } + const child = spawn(PYTHON, [CAPTURE_SCRIPT, ...args], { + env: process.env, + // Detach so the plugin's event handler returns immediately; the + // POST happens asynchronously. We don't await success because + // opencode may dispose the plugin host before /silent-save returns + // (especially in `opencode run` mode). + detached: true, + stdio: ['ignore', outStream, outStream], + }); + child.unref(); + log(`spawned capture for ${sessionID} (pid=${child.pid})`); +} + +export default async function mempalaceLiveCapturePlugin(input) { + const cwd = input?.worktree || input?.directory || process.cwd(); + log(`loaded; cwd=${cwd}, capture=${CAPTURE_SCRIPT}, python=${PYTHON}`); + + // Per-session debounce. session.idle and session.status[idle] both fire + // around the same moment for one logical idle event; we coalesce them + // (and any reentry within DEBOUNCE_MS) into a single capture call. + // The daemon would dedupe by entry-hash anyway, but reducing the spawn + // count keeps the log + load lighter. + const DEBOUNCE_MS = 1500; + const lastCapture = new Map(); // sessionID -> timestamp + + return { + event: async ({ event }) => { + const t = event?.type; + if ( + t === 'session.idle' || + t === 'session.deleted' || + (t === 'session.status' && event.properties?.status?.type === 'idle') + ) { + const sessionID = + event.properties?.sessionID || + event.properties?.info?.sessionID || + event.properties?.info?.id; + if (!sessionID) return; + const now = Date.now(); + const last = lastCapture.get(sessionID) || 0; + if (now - last < DEBOUNCE_MS) { + log(`debounced ${t} for ${sessionID} (${now - last}ms since last)`); + return; + } + lastCapture.set(sessionID, now); + captureSession(sessionID, cwd); + } + }, + }; +} diff --git a/examples/opencode/opencode.jsonc.example b/examples/opencode/opencode.jsonc.example new file mode 100644 index 0000000000..1897b4b9db --- /dev/null +++ b/examples/opencode/opencode.jsonc.example @@ -0,0 +1,32 @@ +{ + "$schema": "https://opencode.ai/config.json", + + // Provider/model — set to whatever you actually use; the example uses + // bedrock as that's how this fork's author runs it. + "provider": { + "amazon-bedrock": { + "options": { + "region": "us-west-1" + } + } + }, + "model": "amazon-bedrock/us.anthropic.claude-opus-4-7", + + // === MemPalace via palace-daemon === + // Read-side MCP — agents can call mempalace_search / mempalace_kg_query / etc. + // The wrapper sources ~/.config/palace-daemon/env (mode 600) before exec'ing + // the daemon's stdio bridge, so PALACE_API_KEY never lands in this config. + "mcp": { + "mempalace": { + "type": "local", + "command": ["/home//Projects/palace-daemon/clients/mempalace-mcp-wrapper.sh"], + "enabled": true + } + }, + + // Push-side live capture — option-K's npm plugin. After install: + // npm install -g opencode-plugin-mempalace + // Optional: pass { "threshold": N } as the second tuple element to override + // the default 15-message mining cadence. + "plugin": ["opencode-plugin-mempalace"] +} diff --git a/examples/opencode/option-k-plugin-daemon-routing.patch b/examples/opencode/option-k-plugin-daemon-routing.patch new file mode 100644 index 0000000000..b4133d3bb9 --- /dev/null +++ b/examples/opencode/option-k-plugin-daemon-routing.patch @@ -0,0 +1,76 @@ +# Patch for opencode-plugin-mempalace v1.2.1 — Fix 1 of 2 (daemon routing) +# +# File: mempalace-cli.js +# Why: option-K's plugin calls `mempalace status --palace /.mempalace/palace` +# inside isInitialized(), which forces a local-palace lookup and bypasses +# $PALACE_DAEMON_URL routing. On a daemon-routed setup the function always +# returns false, so the plugin re-runs `mempalace init --yes ` on every +# OpenCode start. Wasteful but harmless — init is idempotent against the daemon. +# +# Filed upstream as https://github.com/option-K/opencode-plugin-mempalace/issues/1 +# +# Apply with: +# patch -d ~/.npm-global/lib/node_modules/opencode-plugin-mempalace/dist \ +# < examples/opencode/option-k-plugin-daemon-routing.patch +# +# Re-apply after `npm update opencode-plugin-mempalace`. +# +# Companion patch for the message-event hook bug is at +# examples/opencode/option-k-plugin-message-updated.patch +# +--- mempalace-cli.js.orig 2026-05-21 17:00:00.000000000 -0700 ++++ mempalace-cli.js 2026-05-21 17:00:00.000000000 -0700 +@@ -1,5 +1,26 @@ + import execa from 'execa'; + import path from 'path'; ++import { existsSync } from 'fs'; ++ ++// JP-PATCH 2026-05-21: skip the local-palace heuristics when running against ++// a remote palace-daemon. Detection: $PALACE_DAEMON_URL env var, or the ++// presence of ~/.config/palace-daemon/env (the canonical config file the ++// mempalace-mcp-wrapper.sh sources). ++// ++// Without this patch: ++// isInitialized(dir) calls `mempalace status --palace /.mempalace/palace`, ++// which forces a *local* palace lookup at a path that doesn't exist on a ++// daemon-routed setup, returns false, and the plugin re-runs `mempalace init` ++// on every opencode start. Wasteful but harmless. ++// With this patch: ++// isInitialized(dir) treats daemon-reachable === initialized, no false-init ++// loop. Local-only mode keeps the upstream behavior. ++function isDaemonRouted() { ++ if (process.env.PALACE_DAEMON_URL) return true; ++ const envFile = path.join(process.env.HOME || '', '.config/palace-daemon/env'); ++ return existsSync(envFile); ++} ++ + async function executeMempalace(args, options = {}) { + const defaultOptions = { + timeout: 5000, // 5 seconds timeout to prevent hanging +@@ -26,6 +47,12 @@ + } + export async function isInitialized(dir) { + try { ++ if (isDaemonRouted()) { ++ // Daemon-routed: presence of a reachable daemon === initialized. ++ // No --palace, no per-project local store check. ++ await executeMempalace(['status']); ++ return true; ++ } + const palacePath = path.join(dir, '.mempalace', 'palace'); + await executeMempalace(['status', '--palace', palacePath]); + return true; +@@ -36,6 +63,13 @@ + } + export async function initialize(dir) { + try { ++ if (isDaemonRouted()) { ++ // Daemon-routed: the daemon owns palace state globally. Init is a ++ // no-op from the plugin's perspective — `mempalace init` on a ++ // per-project basis would still hit the daemon, but creates no ++ // useful side effect. ++ return; ++ } + await executeMempalace(['init', '--yes', dir], { input: '\n' }); + } + catch (error) { diff --git a/examples/opencode/option-k-plugin-message-updated.patch b/examples/opencode/option-k-plugin-message-updated.patch new file mode 100644 index 0000000000..369d7c2e1b --- /dev/null +++ b/examples/opencode/option-k-plugin-message-updated.patch @@ -0,0 +1,50 @@ +# Patch for opencode-plugin-mempalace v1.2.1 — Fix 2 of 2 (message hook) +# +# File: index.js +# Why: the plugin registers a `'chat.message'` hook to increment a message +# counter that gates session.idle mining. opencode publishes `message.updated` +# (and `message.part.updated`, `session.updated`, etc.) but **never** +# `chat.message` — verified by inspecting the published bus types in +# ~/.local/share/opencode/log/. With chat.message never firing, the counter +# stays at 0, `hasPendingMessages` returns false on every session.idle, and +# the plugin **never actually mines a drawer**. The plugin appears "working" +# (no errors, MCP connects, LLM responses stream) but writes nothing to the +# palace. Symptom: drawer count baseline == drawer count after a normal +# opencode session. +# +# This patch moves the increment into the `event:` catch-all so it runs on +# `message.updated`. The existing 15-message threshold absorbs the per-token +# streaming chatter (one increment per assistant message update, not per +# token). The dead `chat.message` hook below is left in place — harmless, +# and removing it would create a noisier diff that's harder to reapply. +# +# Filed upstream as https://github.com/option-K/opencode-plugin-mempalace/issues/2 +# +# Apply with: +# patch -d ~/.npm-global/lib/node_modules/opencode-plugin-mempalace/dist \ +# < examples/opencode/option-k-plugin-message-updated.patch +# +# Re-apply after `npm update opencode-plugin-mempalace`. +# +# Companion patch for the daemon-routing bug is at +# examples/opencode/option-k-plugin-daemon-routing.patch +# +--- index.js.orig 2026-05-21 18:00:00.000000000 -0700 ++++ index.js 2026-05-21 18:00:00.000000000 -0700 +@@ -61,6 +61,16 @@ + }); + return { + event: async ({ event }) => { ++ // JP-PATCH 2026-05-21: count opencode's real message events into the ++ // threshold. The plugin's existing 'chat.message' hook (below) is ++ // dead — opencode publishes 'message.updated' not 'chat.message'. ++ // Without this, hasPendingMessages() returns false on every ++ // session.idle and the plugin never mines. ++ if (event.type === 'message.updated') { ++ const sessionID = event.properties?.sessionID || event.properties?.info?.sessionID || event.properties?.info?.id; ++ if (sessionID) ++ stateManager.incrementAndCheck(sessionID); ++ } + if (event.type === 'session.idle' || + event.type === 'session.deleted' || + (event.type === 'session.status' && event.properties?.status?.type === 'idle')) { diff --git a/mempalace/sources/context.py b/mempalace/sources/context.py index c5b8644ddf..457222d01e 100644 --- a/mempalace/sources/context.py +++ b/mempalace/sources/context.py @@ -114,6 +114,16 @@ def skip_current_item(self) -> None: advancing past the item.""" self._skip_requested = True + def is_skip_requested(self) -> bool: + """Return whether :meth:`skip_current_item` was called since the last + time core advanced past an item. Adapters check this between the + ``SourceItemMetadata`` yield and the cost of processing the item — if + core has signaled skip (because :meth:`is_current` returned True), the + adapter can bail out of expensive work (SQL queries, chunking, etc.) + rather than waiting for core to drop drawers downstream. Core resets + the flag on its own; adapters MUST NOT clear it.""" + return self._skip_requested + def emit(self, event: str, **details: Any) -> None: """Invoke each registered progress hook with ``(event, **details)``.""" for hook in self.progress_hooks: diff --git a/mempalace/sources/opencode.py b/mempalace/sources/opencode.py new file mode 100644 index 0000000000..13a0844266 --- /dev/null +++ b/mempalace/sources/opencode.py @@ -0,0 +1,480 @@ +"""OpenCode source adapter (RFC 002). + +Ingests OpenCode AI-coding-CLI session transcripts out of OpenCode's local +SQLite store (default ``~/.local/share/opencode/opencode.db``) into the +palace as :class:`DrawerRecord` instances. + +Each OpenCode session becomes one ``source_file`` of the shape +``opencode://#session=``. The drawers under that +``source_file`` are exchange-pair chunks of the session transcript, +formatted to match the existing ``convo_miner`` shape so downstream +ranking, search, and closet-building behave identically. + +Reverse-engineering credit: the SQLite schema, ``json_extract`` paths, +tool-echo / file-injection skip filters, and same-role merge originated in +@JakobSachs's PR #23 (``feat: add OpenCode SQLite session database +support``). This adapter rebuilds those same primitives on the RFC 002 +contract so it can ship as a registered adapter rather than a normalize.py +branch. +""" + +from __future__ import annotations + +import logging +import sqlite3 +from datetime import datetime, timezone +from pathlib import Path +from typing import Iterator, List, Optional, Tuple + +from ..convo_miner import chunk_exchanges, detect_convo_room +from ..config import normalize_wing_name +from . import transforms as _transforms +from .base import ( + AdapterClosedError, + AdapterSchema, + BaseSourceAdapter, + DrawerRecord, + FieldSpec, + RouteHint, + SourceItemMetadata, + SourceNotFoundError, + SourceRef, + SourceSummary, +) +from .context import PalaceContext + +logger = logging.getLogger(__name__) + + +# Default lookup order for the OpenCode SQLite store. Verified 2026-05-12 on +# opencode-ai 1.14.39 (Linux XDG path); the ``~/.opencode`` legacy location +# is kept for older macOS installs (see PR #23 thread). +_DEFAULT_DB_PATHS: Tuple[str, ...] = ( + "~/.local/share/opencode/opencode.db", + "~/.opencode/opencode.db", +) + + +# Hall-detection helper: defer to convo_miner's cached lookup so an adapter +# instance never re-reads the palace config per drawer. Imported lazily because +# convo_miner module-load imports chromadb on some paths. +def _detect_hall(content: str) -> str: + from ..convo_miner import _detect_hall_cached + + return _detect_hall_cached(content) + + +def _resolve_db(local_path: Optional[str] = None) -> str: + """Resolve a concrete SQLite path for the OpenCode store. + + Order: + 1. ``local_path`` if it points at an existing file (caller chose). + 2. Each entry of :data:`_DEFAULT_DB_PATHS` in declaration order. + + Raises :class:`SourceNotFoundError` if no candidate resolves. + """ + candidates: List[str] = [] + if local_path: + candidates.append(local_path) + candidates.extend(_DEFAULT_DB_PATHS) + for raw in candidates: + p = Path(raw).expanduser() + if p.is_file(): + return str(p.resolve()) + raise SourceNotFoundError( + f"No OpenCode SQLite database found (searched {candidates}). " + f"Pass SourceRef(local_path=) or place the file at one of the " + f"default paths." + ) + + +def _build_source_bytes_per_session(conn: sqlite3.Connection, session_id: str) -> str: + """Return the canonical ``source bytes`` for one OpenCode session. + + The bytes are role-prefixed ``part.data`` JSON values, one per line, in + ``(message.time_created, part.time_created)`` order. This is the shape + the declared OpenCode transformations (``opencode_extract_text_parts`` + onward) consume; the conformance suite uses the same shape so the + declared-transformation round-trip is exact. + """ + rows = conn.execute( + """ + SELECT + json_extract(m.data, '$.role') AS role, + p.data AS part_data + FROM message m + JOIN part p ON p.message_id = m.id + WHERE m.session_id = ? + ORDER BY m.time_created, p.time_created + """, + (session_id,), + ).fetchall() + return "\n".join(f"{role or ''}\t{part}" for role, part in rows) + + +def _extract_session_messages(conn: sqlite3.Connection, session_id: str) -> List[Tuple[str, str]]: + """Walk ``part.data`` for one session and return merged ``(role, text)`` pairs. + + Applies the OpenCode-specific transformations in declaration order using + the reference implementations in :mod:`mempalace.sources.transforms`. + Returns the resulting list of ``(role, body)`` tuples ready for the + exchange-format emit step. + """ + raw = _build_source_bytes_per_session(conn, session_id) + if not raw: + return [] + pipeline = [ + _transforms.opencode_extract_text_parts, + _transforms.opencode_skip_tool_echo, + _transforms.opencode_skip_file_injection, + _transforms.opencode_role_coerce, + _transforms.opencode_same_role_merge, + ] + text = raw + for step in pipeline: + text = step(text) + # Final state is ``role\tbody`` lines; split back into tuples. + pairs: List[Tuple[str, str]] = [] + for line in text.split("\n"): + role, sep, body = line.partition("\t") + if not sep: + continue + pairs.append((role, body)) + return pairs + + +def _session_transcript(messages: List[Tuple[str, str]]) -> str: + """Render merged ``(role, text)`` pairs as exchange-pair markdown. + + Uses the declared ``opencode_format_exchange`` transformation; emits + ``> user-text`` blocks alternating with assistant blocks. + """ + role_lines = "\n".join(f"{r}\t{b}" for r, b in messages) + formatted = _transforms.opencode_format_exchange(role_lines) + formatted = _transforms.newline_normalize(formatted) + return _transforms.whitespace_trim(formatted) + + +def _utc_iso(ms: int) -> str: + """Convert OpenCode millisecond-epoch to ISO-8601 UTC string.""" + return ( + datetime.fromtimestamp(ms / 1000.0, tz=timezone.utc) + .replace(microsecond=0) + .isoformat() + .replace("+00:00", "Z") + ) + + +def session_source_file(db_path: str, session_id: str) -> str: + """Construct the stable per-session ``source_file`` identifier. + + Shape: ``opencode://#session=``. Stable across + re-ingests, used as the ChromaDB ``where={"source_file": ...}`` key and + by ``is_current`` to look up existing drawers. + """ + return f"opencode://{db_path}#session={session_id}" + + +class OpenCodeSourceAdapter(BaseSourceAdapter): + """Mine OpenCode AI-coding-CLI sessions into the palace (RFC 002 §1).""" + + name = "opencode" + adapter_version = "0.1.0" + capabilities = frozenset( + { + "supports_incremental", + "supports_structured_metadata", + "requires_local_tool", # SQLite is python-stdlib but the .db is opencode's + "adapter_owns_routing", + } + ) + supported_modes = frozenset({"chunked_content"}) + declared_transformations = frozenset( + { + "opencode_extract_text_parts", + "opencode_skip_tool_echo", + "opencode_skip_file_injection", + "opencode_role_coerce", + "opencode_same_role_merge", + "opencode_format_exchange", + "newline_normalize", + "whitespace_trim", + } + ) + default_privacy_class = "pii_potential" + + # Order of declared transformations as applied by the adapter. The + # conformance suite walks this list in order, so it MUST mirror the + # actual pipeline in ``_extract_session_messages`` + ``_session_transcript``. + DECLARED_TRANSFORMATION_ORDER: Tuple[str, ...] = ( + "opencode_extract_text_parts", + "opencode_skip_tool_echo", + "opencode_skip_file_injection", + "opencode_role_coerce", + "opencode_same_role_merge", + "opencode_format_exchange", + "newline_normalize", + "whitespace_trim", + ) + + def __init__(self) -> None: + self._closed = False + + # ------------------------------------------------------------------ + # Schema + # ------------------------------------------------------------------ + + def describe_schema(self) -> AdapterSchema: + return AdapterSchema( + version="1.0", + fields={ + "session_id": FieldSpec( + type="string", + required=True, + description="OpenCode session id (e.g. ses_a1b2c3...)", + indexed=True, + ), + "session_title": FieldSpec( + type="string", + required=False, + description="Session title as recorded by OpenCode", + ), + "project_dir": FieldSpec( + type="string", + required=True, + description="Absolute filesystem path where the session was started", + indexed=True, + ), + "session_created_at": FieldSpec( + type="string", + required=True, + description="ISO-8601 UTC of session creation (from time_created ms)", + ), + "message_count": FieldSpec( + type="int", + required=True, + description="Number of merged (role, text) exchange parts in the session", + ), + "extract_mode": FieldSpec( + type="string", + required=True, + description="Always 'exchange' for the OpenCode adapter in v0.1", + ), + "opencode_db_path": FieldSpec( + type="string", + required=True, + description="Absolute path of the OpenCode SQLite database the drawer was extracted from", + ), + }, + ) + + # ------------------------------------------------------------------ + # Ingest + # ------------------------------------------------------------------ + + def ingest( + self, + *, + source: SourceRef, + palace: PalaceContext, + ) -> Iterator[object]: + if self._closed: + raise AdapterClosedError("OpenCodeSourceAdapter is closed") + db_path = _resolve_db(source.local_path) + conn = sqlite3.connect(db_path) + try: + self._verify_schema(conn, db_path) + sessions = conn.execute( + """ + SELECT id, title, directory, time_created, time_updated + FROM session + ORDER BY time_created + """ + ).fetchall() + for sid, title, directory, time_created, time_updated in sessions: + src_file = session_source_file(db_path, sid) + # Yield the lazy-fetch metadata so core can short-circuit when + # the session has not changed since the previous ingest. + yield SourceItemMetadata( + source_file=src_file, + version=str(time_updated or time_created or 0), + size_hint=None, + route_hint=self._route_hint_for(source, directory), + ) + if palace.is_skip_requested(): + continue + + messages = _extract_session_messages(conn, sid) + if len(messages) < 2: + # Skip cancelled / single-turn sessions — matches PR #23 + # behavior and convo_miner's general "skip too-small files" + # heuristic. + logger.debug( + "opencode adapter: skipping session %s (%d messages)", + sid, + len(messages), + ) + continue + + transcript = _session_transcript(messages) + if not transcript: + continue + + chunks = chunk_exchanges(transcript) + if not chunks: + continue + + wing = self._wing_for(source, directory) + room = detect_convo_room(transcript) + created_iso = _utc_iso(time_created or 0) + # Pre-compute once per session so every chunk of the same + # session shares the same filed_at timestamp. + filed_at = ( + datetime.now(timezone.utc) + .replace(microsecond=0) + .isoformat() + .replace("+00:00", "Z") + ) + session_version = str(time_updated or time_created or 0) + for chunk in chunks: + content = chunk["content"] + chunk_index = int(chunk["chunk_index"]) + metadata = { + # Universal §5.1 fields + "source_file": src_file, + "chunk_index": chunk_index, + "filed_at": filed_at, + "added_by": "opencode-adapter", + "wing": wing, + "room": room, + "hall": _detect_hall(content), + "ingest_mode": "chunked_content", + "extract_mode": "exchange", + "privacy_class": self.default_privacy_class, + # Adapter-declared fields (§5.2) + "session_id": sid, + "session_title": title or "", + "project_dir": directory or "", + "session_created_at": created_iso, + "message_count": len(messages), + "opencode_db_path": db_path, + # Required by is_current() for incremental ingest; + # mirrors the SourceItemMetadata.version yielded above. + "opencode_session_version": session_version, + } + yield DrawerRecord( + content=content, + source_file=src_file, + chunk_index=chunk_index, + metadata=metadata, + route_hint=RouteHint(wing=wing, room=room, hall=metadata["hall"]), + ) + finally: + conn.close() + + # ------------------------------------------------------------------ + # Incremental ingest + # ------------------------------------------------------------------ + + def is_current( + self, + *, + item: SourceItemMetadata, + existing_metadata: Optional[dict], + ) -> bool: + if not existing_metadata: + return False + # Existing palace drawers expose either ``session_created_at`` (ISO) + # or an older opaque metadata blob. The cheapest stable comparison is + # the raw ``time_updated`` ms-epoch we encoded into ``version`` — but + # the palace stores ISO. Honor both: if a ``opencode_session_version`` + # field is present (future-proof), use it; otherwise fall back to the + # ISO-vs-ISO comparison against ``session_created_at``. + stored_version = existing_metadata.get("opencode_session_version") + if stored_version is not None: + return str(stored_version) == item.version + # Fall back to "we have drawers for this source_file" → assume current. + # Safer than a default of "always re-extract" because OpenCode session + # rows are append-only: an existing drawer for a session_id means we + # already mined the messages that existed at last extraction. + return True + + def source_summary(self, *, source: SourceRef) -> SourceSummary: + try: + db_path = _resolve_db(source.local_path) + except SourceNotFoundError: + return SourceSummary(description="OpenCode database not found", item_count=0) + conn = sqlite3.connect(db_path) + try: + self._verify_schema(conn, db_path) + (count,) = conn.execute("SELECT COUNT(*) FROM session").fetchone() + finally: + conn.close() + return SourceSummary( + description=f"OpenCode database at {db_path}", + item_count=int(count), + ) + + def close(self) -> None: + self._closed = True + + # ------------------------------------------------------------------ + # Internal helpers + # ------------------------------------------------------------------ + + @staticmethod + def _verify_schema(conn: sqlite3.Connection, db_path: str) -> None: + """Confirm the SQLite has the tables the adapter relies on. + + ``json_extract`` is a SQLite JSON1 feature — usually built into + modern Python/SQLite shipments but we sanity-check it once so we + raise a clear error instead of an opaque OperationalError mid-fetch. + """ + tables = { + r[0] + for r in conn.execute("SELECT name FROM sqlite_master WHERE type='table'").fetchall() + } + required = {"session", "message", "part"} + missing = required - tables + if missing: + raise SourceNotFoundError( + f"OpenCode database at {db_path} is missing tables: {sorted(missing)}" + ) + try: + conn.execute("SELECT json_extract('{}', '$')").fetchone() + except sqlite3.OperationalError as e: + raise SourceNotFoundError( + f"SQLite at {db_path} lacks JSON1 (json_extract) — upgrade SQLite/Python: {e}" + ) from e + + def _wing_for(self, source: SourceRef, directory: Optional[str]) -> str: + """Resolve the wing for a session, RFC 002 §2.5 precedence: + + 1. Explicit ``options["wing"]`` from the SourceRef + 2. Project directory basename (the session's ``directory`` column) + 3. Adapter fallback: ``"opencode_general"`` + """ + explicit = (source.options or {}).get("wing") + if explicit: + return normalize_wing_name(str(explicit)) + if directory and directory != "/": + base = Path(directory).name + if base: + return normalize_wing_name(base) + return "opencode_general" + + def _route_hint_for(self, source: SourceRef, directory: Optional[str]) -> Optional[RouteHint]: + # Same wing precedence as _wing_for (RFC 002 §2.5) so the + # SourceItemMetadata route_hint matches the DrawerRecord wing + # when the caller passed options={"wing": ...} — otherwise core + # could make wrong skip/routing decisions on the gap. + wing = self._wing_for(source, directory) + # Room is content-dependent so we leave it None at the lazy-fetch stage; + # the eager DrawerRecord emit fills it in per chunk. + return RouteHint(wing=wing, room=None, hall=None) + + +__all__ = [ + "OpenCodeSourceAdapter", + "session_source_file", +] diff --git a/mempalace/sources/transforms.py b/mempalace/sources/transforms.py index 6fb702bfbe..9ff07b691e 100644 --- a/mempalace/sources/transforms.py +++ b/mempalace/sources/transforms.py @@ -23,6 +23,8 @@ from __future__ import annotations +import json as _json + import re from typing import Protocol, Union @@ -153,6 +155,145 @@ def speaker_role_assignment(text: str) -> str: return text +# --------------------------------------------------------------------------- +# Adapter-namespaced reference implementations +# --------------------------------------------------------------------------- +# +# Per RFC 002 §7.3, custom (non-reserved) transformations declared by an +# adapter MUST expose a reference implementation under +# ``mempalace.sources.transforms._`` so the +# conformance suite can locate and apply them by attribute lookup. The +# implementations below are the OpenCode adapter's; future adapters add +# their own under their own ``_`` prefix. +# +# The OpenCode adapter's canonical source bytes for one session are the +# newline-joined JSON-encoded ``part.data`` rows in +# ``(message.time_created, part.time_created)`` order, each row prefixed with +# its message's ``role`` field as ``"\t"``. That format is +# what ``canonical_source_bytes`` returns to the conformance suite, and is +# what the chain of transformations below collapses into the drawer content. + + +def opencode_extract_text_parts(text: str) -> str: + """Pluck ``data.text`` from each JSON-blob line where ``data.type == "text"``. + + Input lines are ``"\\t"``. The output preserves the + ``\\t`` prefix on each kept line so downstream merge and format + transformations can still see roles. Tool-input, tool-output, and + whitespace-only ``text`` parts are dropped — same skip the live + extraction path applies. + """ + out: list[str] = [] + for line in text.split("\n"): + if not line: + continue + # Split role from JSON; first \t is the separator. + role, _, part_json = line.partition("\t") + if not part_json: + continue + try: + obj = _json.loads(part_json) + except (ValueError, TypeError): + continue + if obj.get("type") != "text": + continue + body = obj.get("text") or "" + if not body.strip(): + continue + out.append(f"{role}\t{body}") + return "\n".join(out) + + +_TOOL_ECHO_NEEDLE = " tool with the following input" + + +def opencode_skip_tool_echo(text: str) -> str: + """Drop user-turn echoes of tool invocations (``Called the X tool ...``). + + Operates on the role-prefixed line stream produced by + :func:`opencode_extract_text_parts`. A line whose body (everything after + the first tab) opens with ``Called the … tool with the following input`` + is dropped wholesale. + """ + kept: list[str] = [] + for line in text.split("\n"): + _, _, body = line.partition("\t") + body_lstripped = body.lstrip() + if body_lstripped.startswith("Called the ") and _TOOL_ECHO_NEEDLE in body_lstripped: + continue + kept.append(line) + return "\n".join(kept) + + +def opencode_skip_file_injection(text: str) -> str: + """Drop ``…`` file-context injections wrapped around context.""" + kept: list[str] = [] + for line in text.split("\n"): + _, _, body = line.partition("\t") + body_lstripped = body.lstrip() + if body_lstripped.startswith("") and "file" in body_lstripped: + continue + kept.append(line) + return "\n".join(kept) + + +def opencode_role_coerce(text: str) -> str: + """Coerce non-``user`` roles to ``assistant``. + + OpenCode emits a small handful of role values (``user``, ``assistant``, + occasionally ``system`` or ``tool``). For transcript-shaped storage the + adapter only distinguishes ``user`` from everything-else. + """ + out: list[str] = [] + for line in text.split("\n"): + role, sep, body = line.partition("\t") + if not sep: + out.append(line) + continue + coerced = "user" if role == "user" else "assistant" + out.append(f"{coerced}\t{body}") + return "\n".join(out) + + +def opencode_same_role_merge(text: str) -> str: + """Merge consecutive same-role lines into a single line with ``\\n\\n`` joiner.""" + out: list[tuple[str, str]] = [] + for line in text.split("\n"): + role, sep, body = line.partition("\t") + if not sep: + continue + if out and out[-1][0] == role: + prev_role, prev_body = out[-1] + out[-1] = (prev_role, prev_body + "\n\n" + body) + else: + out.append((role, body)) + return "\n".join(f"{r}\t{b}" for r, b in out) + + +def opencode_format_exchange(text: str) -> str: + """Reformat role-prefixed lines as ``convo_miner`` exchange-pair markdown. + + ``user`` lines become ``> ``; ``assistant`` lines become the body + on its own paragraph. Pairs are separated by blank lines. The output of + this is what ``mempalace.convo_miner.chunk_exchanges`` recognises as an + exchange transcript. + """ + blocks: list[str] = [] + for line in text.split("\n"): + role, sep, body = line.partition("\t") + if not sep: + continue + body = body.strip() + if not body: + continue + if role == "user": + quoted = "\n".join(f"> {ln}" for ln in body.split("\n")) + blocks.append(quoted) + else: + blocks.append(body) + return "\n\n".join(blocks) + + # --------------------------------------------------------------------------- # Registry # --------------------------------------------------------------------------- diff --git a/pyproject.toml b/pyproject.toml index 2da06799ab..a420cfa312 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -45,11 +45,12 @@ mempalace-mcp = "mempalace.mcp_server:main" chroma = "mempalace.backends.chroma:ChromaBackend" postgres = "mempalace.backends.postgres:PostgresBackend" -# RFC 002 source-adapter entry-point group. Core publishes no first-party -# adapters under this group yet; ``miner.py`` and ``convo_miner.py`` migrate -# onto ``BaseSourceAdapter`` in a follow-up PR. Third-party adapter packages -# (``mempalace-source-cursor``, ``mempalace-source-git``, …) register here. +# RFC 002 source-adapter entry-point group. First-party adapters land here as +# they migrate onto ``BaseSourceAdapter``; third-party adapter packages +# (``mempalace-source-cursor``, ``mempalace-source-git``, …) also register +# under this group. [project.entry-points."mempalace.sources"] +opencode = "mempalace.sources.opencode:OpenCodeSourceAdapter" [project.optional-dependencies] dev = ["pytest>=7.0", "pytest-cov>=4.0", "ruff==0.15.9", "psutil>=5.9"] diff --git a/tests/fixtures/opencode/sample_session_2026_05_12/README.md b/tests/fixtures/opencode/sample_session_2026_05_12/README.md new file mode 100644 index 0000000000..064ed8d5b6 --- /dev/null +++ b/tests/fixtures/opencode/sample_session_2026_05_12/README.md @@ -0,0 +1,77 @@ +# OpenCode sample sessions — fixture builder + +This directory does **not** ship a recorded OpenCode `.db` file. The actual +SQLite schema (verified 2026-05-12 against `opencode-ai 1.14.39` at +`~/.local/share/opencode/opencode.db` on JP's Ubuntu desktop) is replicated +verbatim by the builder script `build_fixture.py`, which produces an +in-memory or on-disk SQLite database populated with synthetic-but-realistic +session data the adapter and its tests consume. + +## Why builder, not recorded fixture + +1. OpenCode `.db` files contain the raw text content of every user/assistant + turn including any pasted file paths, tokens, or secrets — committing a + real recording would leak data even after redaction passes. +2. The schema is small (3 tables we touch — `session`, `message`, `part`). + Constructing a fixture in Python is shorter than a serialized binary. +3. Schema fidelity is the property that matters for adapter testing; content + fidelity matters for `convo_miner`-style integration testing, which is + covered by chunking tests on the synthesized transcripts. + +## Schema captured (verbatim from JP's live DB on 2026-05-12) + +``` +CREATE TABLE `session` ( + `id` text PRIMARY KEY, + `project_id` text NOT NULL, + `parent_id` text, + `slug` text NOT NULL, + `directory` text NOT NULL, + `title` text NOT NULL, + `version` text NOT NULL, + `share_url` text, + `summary_additions` integer, + `summary_deletions` integer, + `summary_files` integer, + `summary_diffs` text, + `revert` text, + `permission` text, + `time_created` integer NOT NULL, + `time_updated` integer NOT NULL, + `time_compacting` integer, + `time_archived` integer, + `workspace_id` text, + `path` text, + `agent` text, + `model` text +); + +CREATE TABLE `message` ( + `id` text PRIMARY KEY, + `session_id` text NOT NULL, + `time_created` integer NOT NULL, + `time_updated` integer NOT NULL, + `data` text NOT NULL +); + +CREATE TABLE `part` ( + `id` text PRIMARY KEY, + `message_id` text NOT NULL, + `session_id` text NOT NULL, + `time_created` integer NOT NULL, + `time_updated` integer NOT NULL, + `data` text NOT NULL +); + +CREATE INDEX `message_session_time_created_id_idx` ON `message` (`session_id`,`time_created`,`id`); +CREATE INDEX `part_message_id_id_idx` ON `part` (`message_id`,`id`); +CREATE INDEX `part_session_idx` ON `part` (`session_id`); +CREATE INDEX `session_project_idx` ON `session` (`project_id`); +``` + +`message.data` is JSON containing `{"role": "user|assistant", ...}`. +`part.data` is JSON containing `{"type": "text|tool-input|tool-output|...", "text": "..."}`. + +OpenCode-PR-#23 reverse-engineered the same schema; `session.directory` is +the path the session was started in, used by this adapter to route the +session's drawers to a wing. diff --git a/tests/fixtures/opencode/sample_session_2026_05_12/build_fixture.py b/tests/fixtures/opencode/sample_session_2026_05_12/build_fixture.py new file mode 100644 index 0000000000..a6f4ae64b9 --- /dev/null +++ b/tests/fixtures/opencode/sample_session_2026_05_12/build_fixture.py @@ -0,0 +1,325 @@ +"""Build a synthetic OpenCode SQLite fixture matching the live schema. + +Live schema captured 2026-05-12 from opencode-ai 1.14.39 at +``~/.local/share/opencode/opencode.db``; see README.md in this directory. + +The builder is a *fixture factory* — tests call ``build_fixture(path, +sessions=...)`` to populate any SQLite path with a known shape. No +recorded `.db` files ship in this directory because the content of a +real OpenCode session is unsanitizably user-private. +""" + +from __future__ import annotations + +import json +import sqlite3 +from dataclasses import dataclass, field +from pathlib import Path +from typing import Iterable, List, Optional + + +# Verbatim DDL from JP's live opencode.db on 2026-05-12. +# We only include the columns the adapter actually reads + their indexes; +# additional columns that OpenCode populates (slug, version, agent, model, +# workspace_id) are kept here so a fixture mirrors the real on-disk shape +# instead of a stripped-down minimum. +DDL = [ + """ + CREATE TABLE `session` ( + `id` text PRIMARY KEY, + `project_id` text NOT NULL, + `parent_id` text, + `slug` text NOT NULL, + `directory` text NOT NULL, + `title` text NOT NULL, + `version` text NOT NULL, + `share_url` text, + `summary_additions` integer, + `summary_deletions` integer, + `summary_files` integer, + `summary_diffs` text, + `revert` text, + `permission` text, + `time_created` integer NOT NULL, + `time_updated` integer NOT NULL, + `time_compacting` integer, + `time_archived` integer, + `workspace_id` text, + `path` text, + `agent` text, + `model` text + ); + """, + """ + CREATE TABLE `message` ( + `id` text PRIMARY KEY, + `session_id` text NOT NULL, + `time_created` integer NOT NULL, + `time_updated` integer NOT NULL, + `data` text NOT NULL + ); + """, + """ + CREATE TABLE `part` ( + `id` text PRIMARY KEY, + `message_id` text NOT NULL, + `session_id` text NOT NULL, + `time_created` integer NOT NULL, + `time_updated` integer NOT NULL, + `data` text NOT NULL + ); + """, + "CREATE INDEX `message_session_time_created_id_idx` ON `message` (`session_id`,`time_created`,`id`);", + "CREATE INDEX `part_message_id_id_idx` ON `part` (`message_id`,`id`);", + "CREATE INDEX `part_session_idx` ON `part` (`session_id`);", + "CREATE INDEX `session_project_idx` ON `session` (`project_id`);", +] + + +@dataclass +class SyntheticPart: + """One part of a message — usually one ``type=text`` part per turn.""" + + type: str = "text" + text: str = "" + extra: dict = field(default_factory=dict) + + +@dataclass +class SyntheticMessage: + role: str # "user" or "assistant" + parts: List[SyntheticPart] = field(default_factory=list) + extra: dict = field(default_factory=dict) + + +@dataclass +class SyntheticSession: + session_id: str + project_id: str + directory: str + title: str + messages: List[SyntheticMessage] = field(default_factory=list) + time_created_ms: int = 1_715_000_000_000 + slug: Optional[str] = None + version: str = "0.1.0" + agent: Optional[str] = None + model: Optional[str] = None + workspace_id: Optional[str] = None + + +def build_fixture( + path: str, + *, + sessions: Iterable[SyntheticSession], +) -> str: + """Create a SQLite file at ``path`` populated with the given sessions. + + Returns the path back so callers can chain into ``SourceRef``. + """ + p = Path(path) + if p.exists(): + p.unlink() + p.parent.mkdir(parents=True, exist_ok=True) + conn = sqlite3.connect(str(p)) + try: + for ddl in DDL: + conn.execute(ddl) + for sess in sessions: + conn.execute( + """ + INSERT INTO session ( + id, project_id, parent_id, slug, directory, title, + version, share_url, summary_additions, summary_deletions, + summary_files, summary_diffs, revert, permission, + time_created, time_updated, time_compacting, time_archived, + workspace_id, path, agent, model + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) + """, + ( + sess.session_id, + sess.project_id, + None, + sess.slug or sess.session_id, + sess.directory, + sess.title, + sess.version, + None, + None, + None, + None, + None, + None, + None, + sess.time_created_ms, + sess.time_created_ms, + None, + None, + sess.workspace_id, + sess.directory, + sess.agent, + sess.model, + ), + ) + for msg_idx, msg in enumerate(sess.messages): + msg_id = f"msg_{sess.session_id}_{msg_idx}" + msg_time = sess.time_created_ms + msg_idx * 1000 + msg_data = {"role": msg.role, **msg.extra} + conn.execute( + """INSERT INTO message (id, session_id, time_created, + time_updated, data) + VALUES (?,?,?,?,?)""", + (msg_id, sess.session_id, msg_time, msg_time, json.dumps(msg_data)), + ) + for part_idx, part in enumerate(msg.parts): + part_id = f"part_{sess.session_id}_{msg_idx}_{part_idx}" + part_data = {"type": part.type, "text": part.text, **part.extra} + conn.execute( + """INSERT INTO part (id, message_id, session_id, + time_created, time_updated, data) + VALUES (?,?,?,?,?,?)""", + ( + part_id, + msg_id, + sess.session_id, + msg_time + part_idx, + msg_time + part_idx, + json.dumps(part_data), + ), + ) + conn.commit() + finally: + conn.close() + return str(p) + + +# Canonical fixture used by the adapter unit + conformance tests. +# Three sessions across two project directories, with a mix of: +# * normal user/assistant text exchanges +# * tool-input parts (skipped on extraction) +# * tool-output parts (skipped on extraction) +# * empty-text parts (skipped on extraction) +# * a session with too few real turns (skipped at the session level) +CANONICAL_SESSIONS: List[SyntheticSession] = [ + SyntheticSession( + session_id="ses_aaa111", + project_id="proj_frontend", + directory="/home/jp/Projects/frontend", + title="Refactor TanStack Query wrapper", + agent="opencode", + model="anthropic/claude-sonnet-4-6", + messages=[ + SyntheticMessage( + role="user", + parts=[ + SyntheticPart( + text="Can you refactor the fetch wrapper to use TanStack Query mutations?" + ) + ], + ), + SyntheticMessage( + role="assistant", + parts=[ + SyntheticPart( + text="Sure. The mutation hook would look like this:\n\nuseMutation({ mutationFn: postUser })\n\nThis gives you onSuccess/onError callbacks." + ), + # Tool-input part should be SKIPPED on extraction. + SyntheticPart( + type="tool-input", + text="", + extra={"name": "edit", "input": {"path": "src/api.ts"}}, + ), + # Tool-output part should be SKIPPED on extraction. + SyntheticPart( + type="tool-output", + text="", + extra={"name": "edit", "output": "file edited"}, + ), + ], + ), + SyntheticMessage( + role="user", + parts=[SyntheticPart(text="And how do I invalidate the query cache after?")], + ), + SyntheticMessage( + role="assistant", + parts=[ + SyntheticPart( + text="Call queryClient.invalidateQueries({ queryKey: ['users'] }) inside onSuccess." + ) + ], + ), + ], + ), + SyntheticSession( + session_id="ses_bbb222", + project_id="proj_frontend", + directory="/home/jp/Projects/frontend", + title="Add session login route", + time_created_ms=1_715_001_000_000, + agent="opencode", + model="anthropic/claude-sonnet-4-6", + messages=[ + SyntheticMessage( + role="user", + parts=[SyntheticPart(text="Add a /login route with JWT auth.")], + ), + SyntheticMessage( + role="assistant", + parts=[ + SyntheticPart( + text="Add the route handler in routes/login.ts and verify the token against the JWT secret on each request." + ) + ], + ), + ], + ), + SyntheticSession( + session_id="ses_ccc333", + project_id="proj_backend", + directory="/home/jp/Projects/backend", + title="Alembic migration question", + time_created_ms=1_715_002_000_000, + agent="opencode", + model="anthropic/claude-sonnet-4-6", + messages=[ + SyntheticMessage( + role="user", + parts=[ + SyntheticPart( + text="How do I downgrade an Alembic migration safely in production?" + ) + ], + ), + SyntheticMessage( + role="assistant", + parts=[ + SyntheticPart( + text="Run alembic downgrade -1 against a staging copy first; verify schema, then run the same command against production inside a transaction." + ) + ], + ), + ], + ), + # Sessions with <2 real text parts should be SKIPPED entirely. + SyntheticSession( + session_id="ses_ddd444", + project_id="proj_backend", + directory="/home/jp/Projects/backend", + title="Cancelled session", + time_created_ms=1_715_003_000_000, + messages=[ + SyntheticMessage( + role="user", + parts=[SyntheticPart(text="wait nvm")], + ), + ], + ), +] + + +if __name__ == "__main__": # pragma: no cover - manual fixture generation + import sys + + target = sys.argv[1] if len(sys.argv) > 1 else "/tmp/opencode_fixture.db" + build_fixture(target, sessions=CANONICAL_SESSIONS) + print(f"wrote fixture to {target}") diff --git a/tests/test_corpus_origin_integration.py b/tests/test_corpus_origin_integration.py index a0de3ad945..4480c993c9 100644 --- a/tests/test_corpus_origin_integration.py +++ b/tests/test_corpus_origin_integration.py @@ -1359,6 +1359,7 @@ def test_no_internal_coordination_jargon_in_source_or_tests(): "mempalace/knowledge_graph.py", "mempalace/i18n/", "tests/test_sources.py", + "tests/test_sources_opencode.py", "tests/test_i18n_lang_case.py", ) # Allowlist for self-reference: this test file mentions the leak diff --git a/tests/test_sources_opencode.py b/tests/test_sources_opencode.py new file mode 100644 index 0000000000..a931a34479 --- /dev/null +++ b/tests/test_sources_opencode.py @@ -0,0 +1,564 @@ +"""Tests for the OpenCode source adapter (RFC 002). + +Covers: + * Adapter class identity (capabilities, modes, declared transformations). + * Conformance against the RFC 002 spec — declared-transformation + round-trip, schema-conformance, stable source_file shape. + * Unit tests for the SQLite walk, tool-echo/file-injection skip, + same-role merge, role coerce, transcript formatting, and the chunked + exchange emit through ``convo_miner.chunk_exchanges``. + * Edge cases: empty session, single-message session, sessions with + mixed text + tool parts, multi-session DB across projects. +""" + +from __future__ import annotations + +import importlib.util +import json +import sqlite3 +import sys +from pathlib import Path + +import pytest + +from mempalace.sources import transforms as src_transforms +from mempalace.sources.base import ( + AdapterClosedError, + AdapterSchema, + DrawerRecord, + FieldSpec, + SourceItemMetadata, + SourceNotFoundError, + SourceRef, +) +from mempalace.sources.context import PalaceContext +from mempalace.sources.opencode import ( + OpenCodeSourceAdapter, + _build_source_bytes_per_session, + session_source_file, +) + +# Fixture builder lives next to the fixture data on purpose so it is +# discoverable by anyone investigating the .db format. Loaded via +# importlib so we don't mutate sys.path at module scope (which would +# also trip ruff E402 on the import-not-at-top check). +FIXTURE_DIR = Path(__file__).parent / "fixtures" / "opencode" / "sample_session_2026_05_12" +_fixture_spec = importlib.util.spec_from_file_location( + "build_fixture", FIXTURE_DIR / "build_fixture.py" +) +build_fixture = importlib.util.module_from_spec(_fixture_spec) +# Register in sys.modules so dataclass / typing introspection inside +# build_fixture can resolve back to the module via cls.__module__. +sys.modules["build_fixture"] = build_fixture +_fixture_spec.loader.exec_module(build_fixture) + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +@pytest.fixture +def canonical_db(tmp_path): + """Build a SQLite fixture from the canonical synthetic sessions.""" + path = tmp_path / "opencode.db" + build_fixture.build_fixture(str(path), sessions=build_fixture.CANONICAL_SESSIONS) + return str(path) + + +@pytest.fixture +def adapter(): + return OpenCodeSourceAdapter() + + +class _FakeCollection: + def __init__(self): + self.upserts = [] + + def add(self, **kwargs): + pass + + def upsert(self, **kwargs): + self.upserts.append(kwargs) + + def query(self, **kwargs): + return {} + + def get(self, **kwargs): + return {} + + def delete(self, **kwargs): + pass + + def count(self): + return 0 + + +class _FakeKG: + def add_triple(self, *args, **kwargs): + pass + + +@pytest.fixture +def palace_ctx(): + return PalaceContext( + drawer_collection=_FakeCollection(), + knowledge_graph=_FakeKG(), + palace_path="/tmp/palace", + adapter_name="opencode", + adapter_version="0.1.0", + ) + + +# --------------------------------------------------------------------------- +# Class identity +# --------------------------------------------------------------------------- + + +def test_adapter_identity(): + assert OpenCodeSourceAdapter.name == "opencode" + assert OpenCodeSourceAdapter.spec_version == "1.0" + assert OpenCodeSourceAdapter.adapter_version == "0.1.0" + assert "chunked_content" in OpenCodeSourceAdapter.supported_modes + assert "supports_incremental" in OpenCodeSourceAdapter.capabilities + assert "adapter_owns_routing" in OpenCodeSourceAdapter.capabilities + assert OpenCodeSourceAdapter.default_privacy_class == "pii_potential" + + +def test_declared_transformations_have_reference_impls(): + """Every declared transformation MUST resolve to an attribute on + mempalace.sources.transforms (RFC 002 §7.3).""" + for name in OpenCodeSourceAdapter.declared_transformations: + impl = getattr(src_transforms, name, None) + assert callable(impl), f"declared transformation {name!r} has no callable reference impl" + + +def test_describe_schema_returns_adapter_schema(adapter): + schema = adapter.describe_schema() + assert isinstance(schema, AdapterSchema) + assert schema.version == "1.0" + required = {k for k, v in schema.fields.items() if v.required} + assert { + "session_id", + "project_dir", + "session_created_at", + "message_count", + "extract_mode", + "opencode_db_path", + }.issubset(required) + assert isinstance(schema.fields["session_id"], FieldSpec) + assert schema.fields["session_id"].indexed is True + assert schema.fields["project_dir"].indexed is True + + +# --------------------------------------------------------------------------- +# Source-file identity +# --------------------------------------------------------------------------- + + +def test_session_source_file_shape_is_stable(): + s1 = session_source_file("/tmp/x.db", "ses_abc") + s2 = session_source_file("/tmp/x.db", "ses_abc") + s3 = session_source_file("/tmp/x.db", "ses_def") + assert s1 == s2 + assert s1 != s3 + assert s1 == "opencode:///tmp/x.db#session=ses_abc" + + +# --------------------------------------------------------------------------- +# DB resolution / errors +# --------------------------------------------------------------------------- + + +def test_ingest_raises_source_not_found_for_missing_db(adapter, palace_ctx, tmp_path): + missing = tmp_path / "nope.db" + ref = SourceRef(local_path=str(missing)) + with pytest.raises(SourceNotFoundError): + list(adapter.ingest(source=ref, palace=palace_ctx)) + + +def test_ingest_raises_source_not_found_for_db_without_expected_tables( + adapter, palace_ctx, tmp_path +): + path = tmp_path / "bad.db" + conn = sqlite3.connect(str(path)) + conn.execute("CREATE TABLE other (id INTEGER)") + conn.commit() + conn.close() + ref = SourceRef(local_path=str(path)) + with pytest.raises(SourceNotFoundError): + list(adapter.ingest(source=ref, palace=palace_ctx)) + + +def test_close_then_ingest_raises_adapter_closed(adapter, palace_ctx, canonical_db): + adapter.close() + ref = SourceRef(local_path=canonical_db) + with pytest.raises(AdapterClosedError): + list(adapter.ingest(source=ref, palace=palace_ctx)) + + +# --------------------------------------------------------------------------- +# Source-summary +# --------------------------------------------------------------------------- + + +def test_source_summary_counts_sessions(adapter, canonical_db): + summary = adapter.source_summary(source=SourceRef(local_path=canonical_db)) + # 4 sessions in the canonical fixture; the <2-message one is still counted + # at the summary stage (it's a session, just skipped on ingest). + assert summary.item_count == 4 + assert "OpenCode database at" in summary.description + + +def test_source_summary_for_missing_db(adapter, tmp_path): + summary = adapter.source_summary(source=SourceRef(local_path=str(tmp_path / "absent.db"))) + assert summary.item_count == 0 + assert "not found" in summary.description.lower() + + +# --------------------------------------------------------------------------- +# Ingest shape +# --------------------------------------------------------------------------- + + +def test_ingest_yields_metadata_then_drawers_per_session(adapter, palace_ctx, canonical_db): + results = list(adapter.ingest(source=SourceRef(local_path=canonical_db), palace=palace_ctx)) + metas = [r for r in results if isinstance(r, SourceItemMetadata)] + drawers = [r for r in results if isinstance(r, DrawerRecord)] + # 4 sessions in the fixture, each gets a SourceItemMetadata + assert len(metas) == 4 + # 3 sessions have >=2 real text exchanges; the 4th (ses_ddd444) is skipped + src_files = {d.source_file for d in drawers} + assert len(src_files) == 3 + # Source files all carry the opencode:// prefix + for sf in src_files: + assert sf.startswith("opencode://") + assert "#session=" in sf + + +def test_ingest_skips_cancelled_session_with_too_few_turns(adapter, palace_ctx, canonical_db): + results = list(adapter.ingest(source=SourceRef(local_path=canonical_db), palace=palace_ctx)) + drawers = [r for r in results if isinstance(r, DrawerRecord)] + src_files = {d.source_file for d in drawers} + assert all("ses_ddd444" not in sf for sf in src_files), ( + "single-message cancelled session must be skipped" + ) + + +def test_drawer_metadata_carries_universal_and_schema_fields(adapter, palace_ctx, canonical_db): + results = list(adapter.ingest(source=SourceRef(local_path=canonical_db), palace=palace_ctx)) + drawers = [r for r in results if isinstance(r, DrawerRecord)] + assert drawers, "expected at least one drawer" + schema_keys = set(adapter.describe_schema().fields.keys()) + universal_keys = { + "source_file", + "chunk_index", + "filed_at", + "added_by", + "wing", + "room", + "hall", + "ingest_mode", + "extract_mode", + "privacy_class", + } + for drawer in drawers: + meta = drawer.metadata + assert universal_keys.issubset(meta.keys()), ( + f"missing universal keys: {universal_keys - meta.keys()}" + ) + assert schema_keys.issubset(meta.keys()), ( + f"missing schema keys: {schema_keys - meta.keys()}" + ) + # Flat-scalar invariant — chroma constraint. + for k, v in meta.items(): + assert isinstance(v, (str, int, float, bool)), ( + f"metadata[{k}]={v!r} of type {type(v).__name__} is not flat-scalar" + ) + + +def test_drawer_route_hint_carries_wing(adapter, palace_ctx, canonical_db): + results = list(adapter.ingest(source=SourceRef(local_path=canonical_db), palace=palace_ctx)) + drawers = [r for r in results if isinstance(r, DrawerRecord)] + for d in drawers: + assert d.route_hint is not None + assert d.route_hint.wing # never empty + # OpenCode adapter populates room from convo_miner.detect_convo_room. + assert d.route_hint.room + + +def test_wing_routing_groups_by_session_directory(adapter, palace_ctx, canonical_db): + """Two frontend sessions and one backend session should produce two wings.""" + results = list(adapter.ingest(source=SourceRef(local_path=canonical_db), palace=palace_ctx)) + drawers = [r for r in results if isinstance(r, DrawerRecord)] + wings = {d.metadata["wing"] for d in drawers} + assert "frontend" in wings + assert "backend" in wings + + +def test_explicit_wing_option_wins_over_directory(adapter, palace_ctx, canonical_db): + ref = SourceRef(local_path=canonical_db, options={"wing": "Custom Wing"}) + results = list(adapter.ingest(source=ref, palace=palace_ctx)) + drawers = [r for r in results if isinstance(r, DrawerRecord)] + wings = {d.metadata["wing"] for d in drawers} + assert wings == {"custom_wing"} # normalize_wing_name lower+underscored + + +# --------------------------------------------------------------------------- +# Skip / incremental behavior +# --------------------------------------------------------------------------- + + +def test_skip_current_item_short_circuits_drawer_emit(adapter, palace_ctx, canonical_db): + """When core calls palace.skip_current_item() after a metadata item, the + adapter MUST stop emitting drawers for that item and move on.""" + ref = SourceRef(local_path=canonical_db) + gen = adapter.ingest(source=ref, palace=palace_ctx) + drawers_seen = 0 + for result in gen: + if isinstance(result, SourceItemMetadata): + palace_ctx.skip_current_item() + elif isinstance(result, DrawerRecord): + drawers_seen += 1 + # Every session was skipped after metadata, so zero drawers should emerge. + assert drawers_seen == 0 + + +def test_is_current_uses_version_when_present(adapter): + item = SourceItemMetadata(source_file="opencode:///x#session=ses_a", version="123") + assert adapter.is_current(item=item, existing_metadata=None) is False + assert ( + adapter.is_current(item=item, existing_metadata={"opencode_session_version": "123"}) is True + ) + assert ( + adapter.is_current(item=item, existing_metadata={"opencode_session_version": "999"}) + is False + ) + + +def test_is_current_falls_back_to_presence_when_version_missing(adapter): + """Older drawers may not carry opencode_session_version; presence implies current.""" + item = SourceItemMetadata(source_file="opencode:///x#session=ses_a", version="123") + # any other-metadata-present scenario should return True (we assume we + # already mined this session) — this is safer than always re-extracting. + assert ( + adapter.is_current(item=item, existing_metadata={"session_id": "ses_a", "wing": "x"}) + is True + ) + + +# --------------------------------------------------------------------------- +# Tool/file part skipping +# --------------------------------------------------------------------------- + + +def test_tool_input_and_tool_output_parts_are_skipped(adapter, palace_ctx, canonical_db): + """ses_aaa111 has tool-input + tool-output parts; their content must not + appear in any drawer.""" + results = list(adapter.ingest(source=SourceRef(local_path=canonical_db), palace=palace_ctx)) + drawers = [ + r + for r in results + if isinstance(r, DrawerRecord) and r.source_file.endswith("#session=ses_aaa111") + ] + joined = "\n".join(d.content for d in drawers) + # Sentinels we wrote into tool-input/tool-output parts in the fixture + assert "src/api.ts" not in joined + assert "file edited" not in joined + + +def test_tool_echo_lines_are_stripped(): + """A user turn echoing a tool invocation should be dropped before the + transcript is chunked.""" + raw = "user\t" + json.dumps( + {"type": "text", "text": "Called the read tool with the following input"} + ) + raw += "\nuser\t" + json.dumps({"type": "text", "text": "Actually, what's the answer?"}) + out = src_transforms.opencode_extract_text_parts(raw) + out = src_transforms.opencode_skip_tool_echo(out) + assert "Called the read tool" not in out + assert "Actually, what's the answer?" in out + + +def test_file_injection_lines_are_stripped(): + raw = "user\t" + json.dumps({"type": "text", "text": "foo.pyfile"}) + raw += "\nuser\t" + json.dumps({"type": "text", "text": "Real question follows."}) + out = src_transforms.opencode_extract_text_parts(raw) + out = src_transforms.opencode_skip_file_injection(out) + assert "foo.py" not in out + assert "Real question follows." in out + + +# --------------------------------------------------------------------------- +# Build-source-bytes / declared-transformation round-trip +# --------------------------------------------------------------------------- + + +def test_build_source_bytes_returns_role_tab_part_lines(canonical_db): + conn = sqlite3.connect(canonical_db) + try: + raw = _build_source_bytes_per_session(conn, "ses_aaa111") + finally: + conn.close() + lines = raw.split("\n") + # Every line is "\t" + for line in lines: + role, sep, body = line.partition("\t") + assert sep == "\t" + assert role in {"user", "assistant", ""} + obj = json.loads(body) + assert "type" in obj + + +def test_declared_transformation_round_trip_reproduces_drawer_content( + adapter, palace_ctx, canonical_db +): + """RFC 002 §7.3: applying the declared transformations to canonical source + bytes, in the adapter's declared order, MUST reproduce the chunk content + (modulo chunk_exchanges' chunking step which is applied on top).""" + results = list(adapter.ingest(source=SourceRef(local_path=canonical_db), palace=palace_ctx)) + drawers = [r for r in results if isinstance(r, DrawerRecord)] + # Group drawers by source_file, then verify the transcript reproduces from + # canonical source bytes via the declared pipeline. + by_src: dict[str, list[DrawerRecord]] = {} + for d in drawers: + by_src.setdefault(d.source_file, []).append(d) + + conn = sqlite3.connect(canonical_db) + try: + for src_file, ds in by_src.items(): + # Extract session_id from source_file shape opencode://#session= + sid = src_file.split("#session=", 1)[1] + raw = _build_source_bytes_per_session(conn, sid) + transformed = raw + for name in adapter.DECLARED_TRANSFORMATION_ORDER: + fn = getattr(src_transforms, name) + transformed = fn(transformed) + # `transformed` is the pre-chunking transcript; chunk_exchanges then + # produces the same per-drawer content list. We import locally so + # the test mirrors what the adapter does. + from mempalace.convo_miner import chunk_exchanges + + expected_chunks = chunk_exchanges(transformed) + assert len(expected_chunks) == len(ds), ( + f"chunk count mismatch for {src_file}: " + f"{len(expected_chunks)} expected, {len(ds)} produced" + ) + ds_sorted = sorted(ds, key=lambda d: d.chunk_index) + for c, d in zip(expected_chunks, ds_sorted): + assert c["content"] == d.content + finally: + conn.close() + + +# --------------------------------------------------------------------------- +# Empty / edge cases +# --------------------------------------------------------------------------- + + +def test_empty_db_yields_nothing(adapter, palace_ctx, tmp_path): + """A schema-valid but empty OpenCode DB ingests cleanly with zero records.""" + path = tmp_path / "empty.db" + build_fixture.build_fixture(str(path), sessions=[]) + results = list(adapter.ingest(source=SourceRef(local_path=str(path)), palace=palace_ctx)) + assert results == [] + + +def test_session_with_only_user_turn_is_skipped(adapter, palace_ctx, tmp_path): + """A single-turn session (just one user message) cannot form an exchange + pair, so the adapter skips it and emits no drawer (only the metadata).""" + only_user = build_fixture.SyntheticSession( + session_id="ses_only_user", + project_id="p", + directory="/tmp/p", + title="Only user", + messages=[ + build_fixture.SyntheticMessage( + role="user", parts=[build_fixture.SyntheticPart(text="hi")] + ) + ], + ) + path = tmp_path / "only_user.db" + build_fixture.build_fixture(str(path), sessions=[only_user]) + results = list(adapter.ingest(source=SourceRef(local_path=str(path)), palace=palace_ctx)) + metas = [r for r in results if isinstance(r, SourceItemMetadata)] + drawers = [r for r in results if isinstance(r, DrawerRecord)] + assert len(metas) == 1 + assert drawers == [] + + +def test_unicode_content_preserved_end_to_end(adapter, palace_ctx, tmp_path): + """Unicode (BMP and non-BMP) in user/assistant text MUST survive to + drawer.content unchanged.""" + # Use larger Unicode bodies so chunk_exchanges' MIN_CHUNK_SIZE (30 chars) + # doesn't elide the only exchange in the session. + sess = build_fixture.SyntheticSession( + session_id="ses_uni", + project_id="p", + directory="/tmp/p", + title="Unicode session", + messages=[ + build_fixture.SyntheticMessage( + role="user", + parts=[ + build_fixture.SyntheticPart( + text=( + "日本語テスト 🎯 кириллица — how do I handle UTF-8 in " + "the database column when storing user names with emoji?" + ) + ) + ], + ), + build_fixture.SyntheticMessage( + role="assistant", + parts=[ + build_fixture.SyntheticPart( + text=( + "हिन्दी और عربى — emoji 🚀 ok. Use utf8mb4 in MySQL, " + "TEXT in Postgres (already 4-byte UTF-8), and make sure " + "your connection charset is utf8mb4 too." + ) + ) + ], + ), + ], + ) + path = tmp_path / "uni.db" + build_fixture.build_fixture(str(path), sessions=[sess]) + results = list(adapter.ingest(source=SourceRef(local_path=str(path)), palace=palace_ctx)) + drawers = [r for r in results if isinstance(r, DrawerRecord)] + assert drawers + combined = "\n".join(d.content for d in drawers) + for needle in ("日本語テスト", "🎯", "кириллица", "हिन्दी", "عربى", "🚀"): + assert needle in combined, f"unicode {needle!r} did not survive transcript" + + +# --------------------------------------------------------------------------- +# Registry integration +# --------------------------------------------------------------------------- + + +def test_registry_can_resolve_opencode_when_registered_explicitly(): + """Even without entry-point discovery (pip install -e .) the registry + SHOULD admit explicit registration.""" + from mempalace.sources.registry import ( + available_adapters, + get_adapter, + register, + unregister, + ) + + register("opencode", OpenCodeSourceAdapter) + try: + assert "opencode" in available_adapters() + inst = get_adapter("opencode") + assert isinstance(inst, OpenCodeSourceAdapter) + finally: + unregister("opencode") + + +def test_capability_byte_preserving_is_NOT_advertised(): + """Sanity check: the OpenCode adapter is declared-lossy (transforms + declared but non-empty), so it MUST NOT advertise byte_preserving.""" + assert "byte_preserving" not in OpenCodeSourceAdapter.capabilities + assert len(OpenCodeSourceAdapter.declared_transformations) > 0