feat(cli): #252 Layer 4 — portable JSON-Lines ledger export/import by Knapp-Kevin · Pull Request #258 · BicameralAI/bicameral-mcp

Knapp-Kevin · 2026-05-07T20:20:35Z

Summary

Closes the #252 Layer 4 substrate from the privacy-preserving remediation strategy: a portable JSON-Lines export/import vehicle for the SurrealDB ledger. Substantively closes #222 (GDPR Art. 15 DSAR CLI) and provides the operator escape hatch for BicameralAI/bicameral-daemon#23 (GDPR Art. 17 right-to-erasure).

Verdict: PASS (round-2 audit). Round-1 VETO specification-drift resolved via Path B (DELETE meta tables before data-records pass; preserves source-provenance + singleton invariant).

What ships

cli/ledger_io.py (132 LOC) + cli/_ledger_io_engine.py (197 LOC) — split per Razor mandate
cli/ledger_export_cli.py (22 LOC) + cli/ledger_import_cli.py (47 LOC) — thin shims
server.py — new ledger-export + ledger-import subparsers + dispatch arms
docs/policies/ledger-export.md — operator-readable policy with 4 workflow recipes (backup, GDPR Art. 17 erasure, Art. 15 DSAR, migration vehicle)
45 new functional tests + 2 content-contract tests
README compliance posture bumped 5 → 6 policy files

Path B contract (closed by construction)

bicameral-mcp reset wipes ledger
bicameral-mcp ledger-import --from-file <path>:
- adapter.connect() populates meta tables (destination-side)
- Phase A _validate_records validates every JSONL line
- Phase B step 1 _delete_meta_tables clears destination-side rows
- Phase B step 2 _write_data_records writes source rows (incl. source's meta tables)
- Phase B step 3 _write_edge_records writes RELATE edges
End state: each meta table has exactly the source's row.

Logged deviations

Helper-extraction split executed per round-1 audit mandate (mirror of Layer 3's _diagnose_gather.py precedent).
_maybe_parse_datetime + _rehydrate helpers added during impl: SurrealDB rejects ISO datetime strings for option<datetime> fields after JSON round-trip. ~20 LOC. Doctrine-positive expansion.
Test count expansion 30 → 45: edge cases discovered during impl. Doctrine-positive expansion.

Test plan

All 45 new functional tests pass against memory:// ledger
2 new content-contract tests in test_compliance_policy_docs.py pass
Path B round-trip smoke-test confirms bicameral_meta + schema_meta end with exactly 1 row each
Source's at_first_write provenance survives the round-trip (verified end-to-end)
RELATE edge writes verified via test_import_jsonl_writes_edge_via_relate
Section 4 Razor: all files <250 LOC, longest function ~38 LOC, depth ≤3, no nested ternaries

Substantiation

META_LEDGER entry #48 sealed (commit 0a10147). Plan: plan-252-layer-4-export-import.md.

🤖 Generated with Claude Code

coderabbitai · 2026-05-07T20:20:43Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ab81dabc-b500-45c7-b177-78ff3a3a9b35

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch plan/252-layer-4-export-import

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

jinhongkuan · 2026-05-07T21:37:23Z

Conflicting after #257 merged ahead of this PR (sibling stack — both touch the same files). Could you rebase against fresh dev so this can land in v0.14.1? After rebase the merge should be clean.

Knapp-Kevin · 2026-05-08T02:04:50Z

CI status — held pending dev-baseline fixes (#272)

CI red on this PR is dev-baseline regressions, not Layer 4 code. Full diagnosis: #272.

Failure	Root cause	Layer 4-introduced?
MCP Regression (ubuntu + windows)	`test-summary/action@v2` mutable tag broken upstream (`File not found: index.js`)	No — pytest itself passes 25/26
MCP Regression (ubuntu) M1 step	Pre-existing `IngestResponse.ungrounded_decisions` AttributeError (field renamed to `pending_grounding_decisions`; eval script not updated)	No
e2e Flow 1	`eabe7b5` (sync skill auto-bind, #263) shipped on dev between #257 merge and this PR's rebase; Flow 1 e2e assertion expects old `ingest → ratify` sequence	No

Holding merge until #272's three dev-baseline fixes land. Re-running CI on this PR after those fixes should produce green.

#252 Layer 5 (plan-252-layer-5-diagnose-remediation.md, audit PASS round 2) is queued behind this PR.

…pe substantiated Layer 5 closes the #252 strategy: `bicameral-mcp diagnose` now surfaces the export → reset → import recipe (Layer 4's portable vehicle) as the recommended remediation when drift_status ∈ {drift, unavailable} or when the ledger is >100 MiB. Recipe is operator-facing display text; diagnose never executes it. Per gating-is-observability discipline. Reality matches Promise: - 8 functional tests + 1 content-contract test all pass - _compute_suggestions at 30 LOC (≤40), file at 248 LOC (≤250) - _remediation_recipe helper at 12 LOC - Layer 3 regression: 5/5 _compute_suggestions tests still pass Logged deviations: - Per-branch single-line append compaction beyond plan's example shapes (lands function at 30 LOC vs plan's 42-LOC projection) - bv truthy-guard preserves Layer 3's invariant: version-derivative suggestions skip when bicameral_version is unknown/missing Substantively closes #221 substrate (GDPR Art. 17 right-to-erasure operator workflow) + #222 (GDPR Art. 15 DSAR). Plan: plan-252-layer-5-diagnose-remediation.md (round 2 PASS; gitignored per c206ad4). Audit: PASS round 2 (round 1 razor-overage + spec-drift remediated). Branch caveat: based on plan/252-layer-4-export-import; will rebase onto fresh dev once #258 unblocks via #273 + Jin's Flow 1 fix (#272).

@v2

…residual) Aligns preflight-eval.yml with the test-mcp-regression.yml pin landed in PR #273. Closes the last of #272's three CI-baseline regressions — preflight-eval was the only remaining consumer of the unpinned mutable @v2 tag whose published artifact silently broke between PR #257 and PR #258 (index.js missing from the action's bundled output). Same SHA (31493c76ec9e7aa675f1585d3ed6f1da69269a86, v2.4) used in test-mcp-regression.yml:213 so a future bump is one grep-and-replace. Per docs/policies/install-trust-model.md (OWASP A06 supply-chain discipline): no GitHub Action runs in our CI from a mutable tag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The migration vehicle + GDPR Art. 15 DSAR + Art. 17 right-to-erasure surface in one shape per docs/research-brief-252-privacy-preserving-ledger-remediation.md. Three phases: - Phase 1: cli/ledger_io.py — canonical record shape (_table + _schema_version + _record_version per record); export_jsonl() async generator with (table, created_at, id) deterministic ordering; import_jsonl() with two-phase validate-then-write (Phase A: validate every record; Phase B: data records first, then edges via RELATE); ImportSummary dataclass + ImportError_ exception. - Phase 2: cli/ledger_export_cli.py + cli/ledger_import_cli.py thin shims (~30 + ~50 LOC); server.py subparser + dispatch wiring for bicameral-mcp ledger-export + ledger-import commands. - Phase 3: docs/policies/ledger-export.md operator policy doc covering canonical record format + GDPR workflow recipes (DSAR, erasure, migration vehicle) + two-pass import rationale + privacy posture + error-mode catalog. README compliance-posture row added (6 -> 7). 8 design decisions locked per /qor-plan dialogue: 1. cli/ledger_io.py shared + thin CLI shims 2. JSON-Lines format 3. Per-record _schema_version + _record_version dual stamp 4. Sort by (table, created_at, id) for diff-stable round-trips 5. Edges as standalone records with in/out preserved (two-pass import) 6. NO redaction at export-time (Layer 3 diagnose is the redacted surface) 7. Wipe-first via separate bicameral-mcp reset; no --force shortcut 8. Strict + summary on import errors; no partial-import state ~400 LOC + ~30 functional tests + 2 content-contract tests. No new pip deps. Identity_supersedes documented as data-shaped despite edge semantics. SCHEMA_VERSION fallback when schema_meta row absent. Plan: plan-252-layer-4-export-import.md Strategy: docs/research-brief-252-privacy-preserving-ledger-remediation.md Closes #252 Layer 4 substrate for #221 (GDPR right-to-erasure) + #222 (GDPR Art. 15 DSAR CLI) when implementation lands.

…re-import) (#252) Round 1 VETO: specification-drift (1 binding finding): bicameral_meta + schema_meta round-trip break — plan claimed at_first_write provenance preserved, but mechanism (init_schema + migrate + sentinel run at adapter.connect BEFORE import; then import CREATE produces duplicate rows with different ULID IDs) doesn't deliver. Layer 2's SELECT...LIMIT 1 invariant breaks under duplicate rows. Round 2 amendments (Path B + 2 Razor advisories baked in): 1. New _DELETE_BEFORE_IMPORT = frozenset({"bicameral_meta", "schema_meta"}) constant. Phase B's first step calls _delete_meta_tables(adapter) which DELETEs both tables before the data-records write pass. End state: each meta table has exactly the source's row, preserving at_first_write provenance + source schema version. Layer 2 SELECT...LIMIT 1 invariant survives by construction. 2. import_jsonl decomposed into 5 private helpers + ~15-LOC orchestrator (per-function Razor 40-LOC ceiling honored): _validate_records (Phase A accumulates ALL errors; no first-failure-only), _assert_ledger_empty (gate skipping _DELETE_BEFORE_IMPORT tables from the check), _delete_meta_tables (Phase B step 1), _write_data_records (Phase B step 2), _write_edge_records (Phase B step 3 via RELATE). 3. cli/ledger_io.py split mandated (not advisory): cli/ledger_io.py (constants + dataclass + canonical-record + sort key ≈ 150 LOC) + cli/_ledger_io_engine.py (gather/export/import + private helpers ≈ 130 LOC). Mirror of Layer 3's cli/_diagnose_gather.py precedent. 3 new functional tests added for the Path B contract: - test_import_jsonl_delete_before_import_preserves_source_at_first_write_provenance - test_import_jsonl_delete_before_import_resolves_schema_meta_to_source_version - test_import_jsonl_delete_before_import_yields_exactly_one_row_per_meta_table Policy doc gains explicit "Meta-table special case" section in docs/policies/ledger-export.md showing the DELETE-before-CREATE flow. Plan: plan-252-layer-4-export-import.md (round 2) Strategy: docs/research-brief-252-privacy-preserving-ledger-remediation.md

…r 4) Closes #252 Layer 4 per docs/research-brief-252-privacy-preserving-ledger-remediation.md. Provides the portable JSON-Lines export/import vehicle that doubles as GDPR Art. 15 DSAR + Art. 17 right-to-erasure surface + surrealdb-py wire-format migration vehicle. cli/ledger_io.py (132 LOC): _DATA_TABLES (15) + _EDGE_TABLES (9) + _DELETE_BEFORE_IMPORT (Path B) frozensets + ImportSummary dataclass + custom exceptions + _canonical_record (stamps _table/_schema_version/ _record_version per record) + _record_sort_key for diff-stable (table, created_at, id) ordering. cli/_ledger_io_engine.py (197 LOC; split per Razor mandate): - _gather_table_rows: tolerant SELECT * (returns [] on missing table) - export_jsonl: async generator, data tables before edges, deterministic sort - _validate_records: Phase A; accumulates ALL errors before raising - _assert_ledger_empty: skips _DELETE_BEFORE_IMPORT tables - _delete_meta_tables: Path B Phase B step 1 - _write_data_records: Phase B step 2 via CREATE - _write_edge_records: Phase B step 3 via RELATE - _maybe_parse_datetime + _rehydrate: ISO-string -> datetime cast for SurrealDB option<datetime> fields (round-trip JSON serialization preserves the type contract) - import_jsonl: ~15-LOC orchestrator over the 5 helpers cli/ledger_export_cli.py + cli/ledger_import_cli.py: thin shims; server.py registers ledger-export + ledger-import subparsers + dispatch. docs/policies/ledger-export.md: operator policy with canonical record shape table, 4 workflow recipes (backup, GDPR Art. 17 erasure, Art. 15 DSAR, migration vehicle), two-pass import rationale, meta-table special case explanation, error mode catalog. README.md: Compliance posture bumped 5 -> 6. Tests: 45 functional pass — 9 canonical-record + 7 export + 12 import (incl. 3 Path B contract tests + 1 _assert_ledger_empty skip-set test) + 3 export CLI + 4 import CLI + 2 doc/code drift content-contract. Path B fully verified: meta tables end with exactly the source's row post-import; Layer 2's SELECT...LIMIT 1 invariant survives. Plan: plan-252-layer-4-export-import.md (round 2) Audit: round 1 VETO specification-drift (bicameral_meta + schema_meta round-trip break) -> round 2 PASS via Path B (DELETE meta tables before data-records pass).

…antiated Path B (round-1 audit central finding) verified end-to-end: - _DELETE_BEFORE_IMPORT + _delete_meta_tables Phase B step 1 - Source's at_first_write survives round-trip - Layer 2's SELECT...LIMIT 1 invariant preserved (1 row per meta table) Reality matches Promise: 45 new functional tests pass + 2 content-contract. Logged deviations: helper-extraction split (Razor mandate), datetime rehydration helpers (impl-time discovery), test count expansion (30 -> 45). Substantively closes #222 (GDPR Art. 15 DSAR CLI) + #221 substrate (GDPR Art. 17 right-to-erasure operator workflow). Plan: plan-252-layer-4-export-import.md (round 2)

The plan-*.md gitignore policy (Jin, c206ad4) landed on dev between the plan commits (fd95ea4, 38e3429) and the rebase. Removing the tracked plan file to align; META_LEDGER entry #48 retains the historical plan reference.

devin-ai-integration · 2026-05-14T21:21:13Z

@jinhongkuan — merged to dev. This PR delivers the Layer 4 ledger export/import CLI for #252, substantively addressing #222 (GDPR Art. 15 data-subject access) and BicameralAI/bicameral-daemon#23 (GDPR Art. 17 right-to-erasure). All automated CI gates green (7/7; manual demo recording pending but not a gate). Please review at your convenience.

Knapp-Kevin temporarily deployed to production May 7, 2026 20:20 — with GitHub Actions Inactive

Knapp-Kevin temporarily deployed to ci-test May 7, 2026 20:20 — with GitHub Actions Inactive

Knapp-Kevin had a problem deploying to recording-approval May 7, 2026 20:20 — with GitHub Actions Failure

Knapp-Kevin temporarily deployed to ci-test May 7, 2026 20:20 — with GitHub Actions Inactive

Knapp-Kevin added P1 High: ship this milestone; user-impacting bug or committed feature compliance Compliance / regulatory / security-standard alignment work ledger Decision ledger, persistence, or query surface tool MCP tool or handler surface labels May 7, 2026

jinhongkuan mentioned this pull request May 7, 2026

release(prep): v0.14.1 — SBOM emitter fix + version bumps #264

Merged

5 tasks

Knapp-Kevin force-pushed the plan/252-layer-4-export-import branch from 0a10147 to 96498f4 Compare May 8, 2026 01:29

Knapp-Kevin had a problem deploying to recording-approval May 8, 2026 01:29 — with GitHub Actions Failure

Knapp-Kevin had a problem deploying to production May 8, 2026 01:29 — with GitHub Actions Failure

Knapp-Kevin had a problem deploying to ci-test May 8, 2026 01:29 — with GitHub Actions Failure

Knapp-Kevin mentioned this pull request May 8, 2026

Dev CI baseline regressions blocking #258 merge: test-summary action + M1 eval AttributeError + e2e Flow 1 #272

Closed

5 tasks

This was referenced May 8, 2026

fix(ci): SHA-pin test-summary action + rename eval_decision_relevance attr (#272) #273

Merged

[DRAFT] feat(diagnose): #252 Layer 5 — opt-in remediation recipe #274

Closed

silongtan mentioned this pull request May 8, 2026

refactor(telemetry): consolidate BICAMERAL_TELEMETRY env-var namespace (#192) #250

Merged

6 tasks

Knapp-Kevin added 5 commits May 14, 2026 21:01

devin-ai-integration Bot force-pushed the plan/252-layer-4-export-import branch from 96498f4 to 0d55fed Compare May 14, 2026 21:02

devin-ai-integration Bot temporarily deployed to production May 14, 2026 21:02 Inactive

devin-ai-integration Bot had a problem deploying to recording-approval May 14, 2026 21:02 Failure

devin-ai-integration Bot temporarily deployed to ci-test May 14, 2026 21:02 Inactive

devin-ai-integration Bot merged commit ea0329a into dev May 14, 2026
7 of 8 checks passed

devin-ai-integration Bot requested a review from jinhongkuan May 14, 2026 21:21

devin-ai-integration Bot mentioned this pull request May 14, 2026

research+ideation(team-server): tier v1 compatible-component survey + structured ideation #325

Merged

3 tasks

jinhongkuan mentioned this pull request May 23, 2026

feat(diagnose): #252 Layer 5 — opt-in remediation recipe (rebased from #274) #506

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): #252 Layer 4 — portable JSON-Lines ledger export/import#258

feat(cli): #252 Layer 4 — portable JSON-Lines ledger export/import#258
devin-ai-integration[bot] merged 5 commits into
devfrom
plan/252-layer-4-export-import

Knapp-Kevin commented May 7, 2026

Uh oh!

coderabbitai Bot commented May 7, 2026 •

edited

Loading

Review skipped

Uh oh!

jinhongkuan commented May 7, 2026

Uh oh!

Knapp-Kevin commented May 8, 2026

Uh oh!

Uh oh!

devin-ai-integration Bot commented May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Knapp-Kevin commented May 7, 2026

Summary

What ships

Path B contract (closed by construction)

Logged deviations

Test plan

Substantiation

Uh oh!

coderabbitai Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

jinhongkuan commented May 7, 2026

Uh oh!

Knapp-Kevin commented May 8, 2026

CI status — held pending dev-baseline fixes (#272)

Uh oh!

Uh oh!

devin-ai-integration Bot commented May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented May 7, 2026 •

edited

Loading