feat: integrate lean-md as an external lean-ctx addon (+ LSP formatter routing)

[dasTholo]

feat: integrate lean-md as an external lean-ctx addon (+ LSP formatter routing)

Summary

Integrate lean-md — a directive/macro Markdown
renderer for agent plans — into lean-ctx as an external addon, not in-tree code. The
renderer engine (developed in-tree across phases 1–9) is reverse-cut before merge: it
never reaches main. What lands here is only the thin surface lean-ctx needs to host and
reach the addon.

Why lean-md. Agents re-describe the same plan scaffolding every turn. lean-md gives them
token-lean, cache-safe, reusable plans — @define/@call macros, @if/@consumer
phase-isolation, @render pipes — single-sourced across a plan. Its code-intel directives
(@read, @refactor, @search, …) call back into lean-ctx over the wire through the stable
ctx_* surface: lean-md brings the plan grammar, lean-ctx brings the code intelligence.

Why an addon (not in-tree, not a fork). The renderer is standalone (rushdown +
evalexpr) with no lean-ctx crate dependency, and lean-ctx ships no render-engine dependency
in return. Keeping it out of tree means an independent release cycle, no coupling of the core
binary to the render engine, and a small, auditable contract — and it reuses the addon
ecosystem that matured in 3.8.x instead of forking/recompiling lean-ctx.

What this PR actually changes (10 commits, 12 files):

• .lmd.md is read raw by ctx_read — like any other file. There is no .lmd.md
  special-casing anywhere in ctx_read.rs; rendering is the addon's job, reached explicitly
  through its ctx_md_render / ctx_md_check tools once installed. A gate test
  (rust/tests/ctx_read_lmd_md_raw.rs) pins this.
• addon_registry.json: the lmd placeholder becomes the lean-md entry, and stays
  listed (command: "", no [install] block ⇒ is_installable() == false). It exists for
  addon search discovery; installation goes through the hosted pack
  (addon add @dasTholo/lean-md). Three registry tests move to the new slug with it.
• Generic extension_registry::RenderTransform trait + registry — infra for
  @render type=<name>, not lmd-exclusive.
• A ctx_refactor / LSP improvement rides along: extension-based reformat routing
  (.rs → rustfmt, else IDE) with honest BLAKE3 change detection, and cache invalidation of
  all changed paths.
• Reverse-cut gates (rust/tests/reverse_cut_gate.rs) fail the build if any engine
  symbol/dep/seed-dir leaks back into rust/src. ctx_read.rs is deliberately not excluded
  from the symbol scan — after the cut it carries no lmd knowledge, so the gate scans it like any
  other file.

This branch was rebuilt from upstream/main and force-pushed. It no longer deletes any
docs/superpowers/* file and no longer touches rust/.config/nextest.toml — both were
artefacts of the previous branch history and are gone. It is also file-disjoint from #780
(the #727 follow-up fixes), so the two can land in either order.

Test plan

• cd rust && cargo test — run as cargo nextest run (this repo's runner): 8762 passed,
  21 skipped, 0 failed
• cd rust && cargo clippy --all-targets --all-features -- -D warnings — clean
• cd rust && cargo fmt --check — clean
• If cookbook/packages changed: relevant npm test / build steps — n/a, none changed

Targeted gates:

• cargo nextest run --test reverse_cut_gate — no engine symbols in rust/src, no render-engine
  deps, seed dir removed, the ctx_* outbound surface survives, ctx_md_render/ctx_md_check
  absent from the core
• cargo nextest run --test ctx_read_lmd_md_raw — a .lmd.md read returns raw source
• cargo nextest run -E 'test(/registry/)' — 199 passed, including
  bundled_registry_passes_security_validator, bundled_addon_registry_is_canonical and
  every_runnable_bundled_addon_declares_scrubbing_capabilities (which skips listed entries)

The live-addon roundtrip (addon add @dasTholo/lean-md → ctx_tools list → ctx_md_render) is
not exercised in CI — no addon is installed there. It depends on the distribution follow-up
listed below.

Notes for reviewers

• Risk areas / edge cases:
  • rust/src/tools/registered/ctx_read.rs is byte-identical to upstream/main on this
    branch (git diff upstream/main -- rust/src/tools/registered/ctx_read.rs is empty). The
    earlier auto-render delegation hook was never re-applied. Zero risk there.
  • rust/src/lsp/format/mod.rs + rust/src/tools/ctx_refactor/ops.rs — the reformat path now
    resolves a formatter by extension and reports a real changed/unchanged via before/after
    BLAKE3 instead of assuming a live IDE. It invalidates the cache for every changed path, not
    just the first; tests_ops.rs guards that regression.
  • rust/data/addon_registry.json is a generated file (bundled_addon_registry_is_canonical
    enforces canonical form). The lmd→lean-md rename shifted the entry's alphabetical sort
    position, so the generator re-sorted it: lean-md now sorts before letta. The diff shows
    that swap; both entries are listed with identical empty [mcp] blocks, so nothing was
    exchanged between them.
• Backwards compatibility:
  • The registry change is entry-only — no code depends on the old lmd slug. The sole
    remaining "lmd" string in the tree is an is_slug literal test in core/addons/manifest.rs.
  • min_lean_ctx on the entry is 3.9.4; that is a metadata statement about the addon, not a
    gate on this PR.
  • No [capabilities] requirement is imposed on existing manifests.
• Docs updated (links/files):
  • docs/reference/21-lean-md.md (new) — addon integration reference: raw read, listed entry,
    the ctx_* callback surface, and the decoupling rationale
  • docs/reference/appendix-mcp-tools.md — records that .lmd.md is returned raw and that
    ctx_md_render / ctx_md_check are addon-provided

Open follow-ups (not in this PR)

• Distribution. Making the addon one-click installable means publishing lean-md and
  switching the entry to a real install path. This PR intentionally lands the integration surface
  only, and says so honestly in the entry (listed, not installable).
• Splitting rust/src/lsp/format/mod.rs further, if you want it smaller than it landed.
• Transparent tool namespace (gateway enforcing a lean-md:: prefix).
• Host callback injection, so the addon can reach the host MCP without a separate lean-ctx call.
• A CI job for the live-addon gates, once an addon can be installed in CI.

Contributor License Agreement

• I have read the CLA Document and I hereby sign the CLA.

[yvgude]

Thorough first-pass review while this is still draft — overall this is exactly the shape we asked for: the reverse-cut discipline is real (both sides of the invariant gated: outbound ctx_* surface must survive, ctx_md_render/ctx_md_check must be gone), the .lmd.md raw-read e2e test is properly hermetic (private LEAN_CTX_DATA_DIR + --fresh), the registry migration cleanly retires the lmd placeholder, and the headless symbols_overview fallback is a genuine improvement beyond the addon itself. CI is green across the board (Windows job still pending as I write this).

Three points need fixing before ready-for-review, plus a few nits.

Blockers

B1 — Unrelated upstream docs are deleted. The PR removes six maintainer planning/spec documents that have nothing to do with lean-md:

• docs/superpowers/plans/2026-06-17-dependency-auto-update-ci.md
• docs/superpowers/specs/2026-06-17-dependency-auto-update-ci-design.md
• docs/superpowers/specs/2026-06-17-dependency-upgrades-plan-a-design.md
• docs/superpowers/specs/2026-06-17-dependency-upgrades-plan-b-design.md
• docs/superpowers/specs/2026-06-19-prose-model-spike.md
• docs/superpowers/specs/2026-06-19-token-company-competitive-improvements-design.md

Looks like an artifact of the reverse-cut script or a merge resolution. Please restore them (git checkout upstream/main -- docs/superpowers/).

B2 — Directory-scope reformat regresses to "unchanged" + stale caches. The new handle_reformat_refactor computes blake3_of(&abs_path) before/after and invalidates only that one path. Two problems:

1. For scope=Directory, abs_path is a directory — std::fs::read(dir) errors, .ok() yields None on both sides, before == after → the tool reports "unchanged" even when the IDE reformatted files, and no cache entry is invalidated.
2. The old render_reformat iterated res.changed_paths from the JetBrains backend and invalidated every changed file; run_jetbrains_reformat now discards them (.map(|_| ())). Even for file scope, an optimize_imports pass can touch more than the target.

Suggestion: keep the BLAKE3 changed/unchanged honesty for the Formatter::Command path (single file, it's a real improvement), but let the JetBrains arm return changed_paths and invalidate them all, reporting changed files: N as before.

B3 — nextest.toml: dropping test-threads = -4 is out of scope. The comment right above it explains why parallelism is capped (shared-state flakiness). If the cap genuinely blocks your suite, make that case in its own PR — silently changing global test parallelism inside an addon-integration PR is how flaky CI comes back.

Nits

• docs/reference/21-lean-md.md §5 links docs/lean-md/specs/2026-06-25-lmd-v2-addon-decoupling-design.md, which doesn't exist upstream and isn't added by this PR — dead link; point it at the addon repo instead.
• ctx_read.rs hunk is whitespace-only inside a tracing::debug! string continuation (semantically neutral, but it's rustfmt drift — please drop it).
• reverse_cut_gate.rs Test 1's doc header still lists ctx_read.rs as the "auto-render delegation hook" survivor — stale now that the delegation was itself reverse-cut and reads are raw.
• The tiktoken warmup in cache.rs::hebbian_eviction_bonus_is_wired is well-reasoned and well-documented, but it's an unrelated test-hardening ride-along; fine to keep, just flagging it for the changelog.

On the design itself

The RenderTransform trait is appropriately minimal, and capabilities.exec: ["lean-ctx"] in the registry entry matches the outbound-contract story in §3.3 of the new doc. min_lean_ctx: "3.8.12" is correct for the manifest/registry features you rely on. Once B1–B3 land I'm happy to take this out of draft — the skills-as-phased-render payload is a compelling first real consumer of the addon surface.

[yvgude]

Follow-up on the distribution question (how does the lean-md binary reach users): this now has an accepted roadmap — epic #724 (Unified Distribution — ctxpkg as the single package layer), design in docs/specs/unified-distribution-v1.md.

What it means for this PR, concretely:

• Now (Phase 0, no waiting): publish lean-md to crates.io and add the already-supported [install] block to the registry entry (manager = "cargo", package = "lean-md", version = "<pinned>"). The shipped bootstrap engine provisions it on addon add — pinned, shell-free. This PR only needs the one-line manifest addition once your crate is up.
• Phase 1 (Unified Distribution Phase 1: kind field + unified artifact installer (managed addon binaries) #725, v3.10): prebuilt, signed, per-platform binaries — addon add lean-md downloads + SHA-256-verifies your CI release artifacts into a managed dir (never on PATH), auto-pins the spawn hash. Your existing release matrix feeds it directly.
• Phase 3 (Unified Distribution Phase 3: kind=skills + cross-kind dependency resolution (lean-md reference case) #727): your skills stop being include_str! payload — they become a kind=skills pack that the addon declares as a dependency. Binary shrinks, skills update without binary releases, and the whole thing stays signed end-to-end (the trust story you asked for via ctxpkg).

So: no side-loading hacks needed, and crates.io is the bridge, not the destination.

[yvgude]

Update on the distribution story discussed above: the infrastructure for prebuilt binaries just landed as PR #729 (Phase 1 of #724).

Concretely for lean-md, once that merges you can add an [artifacts] block to your registry entry alongside the crates.io [install] fallback:

[artifacts.aarch64-apple-darwin]
filename = "lean-md-aarch64-apple-darwin"
url = "https://github.com/dasTholo/lean-md/releases/download/v0.2.0/lean-md-aarch64-apple-darwin"
sha256 = "…"
# one entry per platform you build in CI

lean-ctx addon add lean-md then downloads the binary for the user's platform into a managed dir (never PATH), verifies the SHA-256 before install, pins it as the spawn-time binary hash, and addon update lean-md handles upgrades side-by-side with a health check. Platforms you don't prebuild for fall through to cargo install via the [install] block — so shipping both gives instant installs where you have CI coverage and a working fallback everywhere else. No action needed on this PR; crates.io publishing as discussed stays the Phase 0 path.

[yvgude]

Phase 2 of #724 just merged to main (#734) — the self-service path promised above is now real, end to end:

# once, on your machine
lean-ctx account login && lean-ctx account claim dastholo

# from the lean-md repo (next to lean-ctx-addon.toml)
lean-ctx addon publish --namespace dastholo --check   # all gates offline, no upload
lean-ctx addon publish --namespace dastholo           # sign + upload to ctxpkg.com

After that, anyone can run lean-ctx addon add dastholo/lean-md — signature-verified download, your manifest's [artifacts]/[install] handled exactly like a curated entry (consent prompt, preflight, health probe), and addon update re-resolves new versions from the registry. Hosted packs also show up automatically on leanctx.com/addons.

So for this PR that means: no waiting on a curated-registry MR round-trip — you can ship lean-md yourself the moment the manifest is ready. The curated entry can still follow later for extra visibility (it stays the reviewed tier). Happy to sanity-check your lean-ctx-addon.toml + [artifacts] hashes whenever you want a second pair of eyes.

[dasTholo]

oh a lot to do! I have to take a closer look! I'll sit down right away. But it will probably only become something on Wednesday

[yvgude]

Review — PR #721: lean-md addon integration

Thank you for this thorough contribution! The reverse-cut approach is well-reasoned and the PR is cleanly structured. A few items before merge:

---

1. CLA ⚠️

The CLA checkbox is unchecked — please sign the Contributor License Agreement by checking the box in the PR description. We cannot merge without it.

2. Technical feedback

Positive:

• The reverse-cut gate tests (`reverse_cut_gate.rs`) are a great pattern — they prevent engine symbols from leaking back into the core binary. Would love to see this as a reusable CI template.
• The `RenderTransform` trait in `extension_registry.rs` is well-designed as a generic hook point.
• The `.lmd.md` raw-read decision (no auto-render delegation) is the right call — it keeps `ctx_read` simple and deterministic.

Requests:

• `lsp/format/mod.rs` (254 lines): Please split the formatter resolver (`resolve_formatter_for_extension`) and the command runner (`run_formatter_command`) into separate submodules. Our LOC gate enforces ≤1500 lines for new files, so this is fine now, but the module is doing two distinct things.
• Tests for the formatter routing: The `tests_ops.rs` additions cover the `ctx_refactor` surface, but there are no unit tests for the new `format/mod.rs` itself (edge cases: missing `rustfmt`, fallback to IDE, BLAKE3 unchanged detection). Please add a few.
• Doc cleanup: `21-lean-md.md` references `WasmRenderTransform` in the decoupling table — this was removed in the later commits but the doc still mentions it. Please verify the doc matches the final code state.

3. Merge conflicts

The PR was last rebased against v3.9.3 `main`. We've since merged several proxy and config changes. Please rebase onto current `main` and verify the conflict resolution in `ops.rs` and `integration_tests.rs` is still clean.

---

Summary: Great contribution, well-motivated. Please:

1. ✅ Sign the CLA
2. 🔄 Rebase onto current `main`
3. 📝 Add formatter unit tests + fix doc reference

Looking forward to landing this!

[yvgude]

@dasTholo — friendly follow-up on the review feedback from July 8th.

We've merged 4 PRs into main since then (including the Clippy fix, XDG split-brain fix, and three refactoring PRs), so the merge conflicts will be more significant now. Before you invest time rebasing, here's the priority order for the remaining items:

1. CLA — this is the hard blocker; we can't merge without it. If you have questions about the agreement, please ask.
2. Rebase onto current main — resolve conflicts especially in ops.rs and integration_tests.rs.
3. Formatter unit tests — edge cases for format/mod.rs (missing rustfmt, fallback, BLAKE3 unchanged).
4. Doc cleanup — remove WasmRenderTransform reference from 21-lean-md.md.
5. Split lsp/format/mod.rs — resolver vs. runner into submodules (nice-to-have, can be a follow-up PR).

The architecture is solid and we'd love to land this. Let us know if you need any help with the rebase or have questions about the feedback.

[github-actions]

All contributors have signed the CLA. ✅
_{Posted by the CLA Assistant Lite bot.}

[dasTholo]

I have read the CLA Document and I hereby sign the CLA

[yvgude]

recheck

[dasTholo]

CLA Assist doesn't like me anymore

[yvgude]

@dasTholo — welcome aboard with the CLA signed! 🎉

Since you last rebased, 3 PRs have been merged to main (security fix, build fix, CI hardening). Here's a concrete rebase guide so you don't have to dig through the diff:

Quick rebase steps

git fetch upstream main
git rebase upstream/main

Expected conflict areas (based on the merged diffs):

• rust/src/core/config/mod.rs — 3 new fields added to strip_sensitive_overrides() at line ~906. Your branch shouldn't touch this function, so accept upstream/main's version.
• rust/Cargo.toml / Cargo.lock — rusqlite downgraded from 0.40 to 0.39. Accept upstream and re-lock: cargo update -p rusqlite
• .github/workflows/ci.yml — concurrency group + tarpaulin change. Accept upstream.

None of these should conflict with your actual lean-md changes (registry, format/mod.rs, reverse_cut_gate, ctx_read test).

Remaining review items (from the earlier feedback)

1. Formatter unit tests for format/mod.rs (edge cases: missing rustfmt, fallback to IDE, BLAKE3 unchanged detection)
2. Doc cleanup — remove WasmRenderTransform reference from docs/reference/21-lean-md.md
3. Split lsp/format/mod.rs — resolver vs runner into submodules (can be a follow-up PR if you prefer)

Let me know if you need help with the rebase — happy to do it on your behalf if time is tight.

[dasTholo]

Brought the branch up to current main (v3.9.7) via merge. GitHub reports the PR as MERGEABLE — the lean-md changes have zero file overlap with recent main churn, so the conflict areas anticipated earlier (ops.rs,
integration_tests.rs, config/mod.rs, Cargo.toml, ci.yml) don't materialize. Happy to swap the merge for a linear rebase if you prefer that history.

Two CI failures surfaced purely from the merge with current main, both now fixed:

• fix(clippy) (77761cf) — rust-1.97.0 clippy flags the inline Vec<(&str, fn(&str) -> Result<Snapshot, String>)> in src/bin/gen_registry.rs (from feat(addons): unified distribution Phase 2 — addon publish, hosted installs, registry snapshots #734) as type_complexity under -D warnings, breaking the gen_registry example
  build. Factored the fn-ptr into a Canonicalize type alias, exactly as clippy suggests. Verified with cargo clippy --all-targets --all-features -- -D warnings.
• test(reverse-cut) (f9a0bfd) — the Unified Distribution Phase 3: kind=skills + cross-kind dependency resolution (lean-md reference case) #727/fix(addons): #727 follow-up fixes — pack_dir expansion, declared deps, min_lean_ctx preflight #780 addon-dependency work merged from main references the addon by name in its test fixtures (addon_deps.rs: @dasTholo/lean-md; pack_env.rs: @dasTholo/lean-md-skills). These are
  addon-name/slug references in #[test] code — the exact survivor category the gate already allows — so I added both paths to the no_lmd_symbols_outside_docs_and_hook allowlist and the survivor doc list. All 5 reverse_cut_gate
  tests pass.

On the review items from your last note:

• Formatter unit tests — already in the PR (rust/src/lsp/format/mod.rs), covering exactly the edge cases you flagged: run_command_missing_binary_errors (missing rustfmt), md_and_unknown_and_no_ext_default_to_jetbrains (IDE
  fallback), and blake3_detects_change / blake3_missing_file_errors (BLAKE3 changed/unchanged).
• WasmRenderTransform doc reference — already removed; zero matches repo-wide in docs/reference/21-lean-md.md.
• Split lsp/format/mod.rs — deferred to a follow-up as you okayed (254 lines, well under the LOC gate).