feat: integrate lean-md as an external lean-ctx addon (+ LSP formatter routing)

docs/reference/21-lean-md.md

@@ -0,0 +1,91 @@
+# Journey 21 — lean-md (Addon Integration)
+
+> lean-md is an **external lean-ctx addon** — a macro/directive Markdown renderer.
+> It lives in its own repository (`dasTholo/lean-md`) with its own release cycle.
+> This page documents how lean-ctx **integrates** the addon. The full `@directive`
+> catalog, engine spec, and E-constructs live in the addon repo, not here.
+
+---
+
+## 1. What lean-md is
+
+lean-md renders `.lmd.md` / `.lean-md` files: `@directive` calls plus a macro
+engine (`@define`/`@call`), container gating (`@if`/`@consumer`), and pipes
+(`@render`). Code-intel directives (`@read`/`@refactor`/`@search`/…) call lean-ctx
+`ctx_*` tools **over the wire** (CLI/MCP); the renderer itself is standalone
+(`rushdown` + `evalexpr`) with **no** lean-ctx crate dependency.
+
+Engine, full directive catalog, and spec: **https://github.com/dasTholo/lean-md**.
+
+## 2. Installation
+
+```bash
+lean-ctx addon add @dasTholo/lean-md        # hosted pack (ctxpkg.com)
+lean-ctx addon add ./lean-ctx-addon.toml    # local manifest (dev/test)
+```
+
+`addon add` resolves a local manifest first, then a hosted `ns/slug` pack, then the
+bundled registry slug. The bundled `lean-md` entry is **listed** — it makes the addon
+discoverable through `lean-ctx addon search`, it is not an install path.
+
+After install, restart the MCP client so the gateway catalog is re-read. The addon
+is spawned as a stdio gateway child; its tools (`ctx_md_render`, `ctx_md_check`)
+become reachable through the lean-ctx server.
+
+## 3. Integration points in lean-ctx
+
+lean-ctx keeps its lmd surface deliberately small: `.lmd.md` is read **raw** (§3.1),
+the addon ships as a registry entry (§3.2), and the addon calls back through the
+stable `ctx_*` surface (§3.3). Everything else is the addon's.
+
+### 3.1 Raw `.lmd.md` read (no in-tree rendering)
+
+`ctx_read` treats `.lmd.md` like any other file: it returns the **raw** bytes and
+never renders (a half-rendered body would be worse than none). Rendering is the
+addon's job, reached explicitly through its `ctx_md_render` / `ctx_md_check` tools
+once installed. lean-ctx carries **no** `.lmd.md` special-casing in `ctx_read`; the
+earlier auto-render delegation hook was reverse-cut before merge.
+
+Source: `rust/src/tools/registered/ctx_read.rs` (no lmd branch),
+gate test `rust/tests/ctx_read_lmd_md_raw.rs`.
+
+### 3.2 Addon registry entry
+
+`rust/data/addon_registry.json` carries the **listed** `lean-md` entry (no runnable
+`[mcp]` command, no `[install]` block), so `core::addons::manifest::is_installable`
+reports `false` and the entry serves discovery only. The validator
+(`core::addons::registry::validate_entries`) requires a homepage for a listed entry.
+
+### 3.3 ctx_* outbound surface = addon contract
+
+Every lean-md code-intel directive calls back into lean-ctx via
+`backend.call("ctx_*", …)`. That tool set (`ctx_read`, `ctx_refactor`,
+`ctx_search`, `ctx_outline`, `ctx_impact`, `ctx_repomap`, `ctx_review`,
+`ctx_routes`, `ctx_smells`, `ctx_architecture`, `ctx_graph`, `ctx_callgraph`,
+`ctx_knowledge`, `ctx_handoff`, `ctx_agent`, …) is a stable **outbound contract**
+and must stay registered. Only `ctx_md_render` / `ctx_md_check` are addon-provided
+and absent from lean-ctx.
+
+## 4. Decoupling rationale (vs. main)
+
+lean-md was developed in-tree (phases 1–9) and then **reverse-cut** before merge:
+the in-tree engine never reaches `main`. The lmd-related deltas this branch lands
+in lean-ctx are integration-only.
+
+| Class   | Change (vs. main)                                                       | Why                                                |
+|---------|-------------------------------------------------------------------------|--------------------------------------------------|
+| removed | `.lmd.md` auto-render delegation in `ctx_read.rs` → **raw read**         | no in-tree engine renders; the addon renders on request |
+| changed | `addon_registry.json`: `lmd` placeholder → **listed** `lean-md` entry    | discoverability; install goes through the hosted pack   |
+| added   | generic `extension_registry::RenderTransform` trait + registry           | infra for `@render type=<name>`, not lmd-exclusive      |
+| kept    | ctx_* outbound tool surface                                             | the addon calls them over the wire                 |
+| added   | gate tests `reverse_cut_gate.rs`, `ctx_read_lmd_md_raw.rs`              | enforce the cut invariant + raw read               |
+
+The engine, full `@directive` catalog, E-constructs, and spec now live in
+`dasTholo/lean-md` and are **not** mirrored here.
+
+## 5. See also
+
+- Addon repo (engine + full directive reference): https://github.com/dasTholo/lean-md
+- Addon manifest contract: `docs/contracts/addon-manifest-v1.md` (upstream)
+- MCP tool catalog: [`appendix-mcp-tools.md`](appendix-mcp-tools.md)
+- Decoupling design: https://github.com/dasTholo/lean-md (addon repo — hosts engine, spec & decoupling design)

docs/reference/appendix-mcp-tools.md

@@ -146,3 +146,17 @@ shows the smallest tool profile that exposes the tool (`M` minimal, `S` standard
    `metrics`, `session`) independently of the static profile filter.
 3. Lazy clients use `ctx_call` + `ctx_discover_tools` + `ctx_load_tools` to reach
    tools not in their active profile without listing all 79 upfront.
+
+---
+
+## lean-md addon (`.lmd.md` render pipeline)
+
+`.lmd.md` / `.lean-md` rendering is provided by the **external lean-md addon**
+(`dasTholo/lean-md`), not by lean-ctx itself. `ctx_md_render` / `ctx_md_check` are
+exposed by the addon's MCP server once installed (`lean-ctx addon add @dasTholo/lean-md`); a
+`.lmd.md` passed to `ctx_read` is returned **raw** — lean-ctx never renders it
+(rendering is an explicit addon call). The `@directive` catalog and `@lean-md` header fields live in the
+addon repo.
+
+- **Integration reference:** [`21-lean-md.md`](21-lean-md.md)
+- **Addon repo:** https://github.com/dasTholo/lean-md

rust/data/addon_registry.json

@@ -388,26 +388,27 @@
     },
     {
       "addon": {
-        "name": "letta",
-        "display_name": "Letta (MemGPT)",
-        "version": "",
-        "description": "Stateful agent runtime that manages its own memory like an operating system (core + archival tiers). Exposes MCP-compatible tools, but needs a running server + database, so it is listed rather than auto-installed.",
-        "author": "letta-ai",
-        "homepage": "https://github.com/letta-ai/letta",
+        "name": "lean-md",
+        "display_name": "Lean-MarkDown (LMD)",
+        "version": "0.2.0",
+        "description": "Directive-driven Markdown for agent plans. Reusable macros, phase-isolation and single-sourcing keep plans token-lean and cache-safe — written hard against lean-ctx, inspired by MarkdownAI.",
+        "author": "dasTholo",
+        "homepage": "https://github.com/dasTholo/lean-md",
         "license": "Apache-2.0",
         "categories": [
-          "memory"
+          "plans",
+          "workflow"
         ],
-        "integration": "memory",
+        "integration": "mcp",
         "keywords": [
-          "memory",
-          "agent-runtime",
-          "stateful",
-          "memgpt",
-          "competitor",
-          "mcp"
+          "markdown",
+          "plans",
+          "macros",
+          "directives",
+          "phase-isolation",
+          "subagents"
         ],
-        "min_lean_ctx": "3.8.0",
+        "min_lean_ctx": "3.9.6",
         "verified": false
       },
       "mcp": {
@@ -422,25 +423,24 @@
     },
     {
       "addon": {
-        "name": "lmd",
-        "display_name": "Lean-MarkDown (LMD)",
+        "name": "letta",
+        "display_name": "Letta (MemGPT)",
         "version": "",
-        "description": "Directive-driven Markdown for agent plans. Reusable macros, phase-isolation and single-sourcing keep plans token-lean and cache-safe — written hard against lean-ctx, inspired by MarkdownAI.",
-        "author": "dasTholo",
-        "homepage": "https://github.com/dasTholo/lean-ctx/tree/feat-lmd-v1",
+        "description": "Stateful agent runtime that manages its own memory like an operating system (core + archival tiers). Exposes MCP-compatible tools, but needs a running server + database, so it is listed rather than auto-installed.",
+        "author": "letta-ai",
+        "homepage": "https://github.com/letta-ai/letta",
         "license": "Apache-2.0",
         "categories": [
-          "plans",
-          "workflow"
+          "memory"
         ],
-        "integration": "none",
+        "integration": "memory",
         "keywords": [
-          "markdown",
-          "plans",
-          "macros",
-          "directives",
-          "phase-isolation",
-          "subagents"
+          "memory",
+          "agent-runtime",
+          "stateful",
+          "memgpt",
+          "competitor",
+          "mcp"
         ],
         "min_lean_ctx": "3.8.0",
         "verified": false

rust/src/bin/gen_registry.rs

@@ -31,7 +31,7 @@ fn main() {
     let data_dir = repo_data_dir();
     let mut failed = false;
 
-    let targets: Vec<(&str, fn(&str) -> Result<Snapshot, String>)> = vec![
+    let targets: Vec<(&str, Canonicalize)> = vec![
         ("addon_registry.json", canonical_addon),
         #[cfg(feature = "tree-sitter")]
         ("grammar_registry.json", canonical_grammar),
@@ -85,6 +85,9 @@ fn main() {
 
 use lean_ctx::core::addons::registry_snapshot::{Snapshot, canonical_addon_registry};
 
+/// Canonicalizer for one registry file: raw JSON text -> validated `Snapshot`.
+type Canonicalize = fn(&str) -> Result<Snapshot, String>;
+
 fn canonical_addon(text: &str) -> Result<Snapshot, String> {
     canonical_addon_registry(text)
 }

rust/src/core/addons/registry.rs

@@ -404,25 +404,26 @@ mod tests {
     }
 
     #[test]
-    fn flagship_lmd_is_listed() {
-        let lmd = get("lmd").expect("lmd in registry");
+    fn flagship_lean_md_is_listed() {
+        let lmd = get("lean-md").expect("lean-md in registry");
         assert_eq!(lmd.addon.author, "dasTholo");
         assert!(!lmd.addon.homepage.is_empty());
-        // Listed-only until it publishes an MCP endpoint — never fabricated.
+        // Listed-only: the addon ships as a hosted pack (`addon add @dasTholo/lean-md`),
+        // the bundled entry is discovery only — never a fabricated endpoint.
         assert!(!lmd.is_installable());
     }
 
     #[test]
     fn search_matches_keywords_and_categories() {
-        assert!(search("markdown").iter().any(|m| m.addon.name == "lmd"));
-        assert!(search("plans").iter().any(|m| m.addon.name == "lmd"));
-        assert!(search("").iter().any(|m| m.addon.name == "lmd"));
+        assert!(search("markdown").iter().any(|m| m.addon.name == "lean-md"));
+        assert!(search("plans").iter().any(|m| m.addon.name == "lean-md"));
+        assert!(search("").iter().any(|m| m.addon.name == "lean-md"));
         assert!(search("definitely-no-such-term").is_empty());
     }
 
     #[test]
     fn get_is_case_insensitive() {
-        assert!(get("LMD").is_some());
-        assert!(get("  lmd ").is_some());
+        assert!(get("LEAN-MD").is_some());
+        assert!(get("  lean-md ").is_some());
     }
 }

rust/src/core/cache.rs

@@ -1200,6 +1200,13 @@ mod tests {
     fn hebbian_eviction_bonus_is_wired() {
         // #3: files read together build a Hebbian association via store()'s
         // recording, and that association must feed the eviction bonus.
+        //
+        // Warm up tiktoken first: the very first count_tokens() in the process
+        // lazily loads the BPE tables (can exceed the 500ms co-access burst
+        // window). store() calls count_tokens() internally, so without warming
+        // up, the two store() calls below straddle that window and never
+        // associate — a flaky-empty bonus. Warming up keeps them in one burst.
+        let _ = count_tokens("warmup");
         let mut cache = SessionCache::new();
         cache.store("/a.rs", "fn a() {}");
         cache.store("/b.rs", "fn b() {}");

rust/src/core/extension_registry.rs

@@ -42,12 +42,20 @@ pub trait ReadMode: Send + Sync {
     fn render(&self, source: &str, path: &str) -> String;
 }
 
+/// `@render type=<name>` — a WASM-backed render transform.
+/// `hint`: consumer hint (ai = 0, human = 1).
+pub trait RenderTransform: Send + Sync {
+    fn name(&self) -> &str;
+    fn render(&self, input: &str, hint: i32) -> String;
+}
+
 /// Registry of pluggable read-modes, compressors, and chunkers.
 #[derive(Default)]
 pub struct ExtensionRegistry {
     read_modes: BTreeMap<String, Arc<dyn ReadMode>>,
     compressors: BTreeMap<String, Arc<dyn Compressor>>,
     chunkers: BTreeMap<String, Arc<dyn Chunker>>,
+    render_transforms: BTreeMap<String, Arc<dyn RenderTransform>>,
 }
 
 impl ExtensionRegistry {
@@ -134,6 +142,24 @@ impl ExtensionRegistry {
     pub fn chunker_names(&self) -> Vec<String> {
         self.chunkers.keys().cloned().collect()
     }
+
+    /// Register (or replace) a render transform by its name.
+    pub fn register_render_transform(&mut self, handler: Arc<dyn RenderTransform>) {
+        self.render_transforms
+            .insert(handler.name().to_string(), handler);
+    }
+
+    /// Look up a render transform by name.
+    #[must_use]
+    pub fn render_transform(&self, name: &str) -> Option<Arc<dyn RenderTransform>> {
+        self.render_transforms.get(name).cloned()
+    }
+
+    /// Registered render transform names (sorted).
+    #[must_use]
+    pub fn render_transform_names(&self) -> Vec<String> {
+        self.render_transforms.keys().cloned().collect()
+    }
 }
 
 /// Process-global registry, seeded with built-ins on first access.
@@ -319,6 +345,25 @@ mod tests {
         assert_eq!(c.compress("hi", None), "HI");
     }
 
+    struct UpperRender;
+    impl RenderTransform for UpperRender {
+        fn name(&self) -> &str {
+            "upper"
+        }
+        fn render(&self, input: &str, hint: i32) -> String {
+            format!("{}:{}", hint, input.to_uppercase())
+        }
+    }
+
+    #[test]
+    fn render_transform_registers_and_resolves_with_hint() {
+        let mut reg = ExtensionRegistry::with_builtins();
+        reg.register_render_transform(Arc::new(UpperRender));
+        let r = reg.render_transform("upper").unwrap();
+        assert_eq!(r.render("hi", 1), "1:HI");
+        assert!(reg.render_transform_names().contains(&"upper".to_string()));
+    }
+
     #[test]
     fn global_registry_seeds_builtins() {
         let reg = global().read().unwrap();