v2.1.1 — Local embeddings (semantic search)

.github/PULL_REQUEST_TEMPLATE.md

@@ -25,6 +25,8 @@
 - [ ] New IPC handlers wrapped in `handle()` and return `IpcResult<T>`
 - [ ] New DB migrations appended (not edited) in `schema.ts`
 - [ ] New SQL goes in `electron/db/queries.ts` — single source of truth (imported by both Electron main process and MCP server); never construct a `Database` instance outside `db/client.ts` (Electron) or `mcp-server.ts` (MCP runtime)
+- [ ] New `dependencies` or `devDependencies` added to `ROLE_MAP` in `scripts/generate-licenses.js` and `licenses.json` regenerated
+- [ ] New MCP tools registered in `electron/mcp/tools/index.ts` dispatch + `electron/lib/tool-schemas.ts` Zod schema
 
 ## Notes for reviewer
 

changelogs/v2.1.1.md

@@ -0,0 +1,86 @@
+## What's new
+
+### Features
+
+- **Local Embeddings (Semantic Search)**: Notes now have semantic neighbour search powered by a packaged local embedding model (`nomic-ai/nomic-embed-text-v1.5`, int8 quantised, ~131 MB). A background HTTP worker (Node-based, spawnable also as a standalone `cairn-embeddings` binary via `@yao-pkg/pkg`) handles inference through `@xenova/transformers` tokenizers + `onnxruntime-node` native CPU runtime. Embeddings live in the `note_embeddings` SQLite table (section-based, v18 schema). The Knowledge Graph gains a `"semantic"` edge type (top-5 nearest neighbours per note, 0.55 floor) with an inline opacity slider in the toolbar, and the Notes editor's Backlinks panel gains a "Semantic" section showing the top-5 most similar notes with score bars and matching section titles. Toggle via the Sparkles icon in the editor header.
+
+- **Section-Based Embeddings**: Notes are now split into sections by `#`/`##` headers, with each section receiving its own embedding vector. Multi-topic notes connect to different topic clusters via different sections instead of diluting signal into a single averaged vector. The `note_embeddings` table PK changed from `(note_id)` to `(note_id, section_idx)` with a `section_title` column (v18 migration); `relationship_cache` gains `source_section_title`/`target_section_title` columns (v19 failsafe migration). Semantic edges store the matching section titles for provenance.
+
+- **Top-K Semantic Edges**: Replaced the absolute 0.78 cosine threshold with top-5 nearest neighbours per note (floor 0.55). The old threshold only captured near-duplicates; `nomic-embed-text-v1.5` cosines for related-but-not-duplicate notes typically land 0.60–0.75. Multi-section notes get edges from all sections, with the best weight per note-pair winning.
+
+- **MCP Tool `cairn_get_semantic_neighbors`**: New MCP tool exposes pre-computed semantic edges to the agent. Given a `noteId`, returns related notes with similarity weights and matching section titles — enabling the agent to discover connections that keyword search or wikilinks would miss.
+
+- **KG Edge Hover Tooltips**: Hovering a semantic edge in the Knowledge Graph (Force or Radial layout) shows a tooltip with `"NoteA › SectionA ↔ NoteB › SectionB · 87%"` — the note titles, matching section titles, and similarity percentage.
+
+- **Section-Level Backlinks Panel**: The Backlinks panel now tracks which `##` section the cursor is in during editing and runs a second semantic search (k=3) on the section's text. Results appear in a "Related to 'SectionName'" group, deduplicated against the whole-note semantic hits. Section tracking updates live as the cursor moves.
+
+- **Native ONNX Runtime**: Switched from xenova's `pipeline()` (which silently fell back to `onnxruntime-web` WASM with a 17.4 GB WebAssembly.Memory arena) to native `onnxruntime-node` CPU backend via `AutoTokenizer.from_pretrained` + `ort.InferenceSession.create`. Idle RSS dropped to ~444 MB with ~2× latency improvement.
+
+- **KG Node Z-Ordering on Hover**: Hovered/selected nodes now render on top of all other nodes and edges. ForceGraph uses `onRenderFramePost` for a final paint pass; RadialTree re-appends highlighted node groups to the end of the SVG DOM. Full titles (up to 60 chars) are shown for highlighted nodes in both layouts.
+
+- **Incremental Auto-Reindex on Save**: Editing a note now triggers a single-note embed reindex in the background (skipped if `content_hash` is unchanged), so the Semantic backlinks section stays current without ever visiting Settings. First-time users run **Reindex notes (embed + search)** once from Settings → Embeddings to backfill existing notes; afterward, the index stays fresh on its own.
+
+- **Long-Document Chunking**: Notes longer than 4000 characters are now split into overlapping chunks (~200 char overlap, breaking on paragraph/sentence/word boundaries) and their embeddings averaged after L2 normalisation. Previously long notes silently truncated at the tokenizer's 8192-token limit, losing signal.
+
+- **Query Vector Cache**: Repeat searches for the same note content (post-debounce) now hit a 64-entry LRU cache in the main process keyed by `sha256("q|" + model + "|" + queryText)` — skipping the worker round-trip entirely on cache hit.
+
+- **Packaged Worker Binary**: `scripts/build-embeddings-binary.js` bundles `electron/embeddings/server.ts` into a standalone `cairn-embeddings` executable via `@yao-pkg/pkg` (mirrors `build-mcp-binary.js`). In dev, the worker is spawned via the system Node binary (`which node`) with `--max-old-space-size=512` + `--cache-dir=${MODELS_DIR}` to sidestep Electron-as-Node native module edge cases; in packaged builds, the binary is loaded from `app.asar.unpacked/dist-embeddings/`.
+
+### Fixes
+
+- **`chunkLongText` Infinite Loop → OOM Crash**: When `end` reached `text.length` exactly, `start = end - CHUNK_OVERLAP` (200) still satisfied `start < text.length`, so the loop pushed the same chunk forever, exhausting the V8 heap and crashing the worker with SIGTRAP. Now breaks out of the loop the moment `end >= text.length`, plus defensive guards for `end <= start` and `nextStart <= start` (no forward progress).
+
+- **Search Returned No Results Despite Reindex**: `recomputeProjections` re-embedded every note with `task='clustering'`, which — because `note_embeddings.note_id` is the primary key with `ON CONFLICT DO UPDATE` — silently overwrote the `search_document` rows that `searchAdjacent` filters by. Recompute now reads existing `search_document` embeddings and only runs UMAP (no re-embed); the two maintenance buttons are clearly labelled ("Reindex notes (embed + search)" vs "Recompute graph clusters (UMAP only)") to prevent confusion.
+
+- **Model Install Never Detected as Installed**: `modelFilesPresent` looked for HuggingFace cache slugs (`nomic-ai--nomic-embed-text-v1.5`) but `@xenova/transformers` stores model files in a nested directory layout (`nomic-ai/nomic-embed-text-v1.5/config.json` + `onnx/*.onnx`). Now checks the actual layout, so a previously-downloaded model shows as installed immediately.
+
+- **Worker Killed During In-Flight Requests**: `stopWorker` would SIGTERM the child even when an embed request was in flight, surfacing as `socket hang up` errors on the renderer. Now refuses to kill when `inFlightEmbeds > 0` (logs the call stack); only `dispose()` on app quit forces a kill.
+
+- **Progress State Lost on Settings Remount**: `reindexProgress` (the `done/total` bar) was renderer-only state, reset whenever the Settings view unmounted. Now `getStatus()` exposes `lastReindexDone/Total` + `lastRecomputeDone/Total` from the main-process lock slots, so the bar restores on remount.
+
+- **Duplicate Concurrent Reindex Calls**: No mutex existed, so rapid clicks or cross-window triggers could kick off overlapping reindex passes. Now `withLock` enforces a single concurrent `reindex` and `recomputeProjections`; duplicates are rejected with `Error: "Reindex already in progress"` and surfaced as `Blocked: ...` in the UI.
+
+- **Notes With Zero Backlinks Hid Semantic Section**: `BacklinksPanel` returned `null` when `total === 0`, so enabling the Sparkles toggle on a note with no backlinks produced no feedback. Now stays visible whenever semantic is enabled, showing "searching…" while loading and "No similar notes found. Reindex from Settings → Embeddings to refresh." when search returns 0 hits.
+
+- **Worker Spawn Used Electron-as-Node**: Native ONNX runtime binaries under `ELECTRON_RUN_AS_NODE=1` can crash or fail to load on some platforms. Dev mode now resolves the system Node binary via `which node` / `where node` and spawns the worker with that instead, with `--max-old-space-size=4096` for the model-load phase.
+
+- **Worker Process Had No Uncaught Handlers**: Native crashes inside `onnxruntime-node` surfaced as `worker exited (code=null)` with no signal name or stderr. Now logs `code` + `signal` on exit, dumps the last 1KB of stderr, and the worker itself emits `uncaughtException` / `unhandledRejection` as `{kind:"error", msg:"..."}` JSON-line events before exiting.
+
+- **`computeAutoRelationships` Wiped Semantic Edges**: The "Recompute auto relationships" button deleted all auto-discovered edges including semantic ones. Now the delete statement filters `AND type != 'semantic'` so semantic edges survive auto-relationship recompute.
+
+- **Model-Aware Pipeline Cache**: Concurrent requests for different embedding models could reuse a cached in-flight pipeline for the wrong model. Now `_pipelinePromiseModelId` tracks which model the cached promise is for; mismatched requests get a fresh pipeline.
+
+- **Tokenizer Padding + Truncation**: Varying-length texts caused ONNX tensor shape errors. Tokenizer now runs with `padding: true, truncation: true` to ensure consistent tensor shapes.
+
+- **Sequential Section Embedding**: Batched section embedding created a `[16, 12, 1000, 1000]` attention matrix spiking to 4 GB. Now embeds one section text per HTTP call — 15× faster for 4 KB texts, peak RSS under ~750 MB.
+
+### Changes
+
+- **Single embedding task**: All embedding operations now store under `task='search_document'`. The previous `clustering` task is gone — `recomputeProjections` reuses `search_document` rows for UMAP, halving DB rows and removing the upsert-overwrite footgun. A `pruneOrphanedClusteringRows` query runs at the start of any full reindex to clean up legacy rows from prior versions.
+
+- **Sequential embedding (one text per HTTP call)**: The worker embeds one section text per HTTP request instead of batching. Batched calls created a `[16, 12, 1000, 1000]` attention matrix spiking to 4 GB; sequential is 15× faster for 4 KB texts and keeps peak RSS under ~750 MB.
+
+- **Worker `--max-old-space-size=512`**: JS heap limit set to 512 MB (ONNX C++ memory is separate). Idle RSS ~444 MB.
+
+- **UI: Semantic folded into Backlinks**: The right-hand `SemanticHubsPanel` (separate `w-72` column) is removed; semantic results now render as the first section inside the bottom Backlinks panel, auto-opening it when results arrive. Header reads e.g. `"3 similar + 2 backlinks"`. The Sparkles icon in the editor header still toggles visibility.
+
+- **Embeddings Settings (`src/components/settings/EmbeddingsSettings.tsx`)**: New section with model install/remove/setDefault controls, real-time download progress bars, worker status display, and maintenance buttons (Reindex / Recompute) with progress bars that survive view switches.
+
+- **Build pipeline**: `package.json` adds `--external:@xenova/transformers --external:onnxruntime-node --external:umap-js` to `compile`/`compile:watch` (dev) and a new `compile` target for `embeddings-server.bundle.js`. New script `build:embeddings-binary` runs `scripts/build-embeddings-binary.js`.
+
+- **`main.ts` cleanup**: Imports `dispose as disposeEmbeddingsWorker` from `./embeddings/client` and calls it in `before-quit` so the worker never lingers.
+
+---
+
+## Changed files
+
+| Area | Files | Change |
+|------|-------|--------|
+| Core embeddings | `electron/embeddings/{types,cosine,nomic,projection,pipeline,port,server,client,manifest,service,sections}.ts` | New module: types/Zod schemas, cosine + topK, nomic prefix, UMAP projection, native onnxruntime-node pipeline, HTTP worker, spawn/health/dispose lifecycle, model manifest, reindex/search/recompute service layer, section splitter |
+| DB layer | `electron/db/schema.ts`, `electron/db/queries.ts`, `electron/db/graph-queries.ts`, `electron/mcp/db.ts`, `electron/mcp-server.ts` | v17→v19 migrations (section-based `note_embeddings`, `relationship_cache` section title columns); query helpers; `computeSemanticRelationships` (top-K=5, section-level comparison); `getSemanticNeighbors` MCP query; `ensureEmbeddingsTable` wired into MCP bootstrap |
+| Graph | `electron/db/graph-queries.ts`, `src/types/index.ts`, `src/store/slices/graph.ts`, `src/components/graph/{ForceGraphCanvas,RadialTreeCanvas,KnowledgeGraphView}.tsx` | `"semantic"` edge type with section titles; `onRenderFramePost` z-ordering in ForceGraph; DOM re-append in RadialTree; edge hover tooltips in both canvases; semantic slider in KG toolbar |
+| MCP tools | `electron/mcp/tools/{graph,index}.ts`, `electron/lib/tool-schemas.ts` | `cairn_get_semantic_neighbors(noteId)` tool — queries pre-computed semantic edges from `relationship_cache` |
+| IPC + preload | `electron/ipc/{embeddings-handlers,settings-handlers,handlers,registry,db-handlers}.ts`, `electron/preload.ts`, `electron/lib/config-cache.ts` | New `db:embeddings:*` + `embeddings:*` handlers with mutex + progress broadcasts; preload `embeddings` namespace with `onProgress`; `app:{get,save}EmbeddingsSettings`; cached config extension; incremental reindex hook on note save; `computeSemanticRelationships` called after bulk reindex |
+| UI | `src/components/settings/{EmbeddingsSettings,settings-view}.tsx`, `src/components/notes/{note-editor,BacklinksPanel,markdown-editor,toc-utils}.tsx`, `src/hooks/useDebouncedValue.ts` | Embeddings settings UI; semantic + section-level results in BacklinksPanel; Sparkles toggle in editor header; cursor section tracking via `onCursorActivity`; `findSectionTitleAtOffset`/`extractSectionTextAtOffset` helpers; debounced query hook |
+| Build | `package.json`, `scripts/build-embeddings-binary.js`, `electron/main.ts` | deps (`@xenova/transformers`, `onnxruntime-node`, `umap-js`); esbuild externals + new compile target for worker bundle; `build:embeddings-binary` script; worker dispose on app quit |
+| Tests | `electron/embeddings/{cosine,projection,chunking,query-cache,sections}.test.ts`, `electron/db/graph-queries.test.ts` | New unit tests for cosine similarity, topK, UMAP projection (deterministic seeded), chunking (terminates on long input, correct chunk count), query cache (LRU eviction, hash-keyed hit), section splitter (12 scenarios), graph-queries (25 tests incl. multi-section semantic edges) |