feat(collections): Phase 1 docs collections (create/index/link/grants, wires loaders + gitignore walker)

.gitignore

@@ -8,6 +8,8 @@ data/
 !benchmarks/data/
 benchmarks/data/*
 !benchmarks/data/README.md
+# tiny pre-registered eval sets are tracked (not datasets)
+!benchmarks/data/collections_eval_questions.json
 *.rknn
 *.rkllm
 models/minilm-onnx/model.onnx

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+Collections, Phase 1 (docs MVP, per `docs/specs/codebase-indexing-collections-design.md`): named containers of content indexed from a folder, queryable by granted agents alongside conversation memory. A collection is a first-class row (`created -> indexing -> ready | error`, plus reversible `archived`) with typed project links (`{type: taos|git, id}`, metadata only, never access-granting) and per-agent grants (`(canonical_id, scope='collection', collection_id)` unique rows, enforced at search time). Indexing wires the previously-unwired loader framework into a real ingest path: a gitignore-aware walker (stdlib rules; VCS/dependency/hidden dirs, binaries, oversized files, and symlink escapes skipped) feeds files a registered loader claims through a zero-dep paragraph chunker into `ingest_batch` under the collection's own agent namespace, with per-chunk content-hash ids so re-index dedups unchanged files; changed and deleted files have their old rows superseded (`valid_to` + marker), never deleted. The feature is off by default: the new `collections.allowed_roots` config list (or `TAOSMD_COLLECTIONS_ALLOWED_ROOTS`) must name the directories collections may index, and `source_path` is containment-checked (`resolve_within`) at create and at every index. Surfaces: HTTP (`POST /collections` and `POST /collections/{id}/index` admin-gated with async 202+poll indexing, `DELETE /collections/{id}` archives; list/get/link/unlink/grants on the data plane; `collection`/`collections`/`collections_only` on search), CLI (`taosmd collections list|create|index|link|unlink|grant|revoke`), and MCP (`memory_list_collections`, `collection` on `memory_search`). Collection hits carry `collection_id`/`file_path`/`source` metadata. A per-collection `embedder` field is stored and returned now (the mechanism for the code-embedder bake-off); Phase 1 always indexes with the global default. `benchmarks/collections_eval.py` pre-registers the file-level Recall@5 eval over the repo's own docs.
+
 Admin token separation (#154, phase 1). Admin operations are now gated by a dedicated `admin_token`, distinct from the data-plane `server_token`. Previously the server token gated every data and A2A endpoint AND the admin surface, so on a token-less deployment the only way to authorize an admin op was to set a server token, which locked out every agent on the data plane for the duration of the admin window (this hit the Pi bus in production for about three minutes during a channel cleanup). Now the admin write routes (`POST /shelves`, `POST /shelves/{id}/archive|unarchive`, `POST /a2a/admin/delete-channel|rename-channel|supersede-message`) are exempt from the data-plane token gate and enforce the admin token themselves. Resolution prefers `admin_token` and falls back to `server_token`: existing token-secured installs keep working unchanged; setting only `admin_token` gates admin while leaving data and A2A endpoints open; with both set the data plane is gated by `server_token` and admin by `admin_token`, so a caller holding only the server token cannot run admin ops; with neither set the admin surface still fails closed (403). Configure via `admin_token` in config, the `TAOSMD_ADMIN_TOKEN` env var, or `taosmd config set-admin-token`. Phase 2 (isolating admin operations from the single service loop so a slow admin op cannot stall data reads/writes) is not part of this change and is tracked separately.
 
 Scoping fix: `reindex` now carries the `project` scope and provenance through when it rebuilds an agent's vector rows from the archive. The rebuild loop stamped only `{"agent": agent}` plus role/timestamp, dropping the `project` tag, the `archive_span_id`, and the nested user metadata (`source_id`/`forget_after`). Losing the project tag was a cross-project scope leak: a project-scoped row came back project-untagged but still agent-tagged, so after a reindex it surfaced in a DIFFERENT project's search for the same agent. `reindex` now mirrors `reconcile`'s metadata reconstruction exactly, so a reindexed row is indistinguishable from a reconcile-repaired one in scope and provenance: it stays visible to its own project's search, stays invisible to other projects, keeps its archive-span linkage for the claims gate, and keeps its `source_id` so a re-POST of the same batch still dedupes. Zero-loss was never at risk (the archive is untouched); this is a scoping-correctness repair.

benchmarks/collections_eval.py

@@ -0,0 +1,128 @@
+#!/usr/bin/env python3
+"""Phase 1 collections eval: file-level Recall@5 over the repo's own docs/.
+
+Pre-registered per the design spec (section 5): index the taosmd repo's
+docs/ folder into a temporary collection and answer the 20-question set in
+``benchmarks/data/collections_eval_questions.json`` (written before the
+index was built, gold labels are file paths). A question scores 1 when any
+of the top-5 results' ``file_path`` metadata matches a gold file. No LLM
+judge anywhere in the loop.
+
+Modes
+-----
+- ``semantic``: the real path; requires a local ONNX embedding model. This
+  is the number that counts against the section-5 kill bar; run it on the
+  bench host where the ONNX embedder is installed.
+- ``lexical``: the fallback smoke for hosts with no local ONNX model. The
+  embedder is stubbed with a deterministic hash vector purely so rows land
+  in the vector store; retrieval then uses the engine's BM25-only mode
+  (``mode="bm25"``), which never touches the stubbed vectors. This
+  exercises the full production walk/chunk/ingest/grant/scope path with a
+  lexical ranker, and validates the walker + container plumbing
+  independently of the embedder (the spec's stated purpose for the Phase 1
+  eval).
+- ``auto`` (default): semantic when an ONNX model is present, else lexical.
+
+Usage::
+
+    python3 benchmarks/collections_eval.py [--mode auto|semantic|lexical]
+        [--docs-dir docs] [--k 5] [--questions benchmarks/data/...json]
+"""
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import sys
+import tempfile
+import time
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(REPO_ROOT))
+
+AGENT = "eval-agent"
+
+
+async def _hash_embed(text: str, task: str = "search_document") -> list[float]:
+    """Deterministic stub (same as the test suite's). BM25 mode ignores it."""
+    h = hash(text) & 0xFFFFFFFF
+    return [((h >> (i * 4)) & 0xFF) / 255.0 for i in range(8)]
+
+
+async def run(docs_dir: Path, questions_path: Path, mode: str, k: int) -> int:
+    from taosmd import api, config
+    from taosmd.collections import CollectionStore, ingest_folder
+
+    spec = json.loads(questions_path.read_text())
+    questions = spec["questions"]
+
+    with tempfile.TemporaryDirectory(prefix="taosmd-collections-eval-") as tmp:
+        data_dir = str(Path(tmp) / "data")
+        Path(data_dir).mkdir()
+        config.set_collections_allowed_roots([str(docs_dir)], data_dir=data_dir)
+
+        stores = await api._ensure_stores(data_dir)
+        if mode == "auto":
+            has_onnx = api._resolve_onnx_path(data_dir) is not None
+            mode = "semantic" if has_onnx else "lexical"
+            if mode == "lexical":
+                print("auto: no local ONNX embedding model found; using the "
+                      "lexical (BM25) fallback. Run the semantic arm on the "
+                      "bench host for the number that counts.")
+        if mode == "lexical":
+            stores["vector"].embed = _hash_embed  # rows must land; BM25 ignores vectors
+        search_opts = {"mode": "bm25"} if mode == "lexical" else {}
+
+        store = CollectionStore(data_dir)
+        col = store.create(name="repo-docs", kind="docs", source_path=str(docs_dir))
+        store.grant(col["id"], AGENT)
+
+        t0 = time.time()
+        stats = await ingest_folder(col["id"], data_dir=data_dir)
+        dt = time.time() - t0
+        print(f"indexed {stats['files_indexed']} files / "
+              f"{stats['chunks_ingested']} chunks in {dt:.1f}s "
+              f"(skipped: unclaimed={stats['skipped_unclaimed']} "
+              f"ignored={stats['skipped_ignored']} binary={stats['skipped_binary']})")
+        if stats.get("degraded"):
+            print("ERROR: embedder unavailable; aborting", file=sys.stderr)
+            return 2
+
+        hits_n = 0
+        for q in questions:
+            results = await api.search(
+                q["question"], agent=AGENT, limit=k,
+                collections=[col["id"]], collections_only=True,
+                data_dir=data_dir, **search_opts,
+            )
+            got_files = [h["metadata"].get("file_path") for h in results]
+            hit = any(f in q["answer_files"] for f in got_files)
+            hits_n += hit
+            mark = "HIT " if hit else "MISS"
+            print(f"  [{mark}] {q['question'][:70]:<70} -> {got_files[:3]}")
+
+        recall = hits_n / len(questions)
+        print(f"\nmode={mode}  file-level Recall@{k}: {hits_n}/{len(questions)} = {recall:.3f}")
+        if mode == "lexical":
+            print("note: lexical fallback smoke. The pre-registered kill bar "
+                  "(Recall@5 >= 0.8) is judged on the semantic arm with the "
+                  "ONNX embedder on the bench host.")
+        store.close()
+    return 0
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    p.add_argument("--docs-dir", default=str(REPO_ROOT / "docs"))
+    p.add_argument("--questions",
+                   default=str(REPO_ROOT / "benchmarks" / "data" / "collections_eval_questions.json"))
+    p.add_argument("--mode", choices=["auto", "semantic", "lexical"], default="auto")
+    p.add_argument("--k", type=int, default=5)
+    args = p.parse_args()
+    return asyncio.run(run(Path(args.docs_dir).resolve(),
+                           Path(args.questions), args.mode, args.k))
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

benchmarks/data/collections_eval_questions.json

@@ -0,0 +1,25 @@
+{
+  "description": "Phase 1 collections eval: 20 questions over the taosmd repo's own docs/ folder, written before the index was built. Gold labels are file paths relative to docs/. Metric: file-level Recall@5 (a question scores 1 if any of the top-5 results' file_path is in answer_files). No LLM judge.",
+  "questions": [
+    {"question": "How do I pause a long benchmark run, reboot the machine, and resume it later without repeating work?", "answer_files": ["bench-pause-resume.md"]},
+    {"question": "Can several agents share a single taosmd install while keeping their memories isolated?", "answer_files": ["multi-agent.md"]},
+    {"question": "How do I keep the taosmd HTTP server running in the background so it survives logout and starts at login?", "answer_files": ["serve-service.md"]},
+    {"question": "Which memory controls should the taOS settings UI expose and what scope does each control have?", "answer_files": ["INTEGRATION-memory-config.md"]},
+    {"question": "How do agent trace events and memory operations map onto OpenTelemetry GenAI spans and attributes?", "answer_files": ["otel-genai-mapping.md"]},
+    {"question": "What prompt do I paste into a tool-capable agent to install taosmd for me?", "answer_files": ["INSTALL-AGENT-PROMPT.md"]},
+    {"question": "What Recall@5 and judge scores has taosmd published on LongMemEval?", "answer_files": ["benchmarks.md", "research-report.md"]},
+    {"question": "Where are negative experiment results and methodology notes recorded?", "answer_files": ["research-report.md"]},
+    {"question": "What is the design for feeding codebases and folders into named collections agents can query?", "answer_files": ["specs/codebase-indexing-collections-design.md", "collections.md"]},
+    {"question": "What is the proposal for routing retrieval by semantic category?", "answer_files": ["specs/category-routed-retrieval-design.md"]},
+    {"question": "How does the nightly session catalog pipeline enrich and crystallize conversations?", "answer_files": ["specs/session-catalog-pipeline.md", "plans/session-catalog-pipeline.md"]},
+    {"question": "What is the librarian and how does it decide what to enrich at ingest time?", "answer_files": ["specs/2026-04-15-librarian-design.md"]},
+    {"question": "What was the plan for running retrieval sources in parallel and reranking their results?", "answer_files": ["specs/parallel-retrieval-reranking.md", "plans/parallel-retrieval-reranking.md"]},
+    {"question": "What is the spec for the memory management app?", "answer_files": ["specs/memory-management-app.md", "plans/memory-management-app.md"]},
+    {"question": "How are tasks, edges, and the ready queue designed in the task graph?", "answer_files": ["superpowers/specs/2026-06-10-task-graph-design.md"]},
+    {"question": "What is the long-term vision for the memory cockpit dashboard?", "answer_files": ["superpowers/specs/2026-06-21-memory-cockpit-vision.md"]},
+    {"question": "How does the smart installer pick what to install for a machine?", "answer_files": ["superpowers/specs/2026-06-16-smart-installer-design.md", "superpowers/plans/2026-06-17-smart-installer.md"]},
+    {"question": "What were the LoCoMo scorecard results per conversation?", "answer_files": ["specs/2026-04-19-locomo-scorecards.md"]},
+    {"question": "What went wrong with the cross-encoder model path and how was it fixed?", "answer_files": ["agent-jobs/job-002-cross-encoder-path-fix.md"]},
+    {"question": "How do grants control which agents may query an indexed folder collection?", "answer_files": ["collections.md", "specs/codebase-indexing-collections-design.md"]}
+  ]
+}

docs/collections.md

@@ -0,0 +1,144 @@
+# Collections: feed folders to your agents
+
+A collection is a named container of content indexed from one folder. Point
+it at a repo's `docs/` (or any documentation folder inside an allowed root),
+index it, grant agents access, and they can query it alongside their
+conversation memory. Phase 1 indexes docs-shaped files (md, txt, markdown,
+rst, plus anything another registered loader claims, such as `*.chat.json`);
+the code path is Phase 2.
+
+## Enable the feature (off by default)
+
+Collections read the server's filesystem, so they are disabled until an
+operator opts in by listing the directories collections may be created
+under:
+
+```json
+// ~/.taosmd/config.json
+{ "collections": { "allowed_roots": ["/srv/docs", "/home/jay/repos"] } }
+```
+
+Or via the environment: `TAOSMD_COLLECTIONS_ALLOWED_ROOTS=/srv/docs:/home/jay/repos`.
+A collection's `source_path` must resolve inside one of these roots
+(symlink escapes are rejected), checked at create time and again at every
+index. An empty list means collections are off.
+
+## Lifecycle
+
+```bash
+# create (admin operation over HTTP; local CLI works directly)
+taosmd collections create --name "repo docs" --kind docs --source /srv/docs/myrepo
+
+# index (walks the folder, chunks, embeds; re-run any time)
+taosmd collections index col-ab12cd34ef56
+
+# give an agent query access
+taosmd collections grant col-ab12cd34ef56 my-agent
+
+# attach to a project for discovery (taOS prj-* id or git fingerprint)
+taosmd collections link col-ab12cd34ef56 --type git --id abc123def456
+
+taosmd collections list
+```
+
+Statuses: `created -> indexing -> ready | error` (and `archived` after a
+delete). Re-indexing is incremental: unchanged files are skipped by content
+hash, changed and deleted files have their old rows superseded (hidden from
+recall, never destroyed; the archive keeps every version). A file that still
+exists but whose content has been emptied is retired the same way, and
+reported separately as `files_emptied` rather than `files_deleted`, so a
+blanked file is never mistaken for one that vanished from disk.
+
+Index stats (on the collection row, and in the `GET /collections/{id}`
+response) always carry the same keys:
+
+```text
+files_indexed      files whose content was chunked and ingested this pass
+files_unchanged    files skipped because their content hash matched
+files_deleted      previously-indexed files no longer present on disk
+files_emptied      previously-indexed files still present but now empty
+files_total        files currently tracked in the collection's hash state
+chunks_ingested    chunks written this pass
+chunks_skipped     chunks the batch deduped away
+chunks_superseded  active rows retired this pass (never destroyed)
+errors             up to 20 per-file failure strings
+```
+
+`files_deleted` and `files_emptied` count disjoint sets, and both are
+present (as `0`) when nothing was retired.
+
+The walker respects `.gitignore` files (simplified rules), skips VCS and
+dependency directories, hidden directories, binaries, and oversized files.
+A tree with more than 20000 ingestable files errors the index with a clear
+message instead of grinding through it; point the collection at a smaller
+folder (a `docs/` directory, not a monorepo root).
+
+One index runs per collection at a time: starting an index while one is
+already running returns `409` (poll `GET /collections/{id}` until the
+status settles at `ready` or `error`, then retry).
+
+## Querying
+
+Search merges granted collections in with conversation memory:
+
+```bash
+curl -s localhost:7900/search -d '{
+  "query": "how do I configure the widget?",
+  "agent": "my-agent",
+  "collections": ["col-ab12cd34ef56"],
+  "limit": 5
+}'
+```
+
+- `collections` (list) or `collection` (single id) adds collection content.
+- `collections_only: true` restricts the search to the collections.
+- Grants are enforced per requesting agent: a collection the agent holds no
+  grant for contributes nothing (and its existence is not revealed).
+- Collection hits carry `collection_id`, `file_path` (relative to the
+  source root), and `source: "collection"` in their metadata.
+
+Trust model: grants protect collections from ungranted *agents*, not from
+holders of the server token. Grant and revoke live on the data plane, so
+anyone presenting the server's bearer token can manage grants (and could
+grant themselves access); collection access is therefore exactly as strong
+as the server token. Treat the token as the security boundary and grants
+as the per-agent scoping mechanism inside it. Only create/index/delete sit
+behind the separate admin token, because those touch the server's
+filesystem.
+
+Over MCP: `memory_list_collections` lists them; `memory_search` takes a
+`collection` parameter.
+
+## HTTP surface
+
+Data plane (bearer token when one is configured):
+
+```text
+GET    /collections [?project=<id>]
+GET    /collections/{id}
+POST   /collections/{id}/link      {"type": "taos"|"git", "id": "..."}
+POST   /collections/{id}/unlink    same body
+POST   /collections/{id}/grants    {"agent": "..."}
+DELETE /collections/{id}/grants/{agent}
+POST   /search                     with collection/collections/collections_only
+```
+
+Admin (dedicated admin token, fail-closed):
+
+```text
+POST   /collections                {"name", "kind", "source_path", "embedder"?}
+POST   /collections/{id}/index     -> 202; poll GET /collections/{id}
+DELETE /collections/{id}           -> archive (reversible)
+```
+
+The optional `embedder` field is stored and returned per collection (the
+per-collection embedder mechanism); Phase 1 always indexes with the global
+default embedder.
+
+## Zero-loss guarantees
+
+Delete archives, it never destroys: the collection row, its vector rows,
+and its archive entries all stay on disk; archived collections simply stop
+contributing to search. Re-index supersedes replaced rows with the same
+`valid_to` machinery corrections use. Destruction remains exclusive to the
+wipe surface.