diff --git a/CHANGELOG.md b/CHANGELOG.md index 0997bd77..bc610484 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## Unreleased +`GET /version` with a capability list (new `taosmd.capabilities` module). The server now publishes what the running build actually supports, because neither a status code nor a version number could answer that. `taosmd serve` renders the dashboard SPA on unknown non-API paths, so `GET /collections` returns `200 text/html` on a build with no collections code, and an integrator who "verified" a route by checking for a 200 got a confident yes from a server that could not do the thing (this really happened, against the wrong service). Semver does not close the gap either: features land continuously between bumps, and a production box sat a month stale without anyone noticing even though `GET /health` already reported a version. `/version` returns `{"version", "commit", "commit_source", "built_at", "built_at_source", "capabilities"}` and `GET /health` gains the same `capabilities` list alongside its existing `status` and `version` keys, which are unchanged (taOS and the dashboard consume both). Both endpoints are unauthenticated by design, joining `/health` in `_PUBLIC_PATHS`, so monitoring and drift probes keep working on a token-secured box; they expose build identity and capability identifiers only (no paths, no tokens, no configuration). Capabilities are **stable contract identifiers with an explicit version suffix** (`collections.v1`, `grants.v1`, `temporal.v1`, `a2a.v1`, `tasks.v1`, `ingest.v1`, `search.v1`, `graph.v1`, `shelves.v1`), not feature names: a breaking change to a wire contract becomes `collections.v2`, so a client pinned to `collections.v1` sees the capability disappear (a visible break it can act on) rather than `collections` silently meaning something new; additive changes keep the identifier. The list is derived at request time by probing the running build (each identifier is declared next to the module and symbols that implement it, and is advertised only if they resolve), so deleting or renaming an implementation deletes the claim instead of leaving a stale boast, and a divergence test asserts every declared capability's routes exist in the real dispatcher. The commit sha is resolved once at first call and cached, never per request and never by shelling out: `git rev-parse` in a request path can block on a lock or a slow filesystem, so the git plumbing is read directly from the filesystem (`.git/HEAD` -> loose ref or `packed-refs`, including the `gitdir:` indirection used by worktrees and submodules), with an optional packaged `taosmd/_build_info.py` stamp taking precedence for wheel and container builds. Every step degrades to `null` rather than raising, so a pip install with no checkout and no stamp still gets a working endpoint. + Collections, Phase 1 (docs MVP, per `docs/specs/codebase-indexing-collections-design.md`): named containers of content indexed from a folder, queryable by granted agents alongside conversation memory. A collection is a first-class row (`created -> indexing -> ready | error`, plus reversible `archived`) with typed project links (`{type: taos|git, id}`, metadata only, never access-granting) and per-agent grants (`(canonical_id, scope='collection', collection_id)` unique rows, enforced at search time). Indexing wires the previously-unwired loader framework into a real ingest path: a gitignore-aware walker (stdlib rules; VCS/dependency/hidden dirs, binaries, oversized files, and symlink escapes skipped) feeds files a registered loader claims through a zero-dep paragraph chunker into `ingest_batch` under the collection's own agent namespace, with per-chunk content-hash ids so re-index dedups unchanged files; changed and deleted files have their old rows superseded (`valid_to` + marker), never deleted. The feature is off by default: the new `collections.allowed_roots` config list (or `TAOSMD_COLLECTIONS_ALLOWED_ROOTS`) must name the directories collections may index, and `source_path` is containment-checked (`resolve_within`) at create and at every index. Surfaces: HTTP (`POST /collections` and `POST /collections/{id}/index` admin-gated with async 202+poll indexing, `DELETE /collections/{id}` archives; list/get/link/unlink/grants on the data plane; `collection`/`collections`/`collections_only` on search), CLI (`taosmd collections list|create|index|link|unlink|grant|revoke`), and MCP (`memory_list_collections`, `collection` on `memory_search`). Collection hits carry `collection_id`/`file_path`/`source` metadata. A per-collection `embedder` field is stored and returned now (the mechanism for the code-embedder bake-off); Phase 1 always indexes with the global default. `benchmarks/collections_eval.py` pre-registers the file-level Recall@5 eval over the repo's own docs. Admin token separation (#154, phase 1). Admin operations are now gated by a dedicated `admin_token`, distinct from the data-plane `server_token`. Previously the server token gated every data and A2A endpoint AND the admin surface, so on a token-less deployment the only way to authorize an admin op was to set a server token, which locked out every agent on the data plane for the duration of the admin window (this hit the Pi bus in production for about three minutes during a channel cleanup). Now the admin write routes (`POST /shelves`, `POST /shelves/{id}/archive|unarchive`, `POST /a2a/admin/delete-channel|rename-channel|supersede-message`) are exempt from the data-plane token gate and enforce the admin token themselves. Resolution prefers `admin_token` and falls back to `server_token`: existing token-secured installs keep working unchanged; setting only `admin_token` gates admin while leaving data and A2A endpoints open; with both set the data plane is gated by `server_token` and admin by `admin_token`, so a caller holding only the server token cannot run admin ops; with neither set the admin surface still fails closed (403). Configure via `admin_token` in config, the `TAOSMD_ADMIN_TOKEN` env var, or `taosmd config set-admin-token`. Phase 2 (isolating admin operations from the single service loop so a slow admin op cannot stall data reads/writes) is not part of this change and is tracked separately. diff --git a/README.md b/README.md index 3e933cb6..2d0b97b0 100644 --- a/README.md +++ b/README.md @@ -560,7 +560,7 @@ taosmd config set-token export TAOSMD_TOKEN= ``` -The token is sent as `Authorization: Bearer ` on every request. `GET /health` and the web inspector (`GET /`) are always public so monitoring probes keep working. Never commit the token to version control. +The token is sent as `Authorization: Bearer ` on every request. `GET /health`, `GET /version`, and the web inspector (`GET /`) are always public so monitoring and capability probes keep working. Never commit the token to version control. ### How the Python API and MCP server interact with remote mode @@ -698,7 +698,8 @@ ollama pull qwen3:4b # Same model as the smaller node, same quality | Method | Path | Request | Response | |--------|------|---------|----------| -| `GET` | `/health` | (none) | `{"status": "ok", "version": }` | +| `GET` | `/health` | (none) | `{"status": "ok", "version": , "capabilities": []}` | +| `GET` | `/version` | (none) | `{"version", "commit", "commit_source", "built_at", "built_at_source", "capabilities"}` | | `POST` | `/ingest` | `{"text": str, "agent": str, "project"?: str}` | `{"archived": int, "agent": str, "project": str\|null, "data_dir": str, ...}` (adds `"vector_failures": int` and `"degraded": true` when the embedder fails) | | `POST` | `/ingest/batch` | `{"items": [{"text": str, "id"?: str, "metadata"?: obj}], "agent": str, "project"?: str}` | `{"ingested": int, "skipped": int, ...}` | | `POST` | `/search` | `{"query": str, "agent": str, "limit"?: int, "project"?: str, "also_include"?: [str], "mode"?: "bm25"}` | `{"hits": [...]}` | @@ -718,6 +719,47 @@ Each hit in `/search` results has the agent-rules contract shape: `{text, source `/ingest/batch` is the bulk-import path: each item can carry a stable `id` (your content hash), preserved as `source_id` and used to skip already-imported items, so the whole batch can be re-POSTed safely after a partial migration. `mode=bm25` on `/search` skips query embedding entirely and returns keyword-ranked hits in about 10ms, built for search-as-you-type UIs over short-form memory; the default mode remains the full recipe-driven retrieval. +### Version and capability discovery + +Do not probe for a feature with a status code. `taosmd serve` answers unknown non-API paths with the dashboard SPA, so `GET /collections` returns `200 text/html` even on a build that has no collections code at all. Ask `GET /version` instead: + +```bash +curl -s http://127.0.0.1:7900/version +``` + +```json +{ + "version": "0.4.0", + "commit": "76f72ffef139a9cc08c76d7348b9b25849c845a6", + "commit_source": "git", + "built_at": "2026-07-21T11:38:52Z", + "built_at_source": "install", + "capabilities": [ + "a2a.v1", "collections.v1", "grants.v1", "graph.v1", + "ingest.v1", "search.v1", "shelves.v1", "tasks.v1", "temporal.v1" + ] +} +``` + +`capabilities` is a list of **stable contract identifiers**, not feature names. The `.vN` suffix is the contract: when a wire contract changes in a way that breaks existing callers, the identifier becomes `collections.v2`, so a client pinned to `collections.v1` sees the capability disappear (a visible break it can act on) instead of `collections` quietly meaning something new. Additive changes keep the same identifier. The right client check is membership: + +```python +caps = set(requests.get(f"{base}/version").json()["capabilities"]) +if "collections.v1" not in caps: + raise RuntimeError("this taOSmd build does not speak collections.v1") +``` + +The list is derived at runtime by probing the running build (see `taosmd/capabilities.py`), so it cannot advertise a feature whose code is absent. `commit` and `built_at` are best-effort and may be `null` (see the table below); they are there so an operator can spot a box running a stale build. `/version` is unauthenticated and cheap, like `/health`, and exposes nothing beyond build identity and capability identifiers (no paths, no tokens, no configuration). + +`GET /health` returns the same `capabilities` list alongside its existing `{"status", "version"}` keys, which are unchanged. + +| Field | Meaning | +|-------|---------| +| `commit` | 40-char sha of the build, or `null` | +| `commit_source` | `"git"` (resolved from the checkout), `"build-stamp"` (packaged `taosmd/_build_info.py`), or `null` | +| `built_at` | ISO 8601 UTC build or install time, or `null` | +| `built_at_source` | `"build-stamp"`, `"install"` (dist-info mtime), or `null` | + ### Agent-to-agent (A2A) bus `taosmd serve` also exposes a lightweight message bus for agent-to-agent communication on the same port: diff --git a/docs/collections.md b/docs/collections.md index b665ca9d..0a52cce3 100644 --- a/docs/collections.md +++ b/docs/collections.md @@ -111,6 +111,39 @@ Over MCP: `memory_list_collections` lists them; `memory_search` takes a ## HTTP surface +### Check the server actually speaks collections first + +Collections landed after several releases of `taosmd serve`, so an integrator +pointing at an existing deployment cannot assume the routes are there. Do not +check by requesting `/collections` and looking at the status code: the server +answers unknown non-API paths with the dashboard SPA, so `GET /collections` +returns `200 text/html` on a build with no collections code at all. That check +has already sent one integration to the wrong service. + +Ask `GET /version` (public, no token needed) and test capability membership: + +```bash +curl -s http://127.0.0.1:7900/version | jq -r '.capabilities[]' +# a2a.v1 +# collections.v1 +# grants.v1 +# ... +``` + +```python +caps = set(httpx.get(f"{base}/version").json()["capabilities"]) +if "collections.v1" not in caps: + raise RuntimeError("server does not speak collections.v1") +if "grants.v1" not in caps: + raise RuntimeError("server cannot grant collection access") +``` + +`collections.v1` covers list/get/create/index/link/unlink/archive; `grants.v1` +covers the per-agent grant and revoke routes. The identifiers are derived from +the running build, so a server cannot advertise a capability it lacks, and a +breaking change to the collections wire contract will appear as +`collections.v2` rather than silently redefining `collections.v1`. + Data plane (bearer token when one is configured): ```text diff --git a/docs/serve-service.md b/docs/serve-service.md index 2c1b03a5..666d3c4b 100644 --- a/docs/serve-service.md +++ b/docs/serve-service.md @@ -170,8 +170,9 @@ the same machine can reach the API. Authentication is **off by default**, but a bearer token is available: when `server_token` is set in the server config (via `taosmd config set-token`) or the `TAOSMD_TOKEN` environment variable is set, every data and A2A endpoint requires an `Authorization: Bearer ` -header and returns `401` otherwise. The `/health`, `/`, and `/ui` endpoints -always stay public so monitoring probes and the browser dashboard keep working. +header and returns `401` otherwise. The `/health`, `/version`, `/`, and `/ui` +endpoints always stay public so monitoring probes, capability probes, and the +browser dashboard keep working. On a trusted private network (a home LAN, a Tailscale network), the network boundary is typically sufficient as the access control. For any public-facing or @@ -188,7 +189,23 @@ whether you run `taosmd serve` in the foreground or as a background service. ```bash curl http://127.0.0.1:7900/health -# Expected: {"status": "ok", "version": "..."} +# Expected: {"status": "ok", "version": "...", "capabilities": [...]} ``` +To check which build is running and what it actually supports (useful for +spotting a box left on a stale build): + +```bash +curl http://127.0.0.1:7900/version +# {"version": "...", "commit": "...", "commit_source": "git", +# "built_at": "...", "built_at_source": "install", +# "capabilities": ["a2a.v1", "collections.v1", ...]} +``` + +`capabilities` holds stable contract identifiers derived from what the running +build implements. Test membership (`"collections.v1" in capabilities`) rather +than probing a route's status code: unknown non-API paths serve the dashboard +SPA, so a `200` there says nothing about what the server supports. See the +"Version and capability discovery" section of the README for the `.vN` contract. + The read-only inspection UI is at `http://127.0.0.1:7900/` in your browser. diff --git a/taosmd/capabilities.py b/taosmd/capabilities.py new file mode 100644 index 00000000..50b4d29b --- /dev/null +++ b/taosmd/capabilities.py @@ -0,0 +1,390 @@ +"""Build identity and capability advertisement for ``GET /version``. + +Why this module exists +---------------------- +A taOSmd server answers unknown non-API paths with the dashboard SPA, so a +``GET /collections`` against a build with no collections code returns +``200 text/html``. An integrator checking "did the route exist?" by status code +therefore gets a confident yes from a server that cannot do the thing. And a +semver alone cannot answer "does this box actually speak collections", because +features land continuously between version bumps and a production box can sit a +month stale without anyone noticing. + +So the server publishes a **capability list**: stable contract identifiers a +consumer can test membership against. + +The naming contract +------------------- +Every identifier is ``.v``, for example ``collections.v1``. The +suffix is the whole point: when the collections wire contract changes in a way +that breaks existing callers, the advertised identifier becomes +``collections.v2``. A consumer pinned to ``collections.v1`` sees the capability +disappear (a visible, actionable break) instead of ``collections`` silently +meaning something new. Additive, backwards-compatible changes keep the same +identifier. A build may advertise several versions of one contract at once +during a migration window. + +Keeping the list honest +----------------------- +The list is never a hand-maintained constant of feature names. Each identifier +is declared next to a *probe*: the module and the symbols that implement it, +plus the route markers that expose it over HTTP. A capability is advertised +only if its probe resolves on the running build, so deleting or renaming the +implementation deletes the claim rather than leaving a stale boast. The route +markers are asserted against the real dispatcher by +``tests/test_version_capabilities.py``, so a declaration cannot drift away from +the surface it describes. +""" + +from __future__ import annotations + +import functools +import importlib +import os +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path + +__all__ = [ + "CAPABILITY_PROBES", + "all_probes", + "build_info", + "capabilities", + "probe_for", + "reset_caches", + "resolve_commit", + "version_payload", +] + + +@dataclass(frozen=True) +class CapabilityProbe: + """A contract identifier bound to the code that must exist for it to be true. + + ``module``/``symbols`` : imported and attribute-checked at advertisement + time. All symbols must be present. + ``route_markers`` : substrings that must appear in the HTTP + dispatcher for this capability to be reachable. + Asserted by the divergence test, not at runtime. + """ + + name: str + module: str + symbols: tuple[str, ...] + route_markers: tuple[str, ...] + + +# The capability table. Each entry sits immediately next to the symbols that +# make it true; nothing here is advertised without those symbols resolving. +# Adding a capability means adding its probe, and the divergence test in +# tests/test_version_capabilities.py fails if the routes named here are not in +# the dispatcher. When a contract breaks, add a new `.vN+1` entry (and keep or +# drop the old one depending on whether the build still serves it). +CAPABILITY_PROBES: tuple[CapabilityProbe, ...] = ( + CapabilityProbe( + name="a2a.v1", + module="taosmd.service", + symbols=("a2a_send", "a2a_feed", "a2a_channels", "a2a_members"), + route_markers=( + '"/a2a/send"', + '"/a2a/messages"', + '"/a2a/stream"', + '"/a2a/channels"', + '"/a2a/members"', + ), + ), + CapabilityProbe( + name="collections.v1", + module="taosmd.service", + symbols=( + "collections_create", + "collections_list", + "collections_get", + "collections_index_start", + "collections_link", + "collections_unlink", + "collections_archive", + ), + route_markers=('"/collections"', '"/collections/"', '"/index"', '"/link"'), + ), + CapabilityProbe( + name="grants.v1", + module="taosmd.service", + symbols=("collections_grant", "collections_revoke"), + route_markers=('"/grants"', '"/grants/"'), + ), + CapabilityProbe( + name="graph.v1", + module="taosmd.service", + symbols=("graph", "graph_activations"), + route_markers=('"/graph"', '"/graph/activations"'), + ), + CapabilityProbe( + name="ingest.v1", + module="taosmd.service", + symbols=("ingest", "ingest_batch"), + route_markers=('"/ingest"', '"/ingest/batch"'), + ), + CapabilityProbe( + name="search.v1", + module="taosmd.service", + symbols=("search",), + route_markers=('"/search"',), + ), + CapabilityProbe( + name="shelves.v1", + module="taosmd.service", + symbols=( + "list_shelves", + "admin_shelf_create", + "admin_shelf_archive", + "admin_shelf_unarchive", + ), + route_markers=('"/shelves"', '"/shelves/"'), + ), + CapabilityProbe( + name="tasks.v1", + module="taosmd.service", + symbols=( + "task_create", + "task_list", + "task_ready", + "task_prime", + "task_update", + "task_add_edge", + "task_remove_edge", + ), + route_markers=('"/tasks"', '"/tasks/ready"', '"/tasks/prime"'), + ), + CapabilityProbe( + # Time-travel over the temporal KG: ?as_of= on GET /graph, plus the + # temporal parsing/filtering stage behind search. + name="temporal.v1", + module="taosmd.temporal", + symbols=( + "parse_temporal_expression", + "extract_temporal_expression", + "apply_temporal_stage", + ), + route_markers=('"as_of"', '"/graph"'), + ), +) + + +def all_probes() -> tuple[CapabilityProbe, ...]: + """Every declared capability probe, in table order.""" + return CAPABILITY_PROBES + + +def probe_for(name: str) -> CapabilityProbe: + """Return the probe declaring ``name``; raises ``KeyError`` if undeclared.""" + for probe in CAPABILITY_PROBES: + if probe.name == name: + return probe + raise KeyError(name) + + +def _resolves(probe: CapabilityProbe) -> bool: + """True when the running build really implements ``probe``.""" + try: + module = importlib.import_module(probe.module) + except Exception: # noqa: BLE001 - a missing/broken feature is just absent + return False + return all(hasattr(module, symbol) for symbol in probe.symbols) + + +@functools.lru_cache(maxsize=1) +def _resolved_capabilities() -> tuple[str, ...]: + return tuple(sorted(probe.name for probe in CAPABILITY_PROBES if _resolves(probe))) + + +def capabilities() -> list[str]: + """Contract identifiers this build actually implements, sorted. + + The probe result is cached (a public endpoint should not re-import on every + request) but a fresh list is handed out each call so no caller can mutate + the cached answer. Tests that mutate the probed modules call + :func:`reset_caches`. + """ + return list(_resolved_capabilities()) + + +def reset_caches() -> None: + """Drop the cached capability and build-identity answers (tests only).""" + _resolved_capabilities.cache_clear() + _resolved_build_info.cache_clear() + + +# --------------------------------------------------------------------------- +# build identity: commit + build/install time +# --------------------------------------------------------------------------- +# +# Resolved once, at import, and cached. Two hard rules: +# 1. never shell out. `git rev-parse` in a request path can block on a lock, +# a slow filesystem, or a missing binary, and a monitoring endpoint must +# not be able to hang. The git plumbing we need (HEAD -> ref -> sha) is +# plain file reads, so we read the files directly. +# 2. never raise. A build with no resolvable commit reports null; it does not +# turn /version into a 500. + + +def _build_stamp() -> dict | None: + """Optional build-time stamp written by a packaging step. + + A wheel or container build may drop a ``taosmd/_build_info.py`` exporting + ``COMMIT`` and/or ``BUILT_AT`` (ISO 8601). It is absent from a plain source + checkout, which is why the git reader below exists. + """ + try: + module = importlib.import_module("taosmd._build_info") + except Exception: # noqa: BLE001 - unstamped build is the normal case + return None + commit = getattr(module, "COMMIT", None) + built_at = getattr(module, "BUILT_AT", None) + if not isinstance(commit, str) or not commit.strip(): + commit = None + if not isinstance(built_at, str) or not built_at.strip(): + built_at = None + if commit is None and built_at is None: + return None + return {"commit": commit, "built_at": built_at} + + +def _git_dir(package_dir: Path) -> Path | None: + """Locate the ``.git`` directory for a source checkout, or None. + + Handles the worktree/submodule form where ``.git`` is a file containing + ``gitdir: `` rather than a directory. + """ + dot_git = package_dir.parent / ".git" + if dot_git.is_dir(): + return dot_git + if dot_git.is_file(): + text = dot_git.read_text(encoding="utf-8", errors="replace").strip() + if text.startswith("gitdir:"): + target = Path(text[len("gitdir:") :].strip()) + if not target.is_absolute(): + target = (dot_git.parent / target).resolve() + if target.is_dir(): + return target + return None + + +def _read_ref(git_dir: Path, ref: str) -> str | None: + """Resolve a ref name to a sha via loose ref, then packed-refs.""" + search_dirs = [git_dir] + # A linked worktree keeps refs in the main repo, named by commondir. + common = git_dir / "commondir" + if common.is_file(): + rel = common.read_text(encoding="utf-8", errors="replace").strip() + if rel: + resolved = (git_dir / rel).resolve() + if resolved.is_dir(): + search_dirs.append(resolved) + + for base in search_dirs: + loose = base / ref + if loose.is_file(): + value = loose.read_text(encoding="utf-8", errors="replace").strip() + if _looks_like_sha(value): + return value + packed = base / "packed-refs" + if packed.is_file(): + for line in packed.read_text(encoding="utf-8", errors="replace").splitlines(): + line = line.strip() + if not line or line.startswith(("#", "^")): + continue + sha, _, name = line.partition(" ") + if name.strip() == ref and _looks_like_sha(sha): + return sha + return None + + +def _looks_like_sha(value: str) -> bool: + return len(value) == 40 and all(c in "0123456789abcdef" for c in value.lower()) + + +def resolve_commit(package_dir: Path) -> tuple[str | None, str | None]: + """Best-effort ``(commit, source)`` for the build rooted at ``package_dir``. + + ``source`` is ``"build-stamp"``, ``"git"``, or None when unresolvable. + Never raises and never spawns a process. + """ + try: + stamp = _build_stamp() + if stamp and stamp.get("commit"): + return stamp["commit"], "build-stamp" + + git_dir = _git_dir(Path(package_dir)) + if git_dir is None: + return None, None + head_file = git_dir / "HEAD" + if not head_file.is_file(): + return None, None + head = head_file.read_text(encoding="utf-8", errors="replace").strip() + if head.startswith("ref:"): + sha = _read_ref(git_dir, head[len("ref:") :].strip()) + return (sha, "git") if sha else (None, None) + if _looks_like_sha(head): + return head, "git" + except Exception: # noqa: BLE001 - identity is best-effort, never fatal + return None, None + return None, None + + +def _resolve_built_at(package_dir: Path) -> tuple[str | None, str | None]: + """Best-effort ``(timestamp, source)``: build stamp, else install date.""" + try: + stamp = _build_stamp() + if stamp and stamp.get("built_at"): + return stamp["built_at"], "build-stamp" + + # Installed distributions: the dist-info directory's mtime is when this + # copy was installed, which is what an operator chasing a stale box + # actually wants to know. + from importlib import metadata # noqa: PLC0415 - optional, import-time only + + dist = metadata.distribution("taosmd") + dist_path = getattr(dist, "_path", None) + candidate = Path(dist_path) if dist_path else None + if candidate is None or not candidate.exists(): + candidate = Path(package_dir) + stamped = datetime.fromtimestamp(os.stat(candidate).st_mtime, tz=timezone.utc) + return stamped.strftime("%Y-%m-%dT%H:%M:%SZ"), "install" + except Exception: # noqa: BLE001 - best-effort + return None, None + + +@functools.lru_cache(maxsize=1) +def _resolved_build_info() -> tuple[tuple[str, str | None], ...]: + package_dir = Path(__file__).resolve().parent + commit, commit_source = resolve_commit(package_dir) + built_at, built_at_source = _resolve_built_at(package_dir) + return ( + ("commit", commit), + ("commit_source", commit_source), + ("built_at", built_at), + ("built_at_source", built_at_source), + ) + + +def build_info() -> dict: + """Build identity: commit, its source, build/install time, its source. + + Resolved once on first call (server startup) so no request ever pays for the + filesystem work; a fresh dict is returned each call so callers cannot mutate + the cached answer. + """ + return dict(_resolved_build_info()) + + +def version_payload() -> dict: + """The full ``GET /version`` body. + + Deliberately narrow: build identity and capability identifiers only. No + paths, no tokens, no configuration, because this endpoint is unauthenticated + by design so monitoring and drift probes keep working on a token-secured box. + """ + from . import __version__ # noqa: PLC0415 - avoid a circular import at module load + + return {"version": __version__, **build_info(), "capabilities": capabilities()} diff --git a/taosmd/http_server.py b/taosmd/http_server.py index a979dee3..dafee945 100644 --- a/taosmd/http_server.py +++ b/taosmd/http_server.py @@ -62,7 +62,22 @@ --------- ``GET /`` -> the read-only inspection UI (``text/html``) ``GET /ui`` -> alias of ``GET /`` -``GET /health`` -> ``{"status": "ok", "version": }`` +``GET /health`` -> ``{"status": "ok", "version": , "capabilities": []}`` + ``status`` and ``version`` are the long-standing contract + (taOS and the dashboard consume both); ``capabilities`` is + additive, the same list ``GET /version`` returns. +``GET /version`` -> ``{"version", "commit", "commit_source", "built_at", + "built_at_source", "capabilities"}`` + Public (like ``/health``), cheap, cacheable. ``capabilities`` + is a list of stable contract identifiers (``collections.v1``, + ``grants.v1``, ``temporal.v1``, ``a2a.v1``, ``tasks.v1``, ...), + derived by probing the running build so it cannot claim a + feature the build lacks. A breaking change to a contract + becomes ``.v2``, never a silent redefinition of ``.v1``. + Use it instead of status-code probing: unknown non-API paths + serve the dashboard SPA, so a 200 proves nothing. + ``commit``/``built_at`` are best-effort and may be ``null``. + See :mod:`taosmd.capabilities`. ``POST /ingest`` ``{"text", "agent", "project"?}`` -> ingest result (does not accept per-turn user metadata; use ``/ingest/batch`` for ``forget_after`` and other metadata). @@ -156,7 +171,7 @@ from pathlib import Path from urllib.parse import parse_qs, urlsplit -from . import __version__, config as _config, service +from . import __version__, capabilities, config as _config, service # --------------------------------------------------------------------------- # Static webui helpers @@ -565,8 +580,9 @@ def _make_handler(data_dir, runner: _ServiceLoop, verifier=None, If the server has ``server_token`` set in its own config (or the ``TAOSMD_TOKEN`` env var), every data/A2A JSON endpoint requires a matching ``Authorization: Bearer `` header and returns ``401`` - otherwise. ``GET /health``, ``GET /``, ``GET /ui``, and static assets - are always open so monitoring probes and the inspection UI keep working. + otherwise. ``GET /health``, ``GET /version``, ``GET /``, ``GET /ui``, and + static assets are always open so monitoring probes, capability/drift probes, + and the inspection UI keep working. """ # Read the server-side expected token once at handler-class creation time. # This is the token the *server* checks (not the client's outbound token). @@ -605,7 +621,7 @@ def _make_handler(data_dir, runner: _ServiceLoop, verifier=None, ) # Paths that are always public regardless of the token setting. - _PUBLIC_PATHS = frozenset({"/", "/ui", "/health"}) + _PUBLIC_PATHS = frozenset({"/", "/ui", "/health", "/version"}) class TaosmdHandler(BaseHTTPRequestHandler): server_version = f"taosmd/{__version__}" @@ -881,7 +897,15 @@ def _dispatch(self, method: str) -> None: else: self._send_json(404, {"error": "dashboard disabled (managed_by=taos)"}) elif method == "GET" and path == "/health": - self._send_json(200, {"status": "ok", "version": __version__}) + # "status" and "version" are the existing contract (taOS and + # the dashboard consume both); "capabilities" is additive. + self._send_json(200, { + "status": "ok", + "version": __version__, + "capabilities": capabilities.capabilities(), + }) + elif method == "GET" and path == "/version": + self._send_json(200, capabilities.version_payload()) elif method == "GET" and path == "/controls": self._handle_controls_get() elif method == "POST" and path == "/controls": @@ -2027,7 +2051,7 @@ def serve(host: str = DEFAULT_HOST, port: int = DEFAULT_PORT, data_dir=None) -> where = "localhost only" if bound_host in {"127.0.0.1", "::1"} else "LAN-reachable (no auth)" print(f"taosmd HTTP API listening on http://{bound_host}:{bound_port} ({where})") print(f"Inspection UI (read-only): http://{bound_host}:{bound_port}/") - print("Endpoints: GET /health, POST /ingest, POST /ingest/batch, GET|POST /search, " + print("Endpoints: GET /health, GET /version, POST /ingest, POST /ingest/batch, GET|POST /search, " "GET /projects, GET /shelves, " "GET /pending, POST /pending/resolve, " "POST /a2a/send, GET /a2a/messages, GET /a2a/stream, " diff --git a/tests/test_version_capabilities.py b/tests/test_version_capabilities.py new file mode 100644 index 00000000..dca11ab4 --- /dev/null +++ b/tests/test_version_capabilities.py @@ -0,0 +1,388 @@ +"""Tests for the /version endpoint and the capability contract (taosmd.capabilities). + +Why this exists: a taOSmd server answers unknown non-API paths with the +dashboard SPA (200 text/html), so "the route returned 200" proves nothing about +what a build supports. /version publishes stable contract identifiers so a +consumer can ask "does this box actually speak collections" and get a real +answer, and the tests below exist to make sure the answer cannot be a lie. +""" + +from __future__ import annotations + +import json +import threading +import urllib.error +import urllib.request +from pathlib import Path + +import pytest + +from taosmd import __version__, capabilities +from taosmd import api as taosmd_api +from taosmd import http_server + + +def _patch_embedder(stores: dict) -> None: + """Deterministic 8-dim hash embedder so no ONNX/QMD model is needed.""" + vmem = stores["vector"] + + async def _fake_embed(text: str, task: str = "search_document") -> list[float]: + h = hash(text) & 0xFFFFFFFF + return [((h >> (i * 4)) & 0xFF) / 255.0 for i in range(8)] + + vmem.embed = _fake_embed # type: ignore[assignment] + + +def _get(url: str) -> tuple[int, dict]: + req = urllib.request.Request(url, method="GET") + try: + with urllib.request.urlopen(req, timeout=10) as resp: + return resp.status, json.loads(resp.read().decode()) + except urllib.error.HTTPError as exc: + return exc.code, json.loads(exc.read().decode()) + + +@pytest.fixture +def live_server(tmp_path, monkeypatch): + """Base URL of a running token-less test server.""" + data_dir = tmp_path / "taosmd-data" + data_dir.mkdir() + monkeypatch.setattr(taosmd_api, "_stores_cache", {}) + + httpd = http_server.make_server("127.0.0.1", 0, data_dir=str(data_dir)) + stores = httpd.service_loop.run(taosmd_api._ensure_stores(str(data_dir))) + _patch_embedder(stores) + + host, port = httpd.server_address[:2] + thread = threading.Thread(target=httpd.serve_forever, daemon=True) + thread.start() + try: + yield f"http://{host}:{port}" + finally: + httpd.shutdown() + httpd.server_close() + thread.join(timeout=5) + for s in list(taosmd_api._stores_cache.values()): + for store in (s.get("archive"), s.get("vector"), s.get("kg")): + if store and hasattr(store, "close"): + try: + httpd.service_loop.run(store.close()) + except Exception: + pass + httpd.service_loop.close() + + +# --------------------------------------------------------------------------- +# module-level capability derivation +# --------------------------------------------------------------------------- + + +def test_capabilities_are_contract_identifiers_with_version_suffix(): + """Every advertised capability is `.v`, never a bare feature name. + + The suffix is the contract: a breaking change to collections must surface + as `collections.v2`, a visible break consumers can pin against, rather than + `collections` quietly meaning something new. + """ + caps = capabilities.capabilities() + assert caps, "expected a non-empty capability list" + assert caps == sorted(caps), "capability list must be stably sorted" + assert len(caps) == len(set(caps)), "capability list must not repeat entries" + for cap in caps: + name, _, ver = cap.rpartition(".") + assert name, f"{cap!r} has no contract name" + assert ver.startswith("v") and ver[1:].isdigit(), f"{cap!r} lacks a .vN suffix" + + +def test_capabilities_include_collections_and_grants_on_this_build(): + """This build ships collections, so it must say so.""" + caps = capabilities.capabilities() + assert "collections.v1" in caps + assert "grants.v1" in caps + assert "a2a.v1" in caps + assert "tasks.v1" in caps + assert "temporal.v1" in caps + + +def test_capability_is_dropped_when_its_backing_symbol_is_missing(monkeypatch): + """A capability cannot be advertised by a build that lacks the code. + + This is the anti-drift property: the list is derived by probing the real + implementation, so deleting the code deletes the claim. + """ + from taosmd import service + + probe = capabilities.probe_for("collections.v1") + victim = probe.symbols[0] + assert hasattr(service, victim) + monkeypatch.delattr(service, victim) + capabilities.reset_caches() + + caps = capabilities.capabilities() + assert "collections.v1" not in caps + # Unrelated capabilities are unaffected. + assert "a2a.v1" in caps + + capabilities.reset_caches() + + +def test_capability_declarations_do_not_diverge_from_the_http_surface(): + """Each declared capability's routes must actually exist in the dispatcher. + + The declaration table sits next to nothing but this test; if someone adds a + capability without wiring its routes (or renames a route out from under a + capability) this fails. + """ + source = Path(http_server.__file__).read_text(encoding="utf-8") + dispatch = source[source.index("def _make_handler") :] + for probe in capabilities.all_probes(): + assert probe.route_markers, f"{probe.name} declares no route markers" + for marker in probe.route_markers: + assert marker in dispatch, ( + f"capability {probe.name} claims marker {marker!r} but it is " + "absent from the HTTP dispatcher" + ) + + +def test_every_declared_capability_probe_resolves_on_this_build(): + """No declared capability is unresolvable here (catches typo'd symbols).""" + declared = {p.name for p in capabilities.all_probes()} + assert set(capabilities.capabilities()) == declared + + +# --------------------------------------------------------------------------- +# commit / built_at resolution +# --------------------------------------------------------------------------- + + +def test_commit_is_null_when_the_package_is_not_a_git_checkout(tmp_path): + """A pip install with no build stamp reports commit: null, never an error.""" + pkg = tmp_path / "site-packages" / "taosmd" + pkg.mkdir(parents=True) + commit, source = capabilities.resolve_commit(pkg) + assert commit is None + assert source is None + + +def test_commit_resolves_from_a_git_checkout(tmp_path): + """A checkout with .git/HEAD pointing at a branch resolves that branch's sha.""" + root = tmp_path / "repo" + pkg = root / "taosmd" + pkg.mkdir(parents=True) + git = root / ".git" + (git / "refs" / "heads").mkdir(parents=True) + (git / "HEAD").write_text("ref: refs/heads/master\n") + sha = "0123456789abcdef0123456789abcdef01234567" + (git / "refs" / "heads" / "master").write_text(sha + "\n") + + commit, source = capabilities.resolve_commit(pkg) + assert commit == sha + assert source == "git" + + +def test_commit_resolves_from_packed_refs(tmp_path): + """Branch refs that have been packed still resolve (no loose ref file).""" + root = tmp_path / "repo" + pkg = root / "taosmd" + pkg.mkdir(parents=True) + git = root / ".git" + git.mkdir() + (git / "HEAD").write_text("ref: refs/heads/master\n") + sha = "89abcdef0123456789abcdef0123456789abcdef" + (git / "packed-refs").write_text( + "# pack-refs with: peeled fully-peeled sorted\n" + f"{sha} refs/heads/master\n" + ) + + commit, source = capabilities.resolve_commit(pkg) + assert commit == sha + + +def test_commit_resolves_detached_head(tmp_path): + root = tmp_path / "repo" + pkg = root / "taosmd" + pkg.mkdir(parents=True) + git = root / ".git" + git.mkdir() + sha = "abcdef0123456789abcdef0123456789abcdef01" + (git / "HEAD").write_text(sha + "\n") + + commit, source = capabilities.resolve_commit(pkg) + assert commit == sha + + +def test_commit_resolution_never_raises_on_a_corrupt_git_dir(tmp_path): + """Garbage in .git degrades to null rather than breaking the endpoint.""" + root = tmp_path / "repo" + pkg = root / "taosmd" + pkg.mkdir(parents=True) + git = root / ".git" + git.mkdir() + (git / "HEAD").write_text("ref: refs/heads/gone\n") + + commit, source = capabilities.resolve_commit(pkg) + assert commit is None + assert source is None + + +def test_commit_prefers_a_build_stamp_over_the_checkout(tmp_path, monkeypatch): + """A wheel built with a stamp reports the stamped sha.""" + monkeypatch.setattr( + capabilities, "_build_stamp", lambda: {"commit": "f" * 40, "built_at": None} + ) + commit, source = capabilities.resolve_commit(Path(tmp_path)) + assert commit == "f" * 40 + assert source == "build-stamp" + + +def test_build_info_is_cached_and_shaped(monkeypatch): + """build_info() is resolved once at import and returns the documented keys.""" + info = capabilities.build_info() + assert set(info) == {"commit", "commit_source", "built_at", "built_at_source"} + assert info["commit"] is None or ( + isinstance(info["commit"], str) and len(info["commit"]) == 40 + ) + assert info["commit_source"] in (None, "git", "build-stamp") + assert info["built_at"] is None or info["built_at"].endswith("Z") + # Cached but copy-on-read: a caller cannot poison the cached answer. + info["commit"] = "tampered" + assert capabilities.build_info()["commit"] != "tampered" + + +def test_capabilities_are_copy_on_read(): + """Mutating the returned list cannot poison the cached capability answer.""" + caps = capabilities.capabilities() + caps.append("fake.v9") + assert "fake.v9" not in capabilities.capabilities() + + +# --------------------------------------------------------------------------- +# HTTP surface +# --------------------------------------------------------------------------- + + +def test_version_endpoint_shape(live_server): + status, body = _get(f"{live_server}/version") + assert status == 200, body + assert set(body) == { + "version", + "commit", + "commit_source", + "built_at", + "built_at_source", + "capabilities", + } + assert body["version"] == __version__ + assert isinstance(body["capabilities"], list) + assert all(isinstance(c, str) for c in body["capabilities"]) + assert "collections.v1" in body["capabilities"] + assert body["commit"] is None or isinstance(body["commit"], str) + assert body["built_at"] is None or isinstance(body["built_at"], str) + + +def test_version_matches_the_module_derivation(live_server): + """The endpoint must not maintain its own idea of the capability list.""" + _, body = _get(f"{live_server}/version") + assert body["capabilities"] == capabilities.capabilities() + + +def test_health_keeps_its_existing_contract(live_server): + """taOS and the dashboard already consume status+version; ADD only. + + This assertion is deliberately explicit so a future change cannot silently + break the existing consumers. + """ + status, body = _get(f"{live_server}/health") + assert status == 200 + assert body["status"] == "ok" + assert isinstance(body["version"], str) and body["version"] + assert body["version"] == __version__ + + +def test_health_gains_the_capability_list(live_server): + _, body = _get(f"{live_server}/health") + assert body["capabilities"] == capabilities.capabilities() + + +def test_version_and_health_leak_nothing_sensitive(live_server): + """No paths, tokens, or config values beyond version + capabilities.""" + allowed_version = { + "version", + "commit", + "commit_source", + "built_at", + "built_at_source", + "capabilities", + } + _, version_body = _get(f"{live_server}/version") + _, health_body = _get(f"{live_server}/health") + assert set(version_body) <= allowed_version + assert set(health_body) <= {"status", "version", "capabilities"} + + for body in (version_body, health_body): + blob = json.dumps(body) + assert "/" not in blob.replace("\\/", ""), f"looks like a path leaked: {blob}" + for banned in ("token", "data_dir", "home", "secret", "password"): + assert banned not in blob.lower(), f"{banned!r} leaked in {blob}" + + +# --------------------------------------------------------------------------- +# public by design, even with a server token configured +# --------------------------------------------------------------------------- + + +@pytest.fixture +def token_server(tmp_path, monkeypatch): + """Server with a server_token configured; yields (base_url, token).""" + data_dir = tmp_path / "token-data" + data_dir.mkdir() + token = "test-bearer-token-version" + (data_dir / "config.json").write_text(json.dumps({"server_token": token})) + + monkeypatch.setattr(taosmd_api, "_stores_cache", {}) + httpd = http_server.make_server("127.0.0.1", 0, data_dir=str(data_dir)) + stores = httpd.service_loop.run(taosmd_api._ensure_stores(str(data_dir))) + _patch_embedder(stores) + + host, port = httpd.server_address[:2] + thread = threading.Thread(target=httpd.serve_forever, daemon=True) + thread.start() + try: + yield f"http://{host}:{port}", token + finally: + httpd.shutdown() + httpd.server_close() + thread.join(timeout=5) + httpd.service_loop.close() + + +def _raw_get(url: str) -> tuple[int, dict]: + req = urllib.request.Request(url, method="GET") + try: + with urllib.request.urlopen(req, timeout=5) as resp: + return resp.status, json.loads(resp.read().decode()) + except urllib.error.HTTPError as exc: + return exc.code, json.loads(exc.read().decode()) + + +def test_version_is_public_when_a_server_token_is_configured(token_server): + base_url, _ = token_server + status, body = _raw_get(f"{base_url}/version") + assert status == 200, body + assert body["version"] == __version__ + assert "collections.v1" in body["capabilities"] + + +def test_health_stays_public_when_a_server_token_is_configured(token_server): + base_url, _ = token_server + status, body = _raw_get(f"{base_url}/health") + assert status == 200 + assert body["status"] == "ok" + assert "capabilities" in body + + +def test_data_plane_is_still_gated_alongside_the_public_version(token_server): + """Sanity: making /version public did not open the data plane.""" + base_url, _ = token_server + status, _ = _raw_get(f"{base_url}/projects") + assert status == 401