feat(mcp): gateway preflight + KVM-primary topology in the runbook

pmoves/docs/operations/MCP_TOOLKIT.md

@@ -6,6 +6,67 @@ This is the operational guide for running the **Docker MCP Toolkit** across ever
 
 ---
 
+## 0. Topology — the KVM gateway is primary, Docker Desktop is the fallback
+
+**Refreshed 2026-08-28.** Sections 2-4 below still describe the per-node Docker
+Desktop bootstrap as though it were the whole story. It is not. It is the
+FALLBACK, and it is the one that breaks.
+
+| Layer | Role | Status |
+|---|---|---|
+| **KVM-hosted gateway** | **PRIMARY.** One self-hosted endpoint every node reaches over the tailnet, so a node needs no local Toolkit to have MCP. | **NOT YET DEPLOYED** — verified 2026-08-27, no MCP server runs on any KVM VPS (25+ ports probed across kvm2, kvm4-1, kvm4-2). See `pmoves/docs/handoffs/infra_mcp_hosting_analysis_2026-08-27.md`. |
+| **Docker Desktop Toolkit** | **FALLBACK / workstation dev.** Per-node, per-operator, needs a GUI. | Live on the workstation nodes. This is what sections 2-4 document. |
+| **Direct CLI + MCP** | Hostinger, Cloudflare and Tailscale are reachable directly, with their own CLIs and MCP servers. | Live. Does not depend on either gateway. |
+
+**Why KVM primary.** The Toolkit gateway is a workstation product: it needs
+Docker Desktop, a logged-in desktop session, and a per-node secret keychain that
+is a per-node point of failure (see § 5.1 — it wedged on the 4090 on 2026-08-28
+and survived a full Desktop restart). A KVM-hosted gateway is reachable by every
+node including the arm64 and headless ones, is provisioned by the same funnel
+that provisions everything else, and does not require anyone to be logged into a
+GUI for an agent to have tools.
+
+**Placement, from the 2026-08-27 capacity probe** (do not re-derive; re-measure
+before acting on it):
+
+- **kvm4-1 — best fit for the fleet-facing gateway.** 8C/16GB with roughly 6GB
+  usable headroom, already the API/agent tier, egress-separated.
+- **kvm2 — right host for light public SSE surfaces** behind its nginx; it has
+  the most free RAM (~7.5GB).
+- **kvm4-2 — add nothing.** Over-subscribed: ~29GB of declared container limits
+  on a 16GB host. Fix that before it hosts anything else.
+
+**Which gateway to clone.** The substrate decision is already made: the **Docker
+MCP Gateway** on port **8189** with `--static`, auth through the CHIT secrets
+pipeline, per #2656 / #2665 / #2681, with a verified 23-tool bring-up. The BoTZ
+MCP gateway (`:8052`) draft is **superseded for federation** — it is K8s-only and
+cannot federate existing servers. New gateway work clones #2665's pipeline.
+
+**Prerequisite that is easy to miss:** `up-mcp-gateway` depends on
+`mcp-gateway-preflight`, which fails closed unless the `pmoves_pmoves_app`
+network exists AND `pmoves-botz-mcp-bridge` is running — the bridge is the only
+catalogued server, and the network is created by the PMOVES-BoTZ project, not
+this repo. Run `make -C pmoves up-botz-mcp-bridge` FIRST. `pmoves/mk/infra.mk`
+records why this is called out: a previous verification passed on B850 only
+because that stack happened to already be up.
+
+### 0.1 Profile naming is drifting from the topology
+
+`docker.io/darkxside/pmoves_5090_web:latest` is the artifact every node
+bootstraps to — `scripts/mcp-toolkit-connect.sh:31` and
+`scripts/mcp-toolkit-bootstrap.sh:14-15` both default to it. **5090 is a node.**
+So every node in the fleet runs a profile named after one workstation, and on
+the 4090 that means the node's own `pmoves_4090_web` (24 servers) sits unused
+while it serves the 5090's (25 servers, and the only one carrying
+`github-official`).
+
+It works — the artifact really is the shared bundle — but the name says
+otherwise, and a fleet-wide bundle belongs with the fleet-wide gateway on a KVM,
+not under a workstation's name. Renaming is an operator decision; this section
+exists so the next reader does not mistake the current name for the intent.
+
+---
+
 ## 1. What Docker MCP Toolkit is, in PMOVES vocabulary
 
 | Toolkit term | PMOVES analog | What it actually is |
@@ -152,6 +213,86 @@ Cloudflare's 13 OAuth-mediated servers need a one-time browser-mediated `docker
 
 ---
 
+### 5.1 Recovery — when the secret resolver wedges
+
+**The symptom.** Everything else works and only secrets fail:
+
+```
+$ docker mcp secret ls
+deadline_exceeded: Post "http://unix/resolver.v1.ResolverService/GetSecrets": net/http: timeout awaiting response headers
+
+$ docker mcp profile ls    # OK
+$ docker mcp client ls     # OK
+$ docker mcp catalog ls    # OK
+$ docker mcp feature ls    # OK
+```
+
+Four of five subsystems answer; only the resolver is down. Docker itself is
+healthy — this is not a Docker outage.
+
+**Why it matters more than it looks.** A server whose secret cannot be fetched
+starts anyway, with an EMPTY credential env var. It then fails at call time with
+a 401, in whatever tool called it, with nothing pointing back here. On
+2026-08-28 that presented as "the GitHub token is missing" when the token was in
+fact present in every place the funnel routes it.
+
+**Recorded occurrences.** 2026-08-17 on the 5090 (fresh `mcp-toolkit.db`, stuck
+`.mcp-toolkit-migration.lock`, after a Docker Desktop VMM/backend migration) and
+2026-08-28 on the 4090. The 4090 case **survived a full Docker Desktop
+restart**, so a restart alone is not the fix.
+
+**Check for it before it bites:**
+
+```bash
+python pmoves/tools/mcp_toolkit_preflight.py --profile <id>
+#   0  ready
+#   1  resolver down — servers WILL start unauthenticated (it names which)
+#   3  could not measure — NOT a pass
+```
+
+`make -C pmoves mcp-toolkit-gateway-start` runs this automatically and reports
+non-zero as advisory. Set `PMOVES_MCP_STRICT=1` to refuse to start instead —
+which is what CI and unattended bring-up should do.
+
+**Recovery, in order:**
+
+1. Restart Docker Desktop. Re-check with `docker mcp secret ls`. If it answers,
+   go to step 4.
+2. If it still wedges, the migration state is stuck: stop Docker Desktop and
+   clear the stale `mcp-toolkit.db` / `.mcp-toolkit-migration.lock` under the
+   Docker MCP config directory, then start it again. **Operator step** — that
+   directory holds credentials and is out of scope for agents.
+3. Re-check. Do not proceed while `secret ls` still times out — every write in
+   step 4 goes through the same resolver.
+4. Re-provision without re-typing anything:
+   ```bash
+   make -C pmoves docker-mcp-secrets-hydrate DRY_RUN=1   # preview
+   make -C pmoves docker-mcp-secrets-hydrate             # push
+   ```
+   Values come from the funnel, so nothing is rotated and nothing is entered by
+   hand. Add `PROFILE=<id>` to force a profile; otherwise it is discovered from
+   `.mcp.json`, then `PMOVES_MCP_PROFILE_ID`.
+5. Re-run the preflight. Expect exit 0.
+
+**What hydrate does NOT cover:** the 13 Cloudflare servers are OAuth-mediated,
+not API-key. They need a one-time interactive `docker mcp oauth authorize
+<server>` per node and will still show unauthenticated after any hydrate. That
+is expected, not a regression.
+
+**If a secret is reported as a funnel gap** rather than pushed, the key exists in
+the map but has no value on this node. `GITHUB_PAT` materializes into
+`env.tier-agent`, and the hydrator reads the `env.shared` aggregate, so a stale
+aggregate can hide a populated key — point it at the tier file to tell the two
+apart:
+
+```bash
+python pmoves/tools/docker_mcp_secrets_hydrate.py --dry-run --env-shared pmoves/env.tier-agent
+```
+
+Only a value missing from BOTH needs `secrets-rotate`.
+
+---
+
 ## 6. E2B + sandboxed agents
 
 **Architecture decision (2026-05-20): D-Proxy.** Rather than installing the full Toolkit inside every sandbox, the canonical pattern is:

pmoves/scripts/mcp-toolkit-gateway-listen.sh

@@ -75,6 +75,31 @@ if ! docker mcp profile ls 2>/dev/null | awk 'NR>1 {print $1}' | grep -Fxq "${PR
   fail "Profile '${PROFILE_ID}' not imported. Run: make mcp-toolkit-bootstrap"
 fi
 
+# Preflight. ADVISORY on purpose -- see tools/mcp_toolkit_preflight.py for why a
+# wedged resolver should not refuse to start 25 servers because 4 cannot
+# authenticate. It exists because this script used to start a gateway whose
+# credentialed servers were guaranteed to 401, and said nothing.
+#   exit 0  ready
+#   exit 1  measured a problem (resolver down) -- reported, not blocked here
+#   exit 3  could not measure -- NOT a pass; surfaced so it is not mistaken for one
+# Set PMOVES_MCP_STRICT=1 to refuse to start on anything but 0, which is what
+# CI and unattended bring-up should do.
+PREFLIGHT_PY="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/tools/mcp_toolkit_preflight.py"
+if [ -f "${PREFLIGHT_PY}" ]; then
+  set +e
+  python3 "${PREFLIGHT_PY}" --profile "${PROFILE_ID}"
+  preflight_rc=$?
+  set -e
+  if [ "${preflight_rc}" -ne 0 ]; then
+    if [ "${PMOVES_MCP_STRICT:-0}" = "1" ]; then
+      fail "preflight exit ${preflight_rc} and PMOVES_MCP_STRICT=1 -- refusing to start."
+    fi
+    warn "preflight exit ${preflight_rc}; starting anyway (set PMOVES_MCP_STRICT=1 to gate)."
+  fi
+else
+  warn "preflight tool missing at ${PREFLIGHT_PY} -- starting unchecked."
+fi
+
 # Generate MCP_GATEWAY_AUTH_TOKEN if not already set. Persist to env.shared so
 # clients (E2B sandboxes, BoTZ Gateway, this host's secrets-funnel consumers)
 # can read the same value. The Toolkit gateway reads the env var directly.

pmoves/tests/tools/test_mcp_toolkit_preflight.py

@@ -0,0 +1,147 @@
+"""Offline tests for the MCP Toolkit gateway preflight.
+
+Docker is stubbed so these run on a runner with no daemon and on any arch; what
+is under test is the verdict logic and — most importantly — that an
+unmeasurable gateway exits 3 rather than 0.
+
+Context: mcp-toolkit-gateway-listen.sh checked Docker and the profile, then
+started a gateway. It never checked the SECRET RESOLVER. On 2026-08-28 the
+resolver was wedged (`ResolverService/GetSecrets: deadline_exceeded`, surviving
+a Docker Desktop restart) and the gateway started regardless, serving
+`github-official` with an empty token. Every call returned 401, and nothing at
+start time explained why.
+
+Stubbing `_docker` rather than PATH follows docker_host_policy_check.py, and is
+required on Windows besides: Git Bash resolves `docker` to `docker.exe` and
+never sees an extensionless stub, so a PATH stub silently exercises the REAL
+daemon — which is exactly how an earlier draft of this file "passed" against
+live profiles instead of its fixture.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+MODULE = REPO_ROOT / "pmoves" / "tools" / "mcp_toolkit_preflight.py"
+LISTENER = REPO_ROOT / "pmoves" / "scripts" / "mcp-toolkit-gateway-listen.sh"
+
+spec = importlib.util.spec_from_file_location("mcp_toolkit_preflight", MODULE)
+assert spec and spec.loader
+pf = importlib.util.module_from_spec(spec)
+sys.modules["mcp_toolkit_preflight"] = pf
+spec.loader.exec_module(pf)
+
+
+PROFILE_LS = "ID\tName\ndefault\tDefault\npmoves_5090_web\tPMOVES-5090-web\n"
+PROFILE_SHOW = """
+version: 1
+id: pmoves_5090_web
+servers:
+    - type: image
+      snapshot:
+        server:
+            name: github-official
+            secrets:
+                - name: github.personal_access_token
+                  env: GITHUB_PERSONAL_ACCESS_TOKEN
+    - type: image
+      snapshot:
+        server:
+            name: context7
+"""
+WEDGED = (
+    'deadline_exceeded: Post "http://unix/resolver.v1.ResolverService/GetSecrets": '
+    "net/http: timeout awaiting response headers"
+)
+
+
+def _fake_docker(monkeypatch, *, secrets, profiles: str = PROFILE_LS):
+    def fake(*args: str, timeout: int = 45) -> str:
+        joined = " ".join(args)
+        if joined.startswith("mcp profile ls"):
+            return profiles
+        if joined.startswith("mcp secret ls"):
+            if secrets is None:
+                raise pf.Unmeasured("docker mcp secret ls failed: resolver down")
+            return secrets
+        if joined.startswith("mcp profile show"):
+            return PROFILE_SHOW
+        return ""
+
+    monkeypatch.setattr(pf, "_docker", fake)
+
+
+def test_a_healthy_gateway_passes(monkeypatch):
+    _fake_docker(monkeypatch, secrets="NAME\ne2b.api_key\n")
+    assert pf.main(["--profile", "pmoves_5090_web"]) == 0
+
+
+def test_a_wedged_resolver_exits_1(monkeypatch):
+    """A measured failure, distinct from an unmeasurable one."""
+    _fake_docker(monkeypatch, secrets=WEDGED)
+    assert pf.main(["--profile", "pmoves_5090_web"]) == 1
+
+
+def test_a_wedged_resolver_names_the_servers_at_risk(monkeypatch, capsys):
+    """A generic warning is not actionable; the operator needs the server list."""
+    _fake_docker(monkeypatch, secrets=WEDGED)
+    pf.main(["--profile", "pmoves_5090_web"])
+    err = capsys.readouterr().err
+    assert "github-official" in err, err
+    assert "github.personal_access_token" in err, err
+
+
+def test_a_server_with_no_secret_is_not_reported(monkeypatch):
+    """context7 declares none, so it is not 'at risk' and must not be listed."""
+    _fake_docker(monkeypatch, secrets=WEDGED)
+    rows = pf.servers_requiring_secrets("pmoves_5090_web")
+    assert [r["server"] for r in rows] == ["github-official"]
+
+
+def test_a_secret_ls_failure_is_a_finding_not_an_absence(monkeypatch):
+    """`secret ls` failing IS the resolver reporting itself down.
+
+    Docker ran; the answer was "broken". Treating that as unmeasurable would
+    downgrade a real finding into a shrug.
+    """
+    _fake_docker(monkeypatch, secrets=None)
+    assert pf.resolver_healthy() is False
+
+
+def test_a_missing_profile_is_unmeasured_not_a_pass(monkeypatch):
+    """Exit 3. The gateway's readiness was never established."""
+    _fake_docker(monkeypatch, secrets="NAME\n", profiles="ID\tName\ndefault\tD\n")
+    assert pf.main(["--profile", "pmoves_5090_web"]) == 3
+
+
+def test_no_docker_at_all_is_unmeasured(monkeypatch):
+    monkeypatch.setattr(pf.shutil, "which", lambda _: None)
+    assert pf.main(["--profile", "pmoves_5090_web"]) == 3
+
+
+def test_json_mode_says_whether_it_measured(monkeypatch, capsys):
+    """Machine-readable callers must be able to tell 3 from 0 without the code."""
+    _fake_docker(monkeypatch, secrets="NAME\n", profiles="ID\tName\ndefault\tD\n")
+    pf.main(["--profile", "pmoves_5090_web", "--json"])
+    import json as _json
+
+    payload = _json.loads(capsys.readouterr().out)
+    assert payload["measured"] is False
+    assert "not imported" in payload["reason"]
+
+
+def test_the_listener_actually_invokes_the_preflight():
+    """Wiring assertion: a gate nothing calls is decoration.
+
+    The listener is what `make -C pmoves mcp-toolkit-gateway-start` runs, so the
+    preflight has to be on that path or it protects nothing.
+    """
+    body = LISTENER.read_text(encoding="utf-8")
+    assert "mcp_toolkit_preflight" in body, (
+        "mcp-toolkit-gateway-listen.sh does not call the preflight"
+    )