Skip to content

feat(mcp): gateway preflight + KVM-primary topology in the runbook #2806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
POWERFULMOVES merged 6 commits into from
Aug 28, 2026
Merged

feat(mcp): gateway preflight + KVM-primary topology in the runbook #2806

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
eea0673
feat(mcp): gateway-start now checks the resolver before serving 401s
POWERFULMOVES Aug 28, 2026
ef9f555
docs(mcp): KVM gateway is primary, Docker Desktop is the fallback
POWERFULMOVES Aug 28, 2026
2631efa
feat(bootstrap): claude-pmoves now tells the session whether it HAS m…
POWERFULMOVES Aug 28, 2026
3f7b4a9
fix(mcp): preflight ok was the resolver alone; listener ran bare python3
POWERFULMOVES Aug 28, 2026
8a26e33
refactor(mcp): one at-risk report writer, not two identical loops
POWERFULMOVES Aug 28, 2026
97ec4e6
Merge branch 'main' into feat/mcp-gateway-preflight-and-kvm-runbook
POWERFULMOVES Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 173 additions & 0 deletions pmoves/docs/operations/MCP_TOOLKIT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,67 @@ This is the operational guide for running the **Docker MCP Toolkit** across ever

---

## 0. Topology — the KVM gateway is primary, Docker Desktop is the fallback

**Refreshed 2026-08-28.** Sections 2-4 below still describe the per-node Docker
Desktop bootstrap as though it were the whole story. It is not. It is the
FALLBACK, and it is the one that breaks.

| Layer | Role | Status |
|---|---|---|
| **KVM-hosted gateway** | **PRIMARY.** One self-hosted endpoint every node reaches over the tailnet, so a node needs no local Toolkit to have MCP. | **NOT YET DEPLOYED** — verified 2026-08-27, no MCP server runs on any KVM VPS (25+ ports probed across kvm2, kvm4-1, kvm4-2). See `pmoves/docs/handoffs/infra_mcp_hosting_analysis_2026-08-27.md`. |
| **Docker Desktop Toolkit** | **FALLBACK / workstation dev.** Per-node, per-operator, needs a GUI. | Live on the workstation nodes. This is what sections 2-4 document. |
| **Direct CLI + MCP** | Hostinger, Cloudflare and Tailscale are reachable directly, with their own CLIs and MCP servers. | Live. Does not depend on either gateway. |

**Why KVM primary.** The Toolkit gateway is a workstation product: it needs
Docker Desktop, a logged-in desktop session, and a per-node secret keychain that
is a per-node point of failure (see § 5.1 — it wedged on the 4090 on 2026-08-28
and survived a full Desktop restart). A KVM-hosted gateway is reachable by every
node including the arm64 and headless ones, is provisioned by the same funnel
that provisions everything else, and does not require anyone to be logged into a
GUI for an agent to have tools.

**Placement, from the 2026-08-27 capacity probe** (do not re-derive; re-measure
before acting on it):

- **kvm4-1 — best fit for the fleet-facing gateway.** 8C/16GB with roughly 6GB
usable headroom, already the API/agent tier, egress-separated.
- **kvm2 — right host for light public SSE surfaces** behind its nginx; it has
the most free RAM (~7.5GB).
- **kvm4-2 — add nothing.** Over-subscribed: ~29GB of declared container limits
on a 16GB host. Fix that before it hosts anything else.

**Which gateway to clone.** The substrate decision is already made: the **Docker
MCP Gateway** on port **8189** with `--static`, auth through the CHIT secrets
pipeline, per #2656 / #2665 / #2681, with a verified 23-tool bring-up. The BoTZ
MCP gateway (`:8052`) draft is **superseded for federation** — it is K8s-only and
cannot federate existing servers. New gateway work clones #2665's pipeline.

**Prerequisite that is easy to miss:** `up-mcp-gateway` depends on
`mcp-gateway-preflight`, which fails closed unless the `pmoves_pmoves_app`
network exists AND `pmoves-botz-mcp-bridge` is running — the bridge is the only
catalogued server, and the network is created by the PMOVES-BoTZ project, not
this repo. Run `make -C pmoves up-botz-mcp-bridge` FIRST. `pmoves/mk/infra.mk`
records why this is called out: a previous verification passed on B850 only
because that stack happened to already be up.

### 0.1 Profile naming is drifting from the topology

`docker.io/darkxside/pmoves_5090_web:latest` is the artifact every node
bootstraps to — `scripts/mcp-toolkit-connect.sh:31` and
`scripts/mcp-toolkit-bootstrap.sh:14-15` both default to it. **5090 is a node.**
So every node in the fleet runs a profile named after one workstation, and on
the 4090 that means the node's own `pmoves_4090_web` (24 servers) sits unused
while it serves the 5090's (25 servers, and the only one carrying
`github-official`).

It works — the artifact really is the shared bundle — but the name says
otherwise, and a fleet-wide bundle belongs with the fleet-wide gateway on a KVM,
not under a workstation's name. Renaming is an operator decision; this section
exists so the next reader does not mistake the current name for the intent.

---

## 1. What Docker MCP Toolkit is, in PMOVES vocabulary

| Toolkit term | PMOVES analog | What it actually is |
Expand Down Expand Up @@ -152,6 +213,118 @@ Cloudflare's 13 OAuth-mediated servers need a one-time browser-mediated `docker

---

### 5.1 Recovery — when the secret resolver wedges

**The symptom.** Everything else works and only secrets fail:

```
$ docker mcp secret ls
deadline_exceeded: Post "http://unix/resolver.v1.ResolverService/GetSecrets": net/http: timeout awaiting response headers

$ docker mcp profile ls # OK
$ docker mcp client ls # OK
$ docker mcp catalog ls # OK
$ docker mcp feature ls # OK
```

Four of five subsystems answer; only the resolver is down. Docker itself is
healthy — this is not a Docker outage.

**Why it matters more than it looks.** A server whose secret cannot be fetched
starts anyway, with an EMPTY credential env var. It then fails at call time with
a 401, in whatever tool called it, with nothing pointing back here. On
2026-08-28 that presented as "the GitHub token is missing" when the token was in
fact present in every place the funnel routes it.

**Recorded occurrences.** 2026-08-17 on the 5090 (fresh `mcp-toolkit.db`, stuck
`.mcp-toolkit-migration.lock`, after a Docker Desktop VMM/backend migration) and
2026-08-28 on the 4090. The 4090 case **survived a full Docker Desktop
restart**, so a restart alone is not the fix.

**Check for it before it bites:**

```bash
python pmoves/tools/mcp_toolkit_preflight.py --profile <id>
# 0 ready — resolver answers AND every secret the profile requires is in
# the store
# 1 measured a problem — the resolver is down, OR a required secret is
# absent from the store. Either way the affected servers WILL start
# unauthenticated; the report names which servers and which secret NAMES
# 3 could not measure — no docker, no Toolkit, no such profile, no
# interpreter that can run the preflight, or a secret store that could
# not be enumerated. NOT a pass
```

Readiness is the conjunction on purpose. An earlier cut set the verdict from
`resolver_healthy()` alone while already holding, unread, the list of secrets
the profile requires — so a live resolver over a store where a secret had never
been hydrated (or had since been wiped) reported ready, strict startup and CI
passed, and the server still 401'd at call time. Same for the enumeration
itself: a `docker mcp secret ls` that cannot be read is exit 3, never an empty
list of problems.

`make -C pmoves mcp-toolkit-gateway-start` runs this automatically and reports
non-zero as advisory. Set `PMOVES_MCP_STRICT=1` to refuse to start instead —
which is what CI and unattended bring-up should do. Under strict that now also
covers "the preflight could not be run at all" (no interpreter, tool absent),
which used to warn and launch the gateway unchecked.

**Interpreter selection.** The listener picks its Python through
`pmoves/scripts/pm-python.sh` — the canonical discovery shared with
`claude-pmoves.sh` and `crush-pmoves` — probing each candidate for PyYAML,
which the preflight imports. Order: `$PMOVES_PYTHON`, then
`pmoves/.venv-pmoves`, then `python3` / `py -3` / `python`. It previously called
bare `python3`, which exits 127 on nodes where Python is `py -3` or lives only
in the canonical venv, and fails on hosts whose system Python lacks PyYAML even
though that venv has it. Pin explicitly when a node needs it:

```bash
PMOVES_PYTHON="py -3" make -C pmoves mcp-toolkit-gateway-start
```

A node with no equipped interpreter reports exit **3** (could not measure), not
exit 1 — "I could not run the check" and "the check found a problem" are
different answers and no longer share a bucket.

**Recovery, in order:**

1. Restart Docker Desktop. Re-check with `docker mcp secret ls`. If it answers,
go to step 4.
2. If it still wedges, the migration state is stuck: stop Docker Desktop and
clear the stale `mcp-toolkit.db` / `.mcp-toolkit-migration.lock` under the
Docker MCP config directory, then start it again. **Operator step** — that
directory holds credentials and is out of scope for agents.
3. Re-check. Do not proceed while `secret ls` still times out — every write in
step 4 goes through the same resolver.
4. Re-provision without re-typing anything:
```bash
make -C pmoves docker-mcp-secrets-hydrate DRY_RUN=1 # preview
make -C pmoves docker-mcp-secrets-hydrate # push
```
Values come from the funnel, so nothing is rotated and nothing is entered by
hand. Add `PROFILE=<id>` to force a profile; otherwise it is discovered from
`.mcp.json`, then `PMOVES_MCP_PROFILE_ID`.
5. Re-run the preflight. Expect exit 0.

**What hydrate does NOT cover:** the 13 Cloudflare servers are OAuth-mediated,
not API-key. They need a one-time interactive `docker mcp oauth authorize
<server>` per node and will still show unauthenticated after any hydrate. That
is expected, not a regression.

**If a secret is reported as a funnel gap** rather than pushed, the key exists in
the map but has no value on this node. `GITHUB_PAT` materializes into
`env.tier-agent`, and the hydrator reads the `env.shared` aggregate, so a stale
aggregate can hide a populated key — point it at the tier file to tell the two
apart:

```bash
python pmoves/tools/docker_mcp_secrets_hydrate.py --dry-run --env-shared pmoves/env.tier-agent
```

Only a value missing from BOTH needs `secrets-rotate`.

---

## 6. E2B + sandboxed agents

**Architecture decision (2026-05-20): D-Proxy.** Rather than installing the full Toolkit inside every sandbox, the canonical pattern is:
Expand Down
38 changes: 38 additions & 0 deletions pmoves/scripts/claude-pmoves.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,44 @@ elif [ -f "$IDENT_TOOL" ]; then
echo "[claude-pmoves] node identity: no usable python found (tried .venv-pmoves, python3, py -3, python — yaml required); launching without it." >&2
fi

# CIPHER — persistent memory. Same reasoning as the identity block above: the
# agent has to be TOLD, in context, whether it has memory. An MCP server that
# never connects contributes no tools, so a session with no memory looks exactly
# like a session with memory and nothing to recall. That is the silent failure
# this check exists to end.
#
# It reports WHICH endpoint answered, not merely that one did. The roster
# carries a fleet cipher (${TS_Z890}) and a local one, and #2792 exists because
# it once carried only the fleet entry -- "memory that silently wasn't there"
# whenever Z890 was unreachable. "Memory is up" must not quietly mean "someone
# else's memory is up".
#
# NEVER blocks: a session without memory is degraded, not unusable, and refusing
# to launch would be worse than launching informed.
CIPHER_TOOL="$ROOT/pmoves/tools/cipher_preflight.py"
if [ -f "$CIPHER_TOOL" ] && [ ${#IDENT_PY[@]} -gt 0 ]; then
CIPHER_OUT=""
set +e
CIPHER_OUT="$("${IDENT_PY[@]}" "$CIPHER_TOOL" 2>&1)"
cipher_rc=$?
set -e
case "$cipher_rc" in
0)
CIPHER_WHICH="$(printf '%s\n' "$CIPHER_OUT" | awk '/^cipher OK/ {print $3; exit}')"
echo "[claude-pmoves] cipher=up (${CIPHER_WHICH:-unknown endpoint})" >&2
IDENTITY_ARGS+=(--append-system-prompt "Persistent memory IS available this session via the Cipher MCP server '${CIPHER_WHICH:-unknown}'. Use it for recall and for writes; do not fall back to the auto-memory directory while it is up.")
;;
*)
# 1 = every endpoint was reached and none answered. 3 = nothing to measure
# (no cipher entry in the roster at all). Both mean no memory; the agent
# is told which, because the fixes differ.
echo "[claude-pmoves] cipher=DOWN (exit ${cipher_rc}) — session has no persistent memory" >&2
printf '%s\n' "$CIPHER_OUT" >&2
IDENTITY_ARGS+=(--append-system-prompt "Cipher is NOT reachable this session (preflight exit ${cipher_rc}), so you have NO persistent memory. Say so at session start rather than recalling nothing silently, and use the file-based auto-memory directory instead. Recovery: pmoves/docs/operations/MCP_TOOLKIT.md.")
;;
esac
fi

if [ ! -f "$LAUNCHER" ]; then
# Degrade to the pre-delegation behavior rather than failing: the agent still
# loads, MCP creds do not. Warn so the missing half is visible, not silent.
Expand Down
81 changes: 78 additions & 3 deletions pmoves/scripts/mcp-toolkit-gateway-listen.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ Environment overrides:
PMOVES_MCP_GATEWAY_PID (default: /tmp/pmoves-mcp-gateway.pid)
PMOVES_MCP_GATEWAY_LOG (default: /tmp/pmoves-mcp-gateway.log)
PMOVES_MCP_BLOCK_NETWORK (default: 0 — set 1 to add --block-network)
PMOVES_MCP_STRICT (default: 0 — set 1 to refuse to start unless
the preflight returns 0)
PMOVES_PYTHON (operator pin for the preflight interpreter;
space-separated, e.g. "py -3")
MCP_GATEWAY_AUTH_TOKEN (auto-generated and persisted to env.shared
on first run; reused thereafter)

Expand All @@ -62,6 +66,10 @@ PID_FILE="${PMOVES_MCP_GATEWAY_PID:-/tmp/pmoves-mcp-gateway.pid}"
LOG_FILE="${PMOVES_MCP_GATEWAY_LOG:-/tmp/pmoves-mcp-gateway.log}"
BLOCK_NETWORK="${PMOVES_MCP_BLOCK_NETWORK:-0}"

# Resolved once, used by both the preflight and the token persistence below.
PMOVES_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ENV_SHARED="${PMOVES_DIR}/env.shared"

color() { printf '\033[%sm%s\033[0m\n' "$1" "$2"; }
info() { color "1;34" "[mcp-gateway] $*"; }
warn() { color "1;33" "[mcp-gateway] WARN: $*" >&2; }
Expand All @@ -75,12 +83,79 @@ if ! docker mcp profile ls 2>/dev/null | awk 'NR>1 {print $1}' | grep -Fxq "${PR
fail "Profile '${PROFILE_ID}' not imported. Run: make mcp-toolkit-bootstrap"
fi

# Preflight. ADVISORY on purpose -- see tools/mcp_toolkit_preflight.py for why a
# wedged resolver should not refuse to start 25 servers because 4 cannot
# authenticate. It exists because this script used to start a gateway whose
# credentialed servers were guaranteed to 401, and said nothing.
# exit 0 ready
# exit 1 measured a problem (resolver down, or a required secret is absent
# from the store) -- reported, not blocked here
# exit 3 could not measure -- NOT a pass; surfaced so it is not mistaken for
# one. That now includes "no interpreter could run the preflight",
# which used to arrive as a bare 127 and land in the exit-1 bucket
# alongside real findings (Codex P1 on #2806).
# Set PMOVES_MCP_STRICT=1 to refuse to start on anything but 0, which is what
# CI and unattended bring-up should do.
PREFLIGHT_PY="${PMOVES_DIR}/tools/mcp_toolkit_preflight.py"
PM_PYTHON_SH="${PMOVES_DIR}/scripts/pm-python.sh"
preflight_rc=0
preflight_why=""

if [ ! -f "${PREFLIGHT_PY}" ]; then
# A missing gate is an unmeasured gateway, not a clean one. It used to warn
# and start unchecked even under PMOVES_MCP_STRICT=1.
preflight_rc=3
preflight_why="preflight tool missing at ${PREFLIGHT_PY}"
elif [ ! -f "${PM_PYTHON_SH}" ]; then
preflight_rc=3
preflight_why="python discovery missing at ${PM_PYTHON_SH}"
else
# pm-python.sh is the ONE python discovery in this repo (claude-pmoves.sh and
# crush-pmoves both source it). Bare `python3` -- what this line used to be --
# exits 127 on nodes where Python is `py -3` or lives only in .venv-pmoves,
# and a 127 is indistinguishable here from a real preflight failure. The
# `yaml` argument is load-bearing: the preflight imports PyYAML, so a host
# whose system python lacks it is UNEQUIPPED, not failing, and the venv
# branch of pm_pick_python is the one that has it.
# shellcheck source=pmoves/scripts/pm-python.sh
. "${PM_PYTHON_SH}"
if pm_pick_python yaml; then
set +e
"${PM_PY[@]}" "${PREFLIGHT_PY}" --profile "${PROFILE_ID}"
preflight_rc=$?
set -e
if [ "${preflight_rc}" -eq 127 ]; then
# Belt and braces: pm_pick_python already probed this interpreter, so a
# 127 here means it disappeared between probe and use. Still unmeasured.
preflight_rc=3
preflight_why="preflight interpreter vanished between probe and use"
fi
else
preflight_rc=3
preflight_why="no python with PyYAML found (tried \$PMOVES_PYTHON, pmoves/.venv-pmoves, python3, py -3, python) -- run: make -C pmoves preflight, or pin PMOVES_PYTHON"
fi
fi

if [ "${preflight_rc}" -ne 0 ]; then
if [ -z "${preflight_why}" ]; then
case "${preflight_rc}" in
1) preflight_why="measured a problem: resolver down, or a required secret is absent from the store" ;;
3) preflight_why="COULD NOT MEASURE the gateway's readiness" ;;
*) preflight_why="unexpected preflight exit" ;;
esac
fi
if [ "${preflight_rc}" -eq 3 ]; then
preflight_why="${preflight_why} -- this is NOT a pass"
fi
if [ "${PMOVES_MCP_STRICT:-0}" = "1" ]; then
fail "preflight exit ${preflight_rc}: ${preflight_why}. PMOVES_MCP_STRICT=1 -- refusing to start."
fi
warn "preflight exit ${preflight_rc}: ${preflight_why}; starting anyway (set PMOVES_MCP_STRICT=1 to gate)."
fi

# Generate MCP_GATEWAY_AUTH_TOKEN if not already set. Persist to env.shared so
# clients (E2B sandboxes, BoTZ Gateway, this host's secrets-funnel consumers)
# can read the same value. The Toolkit gateway reads the env var directly.
PMOVES_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
ENV_SHARED="${PMOVES_DIR}/env.shared"

if [ -z "${MCP_GATEWAY_AUTH_TOKEN:-}" ]; then
if [ -f "${ENV_SHARED}" ] && grep -q '^MCP_GATEWAY_AUTH_TOKEN=' "${ENV_SHARED}"; then
# shellcheck disable=SC2046
Expand Down
Loading
Loading