feat(tests): add production-shape local harness (Phase 1)

tests/harness/README.md

@@ -0,0 +1,110 @@
+# Production-shape local harness
+
+The harness brings up the SaaS tenant topology on localhost using the
+same `Dockerfile.tenant` image that ships to production. Tests run
+against `http://harness-tenant.localhost:8080` and exercise the
+SAME code path a real tenant takes — including TenantGuard middleware,
+the `/cp/*` reverse proxy, the canvas reverse proxy, and a
+Cloudflare-tunnel-shape header rewrite layer.
+
+## Why this exists
+
+Local `go run ./cmd/server` skips:
+- `TenantGuard` middleware (no `MOLECULE_ORG_ID` env)
+- `/cp/*` reverse proxy mount (no `CP_UPSTREAM_URL` env)
+- `CANVAS_PROXY_URL` (canvas runs separately on `:3000`)
+- Header rewrites that production's CF tunnel + LB perform
+- Strict-auth mode (no live `ADMIN_TOKEN`)
+
+Bugs that survive `go run` and ship to production almost always live
+in one of those layers. The harness activates ALL of them.
+
+## Topology
+
+```
+client
+  ↓
+cf-proxy        nginx, mirrors CF tunnel header rewrites
+  ↓ (Host:harness-tenant.localhost, X-Forwarded-*)
+tenant          workspace-server/Dockerfile.tenant — same image as prod
+  ↓ (CP_UPSTREAM_URL=http://cp-stub:9090, /cp/* proxied)
+cp-stub         minimal Go service, mocks CP wire surface
+postgres        same version as production
+redis           same version as production
+```
+
+## Quickstart
+
+```bash
+cd tests/harness
+./up.sh                 # builds + starts all services
+./seed.sh               # mints admin token, registers two sample workspaces
+./replays/peer-discovery-404.sh
+./replays/buildinfo-stale-image.sh
+./down.sh               # tear down + remove volumes
+```
+
+First-time setup needs an `/etc/hosts` entry so `harness-tenant.localhost`
+resolves to the local cf-proxy:
+
+```bash
+echo "127.0.0.1 harness-tenant.localhost" | sudo tee -a /etc/hosts
+```
+
+(macOS resolves `*.localhost` automatically in some setups; Linux
+typically does not.)
+
+## Replay scripts
+
+Each replay script reproduces a real bug class against the harness so
+fixes can be verified locally before deploy. The bar for adding a
+replay is "this bug shipped to production despite local E2E being
+green" — the script becomes the regression gate that closes that gap.
+
+| Replay | Closes | What it proves |
+|--------|--------|----------------|
+| `peer-discovery-404.sh` | #2397 | tool_list_peers surfaces the actual reason instead of "may be isolated" |
+| `buildinfo-stale-image.sh` | #2395 | GIT_SHA reaches the binary; verify-step comparison logic works |
+
+To add a new replay:
+1. Drop a script under `replays/` named after the issue.
+2. The script's purpose: reproduce the production failure mode against
+   the harness, then assert the fix is present. PASS criterion is the
+   post-fix behavior.
+3. Wire it into the `tests/harness/run-all-replays.sh` runner (TODO,
+   Phase 2).
+
+## Extending the cp-stub
+
+`cp-stub/main.go` serves the minimum surface for the existing replays
+plus a catch-all that returns 501 + a clear message when the tenant
+asks for a route the stub doesn't implement. To add a new CP route:
+
+1. Add a `mux.HandleFunc` in `cp-stub/main.go` for the path.
+2. Return the same wire shape the real CP returns. The contract is
+   "wire compatibility with the staging CP at the time of writing" —
+   document it with a comment pointing at the real CP handler.
+3. Add a replay script that exercises the path.
+
+## What the harness does NOT cover
+
+- Real TLS / cert handling (CF terminates TLS in production; harness is
+  HTTP-only).
+- Cloudflare API edge cases (rate limits, DNS propagation timing).
+- Real EC2 / SSM / EBS behavior (image-cache replay simulates the
+  outcome but not the AWS API surface).
+- Cross-region or multi-AZ topology.
+- Real production data scale.
+
+These are intentional Phase 1 limits. If a bug class hits one of these
+gaps, escalate to staging E2E rather than expanding the harness past
+its mandate of "exercise the tenant binary in production-shape topology."
+
+## Roadmap
+
+- **Phase 1 (this PR):** harness + cp-stub + cf-proxy + 2 replays.
+- **Phase 2:** convert `tests/e2e/test_api.sh` to run against the
+  harness instead of localhost. Make harness-based E2E a required CI
+  check.
+- **Phase 3:** config-coherence lint that diffs harness env list
+  against production CP's env list, fails CI on drift.

tests/harness/cf-proxy/nginx.conf

@@ -0,0 +1,68 @@
+# cf-proxy — Cloudflare-tunnel-shape reverse proxy for the local harness.
+#
+# Production path: agent → CF tunnel → AWS LB → tenant container.
+# This config replays the same header rewrites the CF tunnel does so
+# the tenant sees the same Host + X-Forwarded-* it would in production.
+#
+# The tenant's TenantGuard middleware activates on MOLECULE_ORG_ID; the
+# canvas's same-origin fetches use the Host header for cookie scoping.
+# Both behave correctly in production because CF rewrites Host to the
+# tenant subdomain — this proxy reproduces that locally.
+#
+# How tests reach it:
+#   curl --resolve 'harness-tenant.localhost:8443:127.0.0.1' \
+#        https://harness-tenant.localhost:8443/health
+# or via /etc/hosts (added automatically by ./up.sh on first boot).
+
+worker_processes 1;
+events { worker_connections 256; }
+
+http {
+    # Map the wildcard <slug>.localhost to the tenant container. The
+    # tenant container itself doesn't care which slug routed to it —
+    # what matters is that the Host header it sees matches what
+    # production's CF tunnel sets, so cookie/CORS/TenantGuard logic
+    # exercises the same code path.
+    server {
+        listen 8080;
+        server_name *.localhost localhost;
+
+        # Cap upload at 50MB to mirror the staging tenant nginx limit;
+        # chat upload tests will fail closed if the platform handler
+        # ever silently expands its limit (catches the failure mode
+        # opposite of the chat-files lazy-heal incident).
+        client_max_body_size 50m;
+
+        location / {
+            proxy_pass http://tenant:8080;
+
+            # Header parity with CF tunnel + AWS LB. Production CF sets
+            # X-Forwarded-Proto=https; we keep http here because TLS
+            # termination in compose is unnecessary for testing the
+            # tenant logic — TLS is a CF concern, not a tenant bug
+            # surface. If TLS-specific bugs ever bite, add cert-manager
+            # + listen 8443 ssl here.
+            proxy_set_header Host              $host;
+            proxy_set_header X-Real-IP         $remote_addr;
+            proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Host  $host;
+            proxy_set_header X-Forwarded-Proto $scheme;
+
+            # Streamable HTTP / SSE / WebSocket — the tenant exposes /ws
+            # and /events/stream + MCP /mcp/stream. Disabling buffering
+            # reproduces CF tunnel's pass-through streaming semantics
+            # (CF tunnel = no buffering by default; nginx default IS
+            # buffering, which would mask issue #2397-class streaming
+            # bugs by accumulating output until the client disconnects).
+            proxy_buffering         off;
+            proxy_request_buffering off;
+            proxy_http_version      1.1;
+            proxy_set_header        Connection "";
+
+            # Read timeout — CF tunnel default is 100s. Setting this to
+            # the same value catches "long agent run finishes after the
+            # proxy already closed the upstream" failure mode.
+            proxy_read_timeout      100s;
+        }
+    }
+}

tests/harness/compose.yml

@@ -0,0 +1,132 @@
+# Production-shape harness for local E2E.
+#
+# Reproduces the SaaS tenant topology on localhost using the SAME
+# images that ship to production:
+#
+#   client → cf-proxy (nginx, mimics CF tunnel headers)
+#          → tenant (workspace-server/Dockerfile.tenant — combined platform + canvas)
+#          → cp-stub (control-plane stand-in) for /cp/* and CP-callback paths
+#          → postgres + redis (same versions as production)
+#
+# Why this matters: the workspace-server binary IS identical between
+# local and production. The bugs that survive local E2E are topology
+# bugs — env-gated middleware (TenantGuard, CP proxy, Canvas proxy),
+# auth state, header rewrites, real production image. This harness
+# activates ALL of them.
+#
+# Quickstart:
+#   cd tests/harness && ./up.sh
+#   ./seed.sh
+#   ./replays/peer-discovery-404.sh   # reproduces issue #2397
+#
+# Env config:
+#   GIT_SHA — passed to the tenant build for /buildinfo verification.
+#             Defaults to "harness" so /buildinfo distinguishes the
+#             harness build from any cached image.
+#   CP_STUB_PEERS_MODE — peers failure mode for replay scripts.
+#                       "" / "404" / "401" / "500" / "timeout".
+
+services:
+  postgres:
+    image: postgres:16-alpine
+    environment:
+      POSTGRES_USER: harness
+      POSTGRES_PASSWORD: harness
+      POSTGRES_DB: molecule
+    networks: [harness-net]
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U harness"]
+      interval: 2s
+      timeout: 5s
+      retries: 10
+
+  redis:
+    image: redis:7-alpine
+    networks: [harness-net]
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 2s
+      timeout: 5s
+      retries: 10
+
+  cp-stub:
+    build:
+      context: ./cp-stub
+    environment:
+      PORT: "9090"
+      CP_STUB_PEERS_MODE: "${CP_STUB_PEERS_MODE:-}"
+    networks: [harness-net]
+    healthcheck:
+      test: ["CMD-SHELL", "wget -q -O- http://localhost:9090/healthz || exit 1"]
+      interval: 2s
+      timeout: 5s
+      retries: 10
+
+  # The actual production tenant image — same Dockerfile.tenant CI publishes.
+  # This is the load-bearing part of the harness: every bug class that hides
+  # behind "but it works locally" is reproducible HERE, against this image,
+  # not against `go run ./cmd/server`.
+  tenant:
+    build:
+      context: ../..
+      dockerfile: workspace-server/Dockerfile.tenant
+      args:
+        GIT_SHA: "${GIT_SHA:-harness}"
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+      cp-stub:
+        condition: service_healthy
+    environment:
+      DATABASE_URL: "postgres://harness:harness@postgres:5432/molecule?sslmode=disable"
+      REDIS_URL: "redis://redis:6379"
+      PORT: "8080"
+      PLATFORM_URL: "http://tenant:8080"
+      MOLECULE_ENV: "production"
+      # ADMIN_TOKEN flips the platform into strict-auth mode (matches
+      # production's CP-minted token configuration). Seeded value lets
+      # E2E scripts authenticate without going through CP.
+      ADMIN_TOKEN: "harness-admin-token"
+      # MOLECULE_ORG_ID — activates TenantGuard middleware. Every request
+      # must carry X-Molecule-Org-Id matching this value. Replays bugs
+      # that only fire in SaaS mode.
+      MOLECULE_ORG_ID: "harness-org"
+      # CP_UPSTREAM_URL — activates the /cp/* reverse proxy mount in
+      # router.go. Without this set, /cp/* would 404 and the canvas
+      # bootstrap would silently drift from production behavior.
+      CP_UPSTREAM_URL: "http://cp-stub:9090"
+      RATE_LIMIT: "1000"
+      # Canvas auto-proxy — entrypoint-tenant.sh exports CANVAS_PROXY_URL
+      # by default; keeping it explicit here makes the topology readable.
+      CANVAS_PROXY_URL: "http://localhost:3000"
+    networks: [harness-net]
+    healthcheck:
+      test: ["CMD-SHELL", "wget -q -O- http://localhost:8080/health || exit 1"]
+      interval: 5s
+      timeout: 5s
+      retries: 20
+
+  # Cloudflare-tunnel-shape proxy — strips the :8080 suffix, rewrites
+  # Host to the tenant subdomain, injects X-Forwarded-*. Tests target
+  # http://harness-tenant.localhost:8080 and exercise the production
+  # routing layer.
+  cf-proxy:
+    image: nginx:1.27-alpine
+    depends_on:
+      tenant:
+        condition: service_healthy
+    volumes:
+      - ./cf-proxy/nginx.conf:/etc/nginx/nginx.conf:ro
+    # Bind to 127.0.0.1 only — the harness uses a hardcoded ADMIN_TOKEN
+    # ("harness-admin-token") so binding 0.0.0.0 (compose's default)
+    # would expose admin access to anyone on the local network or VPN.
+    # Loopback-only is safe for E2E and prevents a known-token leak.
+    ports:
+      - "127.0.0.1:8080:8080"
+    networks: [harness-net]
+
+networks:
+  harness-net:
+    name: molecule-harness-net

tests/harness/cp-stub/Dockerfile

@@ -0,0 +1,14 @@
+# cp-stub — minimal CP stand-in for the local production-shape harness.
+# See main.go for the rationale. Self-contained build, no module deps.
+
+FROM golang:1.25-alpine AS builder
+WORKDIR /src
+COPY go.mod ./
+COPY main.go ./
+RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /cp-stub .
+
+FROM alpine:3.20
+RUN apk add --no-cache ca-certificates
+COPY --from=builder /cp-stub /cp-stub
+EXPOSE 9090
+ENTRYPOINT ["/cp-stub"]

tests/harness/cp-stub/go.mod

@@ -0,0 +1,3 @@
+module github.com/Molecule-AI/molecule-monorepo/tests/harness/cp-stub
+
+go 1.25