feat(tests): add production-shape local harness (Phase 1)

[HongmingWang-Rabbit]

Summary

Phase 1 of the production-shape local harness — addresses the topology gap that lets bugs like #2395 (silent stale image) and #2397 (silent peer-discovery failure) ship to production despite green local E2E.

The harness brings up the SaaS tenant topology on localhost using the SAME workspace-server/Dockerfile.tenant image that ships to production:

client
  ↓
cf-proxy   nginx, mirrors CF tunnel + LB header rewrites
  ↓
tenant     Dockerfile.tenant — same image as prod
  ↓
cp-stub    minimal Go CP stand-in for /cp/* paths
postgres + redis

Tests run against http://harness-tenant.localhost:8080. They exercise:

• TenantGuard middleware (MOLECULE_ORG_ID set)
• /cp/* reverse proxy (CP_UPSTREAM_URL=http://cp-stub:9090)
• Canvas reverse proxy (CANVAS_PROXY_URL from entrypoint-tenant.sh)
• Strict-auth mode (live ADMIN_TOKEN)
• CF-tunnel-shape Host rewrite + X-Forwarded-* injection

All env-gated middleware and request-shape rewrites that local go run skips.

How

• cp-stub/main.go: minimal CP stand-in. Serves /cp/auth/me, /cp/admin/tenants/redeploy-fleet, plus /__stub/{peers,mode,state} toggles for replay scripts. Catch-all /cp/* returns 501 with a clear hint when a new CP route appears so drift surfaces fast.
• cf-proxy/nginx.conf: rewrites Host to <slug>.localhost, injects X-Forwarded-*, disables buffering to mirror CF tunnel streaming semantics.
• compose.yml: one service per topology layer; tenant builds from workspace-server/Dockerfile.tenant.
• up.sh / down.sh / seed.sh: lifecycle scripts.
• replays/peer-discovery-404.sh: reproduces list_peers + get_peers swallow non-200 + exceptions, leaving agent + user blind to root cause #2397 + asserts the diagnostic helper from PR feat(workspace): surface peer-discovery failure reason instead of "may be isolated" #2399 surfaces "404" + "registered".
• replays/buildinfo-stale-image.sh: reproduces redeploy-fleet + tenant provision serve stale workspace-server images even with fresh :latest #2395 + asserts /buildinfo wire shape and GIT_SHA injection from PR feat(deploy): verify each tenant /buildinfo matches published SHA after redeploy #2398.

Bar for adding replays

Each replay script reproduces a real bug class against the harness so fixes can be verified locally before deploy. The bar is "this bug shipped to production despite local E2E being green" — the script becomes the regression gate that closes the gap.

Verification

• cp-stub builds (go build ./...).
• cp-stub responds to all stubbed endpoints (smoke-tested locally — see commit message).
• compose.yml passes docker compose config --quiet.
• All shell scripts pass bash -n syntax check.

Out of scope (Phase 2-3, separate PRs)

• Phase 2: Convert tests/e2e/test_api.sh to target the harness URL instead of localhost. Make harness-based replays a required CI gate.
• Phase 3: Config-coherence lint that diffs harness env list against production CP's env list, fails CI on drift.

Test plan

• cd tests/harness && ./up.sh && ./seed.sh — bring up harness, seed sample workspaces.
• ./replays/buildinfo-stale-image.sh — proves /buildinfo shape works post feat(deploy): verify each tenant /buildinfo matches published SHA after redeploy #2398.
• ./replays/peer-discovery-404.sh — proves diagnostic surfacing works post feat(workspace): surface peer-discovery failure reason instead of "may be isolated" #2399.
• ./down.sh — clean up.

🤖 Generated with Claude Code

[HongmingWang-Rabbit]

Five-axis pass. Substantive infra addition (production-shape harness, 11 new files, +772/-0) → comment-only per loop policy.

Correctness. The harness wires up the exact prod topology with minimum-viable stand-ins:

• tenant builds from workspace-server/Dockerfile.tenant — same image CI publishes, not go run ./cmd/server. This is the load-bearing claim and it lands cleanly via build-context ../.. + args: GIT_SHA.
• cp-stub's catch-all returns 501 with an actionable hint when a new /cp/* path appears — drift surfaces fast instead of silently 404'ing the test.
• cf-proxy nginx mirrors CF tunnel semantics (Host rewrite, X-Forwarded-* injection, buffering disabled for streaming).
• __stub/{peers,mode,state} toggles let replay scripts seed scenarios deterministically; atomic.Value storage keeps the stub race-free under concurrent test traffic.
• Replay scripts each cite a specific shipped bug (#2395, #2397) — the regression bar is "this shipped to prod despite green local E2E."

Readability. Top-of-file docstrings explain why this exists, not just what. The cp-stub/main.go header makes the role explicit ("plays that role on localhost so we can exercise the SAME code path"); compose.yml calls out the load-bearing image choice; replays are named by bug class. README is provided. Each service comment is tied to a concrete production behavior (TenantGuard activation, strict-auth mode, CF tunnel header rewrite).

Architecture. Right scope split:

• Phase 1 (this PR) — bring up the topology + two replay scripts
• Phase 2 — convert tests/e2e/test_api.sh to target the harness URL + make replays a required CI gate
• Phase 3 — config-coherence lint that diffs harness env vs prod CP env

Aligns with the harness-pair pattern from RFC #2251 (same-named scripts on each side of the OSS/SaaS boundary, each owning the surface it tests). Mirrors feedback_chase_verification_to_staging discipline at the local-CI layer: the gap that lets bugs ship is exactly the gap this closes.

Security.

• ADMIN_TOKEN: "harness-admin-token" — hardcoded local test token, not a real credential. No leaked-secret regex match (sk-, gh_, AKIA, sk-ant-, sk-cp- — none).
• POSTGRES_PASSWORD: harness — local docker-compose convention, no real DB exposed.
• cp-stub is bound to the harness-net docker network only — nothing exposed beyond compose.
• No production code modified.
• No .github/workflows/* changes (harness will become a CI gate in Phase 2 — separate PR).
• No deleted tests; this is pure test infrastructure addition.

Performance. Test infra only; no prod hot-path impact. Healthcheck intervals 2s with 10-retry budget is reasonable for compose boot ordering. cp-stub's atomic.Int64 for redeployFleetCalls and atomic.Value for state are cheap.

One nit, not blocking: CP_STUB_PEERS_MODE: "timeout" documented as "hang for 60s" — make sure the replay's per-call timeout is long enough to actually hit the 60s vs. just getting cut off by a default httpx 10s timeout (the actual scenario from a2a_client.py uses timeout=10.0). The diagnostic helper from #2399 will return network-exception with that timeout, which is the correct branch for the replay assertion — so this is fine, but worth confirming the replay docs match.

LGTM. CI is minimal (this is pure test infra) — once green, mergeable.

[HongmingWang-Rabbit]

Five-axis pass with two findings — one Critical, several Optional.

Critical: harness won't boot as-written

compose.yml sets MOLECULE_ENV: "production" but doesn't set SECRETS_ENCRYPTION_KEY. Tracing the boot path:

1. crypto.InitStrict() — aes.go:114-118: isProdEnv() returns true for "prod" or "production".
2. aes.go:76-80: when isProdEnv() && !IsEnabled(), returns ErrEncryptionKeyMissing with message "SECRETS_ENCRYPTION_KEY is required in production...".
3. cmd/server/main.go:60-62: log.Fatalf("Secrets encryption: %v", err) → process exits non-zero → container exits → up.sh --wait blocks until the healthcheck retries exhaust.

The entrypoint-tenant.sh doesn't paper over this (verified — no SECRETS_ENCRYPTION_KEY handling there). The container has no baked-in .env (Dockerfile doesn't COPY one and .env is gitignored).

This squares with the PR's verification list — the four checked items are syntax/build only (go build, docker compose config --quiet, bash -n); the four Test-Plan boxes that actually exercise up.sh are unchecked. The harness as written would fail at up.sh --wait.

Two ways to fix:

• (a) Add SECRETS_ENCRYPTION_KEY to compose.yml env. Use a non-secret-shaped fixed value (e.g. a known-test 32-byte hex like 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef) and document in the README why it's safe (loopback-only bind + no real secrets ever encrypted with it). This preserves the prod-shape aspiration.
• (b) Set MOLECULE_ENV: "harness" instead of "production". Trips isProdEnv() == false, the strict gate is bypassed, encryption disabled with a warning. Caveat: this is the least-prod-shape choice; per the PR's own thesis ("local go run skips ... strict-auth mode"), bypassing the strict path defeats some of the harness's purpose — but ADMIN_TOKEN strict-auth is independent of encryption gating, so the canvas-bootstrap / TenantGuard / CP-proxy tests would still fire.

I'd lean (a) — it keeps the harness behaving identically to prod for everything except the encrypted-secrets surface (which the replays don't currently exercise anyway). Add a follow-up replay if/when secrets-encrypted code paths need coverage.

Other findings (non-blocking)

Optional / Consider — env var name inconsistency in the buildinfo replay:

• compose.yml:67 reads ${GIT_SHA:-harness}
• replays/buildinfo-stale-image.sh:36 reads ${HARNESS_GIT_SHA:-harness}
• These should be the same name. The current shape produces a false-positive WARN ("Image may be cached...") whenever an operator overrides GIT_SHA because the replay is reading a different env var.

Optional — seed.sh creates ALPHA_ID + BETA_ID but neither replay actually consumes them. peer-discovery-404.sh uses a fresh ROGUE_ID for the unregistered case. Either drop the seed.sh invocation from the replay, or add a success-path replay that exercises tool_list_peers against the seeded pair.

Optional — seed.sh:37 POSTs \"id\":\"$ALPHA_ID\" to /workspaces. The platform's typical create handler generates the ID server-side; if it ignores the client value, .seed.env would persist a bogus ID. Worth a one-line check: grep "id.*=.*payload" workspace_crud.go to confirm the field is honored.

FYI — cp-stub/ has its own go.mod. Top-level go test ./... won't catch syntax errors in cp-stub. The Dockerfile build will, but only when compose runs. Phase 2 should add a make ci-harness target so cp-stub stays buildable.

FYI — the loopback-bind comment in compose.yml:115-119 is right next to the port, which is the best location for it. A future contributor changing 127.0.0.1:8080:8080 to 8080:8080 would silently expose harness-admin-token to the LAN/VPN. Optional follow-up: add a one-line shell test in CI that greps compose.yml for ^[^#]*ports: blocks and asserts 127.0.0.1: prefix on the admin-bound services.

Other axes

Readability ✓ — README + per-file commentary are exemplary; every WHY is captured. Replay scripts have step-numbered phases.
Architecture ✓ — clean module boundaries (cp-stub isolated go.mod, nginx config in its own dir, service-per-layer compose). Reuse of the actual Dockerfile.tenant is the right call.
Security ✓ — hardcoded admin token is intentional and bound to loopback. No real-data exposure surface.
Performance ✓ — no concerns; healthcheck cadence reasonable.

Verdict

Request changes — the Critical boot bug needs to land before this is useful. Once that's fixed and you've actually run the four Test-Plan steps end-to-end (so the boxes turn checked), this is a strong addition to the test infrastructure.