ci: gate PRs on tests/harness/run-all-replays.sh

[HongmingWang-Rabbit]

Summary

Phase 2 of the harness roadmap (per tests/harness/README.md): make harness-based replays a required CI check on every PR touching the tenant binary or the harness itself.

Triggers tests/harness/run-all-replays.sh on push + pull_request to staging+main, paths-filtered to workspace-server/**, canvas/**, tests/harness/**, and this workflow.

Why now

2026-04-30 we shipped #2398 which added /buildinfo as a public route in router.go but never added it to TenantGuard's allowlist. The handler-level test in buildinfo_test.go constructs a minimal gin engine without TenantGuard — green. The harness's buildinfo-stale-image.sh replay would have caught it (cf-proxy doesn't inject X-Molecule-Org-Id, so the curl path is identical to production's redeploy verifier). Nobody ran the harness pre-merge.

Result: bug shipped, the redeploy verifier silently soft-warned every tenant as "unreachable" for ~1 day before it was traced (PR #2409 fixes the allowlist).

This gate makes "did you actually run the harness?" a CI invariant instead of memory discipline.

Pipeline

detect-changes (paths filter)
  └ harness-replays (always)
      ├ no-op pass when paths-filter says no relevant change
      └ otherwise:
          ├ checkout PR source
          ├ checkout sibling Molecule-AI/molecule-ai-plugin-github-app-auth (Dockerfile.tenant needs it)
          ├ /etc/hosts entry for harness-tenant.localhost (CI ubuntu doesn't auto-resolve *.localhost)
          ├ run tests/harness/run-all-replays.sh
          ├ dump compose logs (tenant + cp-stub + cf-proxy + postgres) on failure
          └ force teardown (always; belt-and-suspenders against trap-bypass kills)

Single-job-with-conditional-steps pattern (matches e2e-api.yml). One check run regardless of paths-filter outcome — branch-protection-clean per the PR #2264 SKIPPED-in-set finding.

merge_group trigger included so this is branch-protectable when ready.

What runs today

• replays/buildinfo-stale-image.sh — would have caught feat(deploy): verify each tenant /buildinfo matches published SHA after redeploy #2398's TenantGuard gap
• replays/peer-discovery-404.sh — closes list_peers + get_peers swallow non-200 + exceptions, leaving agent + user blind to root cause #2397

Both run automatically. Adding a new replay needs no per-replay registration — run-all-replays.sh picks up every replays/*.sh.

Build cost

The tenant image is the full Dockerfile.tenant (Go + Next.js + template clones from GitHub). Each PR currently rebuilds from scratch — timeout-minutes: 30 ceiling. Acceptable as a Phase 2 floor; can wire buildx GHA cache later if it becomes painful.

Follow-ups (separate PRs)

• Add Harness Replays to staging branch protection once the workflow has been green for several PRs (the new-workflow-instability hedge other gates followed).
• buildx GHA cache for tenant build to drop CI time once stable.

Test plan

• CI on this PR runs harness-replays end-to-end and goes green
• After merge, next PR touching workspace-server/ triggers this workflow
• An intentionally-broken replay change (local experiment) makes the gate fail

🤖 Generated with Claude Code

[HongmingWang-Rabbit]

Five-axis review (workflow-touching → comment-only per spec):

Correctness ✓ — Single-job-with-per-step-if: pattern is the right shape. The 2026-04-29 corrected memory (feedback_branch_protection_check_name_parity) makes that explicit: two-jobs-sharing-name leaves SKIPPED in the set, which still blocks branch protection. This workflow does the right thing — harness-replays always runs, paths-filter gates the real work.

Readability ✓ — The header comment grounds the gate in #2398's outage cleanly. Inline annotations on /etc/hosts, PLUGIN_REPO_PAT fallback, and trap teardown all earn their lines. Future readers won't have to re-derive why this exists.

Architecture ✓ — Per-SHA concurrency (harness-replays-${{ ... .head.sha || github.sha }}) avoids the auto-promote cancellation deadlock from feedback_concurrency_group_per_sha. cancel-in-progress: false is the correct inverse of the per-ref hazard. merge_group: types: [checks_requested] is present, so the Required workflows have merge_group trigger check is satisfied.

Security ✓ — Both third-party actions SHA-pinned (actions/checkout@34e114876b..., dorny/paths-filter@fbd0ab8f3e...) per feedback#15. secrets.PLUGIN_REPO_PAT || secrets.GITHUB_TOKEN fallback is OK for PR triggers — if molecule-ai-plugin-github-app-auth is private and the PAT is missing, sibling checkout hard-fails (clean), not silent-succeeds. No log-injection surface in the dump-on-failure step.

Performance ✓ — 30min timeout is generous; trap + belt-and-suspenders down.sh prevents leaked compose networks across runner reuse. Paths filter excludes doc-only changes, so cost stays tied to what actually needs the harness.

Observation (non-blocking) — The "Dump compose logs on failure" step swallows individual docker compose logs <svc> failures with || true. If the compose network itself is down (the failure mode that motivates the dump), all four service logs will fail in series and you'll only see four "Error: No such service" lines. Not blocking — just consider a single docker compose -f compose.yml logs --no-color as the lead, with per-service tails as fallback. Defer to author judgment.

LGTM in concept. Workflow-touch → no auto-approval per spec.

[HongmingWang-Rabbit]

Self-review caught the harness-tenant unhealthy failure on the workflow's first run — same root cause as the deferred CRITICAL from PR #2401 review:

2026/04/30 20:18:22 Secrets encryption: SECRETS_ENCRYPTION_KEY is required in production
  (MOLECULE_ENV=prod) and must be 32 bytes raw or base64-encoded:
  refusing to boot without encryption in production

The harness sets MOLECULE_ENV: production (intentionally, to replay prod-shape bugs) but never seeded an encryption key. Pushed 630dd0da to seed a clearly-test 32-byte sentinel inline in the compose file. Keeping MOLECULE_ENV=production preserves the harness's value as a production-shape replay — now it exercises the strict-encryption boot path too instead of skirting it.

CI re-running on the new SHA. Will approve+merge once green.

Cross-reference: this closes the gap I flagged on #2401 — the harness was a missing instance of that check (production binary refuses to boot without the key, but no test invoked production-mode boot until this PR added one).