Skip to content

feat(deploy): verify each tenant /buildinfo matches published SHA after redeploy - #2398

Merged
HongmingWang-Rabbit merged 1 commit into
stagingfrom
auto/buildinfo-deploy-verification
Apr 30, 2026
Merged

feat(deploy): verify each tenant /buildinfo matches published SHA after redeploy#2398
HongmingWang-Rabbit merged 1 commit into
stagingfrom
auto/buildinfo-deploy-verification

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Summary

Closes #2395. The redeploy-fleet workflows currently report ssm_status=Success based on the SSM RPC return code alone — but on hongmingwang.moleculesai.app's tenant, that "Success" was firing 30+ minutes after every push while the EC2 instance kept serving the previous :latest digest. Root cause: docker compose up -d is a no-op when the local tag already resolves locally; without an explicit docker pull, "the SSM RPC didn't error" and "the new code is actually running" are different statements.

This PR makes them the same statement.

How

  • New internal/buildinfo package exposes GitSHA, set at link time via -ldflags "-X .../buildinfo.GitSHA=$GIT_SHA". Default value "dev" so test runs without ldflags fail closed against an unset deploy rather than rounding-tripping "".
  • Router exposes GET /buildinfo → {git_sha}. Public, no auth, cheap enough to curl from CI for every tenant.
  • Both Dockerfiles thread GIT_SHA from a build-arg into the Go build line.
  • publish-workspace-server-image.yml passes GIT_SHA=${{ github.sha }} for both images.
  • redeploy-tenants-on-main.yml and redeploy-tenants-on-staging.yml add a "Verify each tenant /buildinfo matches published SHA" step that curls every tenant after the redeploy RPC and fails the workflow on mismatch. Staging treats both :latest and :staging-latest as moving tags. Verification is skipped only when the operator pinned a specific tag via workflow_dispatch.

Why a verify step rather than docker pull in the SSM script

Because the verify step also catches:

  • partial fleet drift (one EC2 docker daemon hung; the rest pulled fine)
  • container restart races (pull succeeded, container exited and restarted on the old image)
  • tag-name typos in the deploy script
  • the entire class of "the new code crashed on startup and supervisord rolled back"

Adding docker pull would only address one of the four. The verify step is the load-bearing fix.

Tests

  • TestGitSHA_DefaultDevSentinel pins the dev default — guarantees an unset deploy always fails closed.
  • TestBuildInfoEndpoint_ReturnsGitSHA pins the wire shape ({"git_sha": "..."}) that the workflow's jq lookup depends on.
  • go build ./..., go vet ./..., both touched packages green.
  • All three workflow YAMLs validated.

Test plan

  • Merge to staging
  • Watch publish-workspace-server-image build with the new GIT_SHA build-arg
  • Watch redeploy-tenants-on-staging — should now hit the verify step and either green or fail with a useful diff
  • On green, verify hongmingwang.moleculesai.app/buildinfo returns the new SHA

🤖 Generated with Claude Code

…er redeploy

Closes the gap that let issue #2395 ship: redeploy-fleet workflows reported
ssm_status=Success based on SSM RPC return code alone, while EC2 tenants
silently kept serving the previous :latest digest because docker compose up
without an explicit pull is a no-op when the local tag already exists.

Wire:
  - new buildinfo package exposes GitSHA, set at link time via -ldflags from
    the GIT_SHA build-arg (default "dev" so test runs without ldflags fail
    closed against an unset deploy)
  - router exposes GET /buildinfo returning {git_sha} — public, no auth,
    cheap enough to curl from CI for every tenant
  - both Dockerfiles thread GIT_SHA into the Go build
  - publish-workspace-server-image.yml passes GIT_SHA=github.sha for both
    images
  - redeploy-tenants-on-main.yml + redeploy-tenants-on-staging.yml curl each
    tenant's /buildinfo after the redeploy SSM RPC and fail the workflow on
    digest mismatch; staging treats both :latest and :staging-latest as
    moving tags; verification is skipped only when an operator pinned a
    specific tag via workflow_dispatch

Tests:
  - TestGitSHA_DefaultDevSentinel pins the dev default
  - TestBuildInfoEndpoint_ReturnsGitSHA pins the wire shape that the
    workflow's jq lookup depends on

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@HongmingWang-Rabbit
HongmingWang-Rabbit added this pull request to the merge queue Apr 30, 2026
Merged via the queue into staging with commit dea306d Apr 30, 2026
19 checks passed
@HongmingWang-Rabbit
HongmingWang-Rabbit deleted the auto/buildinfo-deploy-verification branch April 30, 2026 18:04
HongmingWang-Rabbit pushed a commit that referenced this pull request Apr 30, 2026
Workspace-server has GET /buildinfo (PR #2398) — `curl https://<slug>.
moleculesai.app/buildinfo` returns the live git SHA. Canvas had no
parallel: debugging "is this the deployed code?" required reading
Vercel's UI or response headers (deployment ID, not git SHA).

Add canvas /api/buildinfo returning {git_sha, git_ref, vercel_env}
sourced from VERCEL_GIT_COMMIT_SHA / _REF / VERCEL_ENV — Vercel injects
these at build time from the deploying commit. Outside Vercel (local
`next dev`, harness) all three are unset and the endpoint returns
`git_sha: "dev"`, the same sentinel workspace-server uses pre-ldflags-
injection.

Now both surfaces speak the same vocabulary:

  curl https://<slug>.moleculesai.app/buildinfo
  curl https://canvas.moleculesai.app/api/buildinfo

3 tests cover dev-fallback, Vercel-injected SHA pass-through, and JSON
content type.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…ce/scope/namespace context (#2398)

Replace the generic log.Printf with operator-diagnosis context:
- workspace=%s — the calling workspace
- scope=%s — requested scope (LOCAL|TEAM|GLOBAL)
- namespace=%s — resolved v2 plugin namespace
- err_class=%T — Go concrete type for log-aggregator filtering
- err=%q — full quoted error message

The HTTP 500 response body is unchanged ('failed to store memory')
to preserve the no-client-leak discipline.

Fixes #2398.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…ror log with workspace/scope/namespace context (#2398)' (#2402) from fix/2398-enrich-commit-memory-log into main
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…2398)

Replace the bare log.Printf("Commit memory error (plugin): %v", err) with
operator-diagnosis context:

  workspace=%s scope=%s namespace=%s err_class=%T err=%q

The HTTP 500 response body stays the generic literal
"failed to store memory" — zero client-side leak. Only the server-side
log is enriched so recurring incidents can be distinguished in the log
aggregator.

Closes #2398
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…mory plugin error (#2398)' (#2415) from fix/2398-enrich-commitmemory-log into main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

redeploy-fleet + tenant provision serve stale workspace-server images even with fresh :latest

1 participant