feat(canary): gate :latest tag promotion on canary verify green (Phase 3) - #988
Merged
Merged
Conversation
…e 3) Completes the canary release train. Before this, publish-workspace- server-image.yml pushed both :staging-<sha> and :latest on every main merge — meaning the prod tenant fleet auto-pulled every image immediately, before any post-deploy smoke test. A broken image (think: this morning's E2E current_task drift, but shipped at 3am instead of caught in CI) would have fanned out to every running tenant within 5 min. Now: - publish workflow pushes :staging-<sha> ONLY - canary tenants are configured to track :staging-<sha>; they pick up the new image on their next auto-update cycle - canary-verify.yml runs the smoke suite (Phase 2) after the sleep - on green: a new promote-to-latest job uses crane to remotely retag :staging-<sha> → :latest for both platform and tenant images - prod tenants auto-update to the newly-retagged :latest within their usual 5-min window - on red: :latest stays frozen on prior good digest; prod is untouched crane is pulled onto the runner (~4 MB, GitHub release) rather than docker-daemon retag so the workflow doesn't need a privileged runner. Rollback: if canary passed but something surfaces post-promotion, operator runs "crane tag ghcr.io/molecule-ai/platform:<prior-good-sha> latest" manually. A follow-up can wrap that in a Phase 4 admin endpoint / script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
feat(canary): gate :latest tag promotion on canary verify green (Phase 3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the canary release loop. Before this, `publish-workspace-server-image.yml` pushed both `:staging-` and `:latest` on every main merge — so the prod tenant fleet auto-pulled every image immediately, before any post-deploy smoke test. A bad image (e.g. today's E2E `current_task` drift, but shipped at 3am instead of caught in CI) would have fanned out to every running tenant within 5 min.
New flow
Why crane
~4 MB static binary pulled onto the runner, no Docker daemon needed — just a registry API client. A single `crane tag` call per image retags remotely without pulling or pushing layers.
Rollback
If canary passed but something surfaces post-promotion, operator runs `crane tag ghcr.io/molecule-ai/platform: latest` manually. Phase 4 will wrap that in an admin endpoint / `scripts/rollback-latest.sh`.
Test plan
🤖 Generated with Claude Code