ci(smoke): make prod smoke non-blocking (alert, not auto-rollback) - #294
Conversation
The suite is genuinely flaky against cold-start prod — across 8 runs it intermittently failed log-habit (Today-list re-render makes the status toggle hit a stale element) and paywall (plans query loads in 2s on a warm hit, never on a cold one). Its retries=0 auto-rollback therefore reverted ~6 healthy deploys and repeatedly paused production. Per the repo's own fix-or-delete-flaky policy, gate-by-flaky-rollback is worse than no gate: run the smoke as a non-blocking post-deploy check that alerts, and recover with the Promote Production workflow if prod is genuinely bad. Re-enable blocking once the two flows are hardened (wait for create-persistence before logging; make plans-load deterministic). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review: PR #294 — ci(smoke): make prod smoke non-blocking
Recommendation: NEEDS WORK
Summary
This PR disables the auto-rollback on smoke failure and converts the smoke step to non-blocking (continue-on-error: true), which is the right call given the documented flakiness. However, the "Capture previous production deployment for rollback" step is now dead operational code — it still runs on every push to main, makes a Vercel API call, and produces an output (previous_url) that nothing downstream consumes.
Findings
Critical
None
High
[HIGH] Dead "Capture previous production deployment for rollback" step still executes
- dimension: Dead / stale code (rubric #2)
- location:
.github/workflows/smoke-prod.yml:86-100 - issue: The
prev_deploystep capturessteps.prev_deploy.outputs.previous_urlbut nothing references that output after this PR removes the rollback step. The step has noif:guard, so it runs on every push to main and makes a Vercel API call for no purpose. - risk: Wasted CI time and an unnecessary Vercel API call on every prod deploy; the dead output is a permanent source of confusion for whoever next reads the workflow.
- fix: Delete the entire "Capture previous production deployment for rollback" step (lines 86-100 in the post-merge file). If rollback is re-enabled later, reintroduce it at that point.
- reference: CLAUDE.md rule 2 — "Delete unused code immediately. No just-in-case exports, dead branches, stub functions."
Medium
None
Low / Info
None (job name Post-deploy smoke + rollback on red is now stale — updating to Post-deploy smoke (alert-only) is cheap and avoids misleading the GitHub Actions UI)
Subagents
| Agent | Verdict |
|---|---|
| parity-checker | N/A |
| i18n-syncer | N/A |
| contract-aligner | N/A |
| security-reviewer | N/A |
Validation
| Check | Result |
|---|---|
| Lint | N/A — CI YAML only |
| Type check | N/A — CI YAML only |
| Tests | N/A — CI YAML only |
| Build (api) | N/A — CI YAML only |
What is good
continue-on-error: truecorrectly scopes non-blocking to the smoke step only, so upload-artifact and other cleanup steps still run regardless.- The warning message is actionable: it names the two flaky flows, explains why rollback was disabled, and points at the recovery path (Promote Production workflow).
- Removing the
set -euo pipefailblock and the edge-caseexit 1path is a clean simplification.
Recommendation
Delete the now-dead "Capture previous production deployment for rollback" step (lines 86-100 of the post-merge file). That is the only change needed for approval.
|
1 participant
The smoke suite is genuinely flaky against cold-start prod (log-habit Today-list re-render; paywall intermittent plans-load), and its retries=0 auto-rollback reverted ~6 healthy deploys + repeatedly paused prod. Make it a non-blocking post-deploy check that alerts instead of rolling back; recover via the Promote Production workflow if prod is actually bad. Re-enable blocking after the two flows are hardened. .github-only.
🤖 Generated with Claude Code