fix(release): prove the upload became production — the deploy lane called a stale site "deployed" - #225
Conversation
…stale site "deployed" The release lane uploaded a bundle and then "verified" it by probing https://tinystudio.in against a fixed list of ALREADY-merged fixes. Those assertions pass against the OLD site, so an upload that never became the production deployment (preview branch, wrong project, no-op upload, custom domain served elsewhere) still printed: [publish] done: live site matches the verified bundle with production untouched. Reproduced against the pre-change pipeline: with a Cloudflare fake whose canonical_deployment never moves and an acceptance that passes, releasePipeline resolves successfully. The lane now proves promotion between upload and acceptance: the canonical production deployment id must move away from the captured rollback target, and (when Cloudflare reports one) its source commit must be the bundle's. Cloudflare is read-after-write, so promotion is polled 6x/5s before it counts as a failure. Promotion failures are deliberately NOT rollback cases: if production never moved there is nothing to restore, and if it moved to somebody else's deployment a rollback would clobber their release.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 47 minutes Limit details: You’ve used the included review currently available. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (6)
ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe release pipeline now passes bundle source commits to Wrangler, polls for production promotion, validates deployment identity and commit ownership, and runs live acceptance only after promotion proof. Tests cover delayed, missing, unchanged, and foreign production deployments. ChangesRelease promotion verification
Estimated code review effort: 3 (Moderate) | ~25 minutes Merge Risk: ⚪ Minimal · up to The release lane now requires production promotion before declaring success, reducing false-success releases; no actionable merge-blocking risk remains, so the PR is merge-ready after normal checks. Sequence Diagram(s)sequenceDiagram
participant releasePipeline
participant Wrangler
participant CloudflareProduction
participant liveAcceptance
releasePipeline->>Wrangler: Upload bundle with source commit
Wrangler->>CloudflareProduction: Create deployment
releasePipeline->>CloudflareProduction: Poll production deployment
CloudflareProduction-->>releasePipeline: Return deployment identity and commit
releasePipeline->>liveAcceptance: Run checks after promotion proof
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/settings/billing. Comment |
…rally, correct workflow headers that told readers to expect red - publish: a blip on the Cloudflare project read during promotion polling is retried; only an identity still unreadable on the last attempt fails the release. Without this a transient 500 would fail a release that did go live. - test-deploy-public-site-workflow: assert the pipeline proves promotion AFTER the upload and BEFORE the acceptance, so a refactor cannot silently drop it (2 of these checks fail against the pre-change script). - live-site-check.yml said the nightly staleness alarm "is expected to FAIL". That is how a red alarm becomes wallpaper. Red now means act; the site was restored 2026-08-20. - deploy-public-site.yml claimed the Pages token was unprovisioned and the lane would stay red until it landed. It has been provisioned since 2026-08-20 under the secret name CLOUDFLARE. Header now records the real failure history (#208 /client/v4, #209 Node 22, #211 dead acceptance) and the promotion proof. - Lane evidence report.
The defect
The release lane uploaded a bundle and then "verified" it by probing
https://tinystudio.inagainst a fixed list of already-merged fixes(
scripts/check-public-live-deploy.mjs). Those assertions pass against theold site. So an upload that never became the production deployment still
printed:
…with production untouched. That is the mechanism by which "merged public fixes have
not gone live" survives for two months while the lane looks like it is working.
Reproduced against the pre-change pipeline (hermetic; Cloudflare faked so
canonical_deploymentnever moves; acceptance scripted to pass):The fix
Between upload and acceptance the lane now proves promotion
(
verifyProductionPromotion):target, and
bundle's
source_commit.Cloudflare's project read is read-after-write, so promotion is polled (6 × 5s) before it
counts as a failure, and a transient read error is retried rather than failing a release
that did go live.
Promotion failure is deliberately not a rollback case:
Both fail loudly with the reason and the stuck deployment named.
test-deploy-public-site-workflow.mjsadditionally pins the ordering structurally, so afuture refactor cannot drop the proof: promotion must be proven after the upload and
before the acceptance.
Documentation that was actively harmful
live-site-check.ymltold readers the nightly staleness alarm "is expected toFAIL". That is how a red alarm becomes wallpaper. Red now means act.
deploy-public-site.ymlclaimed the Pages token was unprovisioned and the lane wouldstay red until it landed. It has been provisioned since 2026-08-20 (secret name
CLOUDFLARE). The header now records the real failure history (fix(release): CF_API_BASE typo — deploy lane 403'd since creation #208/client/v4,fix(release): deploy lane Node 20 → 22 (wrangler hard requirement) #209 Node 22, fix(release): acceptance checker was dead code (missing import + undefined helpers) #211 dead acceptance section J) and the promotion step.
Status of the underlying incident
The acute outage is already repaired on main by #208/#209/#211; the first fully green
publish was 2026-08-20 16:57Z (
6c3d83f). Independently re-verified here:check-public-live-deploy.mjs→ 240 checks, 0 failures live;origin/mainagainst every live URL: theonly differences are Cloudflare's own email-obfuscation rewrite (
_headers404sbecause Pages consumes it). No stale content.
This PR closes the hole that let the site sit stale under a lane that reported success.
Verification
New checks fail against
origin/main's script (not vacuous): missing exportbundleSourceCommit, and12 checks, 2 failureson the ordering guard.Evidence:
.lane/reports/fix-lane1-deploy-promotion-proof-20260821.md