Skip to content

fix(release): prove the upload became production — the deploy lane called a stale site "deployed" - #225

Merged
nish3451 merged 34 commits into
mainfrom
fix/lane1-deploy-promotion-proof-20260821
Aug 22, 2026
Merged

fix(release): prove the upload became production — the deploy lane called a stale site "deployed"#225
nish3451 merged 34 commits into
mainfrom
fix/lane1-deploy-promotion-proof-20260821

Conversation

@nish3451

@nish3451 nish3451 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

The defect

The release lane uploaded a bundle and then "verified" it by probing
https://tinystudio.in against a fixed list of already-merged fixes
(scripts/check-public-live-deploy.mjs). Those assertions pass against the
old site. So an upload that never became the production deployment still
printed:

[publish] done: live site matches the verified bundle

…with production untouched. That is the mechanism by which "merged public fixes have
not gone live" survives for two months while the lane looks like it is working.

Reproduced against the pre-change pipeline (hermetic; Cloudflare faked so
canonical_deployment never moves; acceptance scripted to pass):

[publish] uploading bundle
[publish] running live acceptance against the neutral merged fixes
[publish] done: live site matches the verified bundle
RED: pre-change pipeline reported SUCCESS while production is still previous-deployment

The fix

Between upload and acceptance the lane now proves promotion
(verifyProductionPromotion):

  • the canonical production deployment id must move away from the captured rollback
    target, and
  • when Cloudflare reports one, the promoted deployment's source commit must be this
    bundle's source_commit.

Cloudflare's project read is read-after-write, so promotion is polled (6 × 5s) before it
counts as a failure, and a transient read error is retried rather than failing a release
that did go live.

Promotion failure is deliberately not a rollback case:

  • production never moved → nothing to restore (it already is the pre-release state);
  • production moved to a foreign deployment → a rollback would clobber someone else's release.

Both fail loudly with the reason and the stuck deployment named.

test-deploy-public-site-workflow.mjs additionally pins the ordering structurally, so a
future refactor cannot drop the proof: promotion must be proven after the upload and
before the acceptance.

Documentation that was actively harmful

Status of the underlying incident

The acute outage is already repaired on main by #208/#209/#211; the first fully green
publish was 2026-08-20 16:57Z (6c3d83f). Independently re-verified here:

  • check-public-live-deploy.mjs240 checks, 0 failures live;
  • byte-level diff of the bundle built from origin/main against every live URL: the
    only differences are Cloudflare's own email-obfuscation rewrite (_headers 404s
    because Pages consumes it). No stale content.

This PR closes the hole that let the site sit stale under a lane that reported success.

Verification

node scripts/test-pages-release.mjs               → 63 checks, 0 failures
node scripts/test-deploy-public-site-workflow.mjs → 12 checks, 0 failures
node scripts/check-public-live-deploy.mjs         → 240 checks, 0 failures (live)
repo-checks (runner, full npm run ci + npm audit) → pass

New checks fail against origin/main's script (not vacuous): missing export
bundleSourceCommit, and 12 checks, 2 failures on the ordering guard.

Evidence: .lane/reports/fix-lane1-deploy-promotion-proof-20260821.md

…stale site "deployed"

The release lane uploaded a bundle and then "verified" it by probing
https://tinystudio.in against a fixed list of ALREADY-merged fixes. Those
assertions pass against the OLD site, so an upload that never became the
production deployment (preview branch, wrong project, no-op upload, custom
domain served elsewhere) still printed:

  [publish] done: live site matches the verified bundle

with production untouched. Reproduced against the pre-change pipeline: with a
Cloudflare fake whose canonical_deployment never moves and an acceptance that
passes, releasePipeline resolves successfully.

The lane now proves promotion between upload and acceptance: the canonical
production deployment id must move away from the captured rollback target,
and (when Cloudflare reports one) its source commit must be the bundle's.
Cloudflare is read-after-write, so promotion is polled 6x/5s before it counts
as a failure. Promotion failures are deliberately NOT rollback cases: if
production never moved there is nothing to restore, and if it moved to
somebody else's deployment a rollback would clobber their release.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@nish3451, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 47 minutes

Limit details: You’ve used the included review currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: 0256a317-e8ac-44b0-b3bd-61cf09e1a639

📥 Commits

Reviewing files that changed from the base of the PR and between e366b32 and 77cb6f1.

📒 Files selected for processing (6)
  • .github/workflows/deploy-public-site.yml
  • .github/workflows/live-site-check.yml
  • .lane/reports/fix-lane1-deploy-promotion-proof-20260821.md
  • scripts/publish-public-site.mjs
  • scripts/test-deploy-public-site-workflow.mjs
  • scripts/test-pages-release.mjs
ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: ed35fc28-fcb5-490c-8526-16f8ecdb83c6

📥 Commits

Reviewing files that changed from the base of the PR and between 6c3d83f and e366b32.

📒 Files selected for processing (2)
  • scripts/publish-public-site.mjs
  • scripts/test-pages-release.mjs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The release pipeline now passes bundle source commits to Wrangler, polls for production promotion, validates deployment identity and commit ownership, and runs live acceptance only after promotion proof. Tests cover delayed, missing, unchanged, and foreign production deployments.

Changes

Release promotion verification

Layer / File(s) Summary
Bundle provenance and polling dependencies
scripts/publish-public-site.mjs, scripts/test-pages-release.mjs
The release script centralizes manifest parsing, extracts non-unknown source commits, compares commit identifiers, and adds bounded polling with injectable sleep support. Tests cover commit formats and manifest extraction.
Production promotion verification
scripts/publish-public-site.mjs, scripts/test-pages-release.mjs
verifyProductionPromotion polls production and validates deployment identity and source commit ownership. Tests cover delayed, missing, unchanged, and foreign deployments.
Release pipeline promotion gate
scripts/publish-public-site.mjs, scripts/test-pages-release.mjs
releasePipeline derives the bundle commit, requires promotion proof, logs the promoted deployment, and runs live acceptance afterward. The release contract documents this requirement.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to e366b

The release lane now requires production promotion before declaring success, reducing false-success releases; no actionable merge-blocking risk remains, so the PR is merge-ready after normal checks.

Sequence Diagram(s)

sequenceDiagram
  participant releasePipeline
  participant Wrangler
  participant CloudflareProduction
  participant liveAcceptance
  releasePipeline->>Wrangler: Upload bundle with source commit
  Wrangler->>CloudflareProduction: Create deployment
  releasePipeline->>CloudflareProduction: Poll production deployment
  CloudflareProduction-->>releasePipeline: Return deployment identity and commit
  releasePipeline->>liveAcceptance: Run checks after promotion proof
Loading

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

minimax-vps and others added 27 commits August 21, 2026 02:21
…rally, correct workflow headers that told readers to expect red

- publish: a blip on the Cloudflare project read during promotion polling is
  retried; only an identity still unreadable on the last attempt fails the
  release. Without this a transient 500 would fail a release that did go live.
- test-deploy-public-site-workflow: assert the pipeline proves promotion AFTER
  the upload and BEFORE the acceptance, so a refactor cannot silently drop it
  (2 of these checks fail against the pre-change script).
- live-site-check.yml said the nightly staleness alarm "is expected to FAIL".
  That is how a red alarm becomes wallpaper. Red now means act; the site was
  restored 2026-08-20.
- deploy-public-site.yml claimed the Pages token was unprovisioned and the lane
  would stay red until it landed. It has been provisioned since 2026-08-20
  under the secret name CLOUDFLARE. Header now records the real failure history
  (#208 /client/v4, #209 Node 22, #211 dead acceptance) and the promotion proof.
- Lane evidence report.
@nish3451
nish3451 merged commit d6b4918 into main Aug 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant