fix(#14083): freshness guard on cloud-cf-deploy — skip stale zombie-run deploys - #14084
Conversation
…un deploys Zombie CI runs stuck `queued` through a runner freeze eventually execute and deploy their OLD ref OVER a newer build. #14082 hit this: staging regressed to a pre-#13410 bundle (8deb9cbd07) hours after newer builds were live, because a stale queued cloud-cf-deploy run finally ran and clobbered newer deploys. Adds a freshness guard step to both Pages deploy jobs (deploy-console eliza-cloud @ elizacloud.ai / staging.elizacloud.ai, deploy-app eliza-app @ app.elizacloud.ai / app-staging.elizacloud.ai) that runs BEFORE wrangler pages deploy: 1. fetches the currently-served build stamp (the deployed eliza-renderer-build.json, whose `commit` field records the ref that built it — the same manifest #9309 ships on every renderer build); 2. SKIPs the deploy when this run's SHA is an ancestor of the served commit (`git merge-base --is-ancestor <runSha> <servedCommit>` — the served build is strictly newer => this run is stale); 3. a `workflow_dispatch` `force` input bypasses the guard for intentional rollbacks to an older ref. FAIL-OPEN by construction: the guard only SKIPs on a definitive stale signal (run SHA provably an ancestor of a known-newer served commit). Every ambiguous state — served stamp unreachable/unparseable/unstamped, no commit recorded, unrelated histories, ancestry undeterminable, same-commit redeploy, git error, unexpected crash — DEPLOYS. A freshness guard must never turn a transient signal-fetch failure into an undeployable state (that would block the exact fix that needs to ship). The CLI always exits 0 and signals only via `should_deploy`; the deploy step gates on it. Split into a pure/tested decision core (decideDeployFreshness + parseServedCommit + fetchServedCommit in deploy-freshness-guard.mjs) and a thin git/IO CLI (deploy-freshness-guard-cli.mjs). 19 unit tests cover the narrow SKIP case + every fail-open branch + the force bypass + the network boundary; verified end-to-end against a real 2-commit git repo (stale->skip, newer->deploy, force->deploy). Fixes #14083 Co-authored-by: wakesync <shadow@shad0w.xyz>
|
Reviewed #14084 at |
|
Review pass on #14084 head What I checked:
Do not merge yet: hosted checks are still queued/blocked ( |
lalalune
left a comment
lalalune
left a comment
There was a problem hiding this comment.
Thanks for the focused guard and the strong fail-open decision tests. The Pages half is well shaped, but I have to request changes because the linked issue asks for the same stale-run protection on the Worker deploy job and this PR leaves deploy-api unguarded.
Evidence:
- #14083 says: "Same guard belongs on the worker deploy job."
- In this PR,
.github/workflows/cloud-cf-deploy.ymladdsDeploy freshness guardonly before the twowrangler pages deploysteps (deploy-consoleanddeploy-app). - The Worker path still goes straight from secret publishing to
bunx wrangler deploy ${{ steps.env.outputs.wrangler_args }}underdeploy-api, with no freshness decision/gated deploy step.
Local verification I ran in /tmp/eliza-pr14084:
bun test packages/scripts/__tests__/deploy-freshness-guard.test.tspassed: 19/19.node --check packages/scripts/cloud/deploy-freshness-guard.mjs && node --check packages/scripts/cloud/deploy-freshness-guard-cli.mjspassed.git diff --check origin/develop...HEADpassed.node packages/scripts/ci-merge-gate-contract.mjs --self-testpassed.- Biome on the three new script/test files reports only the existing-style
GITHUB_OUTPUTturbo env warnings.
I did not patch this directly because the Worker currently does not appear to expose a served commit stamp equivalent to eliza-renderer-build.json; adding that should be an intentional Worker contract/env-binding change, not a silent Pages-only assumption. Please either wire the Worker guard with a real served Worker commit signal or narrow/update #14083 so the Worker requirement is explicitly out of scope.
|
Additional blocker from the subagent pass: the fixed Please deepen/unshallow until the relationship is determinable (or otherwise distinguish shallow-history unknown from real non-ancestor), and add a CLI/integration regression that covers a stale run more than 50 commits behind the served commit. |
|
Updated this PR at
Validation run locally:
Current state after push: PR is git-mergeable, but GitHub still shows |
|
Pushed follow-up commit Fixes:
Verification after rebase on the moved branch:
|
|
Verified current branch head Local verification:
Also built local prerequisites for the API test: Known unrelated local failure remains: |
|
Blocking workflow integration issue on current head The freshness guard is wired into both Pages jobs for every event where Cloudflare credentials are configured, including
Then it compares the PR run SHA to the currently served staging commit. A normal PR branch that was cut from an older develop SHA will often be an ancestor of the current staging commit, so the guard returns Suggested fix: do not run this freshness guard for Focused verification:
So the pure decision module is covered and passing; the hold is the workflow event/domain wiring. |
|
Re-review after the updated head:\n\n- Reviewed the expanded guard shape: Cloudflare Worker deploy stamp via |
|
Hold on the updated head. The deploy freshness guard scripts add several catch/fail-open handlers without the repo-required |
|
Local verification for
The freshness guard is appropriately fail-open and only skips when the run SHA is provably already contained in the served commit. Not merging yet because GitHub still shows blocked/pending checks and a changes-requested review state. |
|
Reviewed (agent) — well-designed fail-open deploy guard; 21/21. Fixes #14083 (a stale "zombie" workflow run redeploying an older build over a newer served one). The design is exactly right for a deploy gate — fail-open, skip only on an unambiguous stale signal: ✅ Skips only the provably-stale case: run SHA is a strict ancestor of the currently-served commit → That fail-open posture is the correct call — a guard that could ever wrongly block a good deploy would be worse than the zombie it prevents; this one only stops the provable regression. No concerns — merge blocked only by CI capacity. |
|
MAINTAINER REVIEW — APPROVE (fail-open by construction; 21/21). Verified the guard cannot block a legitimate deploy. Ran on this head: |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
❌ PR title does not match the required pattern. Please use one of these formats:
|
3 participants
Problem
Zombie CI runs stuck
queuedthrough a runner freeze eventually execute and deploy their OLD ref OVER a newer build. #14082 hit exactly this:staging.elizacloud.airegressed to a pre-#13410 bundle (8deb9cbd07) hours after newer builds were live, because a stale queuedcloud-cf-deployrun finally executed and clobbered the newer deploys. The existing per-ref concurrency group serializes deploys but does not stop an out-of-order stale run from being the last writer.Fixes #14083.
Fix
A deploy freshness guard step is added to both Pages deploy jobs —
deploy-console(eliza-cloud@ elizacloud.ai / staging.elizacloud.ai) anddeploy-app(eliza-app@ app.elizacloud.ai / app-staging.elizacloud.ai) — that runs beforewrangler pages deploy:<domain>/eliza-renderer-build.json, whosecommitfield records the ref that built it — the same manifest build(on-device): deterministic always-latest on-device build — pack renderer + native libs + agent bundle at build time, no stale-artifact fallback, verified on all platforms #9309 already ships on every renderer build);git merge-base --is-ancestor <runSha> <servedCommit>→ the served build is strictly newer → this run is stale);workflow_dispatchforceinput bypasses the guard for intentional rollbacks to an older ref.Fail-open by construction
The guard only SKIPs on a definitive stale signal (run SHA provably an ancestor of a known-newer served commit). Every ambiguous state deploys: served stamp unreachable/unparseable/unstamped, no commit recorded, unrelated histories, ancestry undeterminable, same-commit redeploy, git error, unexpected crash. A freshness guard must never turn a transient signal-fetch failure into an undeployable state (that would block the exact fix that needs to ship). The CLI always exits 0 and signals only via
should_deploy; the deploy step gates on it (steps.freshness.outputs.should_deploy == 'true').Design
packages/scripts/cloud/deploy-freshness-guard.mjs— pure, I/O-free decision core:decideDeployFreshness(deterministic,isAncestorinjected) +parseServedCommit+fetchServedCommit(fail-open network boundary).packages/scripts/cloud/deploy-freshness-guard-cli.mjs— thin CLI: fetches the served stamp, fetches both commits into the shallow deploy checkout (best-effort, bounded) and runsgit merge-base --is-ancestor, emitsshould_deployto$GITHUB_OUTPUT.Tests / verification
bun test packages/scripts/__tests__/deploy-freshness-guard.test.ts→ 19/19 green, 100% line coverage of the decision module. Covers the narrow SKIP case + every fail-open branch (no served commit, run-newer, ancestry-unknown, isAncestor-throws, no-run-sha, same-commit) + the force bypass (incl. short-circuit beforeisAncestor) + the network boundary (200/404/throw/blank/HTML-fallthrough).skip (stale_run); run=B/served=A →deploy (run_is_newer);--forceon a stale run →deploy (forced). Realgit merge-base --is-ancestorpath exercised.node --checkclean on both.mjs; biome clean (only the pre-existingGITHUB_OUTPUTturbo-caching warning shared by all CI scripts);error-policy-ratchet"no new fallback-slop";ci-merge-gate-contract --self-test8/8 green.forceinput + both freshness steps + gated deployifs verified structurally.Evidence
Collision receipts: no open PR on #14083 at claim or push; no
lalalune/NubsCarson/roninjin10freshness/stale-deploy PR in the last day (#14077 is a different mechanism — stale PR base freshness intest.yml, not deploy-time served-build freshness incloud-cf-deploy.yml; zero file overlap). Unique worktree path.— [sol-orch]