fix(redeploy-staging): tolerate e2e-* teardown race in fleet HTTP 500 - #2511
Merged
HongmingWang-Rabbit merged 1 commit intoMay 2, 2026
Merged
Conversation
Recurring failure pattern in redeploy-tenants-on-staging: ##[error]redeploy-fleet returned HTTP 500 ##[error]Process completed with exit code 1. with the per-tenant breakdown in the response body showing the failures were on ephemeral e2e-* tenants (saas/canvas/ext) whose parent E2E run torn them down mid-redeploy — SSM exit=2 because the EC2 was already terminating, or healthz timeout because the CF tunnel was already gone. The actual operator-facing tenants (dryrun-98407, demo-prep, etc) all rolled fine in the same call. This shape repeats every staging push that overlaps an active E2E run. The downstream `Verify each staging tenant /buildinfo matches published SHA` step ALREADY distinguishes STALE vs UNREACHABLE for exactly this reason (per #2402); only the top-level `if HTTP_CODE != 200; exit 1` gate misclassifies the race. Filter: HTTP 500 + every failed slug matches `^e2e-` → soft-warn and fall through to verify. Any non-e2e-* failure or non-500 HTTP remains a hard fail, with the failed non-e2e slugs surfaced in the error so the operator doesn't have to dig the response body out of CI. Verified the gate logic with 6 synthetic CP responses (happy / e2e-only race / mixed real+e2e fail / non-200 / 200+ok=false / all-real-fail) — all behave correctly. prod's redeploy-tenants-on-main is intentionally NOT touched: prod CP serves no e2e-* tenants, so the race can't occur there and the strict gate is the right behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
requested a review
from hongmingwang-moleculeai
as a code owner
May 2, 2026 09:17
HongmingWang-Rabbit
enabled auto-merge
May 2, 2026 09:18
HongmingWang-Rabbit
pushed a commit
that referenced
this pull request
Jun 12, 2026
…or build-and-push (slowest CI job class)' (#2511) from ci/publish-image-registry-layer-cache into main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
`redeploy-tenants-on-staging` fails with:
```
##[error]redeploy-fleet returned HTTP 500
##[error]Process completed with exit code 1.
```
The per-tenant breakdown in the response body shows the failures are on ephemeral `e2e-*` tenants whose parent E2E run tore them down mid-redeploy — operator-facing tenants (`dryrun-98407`, `demo-prep`, etc) rolled fine in the same call.
Two recent occurrences (triage runs #7 and #9 today): runs 25247836697 and 25248307107.
Root cause
CP returns HTTP 500 + `ok:false` whenever ANY tenant in the fleet failed SSM or healthz. The downstream `Verify each staging tenant /buildinfo matches published SHA` step ALREADY distinguishes STALE-vs-UNREACHABLE for exactly this reason (per #2402); only the top-level `if HTTP_CODE != 200; exit 1` gate misclassifies the race as a real fleet failure.
Fix
Filter at the gate:
Test
Replayed the gate against 6 synthetic CP responses:
Scope
Only `redeploy-tenants-on-staging.yml`. `redeploy-tenants-on-main.yml` is intentionally NOT touched: prod CP serves no `e2e-*` tenants, so the race can't occur there and the strict gate is the right behavior.
Test plan