Skip to content

fix(redeploy-staging): tolerate e2e-* teardown race in fleet HTTP 500 - #2511

Merged
HongmingWang-Rabbit merged 1 commit into
stagingfrom
fix/redeploy-tolerate-e2e-teardown-race
May 2, 2026
Merged

HongmingWang-Rabbit merged 1 commit into
stagingfrom
fix/redeploy-tolerate-e2e-teardown-race

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Symptom

`redeploy-tenants-on-staging` fails with:

```
##[error]redeploy-fleet returned HTTP 500
##[error]Process completed with exit code 1.
```

The per-tenant breakdown in the response body shows the failures are on ephemeral `e2e-*` tenants whose parent E2E run tore them down mid-redeploy — operator-facing tenants (`dryrun-98407`, `demo-prep`, etc) rolled fine in the same call.

Two recent occurrences (triage runs #7 and #9 today): runs 25247836697 and 25248307107.

Root cause

CP returns HTTP 500 + `ok:false` whenever ANY tenant in the fleet failed SSM or healthz. The downstream `Verify each staging tenant /buildinfo matches published SHA` step ALREADY distinguishes STALE-vs-UNREACHABLE for exactly this reason (per #2402); only the top-level `if HTTP_CODE != 200; exit 1` gate misclassifies the race as a real fleet failure.

Fix

Filter at the gate:

  • HTTP 500 + every failed slug matches `^e2e-` → soft-warn, fall through to verify
  • Any non-`e2e-*` failure → hard fail, with the failed slugs surfaced in the error message
  • Non-500 HTTP → hard fail (CP-down, network, auth — none of which the e2e filter applies to)

Test

Replayed the gate against 6 synthetic CP responses:

  • happy path → pass through ✓
  • HTTP 500 + only e2e-* failed → soft-warn ✓
  • HTTP 500 + mixed real + e2e fail → hard-fail ✓
  • HTTP 502 → hard-fail ✓
  • HTTP 200 + ok:false → hard-fail ✓
  • HTTP 500 + all real fail → hard-fail ✓

Scope

Only `redeploy-tenants-on-staging.yml`. `redeploy-tenants-on-main.yml` is intentionally NOT touched: prod CP serves no `e2e-*` tenants, so the race can't occur there and the strict gate is the right behavior.

Test plan

  • yaml + shell syntax clean
  • 6-case unit replay of gate logic
  • Next staging push that overlaps an active E2E run reaches verify (no more "redeploy-fleet returned HTTP 500" if the only failures are `e2e-*` slugs)

Recurring failure pattern in redeploy-tenants-on-staging:

  ##[error]redeploy-fleet returned HTTP 500
  ##[error]Process completed with exit code 1.

with the per-tenant breakdown in the response body showing the failures
were on ephemeral e2e-* tenants (saas/canvas/ext) whose parent E2E run
torn them down mid-redeploy — SSM exit=2 because the EC2 was already
terminating, or healthz timeout because the CF tunnel was already gone.
The actual operator-facing tenants (dryrun-98407, demo-prep, etc) all
rolled fine in the same call.

This shape repeats every staging push that overlaps an active E2E run.
The downstream `Verify each staging tenant /buildinfo matches published
SHA` step ALREADY distinguishes STALE vs UNREACHABLE for exactly this
reason (per #2402); only the top-level `if HTTP_CODE != 200; exit 1`
gate misclassifies the race.

Filter: HTTP 500 + every failed slug matches `^e2e-` → soft-warn and
fall through to verify. Any non-e2e-* failure or non-500 HTTP remains
a hard fail, with the failed non-e2e slugs surfaced in the error so
the operator doesn't have to dig the response body out of CI.

Verified the gate logic with 6 synthetic CP responses (happy / e2e-only
race / mixed real+e2e fail / non-200 / 200+ok=false / all-real-fail) —
all behave correctly.

prod's redeploy-tenants-on-main is intentionally NOT touched: prod CP
serves no e2e-* tenants, so the race can't occur there and the strict
gate is the right behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@HongmingWang-Rabbit
HongmingWang-Rabbit added this pull request to the merge queue May 2, 2026
Merged via the queue into staging with commit 3d7b4b7 May 2, 2026
21 checks passed
@HongmingWang-Rabbit
HongmingWang-Rabbit deleted the fix/redeploy-tolerate-e2e-teardown-race branch May 2, 2026 09:23
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…or build-and-push (slowest CI job class)' (#2511) from ci/publish-image-registry-layer-cache into main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant