fix(e2e): teardown patience matches prod cascade duration (~30–90s) - #2201
Merged
Conversation
E2E Staging SaaS has been failing on every cron + push run since
2026-04-27 with `LEAK: org … still present post-teardown (count=1)`,
exit 4. Root cause: the curl timeout on the teardown DELETE was 30s
and the post-DELETE leak check was a single 10s sleep — but the
DELETE handler runs the full GDPR Art. 17 cascade synchronously,
including EC2 termination which AWS reports in 30–60s. Real-world
wall time on a prod-shaped run was 57s on 2026-04-27 (hongmingwang
DELETE); the 30s curl timeout aborted the request mid-cascade and
the 10s post-sleep check found the row still present (status not
yet 'purged').
Two-part fix to match real cascade timing:
1. DELETE curl gets its own --max-time 120 (was 30) so the
synchronous cascade has room to complete in-band.
2. The leak check polls up to 60s for status='purged' instead of
one rigid 10s sleep. Covers two cases:
- DELETE returns 5xx mid-cascade but the cascade finishes anyway
(we still observe a clean state).
- DELETE legitimately exceeds 120s — eventual-consistency catches
the eventual purge instead of false-flagging a leak.
The 5–15s estimate in `molecule-controlplane/internal/handlers/
purge.go`'s comment is the API-call cost only, not the AWS-side
time-to-termination it waits on. The async-purge refactor noted in
that comment would let us drop these timeouts back to ~15s — file
that under future work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit
requested a review
from hongmingwang-moleculeai
as a code owner
April 28, 2026 18:14
3 tasks
HongmingWang-Rabbit
added a commit
that referenced
this pull request
Apr 28, 2026
staging → main: e2e teardown patience (#2201) one-time bridge
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
`E2E Staging SaaS` has been failing every cron + push run since 2026-04-27 with:
```
⚠️ LEAK: org e2e-… still present post-teardown (count=1)
exit code 4
```
Root cause: the teardown gave the cascade 30s (curl timeout) + 10s sleep, but the real prod cascade takes 30–90s — EC2 termination alone waits 30–60s on AWS. Observed example: `hongmingwang` DELETE 2026-04-27 took 57s to complete with HTTP 204. The test's 30s timeout aborted the curl mid-cascade and the 10s post-sleep check found `org_instances.status` not yet 'purged'.
Fix
Two cases now handled:
The 5-15s estimate in `purge.go`'s comment is the API-call cost only, not the AWS-side time-to-termination it waits on. Async purge would let us drop these limits back to ~15s — future work, called out in code.
Test plan
Unblocks
Once green, this is the prereq for wiring auto-promote on green staging E2E (the doc-aligned alternative to standing up a canary fleet at <20 paying tenants). Tracked separately.
🤖 Generated with Claude Code