Skip to content

fix(e2e): bump staging tenant TLS-readiness timeout 3min → 10min - #2099

Merged
HongmingWang-Rabbit merged 1 commit into
stagingfrom
fix/staging-e2e-tls-timeout
Apr 26, 2026
Merged

HongmingWang-Rabbit merged 1 commit into
stagingfrom
fix/staging-e2e-tls-timeout

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

[Molecule-Platform-Evolvement-Manager]

Why

Canvas tabs E2E has been flaking with `Error: tenant TLS: timed out after 180s` across 4+ open PRs over multiple cycles (#2096 today, #2094, #2055, #2079, #2061, #2066 historically).

Root cause

`TLS_TIMEOUT_MS=180s` is too tight. The TLS-readiness step layers three independent waits:

  1. Cloudflare DNS propagation through the edge (1-2 min typical)
  2. Tenant CF Tunnel registering the new hostname (1-2 min)
  3. CF edge ACME cert provisioning + cache (1-3 min)

Each layer can add 1-3 min on its own under heavy staging load. The realistic worst case is past the 3-min cap.

Fix

Bump TLS timeout 3 min → 10 min in both call sites:

  • `canvas/e2e/staging-setup.ts` — `TLS_TIMEOUT_MS = 10 * 60 * 1000`
  • `tests/e2e/test_staging_full_saas.sh` — `TLS_DEADLINE += 600`

Stays inside the 20-min PROVISION envelope (already raised in the prior history at line 42-46) so a genuinely-stuck tenant still fails loud at the earlier provision step rather than masquerading as a TLS issue.

Each site carries an inline rationale comment so the next reviewer sees the layer-by-layer decomposition without re-reading the issue thread.

Test plan

Closes a 4+ cycle Canvas tabs E2E flake pattern that's been blocking
staging→main PRs since 2026-04-24+ (#2096, #2094, #2055, #2079, ...).

Root cause: TLS_TIMEOUT_MS=180s (3 min) is too tight for the layered
realities of staging tenant TLS readiness:

1. Cloudflare DNS propagation through the edge (1-2 min typical)
2. Tenant CF Tunnel registering the new hostname (1-2 min)
3. CF edge ACME cert provisioning + cache (1-3 min)

Each layer can add 1-3 min on its own under heavy staging load — the
realistic worst case is well past the 3-min cap.

Provision and workspace-online timeouts were already raised to 20 min
(staging-setup.ts:42-46 history). The TLS gate was the remaining
under-budgeted step. Bumping to 10 min keeps it inside the 20-min
PROVISION envelope so a genuinely-stuck tenant still fails loud at
the earlier provision step rather than masquerading as a TLS issue.

Both call sites raised together:
- canvas/e2e/staging-setup.ts: TLS_TIMEOUT_MS = 10 * 60 * 1000
- tests/e2e/test_staging_full_saas.sh: TLS_DEADLINE += 600

Each carries an inline rationale comment so the next reviewer sees
the layer-by-layer decomposition without re-reading the issue thread.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@HongmingWang-Rabbit
HongmingWang-Rabbit added this pull request to the merge queue Apr 26, 2026
Merged via the queue into staging with commit dafe084 Apr 26, 2026
14 of 15 checks passed
HongmingWang-Rabbit added a commit that referenced this pull request Apr 26, 2026
@molecule-ai
molecule-ai Bot deleted the fix/staging-e2e-tls-timeout branch May 20, 2026 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant