feat(ops): sweep-cf-tunnels janitor — orphan tunnels accumulate forever - #2328
Merged
Merged
Conversation
…ccumulate
CP's tenant-delete cascade removes the DNS record (with sweep-cf-orphans
as a backstop) but does NOT delete the underlying Cloudflare Tunnel.
Each E2E provision creates one Tunnel named `tenant-<slug>`; without
cleanup these accumulate indefinitely on the account, consuming the
tunnel quota and cluttering the dashboard.
Observed 2026-04-30: dozens of `tenant-e2e-canvas-*` tunnels in Down
state with zero replicas, weeks past their tenant's deletion. Same
class of bug as the DNS-records leak that drove sweep-cf-orphans
(controlplane#239).
Parallel-shape to sweep-cf-orphans:
- Same dry-run-by-default + --execute pattern
- Same MAX_DELETE_PCT safety gate (default 90% — higher than DNS
sweep's 50% because tenant-shaped tunnels are orphans by design)
- Same schedule/dispatch hardening (hard-fail on missing secrets
when scheduled, soft-skip when dispatched)
- Cron offset to :45 to avoid CF API bursts colliding with the DNS
sweep at :15
Decision rules (in order):
1. Name doesn't match `tenant-<slug>` → keep (unknown — never sweep
tunnels that might belong to platform infra).
2. Tunnel has active connections (status=healthy or non-empty
connections array) → keep (defense-in-depth: don't kill a live
tunnel even if CP forgot the org).
3. Slug ∈ {prod_slugs ∪ staging_slugs} → keep.
4. Otherwise → delete (orphan).
Verified by:
- shell syntax check (bash -n)
- YAML lint
- Decide-logic offline smoke (7 cases, all pass)
- End-to-end dry-run smoke with stubbed CP + CF APIs
Required secrets (added to existing org-secrets):
CF_API_TOKEN must include account:cloudflare_tunnel:edit
scope (separate from zone:dns:edit used by
sweep-cf-orphans — same token if scope is
broad, or a new token if narrowly scoped).
CF_ACCOUNT_ID account that owns the tunnels (visible in
dash.cloudflare.com URL path).
CP_PROD_ADMIN_TOKEN reused from sweep-cf-orphans.
CP_STAGING_ADMIN_TOKEN reused from sweep-cf-orphans.
Note: CP-side root cause (tenant-delete should cascade to tunnel
delete) is in molecule-controlplane and worth fixing separately. This
janitor is the operational backstop in the meantime — same pattern
applied to DNS records when the same root cause was unaddressed.
HongmingWang-Rabbit
requested a review
from hongmingwang-moleculeai
as a code owner
April 30, 2026 02:43
HongmingWang-Rabbit
enabled auto-merge
April 30, 2026 02:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CP's tenant-delete cascade removes the DNS record but does not delete the underlying Cloudflare Tunnel. Tunnels named
tenant-<slug>accumulate indefinitely.Same class of leak as the DNS-records issue that drove
sweep-cf-orphans(controlplane#239). This PR adds the parallel-shape janitor for Tunnels.Observed leak
User screenshot of CF dashboard shows dozens of
tenant-e2e-canvas-20260...tunnels in Down state with 0 replicas, surviving weeks past their tenant's deletion.Decision rules
tenant-<slug>Why a separate workflow vs extending sweep-cf-orphans
/zones/<id>/dns_records)/accounts/<id>/cfd_tunnel)CF_API_TOKENmay not includeaccount:cloudflare_tunnel:editVerified
bash -nshell syntaxRequired secrets
Adds these to the existing repo-secrets set:
CF_API_TOKENaccount:cloudflare_tunnel:editscope (separate from thezone:dns:editused by sweep-cf-orphans — same token if scope is broad, or a new token if narrowly scoped)CF_ACCOUNT_IDCP_PROD_ADMIN_TOKENCP_STAGING_ADMIN_TOKENIf the secrets aren't set, the scheduled run fails-loud (red CI) instead of silent-skipping — same hardening as PR #2243 applied to sweep-cf-orphans after the 152/200 zone-records silent-leak incident.
Test plan
gh workflow run sweep-cf-tunnels.yml -f dry_run=trueto confirm secret presence + dry-run outputdry_run=falseto delete the existing orphan tunnels visible in the dashboardRelated
🤖 Generated with Claude Code