Skip to content

ops(#1976): smart-sweep script for orphan Cloudflare DNS records - #1978

Merged
molecule-ai[bot] merged 2 commits into
stagingfrom
ops/smart-cf-sweep-script
Apr 24, 2026
Merged

ops(#1976): smart-sweep script for orphan Cloudflare DNS records#1978
molecule-ai[bot] merged 2 commits into
stagingfrom
ops/smart-cf-sweep-script

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

What

A replacement for the "panic-button at >65 records" manual sweep. The old script nuked every pattern-match unconditionally, which would have deleted live workspaces along with orphans.

This version queries CP + AWS to build a live-set, and only deletes records whose slug/ws-id has no live counterpart.

Why

Session monitoring on 2026-04-24 showed CF DNS records growing at ~10/hour (29 → 65 in 3h) and the existing sweep was unsafe to run without manual review. Issue #1976 filed for the root-cause leak (tenant.Delete + workspace.Delete not cleaning CF records). Until #1976 lands, this is the maintenance tool.

How

  1. `GET /cp/admin/orgs` on prod + staging — live tenant slugs
  2. `aws ec2 describe-instances --filters state=running,pending` — live workspace Name tags
  3. `GET /zones//dns_records` — all CF records
  4. For each record, apply 5 rules in priority order:
    • Platform-core (apex, api, app, etc.) → keep
    • `ws-.(staging.)moleculesai.app` → keep if matching EC2 exists
    • `e2e-.staging.moleculesai.app` → keep if slug in live orgs
    • `.(staging.)moleculesai.app` → keep if slug in live orgs
    • Anything we don't understand → keep (never delete-by-default)
  5. Safety gate: refuses to delete >50% of records

Dry-run output (validated 2026-04-24 04:10Z)

```
[04:10] Fetching CP prod org slugs... prod orgs: 2
[04:10] Fetching CP staging org slugs... staging orgs: 3
[04:10] Fetching live EC2 Name tags... live EC2s: 11
[04:10] Fetching Cloudflare DNS records... CF records: 65

== Sweep plan ==
total CF records: 65
would delete: 30
would keep: 35

delete/orphan-e2e-tenant: 17
delete/orphan-ws: 13

Dry run complete. Pass --execute to actually delete 30 records.
```

Verified both Chloe's (ws-71f855ba) and CEO's (ws-e994e287) live workspaces are not in the delete list.

Follow-up

Related

Replaces the "panic-button at >65 records" manual sweep that nukes
every pattern-match unconditionally (would delete live workspaces
along with orphans).

This version:
- Queries CP prod + staging /admin/orgs for live tenant slugs
- Queries AWS EC2 describe-instances for live workspace Name tags
- Only deletes CF records whose slug/ws-id has no live counterpart
- Dry-run by default (--execute to actually delete)
- Safety gate refuses to delete >50% of records (configurable via
  MAX_DELETE_PCT env var) — catches the "API returned zero orgs, every
  tenant looks orphan" failure mode before it nukes production
- Per-category accounting: orphan-ws / orphan-e2e-tenant / etc.

Usage:
  CF_API_TOKEN=... CF_ZONE_ID=... \
    CP_PROD_ADMIN_TOKEN=... CP_STAGING_ADMIN_TOKEN=... \
    bash scripts/ops/sweep-cf-orphans.sh           # dry-run
  bash scripts/ops/sweep-cf-orphans.sh --execute   # actually delete

Ref: #1976 (root-cause: tenant.Delete + workspace.Delete don't clean
their CF records — until that's fixed, this script is the maintenance
path)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@molecule-ai
molecule-ai Bot enabled auto-merge (squash) April 24, 2026 04:17
@molecule-ai
molecule-ai Bot merged commit 0576e34 into staging Apr 24, 2026
13 checks passed
molecule-ai Bot added a commit that referenced this pull request Apr 24, 2026
…1978)

Replaces the "panic-button at >65 records" manual sweep that nukes
every pattern-match unconditionally (would delete live workspaces
along with orphans).

This version:
- Queries CP prod + staging /admin/orgs for live tenant slugs
- Queries AWS EC2 describe-instances for live workspace Name tags
- Only deletes CF records whose slug/ws-id has no live counterpart
- Dry-run by default (--execute to actually delete)
- Safety gate refuses to delete >50% of records (configurable via
  MAX_DELETE_PCT env var) — catches the "API returned zero orgs, every
  tenant looks orphan" failure mode before it nukes production
- Per-category accounting: orphan-ws / orphan-e2e-tenant / etc.

Usage:
  CF_API_TOKEN=... CF_ZONE_ID=... \
    CP_PROD_ADMIN_TOKEN=... CP_STAGING_ADMIN_TOKEN=... \
    bash scripts/ops/sweep-cf-orphans.sh           # dry-run
  bash scripts/ops/sweep-cf-orphans.sh --execute   # actually delete

Ref: #1976 (root-cause: tenant.Delete + workspace.Delete don't clean
their CF records — until that's fixed, this script is the maintenance
path)

Co-authored-by: Hongming Wang <hongmingwang.rabbit@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: molecule-ai[bot] <276602405+molecule-ai[bot]@users.noreply.github.com>
HongmingWang-Rabbit added a commit that referenced this pull request Jun 12, 2026
…coded billing precedence (#1978 review)

agent-reviewer #7790 (blocking) found that ConfigTab.registryBilling.test.tsx
did not actually pin retire-list #5's core claim — both existing assertions
("platform"→platform_managed, "anthropic-oauth"→byok) return the SAME value
under both the registry-authoritative impl and a regression to the old
hardcoded billingModeForProvider rule, so the test was tautological and a
regression would still pass. The misleading comment on the anthropic-oauth
case claimed it was "a case the hardcoded rule gets WRONG" but the hardcoded
rule actually agrees there too.

This commit adds a genuine disagreement case: a registry provider
"managed-federated" whose registry-served billing_mode is "platform_managed"
even though its name is not "" / "platform" (so the legacy
billingModeForProvider rule would return "byok"). The new test asserts the
two rules disagree on this input (sanity) and then asserts
billingModeForSelectedProvider returns the REGISTRY value
("platform_managed"), which is only reachable by honoring the catalog.

Load-bearing proof: with the registry-first impl, the new test PASSES; when
billingModeForSelectedProvider is temporarily forced to fall through to the
hardcoded rule, the new test (and only the new test) FAILS with
expected 'platform_managed' / received 'byok' — proving it pins the
registry-wins contract.

Also fixes the misleading "hardcoded rule gets WRONG" comment on the
anthropic-oauth case (explicitly annotates it as non-discriminating and
points to the new disagreement case as the registry-WINS proof).

Implementation (billingModeForSelectedProvider) untouched — confirmed
byte-identical to PR #1978 HEAD (f2d7f1da).

Verification:
  - targeted: 5 passed (was 4 — adds the discriminating case)
  - regressed-impl: only the new test fails, others pass (= they are
    non-discriminating as the review found)
  - full canvas vitest: 223 files / 3381 passed | 1 skipped (3382) — +1
    vs the 3380/1 baseline
  - tsc: 0 new errors (touched file clean; pre-existing 223 baseline
    unchanged with my diff stashed)
  - eslint on touched file: 0

Refs: #1978, review #7790, internal#718 P3 retire-list #5.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…/templates, retire hardcoded provider vocab #4/#5 (PR-B; NOT merged)' (#1978) from feat/internal-718-p3b-canvas-consume-registry into main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant