feat(user): run GDPR deletion as a durable admin queue

[eshurakov]

Summary

Replace one-shot Cloud GDPR deletion with a durable admin deletion queue.

• Disable matching Cloud accounts immediately on enqueue.
• Persist a deletion-task catalog in PostgreSQL and drive teardown through cron retries.
• Anonymize only after required tasks succeed or an admin marks a stuck task done.
• Treat deletion-in-progress as gone for owner matching, and keep HMAC/encrypted audit subjects from re-enqueuing already-covered resources.
• Scrub usage prompt prefixes as a paginated queue step so large usage histories cannot timeout deletion.
• Add /admin/deletion-queue for intake, retry, and mark-done.
• Signup/account-link still block on a real in-flight deletion, but unexpected fence lookup errors fail open and are reported to Sentry.

PostHog, Substack, and Pylon remain CSA-handled for this first cut. Add-requests is hidden in the queue UI; start deletion from the user profile or CSA API.

Test plan

• Enqueue a known Cloud user from the user admin GDPR form and confirm the request appears on /admin/deletion-queue.
• Confirm the request persists with the v1 catalog and usage-prompt cleanup is a teardown step.
• Run process-user-deletions until teardown succeeds and anonymize runs only after required tasks complete.
• Retry a stuck provider task from the admin UI; mark-done still requires evidence.
• Cancel a pending request; the email is redacted and the summary count updates after invalidate.
• Soft-delete still pages usage prompt prefixes before anonymizing.

Local E2E on http://localhost:6200/ (offset 3200):

• Fake-admin login can open /admin/deletion-queue. Start deletion from user Admin Tools (handbook checkbox → confirm → queued).
• Queue list, email filter, detail sheet, dedicated request page, cancel, and re-queue after cancel all work.
• GDPR confirm no longer hydrates a <div> inside <p>. Cancel/retry now invalidate the queue summary.
• Existing-user re-login is not fenced; the fence only blocks new signup / account link. Unexpected fence errors fail open.
• Chunked usage: 2500 prompt-prefix rows, batch 1000 → continue, continue, succeeded, then anonymize.
• Chunked CLI sessions: 25 cli_sessions_v2 rows, batch 10. Invalid IDs (ses_chunk_001) returned http_400 because ingest requires ses_ + 26 chars. Valid IDs + admin retry → continue, continue, succeeded.
• Customer.io succeeded locally. KiloClaw and CLI v1 were not applicable on these users.

Not covered locally: PostHog/Substack/Pylon (CSA).

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (2 files)

• apps/web/src/lib/user/deletion-queue/deletion-preflight.ts
• apps/web/src/lib/user/deletion-queue/deletion-outcomes.test.ts

Previous Review Summaries (7 snapshots, latest commit c33e7b4)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit c33e7b4)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

Self-service performGdprRemoval now queues with allowSelf, but preflight still treats the requester as a protected self-delete and parks the request, so requestAccountDeletion reports success while the account stays live.

Overview

Severity	Count
CRITICAL	1
WARNING	0
SUGGESTION	0

Issue Details (click to expand)

CRITICAL

File	Line	Issue
apps/web/src/lib/user/gdpr-removal.ts	28	allowSelf is not persisted; preflight re-runs classifyProtectedIdentity without it and parks the request as protected_self

Files Reviewed (7 files)

• apps/web/src/lib/user/gdpr-removal.ts — 1 CRITICAL
• apps/web/src/lib/user/deletion-queue/deletion-enqueue.ts — allowSelf passed only at enqueue; no issues in the insert path itself
• apps/web/src/lib/user/deletion-queue/deletion-intake.ts — self-guard skip is enqueue-only; no issues
• apps/web/src/lib/user/deletion-queue/deletion-intake.test.ts — covers classify allowSelf; does not cover preflight
• apps/web/src/lib/user/index.ts — assertNoLiveSubscriptionsForSoftDelete re-export only
• apps/web/src/lib/user/deletion-queue/handlers/kiloclaw.ts — previous post-mark continueIfLowTime CRITICAL is fixed
• apps/web/src/lib/user/deletion-queue/deletion-catalog.ts / deletion-outcomes.ts — anonymize manual verification disabled as intended

Fix these issues in Kilo Cloud

Previous review (commit 872f5f7)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

Increment 4e1c77775..872f5f711 fail-opens unexpected deletion-fence errors and is sound, but the prior post-mark time-budget check in handleKiloclawDestroy can still hide an instance after destroyed_at is set and report teardown success without calling client.destroy().

Overview

Severity	Count
CRITICAL	1
WARNING	0
SUGGESTION	0

Issue Details (click to expand)

CRITICAL

File	Line	Issue
apps/web/src/lib/user/deletion-queue/handlers/kiloclaw.ts	77	continueIfLowTime after markActiveInstanceBatchDestroyedForGdpr skips destroy/restore; next run never sees the instance (destroyed_at IS NULL filter) and can mark the step succeeded

Files Reviewed (6 files)

• apps/web/src/lib/user/deletion-queue/deletion-identity-fence.ts — fail-open wrapper reports unexpected errors to Sentry and still blocks ActiveDeletionFenceError; no issues
• apps/web/src/lib/user/index.ts — signup/link now use authPassesDeletionFence; no issues
• apps/web/src/lib/user/deletion-queue/deletion-identity-fence.test.ts — covers block and unexpected-error paths; no issues
• apps/web/src/app/admin/components/UserAdmin/UserAdminGdprRemoval.tsx — DialogDescription asChild hydration fix; no issues
• apps/web/src/app/admin/deletion-queue/[id]/DeletionQueueDetailContent.tsx — summary query invalidation; no issues
• apps/web/src/lib/user/deletion-queue/handlers/kiloclaw.ts — previous CRITICAL re-verified unchanged at HEAD

Fix these issues in Kilo Cloud

Previous review (commit 4e1c777)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

Increment 923a1f494..4e1c77775 removes the blocking kilocode_users index and HMAC key-versioning, but the new post-mark time-budget check in handleKiloclawDestroy can hide an instance after destroyed_at is set and then report teardown success without calling client.destroy().

Overview

Severity	Count
CRITICAL	1
WARNING	0
SUGGESTION	0

Issue Details (click to expand)

CRITICAL

File	Line	Issue
apps/web/src/lib/user/deletion-queue/handlers/kiloclaw.ts	77	continueIfLowTime after markActiveInstanceBatchDestroyedForGdpr skips destroy/restore; next run never sees the instance (destroyed_at IS NULL filter) and can mark the step succeeded

Files Reviewed (58 files)

• packages/db/src/schema.ts, schema-types.ts, schema.test.ts, 0217_slimy_black_crow.sql — previous blocking IDX_kilocode_users_lower_google_user_email removed; paused_at / HMAC key-version columns dropped from unshipped tables; no issues
• apps/web/src/lib/user/deletion-queue/handlers/kiloclaw.ts — 1 CRITICAL
• apps/web/src/lib/user/deletion-queue/deletion-outcomes.ts, deletion-worker.ts, deletion-task-runner.ts, deletion-preflight.ts, deletion-completion.ts — anonymize now runs in persist; pause path removed; no new issues
• apps/web/src/lib/user/deletion-queue/deletion-enqueue.ts, deletion-hmac.ts, deletion-identity-fence.ts, deletion-preview.ts, deletion-audit.ts — single-key HMAC consistent; no issues
• apps/web/src/lib/user/deletion-queue/handlers/anonymize.ts, cli-v1.ts, common.ts — handler timeout/abort wiring; no issues
• apps/web/src/routers/admin/user-deletion-queue-router.ts, admin deletion-queue UI, cron route — pause API/UI removed; cron now returns 500 on worker failure; no issues
• apps/web/src/lib/config.server.ts, ENVIRONMENT.md, .env.local.example, dev/local/setup-env.ts — HMAC env renamed to USER_DELETION_AUDIT_HMAC_KEY; no issues
• Remaining incremental tests and generated migration meta — no issues

Fix these issues in Kilo Cloud

Previous review (commit 923a1f4)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

Increment 4bbbc19ba..923a1f494 introduces a non-concurrent CREATE INDEX on the hot kilocode_users table in migration 0217_tidy_vulcan.sql that will block writes during deploy; the surrounding rework (lease removal, claim-token claims, effects-table retirement, pylon ticket dedupe, progress-json checkpointing) checks out and all six previous fix claims verified as genuine.

Overview

Severity	Count
CRITICAL	1
WARNING	1
SUGGESTION	1

Issue Details (click to expand)

CRITICAL

File	Line	Issue
packages/db/src/migrations/0217_tidy_vulcan.sql	104	New non-concurrent CREATE INDEX on populated kilocode_users blocks all writes during the build; add .concurrently() in schema.ts and regenerate in its own migration

WARNING

File	Line	Issue
apps/web/src/app/admin/api/users/gdpr-removal/route.ts	27	Carried, still applies unchanged at HEAD: unvalidated requestId query param reaches the DB uuid cast unguarded → unhandled 500 (author: admin-only path, leaving as-is)

SUGGESTION

File	Line	Issue
apps/web/src/routers/admin/user-deletion-queue-router.ts	143	Carried, still applies unchanged at HEAD: decodeCursor validates the cursor id as UUID but never validates createdAt; a tampered cursor yields a DB cast error → tRPC 500 (author: parameterized, leaving as-is)

Resolved since last review (verified at HEAD)

• deletion-enqueue.ts unique-violation recovery — fixed and complete: email-HMAC, pylon-ticket, and user_id branches all recover correctly
• deletion-outcomes.ts lock-order inversion — fixed: request row locked before step row, consistent with deletion-completion.ts
• deletion-worker.ts wrong-request fill probe — fixed: now uses wave[0].requestId
• deletion-worker.ts swallowed errors — fixed: captureException with request/step context before continuing
• handlers/kiloclaw.ts transient misclassification — fixed: ownership/empty-batch → needs_attention, all other errors → retry
• handlers/posthog.ts stale reserved checkpoint — fixed: recomputed person IDs are now unconditionally persisted via progress_json on re-run
• deletion-enqueue.test.ts misnamed test — addressed: renamed, plus new pylon-ticket already_active coverage that matches the implementation
• ENVIRONMENT.md / .env.local.example env-catalog and SUPPORT_API_SECRET wording — updated as the author described

Files Reviewed This Increment (45 files)

• packages/db/src/migrations/0217_tidy_vulcan.sql, meta/0217_snapshot.json, meta/_journal.json — dropped pre-merge 0217 tables/effects cleanly; 1 CRITICAL (non-concurrent index on existing table)
• packages/db/src/schema.ts, schema-types.ts, schema.test.ts, user-soft-delete.ts, user-soft-delete-reasons.ts (new), packages/db/package.json — consistent with migration; no dangling references
• apps/web/src/lib/user/deletion-queue/deletion-enqueue.ts, deletion-intake.ts, deletion-types.ts, deletion-constants.ts, deletion-hints.ts, deletion-crypto.ts, deletion-hmac.ts, deletion-identity-fence.ts, deletion-audit.ts — ticket dedupe + recovery ordering correct; no issues
• apps/web/src/lib/user/deletion-queue/deletion-worker.ts, deletion-task-selector.ts, deletion-task-runner.ts, deletion-outcomes.ts, deletion-completion.ts, deletion-preview.ts — lease-free claim/lock ordering sound; no issues
• apps/web/src/lib/user/deletion-queue/handlers/posthog.ts, pylon-reply.ts, pylon-contact.ts (+ tests posthog.test.ts, pylon-reply.test.ts) — idempotent progress-json state machines; no issues
• apps/web/src/app/api/cron/process-user-deletions/route.ts (+test), apps/web/src/lib/config.server.ts, apps/web/src/lib/user/deletion.ts, ENVIRONMENT.md — no issues
• apps/web/src/app/admin/components/UserAdmin/UserAdminGdprRemoval.tsx, UserAdminNotes.tsx, apps/web/src/app/admin/deletion-queue/DeletionQueueContent.tsx, [id]/DeletionQueueDetailContent.tsx — no issues
• apps/web/src/routers/admin/user-deletion-queue-router.ts — carried SUGGESTION re-verified at HEAD (line 143)
• apps/web/src/lib/user/deletion-queue/deletion-enqueue.test.ts, deletion-intake.test.ts, deletion-outcomes.test.ts, deletion-preview.test.ts, deletion-worker.test.ts — new tests exercise real code paths; no issues

Fix these issues in Kilo Cloud

Previous review (commit 4bbbc19)

Status: 4 Issues Found | Recommendation: Address before merge

Executive Summary

Increment 17fa544a0..4bbbc19ba is a pure rebase onto latest main (now including #5300 v2 shared-session JWT access): the PR's own diff is byte-identical to the previously reviewed revision apart from hunk offsets and one context line, no new issues were introduced, and the four previously reported findings remain open on unchanged code (each has an author reply with fix or rationale).

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	2

Issue Details (click to expand)

WARNING

File	Line	Issue
apps/web/src/lib/user/deletion-queue/handlers/posthog.ts	156	Recomputed PostHog person IDs are not persisted when a reserved effect exists, so the checkpoint recorded at reservation time can go stale (author: PostHog is CSA-handled, not in the live catalog)
apps/web/src/app/admin/api/users/gdpr-removal/route.ts	27	Unvalidated requestId query param reaches the DB uuid cast unguarded → unhandled 500 instead of 400 (author: admin-only path, leaving as-is)

SUGGESTION

File	Line	Issue
apps/web/src/lib/user/deletion-queue/deletion-enqueue.test.ts	115	Renamed test still exercises only the in-transaction fast path; the unique-violation user_id recovery branch remains unexercised (author: unreachable in a sequential test)
apps/web/src/routers/admin/user-deletion-queue-router.ts	142	decodeCursor validates the cursor id as UUID but never validates createdAt; a tampered cursor yields a DB cast error → tRPC 500 (author: leaving as-is)

Files Reviewed This Increment (7 files)

• services/session-ingest/src/app.ts — rebase-only context shift; PR content identical to previously reviewed revision; no issues
• services/session-ingest/src/routes/api.ts — rebase-only context shift; no issues
• services/session-ingest/src/routes/api.test.ts — rebase-only; the single differing line is a context line rewritten by main's fix(sessions): re-enable v2 shared session access #5300; no issues
• apps/web/src/lib/user/deletion-queue/handlers/posthog.ts — carried WARNING re-verified unchanged at HEAD — 1 issue
• apps/web/src/app/admin/api/users/gdpr-removal/route.ts — carried WARNING re-verified unchanged at HEAD — 1 issue
• apps/web/src/lib/user/deletion-queue/deletion-enqueue.test.ts — carried SUGGESTION re-verified unchanged at HEAD — 1 issue
• apps/web/src/routers/admin/user-deletion-queue-router.ts — carried SUGGESTION re-verified unchanged at HEAD — 1 issue

Fix these issues in Kilo Cloud

Previous review (commit 17fa544)

Status: 4 Issues Found | Recommendation: Address before merge

Executive Summary

Increment 50937bd40..17fa544a0 is a rebase onto latest main (migration renumbered 0216_thick_sasquatch → 0217_youthful_mongoose, otherwise byte-identical SQL) plus one substantive change: anonymizeCloudUserData now also deletes operation_ledgers and analytics_event_outbox rows, with new regression tests; no new issues were introduced, and the four previously reported findings remain open on unchanged code.

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	2

Issue Details (click to expand)

WARNING

File	Line	Issue
apps/web/src/lib/user/deletion-queue/handlers/posthog.ts	156	Recomputed PostHog person IDs are not persisted when a reserved effect exists, so the checkpoint recorded at reservation time can go stale
apps/web/src/app/admin/api/users/gdpr-removal/route.ts	27	Unvalidated requestId query param reaches the DB uuid cast unguarded (no try/catch in GET) → unhandled 500 instead of 400

SUGGESTION

File	Line	Issue
apps/web/src/lib/user/deletion-queue/deletion-enqueue.test.ts	115	The renamed test still exercises the in-transaction fast path; the unique-violation user_id recovery branch (catch at deletion-enqueue.ts:207) is never exercised
apps/web/src/routers/admin/user-deletion-queue-router.ts	142	decodeCursor validates the cursor id as UUID but never validates createdAt; a tampered cursor yields a DB cast error → tRPC 500

Files Reviewed This Increment (6 files)

• apps/web/src/lib/user/index.ts — anonymizeCloudUserData now deletes operation_ledgers (by kilo_user_id) and analytics_event_outbox (by distinct_id in [originalEmail, userId]); originalEmail is captured before the user row is anonymized and is notNull in the schema, so inArray is safe; no issues
• apps/web/src/lib/user/index.test.ts — new coverage for blocked-user sign-in sync, blocked auto-link, soft-delete precondition leaving usage metadata intact, deletion-queue rows not mutated, and suite cleanup; no issues
• apps/web/vercel.json — rebase context shift only; the process-user-deletions cron entry is unchanged; no issues
• packages/db/src/migrations/0217_youthful_mongoose.sql — byte-identical to previously reviewed 0216_thick_sasquatch.sql, renumbered after rebase; no issues
• packages/db/src/migrations/meta/0217_snapshot.json — generated snapshot, out of review scope
• packages/db/src/migrations/meta/_journal.json — generated journal entry for 0217; out of review scope

Fix these issues in Kilo Cloud

Previous review (commit 50937bd)

Status: 4 Issues Found | Recommendation: Address before merge

Executive Summary

Increment 74c73a908..50937bd40 resolves five previous findings (support requestId UUID validation, retry-route removal, hasDueDeletionWork deletion, SUBSTACK_PUBLICATION_URL docs, and the SUPPORT_API_SECRET leak warning) and introduces no new issues; the PostHog checkpoint persistence warning, the unvalidated admin requestId query param, and two related suggestions remain open on unchanged code.

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	2

Issue Details (click to expand)

WARNING

File	Line	Issue
apps/web/src/lib/user/deletion-queue/handlers/posthog.ts	156	Recomputed PostHog person IDs are not persisted when a reserved effect exists, so the checkpoint recorded at reservation time can go stale
apps/web/src/app/admin/api/users/gdpr-removal/route.ts	27	Unvalidated requestId query param reaches the DB uuid cast unguarded (no try/catch in GET) → unhandled 500 instead of 400

SUGGESTION

File	Line	Issue
apps/web/src/lib/user/deletion-queue/deletion-enqueue.test.ts	115	The renamed test still exercises the in-transaction fast path; the unique-violation user_id recovery branch (catch at deletion-enqueue.ts:207) is never exercised
apps/web/src/routers/admin/user-deletion-queue-router.ts	142	decodeCursor validates the cursor id as UUID but never validates createdAt; a tampered cursor yields a DB cast error → tRPC 500

Files Reviewed This Increment (8 files)

• .env.local.example — restored leak-impact warning above SUPPORT_API_SECRET; advisory comment only, no issues
• ENVIRONMENT.md — documents SUBSTACK_PUBLICATION_URL (verified real usage in deletion-substack-credential.ts / handlers/substack.ts) and expands SUPPORT_API_SECRET leak impact; no issues
• apps/web/src/app/admin/api/users/deletion-steps/retry/route.ts — removed; retry behavior still covered by retryAttentionTask/retryBlockedPreflight in the tRPC admin router; DeletionStepRetryResponse has no remaining importers
• apps/web/src/app/admin/api/users/deletion-steps/retry/route.test.ts — removed with the route; no issues
• apps/web/src/app/api/internal/support/users/gdpr-removal/route.ts — GET now validates requestId as UUID and returns 400 pre-DB; resolves the previous malformed-requestId 500 finding; no issues
• apps/web/src/app/api/internal/support/users/gdpr-removal/route.test.ts — updated expectations plus a new non-UUID case; fixtures remain consistent with RequestIdSchema header semantics; no issues
• apps/web/src/lib/user/deletion-queue/deletion-enqueue.ts — race-condition comment on unique-violation recovery only; no issues
• apps/web/src/lib/user/deletion-queue/deletion-request-selector.ts — hasDueDeletionWork deleted; no remaining references and no orphaned imports; no issues

Fix these issues in Kilo Cloud

---

_{Reviewed by grok-4.6 · Input: 110.3K · Output: 9.1K · Cached: 398.7K}

_{Review guidance: REVIEW.md from base branch main}

[pandemicsyn]

two findings from my local robot inline. Both sound very edge cases (especially the pause race) , so feel free to ignore.

[pandemicsyn]

one inline, seems very edge case though. If a user logs in via work os after having requested deletion at the exact time the deletion happens ? I'm not sure its worth the effort.

[pandemicsyn]

bot: This fence protects the normal linking path, but createOrUpdateUser has a separate same-email WorkOS branch that deletes/reinserts user_auth_provider and rotates web_session_pepper without calling linkAccountToExistingUser or authPassesDeletionFence. An active deletion request can therefore still have its auth identity mutated through a WorkOS login. Please apply the same fence in that transaction (or route WorkOS through this helper) and add an active-deletion + WorkOS auto-link regression test.