feat(exports): add queued user data exports - #5156
Conversation
| @@ -0,0 +1,326 @@ | |||
| import { getWorkerDb, pg } from '@kilocode/db/client'; | |||
There was a problem hiding this comment.
Moderate, Testing: This file contains the most intricate SQL in the PR (claim with lease and generation fencing, the attach CTE with the tombstone fallback, the multi CTE checkpoint, reconcile), and none of it runs against real Postgres in any test. worker.test.ts mocks the state object entirely, and the vitest-pool-workers test only covers HTTP auth and body limits. Meanwhile the web side does have a real database harness (the router and softDeleteUser tests use it).
These queries interact with CHECK constraints, partial unique indexes, and cascade rules defined in this same PR, and mocks cannot validate that interaction; the attach constraint violation flagged above is exactly the kind of failure this gap hides. The state machine here (queued to processing to finalizing to ready, lease takeover, generation fencing) is the correctness core of the feature.
Suggest an integration test that drives claim, attach, checkpoint, complete, and the reconcile sweeps against the real migrated schema, plus the deletion race (soft delete between attach and checkpoint). Reusing the web test database harness for these functions would be enough even if the R2 side stays mocked.
There was a problem hiding this comment.
Not implemented per project direction: test-only feedback was explicitly excluded from this feedback pass. Existing state-machine unit, runtime, migration bootstrap, and database-backed web tests remain in place.
75e1a3c to
7bc6950
Compare
|
|
||
| export async function POST(request: NextRequest): Promise<NextResponse> { | ||
| const secret = request.headers.get('x-internal-api-key'); | ||
| if (!INTERNAL_API_SECRET || !secret || !timingSafeEqual(secret, INTERNAL_API_SECRET)) { |
There was a problem hiding this comment.
Low, Testing: This auth guard has no test coverage. The Worker's equivalent x-internal-api-key guard is tested in test/worker.test.ts, but on the web side only delivery-state.ts is covered; nothing asserts this route returns 401 for a missing key, a wrong key, or an unset INTERNAL_API_SECRET. Since this endpoint triggers outbound email to users, it should get rejection and acceptance coverage in the same PR, consistent with how the other authorization guards in this feature are tested.
There was a problem hiding this comment.
Not implemented per project direction: test-only feedback was explicitly excluded from this feedback pass. The route uses the same timing-safe internal API-key contract as the tested Worker boundary.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Executive SummaryIncremental review of Overview
Issue Details (click to expand)WARNING
Files Reviewed (13 files in incremental diff)Incremental review of
Fix these issues in Kilo Cloud Previous Review Summaries (2 snapshots, latest commit c7d3cfa)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit c7d3cfa)Status: 8 Issues Found | Recommendation: Address before merge Executive SummaryThe export pipeline's keyset pagination truncates Incremental update at Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (2 files in incremental diff)Incremental review of
Unchanged files carrying still-valid prior findings: Fix these issues in Kilo Cloud Previous review (commit 7bc6950)Status: 10 Issues Found | Recommendation: Address before merge Executive SummaryThe export pipeline's keyset pagination truncates Overview
Prior-review reconciliation: 3 of the 4 existing findings are fixed at HEAD (the backwards Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (47 files)
Reviewed by kimi-k3 · Input: 58.7K · Output: 17.8K · Cached: 704.8K Review guidance: REVIEW.md from base branch |
2 participants
Summary
Adds the Phase 1 foundation for Kilo-admin user data exports:
/data-exportsaccount page for requesting exports, monitoring status, and downloading completed artifacts.2026-08-03T00:00:00.000Z; records created after that timestamp are excluded.x-internal-api-key/INTERNAL_API_SECRETcontract for web-to-Worker and Worker-to-web communication.pnpm dev:start data-exportwith worktree-aware ports, generated local URLs, both Hyperdrive overrides, and the shared internal API key.wrangler types --include-runtime=falsegeneration setup.flowchart TB Admin["Kilo admin browser"] subgraph Web["Kilo Web trust boundary"] Page["/data-exports<br/>server page: adminOnly"] TRPC["userExports router<br/>adminProcedure + ctx.user.id"] Notify["Ready-email endpoint<br/>timing-safe internal key check"] end subgraph Worker["Export Worker trust boundary"] Internal["user-data-export.kilosessions.ai<br/>timing-safe x-internal-api-key"] Consumer["Queue consumer<br/>lease + generation fencing"] Signer["Download signer<br/>re-checks exportId + kiloUserId<br/>ready + unexpired"] end subgraph Data["Private data plane"] Primary[("PostgreSQL primary<br/>existing shared Hyperdrive<br/>job state, outbox, object key")] Replica[("Read-only replica role<br/>matched Kilo user row<br/>and approved source tables only")] Queue["Cloudflare Queue + DLQ<br/>IDs/generation only; no prompts"] Bucket[("Private R2 bucket<br/>no public domain")] Tombstone[("Deletion tombstones<br/>survive user/export cascade")] end Mailgun["Mailgun transactional email"] Signed["5-minute presigned GET<br/>attachment; private, no-store"] Admin -->|"authenticated session"| Page --> TRPC TRPC -->|"rows scoped by ctx.user.id"| Primary TRPC -->|"x-internal-api-key"| Internal Internal --> Queue Queue --> Consumer Consumer -->|"SELECT only; user predicate;<br/>cutoff <= 2026-08-03 UTC"| Replica Consumer -->|"state/checkpoints"| Primary Consumer -->|"multipart .jsonl.gz"| Bucket Consumer -->|"x-internal-api-key; exportId only"| Notify --> Mailgun TRPC -->|"exportId + ctx.user.id"| Signer Signer -->|"ownership/readiness lookup"| Primary Signer -->|"sign exact stored key"| Signed --> Admin Primary -->|"account deletion copies key + upload ID"| Tombstone Consumer -->|"abort multipart + delete object"| Tombstone classDef secure fill:#103b2d,stroke:#55d68b,color:#fff classDef private fill:#17233d,stroke:#73a7ff,color:#fff class TRPC,Notify,Internal,Consumer,Signer secure class Primary,Replica,Queue,Bucket privateflowchart LR Request["Admin request"] --> Admit["Single DB transaction<br/>create queued job and generation 0 outbox"] Admit --> Dispatch["Immediate Queue send<br/>plus scheduled outbox recovery"] Dispatch --> Delivery["At-least-once delivery<br/>exportId and generation only"] Delivery --> Claim{"Lease claim matches<br/>current generation?"} Claim -->|"No: stale, duplicate, or terminal"| Ack["Acknowledge safe no-op"] Claim -->|"Yes"| Read["Read bounded keyset pages<br/>for one user and fixed cutoff"] Claim --> Attach["Create multipart and CAS-persist upload ID<br/>abort immediately if row deleted or lease lost"] Attach --> Read Read --> Upload["Stream gzip to R2<br/>uniform multipart parts"] Upload --> Checkpoint["One DB transaction<br/>persist all ETags and cursor<br/>advance generation and outbox"] Checkpoint --> More{"Sources complete?"} More -->|"No"| Dispatch More -->|"Yes"| Final["Complete multipart<br/>HEAD exact object"] Final --> Ready["Lease-fenced ready transition<br/>store key, size, expiry"] Claim -.->|"expired lease"| Recover["Reconciler clears lease<br/>re-arms current outbox"] Recover --> Dispatch Delivery -.->|"configured retry limit"| Failed["DLQ consumer marks failed<br/>reconciler aborts multipart"] Claim -.->|"five consecutive claims expire"| Failed classDef durable fill:#17233d,stroke:#73a7ff,color:#fff classDef guarded fill:#103b2d,stroke:#55d68b,color:#fff class Admit,Dispatch,Checkpoint,Ready,Recover durable class Claim,Attach,Upload,Final guardedVerification
/data-exportsreturns 404 and the navigation item is absent.Visual Changes
Reviewer Notes
PRIMARY_STATE_DBreuses the existing shared production Hyperdrive; it does not provision or deploy Queues, R2, the read-replica Hyperdrive, standard Worker secrets, environment values, or production migrations.wrangler secret putsecrets; this Worker intentionally does not use Secrets Store.2026-08-03T00:00:00.000Z. No WAL replay gate is needed because the workload is intentionally historical and runs against four large read replicas.~/fd-plans/research/data-export.md.