diff --git a/docs/contributing/architecture/data-storage.md b/docs/contributing/architecture/data-storage.md index 3a5138935e..0038ac6a83 100644 --- a/docs/contributing/architecture/data-storage.md +++ b/docs/contributing/architecture/data-storage.md @@ -227,10 +227,12 @@ Durable Object export behavior: RPC; keyset pagination with prefixed cursors over those tables). Manifest counts use `countMailbox`. Phase 1 registers this consumption; phase 2 wires live dual-write for outbound terminals, provider delivery-queue graph repair, - user classification (`service.ts#setEmailMessageClassification`), and - high-risk inbound terminal paths without changing D1 authority — D1 `email_*` - rows remain the live source of truth and are still exported in the `d1` - section. See [Mailbox](#durable-objects-mailbox). + user classification (`service.ts#setEmailMessageClassification`), and user + message-graph paths. USER inbound delivery lifecycle/effect authority is now + Mailbox CAS; D1 `email_delivery_events` is its synchronous compatibility + mirror and graph-write fence. Other D1 `email_*` rows remain live graph + storage and are still exported in the `d1` section. See + [Mailbox](#durable-objects-mailbox). - `RemoteConnectorSession` exposes persisted connector metadata and tool descriptors through an export RPC. - `PackageServiceInstance` uses its status RPC as the stable persisted service @@ -488,17 +490,15 @@ Raw email MIME payloads live in the `EMAIL_BLOBS` R2 bucket instead of D1. `insertEmailMessage` puts the payload to `EMAIL_BLOBS` before the D1 insert and writes only `raw_mime_key` (never `raw_mime`). On R2 put failure the insert throws `EmailRawMimeStorageError` (a `RetryableInboundStorageError`; no D1 row). -The inbound Worker refunds the daily receive charge and rethrows only typed -pre-commit failures so Cloudflare Email Routing retries without burning quota. -The durable commit boundary is message + attachment rows: thread prework, R2 -put, and D1 message/attachment storage are pre-commit; `touchEmailThread` / -`received` delivery-event writes are post-commit and are logged without throwing -(retry would duplicate mail). If attachment insert fails but message cleanup -cannot remove the row — or the residual-row probe itself fails (ambiguous commit -state) — the handler acknowledges the already-created message (logged, -non-retry) rather than risking a duplicate. Outbound messages pass -`rawMime: null` and are unaffected. If D1 insert fails after a successful put, -the blob is best-effort deleted. +The inbound Worker rethrows typed pre-commit failures so Cloudflare Email +Routing retries; UserMeter delivery-id idempotency prevents a second charge. The +message-graph commit boundary is message + attachment rows: thread prework, R2 +put, and D1 message/attachment storage precede Mailbox `received` finalization. +If attachment insert fails but message cleanup cannot remove the row — or the +residual-row probe itself fails (ambiguous commit state) — the handler +acknowledges the already-created message (logged, non-retry) rather than risking +a duplicate. Outbound messages pass `rawMime: null` and are unaffected. If D1 +insert fails after a successful put, the blob is best-effort deleted. **Expand/contract Stage 4b1 (code-only):** the worker does not read or write transitional `email_messages.raw_mime` / `raw_mime_offload_blocked`, does not @@ -712,35 +712,126 @@ owner is also used to validate canonical owner-scoped R2 keys (`emailRawMimeKey` objects install the full DDL; warm v1 objects run `CREATE INDEX IF NOT EXISTS` only. No destructive ALTERs. -**Additive inbound ledger CAS (step 2a — no authority flip):** owner-bound -atomic RPCs in `mailbox-inbound-ledger.ts` / `mailbox-inbound-effect-ledger.ts` -mirror D1 USER transitions from `inbound-delivery.ts` / `inbound-effects.ts` -(get delivery + active fingerprint window; claim/rewrite dedupe; insert charged -pending with explicit `inserted`/`existed`; claim/release storage lease; mark -rejected/received with lease/finalization CAS; prune dedupe; defer reconcile; -claim/complete usage effect; claim/complete/fail subscription effect with -retry/dead-letter/suppression; list due stale/effect work). Mutations keep -promoted columns and `detail_json` in sync and set canonical `updated_at`. -Usage/subscription complete and subscription fail require an exact -`expectedFinalizationToken` match (`event_type`/`state` = `received`); mismatch -is `lease-lost`. Storage claim clears finalization plus in-flight effect -leases/retry (and resets `processing` → `pending`) so reclaim/re-finalization -cannot be completed by a stale effect worker — stronger than D1's `json_set` -patch, required because Mailbox promotes those lease columns. The DO does -**not** perform external usage recording or subscription dispatch. Mirror -`upsertDeliveryEvent` / `upsertDeliveryEvents` remain the compatibility write -API. **These CAS RPCs are not live-wired** — D1 stays sole write authority for -inbound ledger/effects. `system:email` stays on D1 and is never written into -per-user Mailbox objects. - -**Mirror write contract:** `mirrorMessage`, `upsertDeliveryEvent`, and -`upsertDeliveryEvents` take complete snapshots — every persisted field is -explicit (nullable fields use explicit `null`). They are not patch APIs. All -require `ownerId` and apply equal-or-newer `updatedAt` snapshots (stale -snapshots are ignored, not applied). `upsertDeliveryEvents` accepts at most -`mailboxUpsertDeliveryEventsMax` (100) events in one owner-bound batch RPC and -inserts them in caller order (chronological when loaded from D1). The only -omission exception is the `mirrorMessage` `attachments` bundle: omitting it +**USER inbound ledger authority (step 2b):** owner-bound atomic RPCs in +`mailbox-inbound-ledger.ts` / `mailbox-inbound-effect-ledger.ts` are the sole +authority for USER delivery/window/storage/rejection/receive/reconciliation and +effect transitions. Mutations keep promoted columns and `detail_json` in sync +and set canonical `updated_at`. Usage/subscription complete and subscription +fail require an exact `expectedFinalizationToken` match (`event_type`/`state` = +`received`); mismatch is `lease-lost`. Storage claim clears finalization plus +in-flight effect leases/retry (and resets `processing` → `pending`) so +reclaim/re-finalization cannot be completed by a stale effect worker. Cleanup +claim/release and orphan-cleaned tombstones are also owner-bound CAS. + +The DO does **not** perform external usage recording or subscription dispatch: +Workers claim in Mailbox, perform the D1 usage-rollup or package dispatch, then +complete/fail in Mailbox. `inbound-delivery-authority.ts` synchronously upserts +one full Mailbox snapshot into D1 `email_delivery_events` (promoted columns plus +`detail_json`, owner/provider fenced, monotonic and idempotent). Pending is +mirrored before delivery reads; storing is mirrored before D1 +thread/message/attachment fence predicates. Those fence-critical failures fail +closed. A rejected CAS is the deliberate exception: SMTP rejection remains +permanent when its D1 projection fails, and rejected terminal work read-repairs +the projection. Terminal/effect snapshots keep D1 global due-owner discovery +current, but D1 is never read back as ongoing authority. Dedupe pruning projects +only the exact bounded pointer IDs deleted by Mailbox, with D1 owner/provider +fences; it never runs a second independent expiry/limit selection. + +The only reverse path is the deployment bridge: when a USER point lookup misses +in Mailbox and a pre-deploy D1 row exists, one complete D1 snapshot bootstraps +the owner-bound DO row, after which transitions continue through Mailbox CAS. +Scheduled parity partitions those validated legacy lifecycle/dedupe snapshots to +the missing-only `bootstrapDeliveryEvents` RPC; pre-claim rejection audits and +non-inbound events continue through normal `upsertDeliveryEvents`. The bootstrap +RPC accepts at most 100 snapshots, validates owner/provider/detail coherence, +and reports inserted/existing/skipped counts. It never updates an existing +Mailbox row. Normal delivery-event upserts continue rejecting USER inbound +authority snapshots. Legacy lifecycle bootstrap preserves canonical +`reconcileAfter` and orphan-cleaned `cleanupRetryAt` schedules in Mailbox +columns/detail JSON so due work cannot run early; irrelevant dedupe/terminal +schedule columns remain null. A malformed schedule is skipped per row, allowing +the normal audit subset to commit; parity records a count mismatch instead of +treating the mixed page as an RPC failure. `system:email` stays on the existing +D1 implementation and cannot bootstrap a Mailbox. Migration +`0129-email-inbound-mailbox-authority-mirror.sql` is additive: it promotes +compatibility/fence fields and adds the cross-store usage effect idempotency +ledger; it drops no tables. Destructive follow-up work remains gated on the +verified backup whose SHA-256 starts with `7787f8c9`; this change is explicitly +non-destructive. + +**Accepted rollback → roll-forward caveat and manual repair:** a rollback to the +previous Worker can advance USER inbound state in legacy `detail_json` with +`json_set` without advancing `email_delivery_events.updated_at`. A later +roll-forward does not automatically detect that D1 is newer: the bootstrap is +missing-only, an existing Mailbox authority row wins, and the Mailbox → D1 +projection fence treats existing D1 `updated_at >=` the snapshot timestamp as +already current. Rollback-era D1 progress can therefore require operator repair +before redeploy. + +Use this exact owner-by-owner procedure; do **not** run ordinary Mailbox purge +casually: + +1. Gate the repair on the sealed, verified production backup whose D1 SQL + SHA-256 starts with `7787f8c9`. Verify the signed manifest and stored object, + not a copied checksum string. Stop if the prefix or restore drill evidence + does not match. +2. Put the app behind the approved maintenance controls, disable the affected + Cloudflare Email Routing ingress, and pause scheduled/queue consumers that + can write email. Keep the rollback Worker quiesced for the entire inspection, + purge, rebuild, and verification window. +3. For each affected `stable_user_id`, capture D1 `email_threads`, + `email_messages`, `email_attachments`, and `email_delivery_events` counts. + Inspect every inbound lifecycle/dedupe row's `detail_json`, promoted state, + effect/finalization fields, `created_at`, and `updated_at`; compare with + `Mailbox.exportMailbox`/`countMailbox`. Decide that D1 contains the desired + rollback-era progress from operator evidence. The code does not make this + decision. +4. Only after that owner passes inspection, invoke the existing owner-derived + `Mailbox.purge()` RPC through a reviewed production operator script/Worker + using `MAILBOX.idFromName(stable_user_id)`. This is the safe metadata-only + purge: it clears that owner's Mailbox SQLite/alarm state and does not delete + D1 or R2. Never substitute the normal retention/delete surfaces. +5. Reset only that owner's parity state in D1, preserving all `email_*` rows: + + ```sql + UPDATE users + SET mailbox_parity_checked_at = NULL, + mailbox_parity_matching_since = NULL, + mailbox_parity_mismatch_count = 0, + mailbox_parity_last_error = NULL, + mailbox_parity_content_watermark_at = NULL, + mailbox_parity_content_replay_upper_at = NULL, + mailbox_parity_content_replay_cursor_updated_at = NULL, + mailbox_parity_content_replay_cursor_id = NULL, + mailbox_parity_message_backfill_cursor_created_at = NULL, + mailbox_parity_message_backfill_cursor_id = NULL, + mailbox_parity_message_backfill_completed_at = NULL, + mailbox_parity_event_backfill_cursor_created_at = NULL, + mailbox_parity_event_backfill_cursor_id = NULL, + mailbox_parity_event_backfill_completed_at = NULL + WHERE stable_user_id = ? AND deleting_at IS NULL; + ``` + +6. Run `admin_mailbox_maintenance({ action: "reconcile", batch_size: 100 })` + until that owner completes a full D1 → Mailbox rebuild and exact count + compare. Re-export the Mailbox and verify per-row lifecycle state, dedupe + pointers, effect state/leases/retry/dead-letter fields, finalization tokens, + usage fields, and message/attachment counts against the inspected D1 source. + Require zero parity error/mismatch before continuing. +7. Redeploy the roll-forward Worker while writes remain quiesced. Re-run + owner/fleet status and a focused inbound canary, then resume queues, + schedules, Email Routing, and normal ingress in that order. + +**Mirror write contract:** `mirrorMessage`, `upsertDeliveryEvent`, +`upsertDeliveryEvents`, and `bootstrapDeliveryEvents` take complete snapshots — +every persisted field is explicit (nullable fields use explicit `null`). They +are not patch APIs. All require `ownerId`. Normal upserts apply equal-or-newer +`updatedAt` snapshots (stale snapshots are ignored) and reject USER inbound +lifecycle/dedupe authority rows. `bootstrapDeliveryEvents` is their explicit +missing-only exception: it validates legacy USER inbound snapshots and never +updates an existing ID. Both batch RPCs accept at most +`mailboxUpsertDeliveryEventsMax` (100) events and process in caller order. The +only omission exception is the `mirrorMessage` `attachments` bundle: omitting it preserves existing attachment rows; an explicit `attachments: []` clears them. Accepted mirrors validate inbound/outbound `rawMimeKey` and external attachment `storageKey` values against the canonical builders for that `ownerId`. @@ -799,38 +890,40 @@ batch bound without slowing live dual-write. Each returns a structured `MailboxMirrorResult`: `{ status: 'mirrored' }`, `{ status: 'stale' }`, `{ status: 'missing' }`, `{ status: 'timeout' }`, `{ status: 'skipped', reason }` (`system-email` | `mailbox-unconfigured` | -`missing-owner`), or `{ status: 'error', error }`. Single-RPC helpers record one -`mailbox_mirror:` outcome automatically when a user id is known; -`mirrorMailboxDeliveryEventSnapshots` records one -`mailbox_mirror:upsert_delivery_event_batch` outcome per batch (timeout/error -apply uniformly to per-event summary entries). `system:email` is excluded. -Failures log with stable tags (for example `mailbox-mirror-message-failed`) and -never propagate into D1 commit paths. Helpers cover full snapshots -(`mirrorMailboxMessageSnapshot`, `mirrorMailboxDeliveryEventSnapshot`, -`mirrorMailboxDeliveryEventSnapshots` — prefer loading delivery events with -`getMailboxDeliveryEventMirrorInput` or +`missing-owner` | `user-inbound-authority`), or `{ status: 'error', error }`. +Single-RPC helpers record one `mailbox_mirror:` outcome automatically +when a user id is known; `mirrorMailboxDeliveryEventSnapshots` partitions a +mixed page into normal and bootstrap subsets, bounds each non-empty RPC, and +records one aggregate `mailbox_mirror:upsert_delivery_event_batch` outcome for +the original page. Inserted/existing/skipped bootstrap results map to +mirrored/stale/skipped rather than treating idempotency as an error. +`system:email` is excluded. Failures log with stable tags (for example +`mailbox-mirror-message-failed`) and never propagate into D1 commit paths. +Helpers cover full snapshots (`mirrorMailboxMessageSnapshot`, +`mirrorMailboxDeliveryEventSnapshot`, `mirrorMailboxDeliveryEventSnapshots` — +prefer loading delivery events with `getMailboxDeliveryEventMirrorInput` or `listMailboxDeliveryEventMirrorInputsForMessage`) and partial mutations (`mirrorMailboxTouchThread`, `mirrorMailboxUpdateMessageDelivery`, `mirrorMailboxSetMessageClassification`, `mirrorMailboxDeleteMessageMetadata`, `mirrorMailboxDeleteDeliveryEvent`, `mirrorMailboxDeleteThreadIfEmpty`). Partial mutation helpers are library-only; live paths prefer the graph orchestrator -below or parity purge/rebuild for deletes. D1 remains sole authority for all -live mail read/write paths. +below or parity purge/rebuild for deletes. These best-effort helpers remain for +D1-authoritative message graphs; USER inbound delivery events use the +synchronous reverse compatibility mirror described above. **Live graph orchestrator** (`mailbox-live-mirror.ts`): loads a cohesive D1 message graph (optional caller thread, message, attachments, then delivery events) and mirrors it best-effort. `mirrorMailboxMessageGraphFromD1` settles -the message snapshot RPC first, then repairs delivery events with one -owner-bound `upsertDeliveryEvents` batch RPC (not concurrent per-event RPCs to -the same DO). Event load queries newest `max+1` rows from D1, restores -chronological order, and when truncated keeps the newest -`mailboxLiveMirrorMaxEvents` (`mailboxUpsertDeliveryEventsMax`, 100) — dropping -oldest overflow — with a stable warning (`mailbox-live-mirror-events-truncated`; -`userId`, `messageId`, `loaded`, `max`). The batch RPC shares one 1s timeout and -one `mailbox_mirror:upsert_delivery_event_batch` telemetry outcome -(timeout/error apply uniformly to per-event summary entries). Each graph attempt -emits at most two Analytics Engine writes (1 message outcome + 1 batch outcome). -Never throws; returns a bounded summary. **Live callers:** +the message snapshot RPC first, then repairs delivery events with sequential +owner-bound normal/bootstrap batch RPCs (never concurrent per-event RPCs to the +same DO). Event load queries newest `max+1` rows from D1, restores chronological +order, and when truncated keeps the newest `mailboxLiveMirrorMaxEvents` +(`mailboxUpsertDeliveryEventsMax`, 100) — dropping oldest overflow — with a +stable warning (`mailbox-live-mirror-events-truncated`; `userId`, `messageId`, +`loaded`, `max`). Each non-empty subset uses the 1s timeout; the original page +emits one `mailbox_mirror:upsert_delivery_event_batch` telemetry outcome. Each +graph attempt emits at most two Analytics Engine writes (1 message outcome + 1 +batch outcome). Never throws; returns a bounded summary. **Live callers:** - **Outbound terminals** (`outbound.ts`) — after D1 reaches a terminal outbound state (`sent`, attachment-store `failed`, or send `failed`), mirrors the full @@ -848,26 +941,27 @@ Never throws; returns a bounded summary. **Live callers:** transport handlers (`account-email.ts`, `email-message-classify.ts`) delegate here for the D1 mutation + full graph mirror invariant (mirror only after a successful D1 update; failures never change the mutation response). -- **Inbound terminals** (`inbound.ts`) — high-risk user-mail dual-write with - strict ordering: **no Mailbox RPC before** durable D1/R2 message + attachment - storage and `received` finalization win. The winner schedules full graph - repair via `ctx.waitUntil` (`scheduleInboundReceivedTerminalWork`). - **Already-received** Email Routing retries (delivery ledger - `state === 'received'` with an existing message row) idempotently repair the - graph and re-run effect reconciliation without a second charge. **Rejected - terminals** (post-claim parse failure or replay of a claimed `rejected` - delivery) mirror the delivery event only via - `scheduleInboundRejectedTerminalWork` (`mirrorMailboxDeliveryEventFromD1`). - After successful `processInboundDeliveryEffects`, the same received - coordinator task re-mirrors the updated delivery event (usage/subscription - fields). Mirror failures, timeouts, and hangs never affect SMTP - reject/refund/retry/charge semantics. **Pre-claim bounded rejection rows** - (`recordBoundedEmailRejectionEvent` for verification, suspension, - sender-policy, size, entitlement, and system-limit gates before delivery - claim/charge) stay **D1-only on the live path** — the every-5-minute - `mailbox_parity` lane backfills them. **`system:email` stays excluded** (no - per-user Mailbox object). Retention sweeper deletes and other bulk - metadata-delete mirrors are **still not wired** on live paths. Scheduled +- **Inbound terminals** (`inbound.ts`) — USER delivery authority starts in + Mailbox: dedupe claim, UserMeter consume, and charged-pending CAS precede + D1/R2 message-graph storage. Mailbox then finalizes `received` or `rejected`. + Received snapshots synchronously project to D1; rejected snapshots project + best-effort so a compatibility-write outage cannot undo the permanent SMTP + reject. A received winner schedules D1 message-graph repair via + `ctx.waitUntil` (`scheduleInboundReceivedTerminalWork`) without D1 + delivery-event write-back. **Already-received** Email Routing retries + (delivery ledger `state === 'received'` with an existing message row) + idempotently repair the graph and re-run effect reconciliation without a + second charge. **Rejected terminals** (post-claim parse failure or replay of a + claimed `rejected` delivery) read-repair the Mailbox → D1 projection via + `scheduleInboundRejectedTerminalWork`. Effects claim and complete/fail in + Mailbox around external work; terminal snapshots synchronously repair D1. + Terminal coordinator failures are contained after the authoritative CAS. + **Pre-claim bounded rejection rows** (`recordBoundedEmailRejectionEvent` for + verification, suspension, sender-policy, size, entitlement, and system-limit + gates before delivery claim/charge) stay **D1-only on the live path** — the + every-5-minute `mailbox_parity` lane backfills them. **`system:email` stays + excluded** (no per-user Mailbox object). Retention sweeper deletes and other + bulk metadata-delete mirrors are **still not wired** on live paths. Scheduled parity reconcile **is** wired (see below); read cutover is prepared but not flipped. @@ -912,16 +1006,20 @@ Per user, the lane: **every** owner `email_delivery_events` row by `(created_at, id)` into ready `MailboxDeliveryEventInput` snapshots (`listMailboxDeliveryEventMirrorInputsForOwnerKeyset`; not only - `message_id`-null orphans) and mirrors each page with **one** - `upsertDeliveryEvents` batch via `mirrorMailboxDeliveryEventSnapshots` - (`mailboxParityEventPageSize` ≤ DO max, timeout + `message_id`-null orphans). Each page partitions legacy USER inbound + lifecycle/dedupe snapshots to missing-only `bootstrapDeliveryEvents`, and + sends pre-claim rejection audits plus non-inbound rows to normal + `upsertDeliveryEvents`; a legacy authority row cannot roll back the normal + audit batch. Both subsets remain bounded by the original page + (`mailboxParityEventPageSize` ≤ DO max), with timeout `mailboxParityEventMirrorTimeoutMs` ≈ 5s). Production evidence: per-event 1s RPCs repeatedly timed out on a lagging owner and prevented soak under the 10s lane budget; page batches restore convergence while live dual-write keeps the 1s bound. Cursor advances through per-event mirrored/stale/missing results - (equal `created_at` progresses by id); uniform timeout/error/unconfigured - retains the cursor so the next tick reloads the same page. Rows deleted - before the snapshot load simply do not appear. + and USER-inbound bootstrap skips (so count comparison exposes malformed + legacy rows); equal `created_at` progresses by id. Uniform + timeout/error/unconfigured retains the cursor so the next tick reloads the + same page. Rows deleted before the snapshot load simply do not appear. 4. **Durable content watermark replay** — after both creation phases complete, opens a frozen window `(watermark, upper]` (`mailbox_parity_content_replay_upper_at` set once when the window opens; @@ -1045,11 +1143,11 @@ mail content. Account export pages Mailbox state through the `mailbox` section ### Expand/contract phases -This is an expand/contract migration. **Phase 2 live paths are wired (terminal -inbound + parity):** live dual-write covers outbound terminal -message/thread/attachment/event graphs, provider delivery-queue graph repair -(`recorded` / `duplicate` / `stale` with a message, via `waitUntil`), user -classification (full graph repair after D1 update via +This is an expand/contract migration. **Phase 2 live paths are wired (USER +inbound authority + graph dual-write + parity):** live dual-write covers +outbound terminal message/thread/attachment/event graphs, provider +delivery-queue graph repair (`recorded` / `duplicate` / `stale` with a message, +via `waitUntil`), user classification (full graph repair after D1 update via `service.ts#setEmailMessageClassification`), high-risk inbound terminal paths (received graph + rejected delivery-event mirror + post-effects event re-mirror; `waitUntil`; no Mailbox before D1/R2 finalization; pre-claim bounded rejections @@ -1065,12 +1163,14 @@ rejection rows), durable content-watermark replays, compares owner-scoped D1 vs Mailbox counts, persists soak state on `users` (migration `0125`), re-purges the DO when account deletion races the lane, and repairs delete drift via purge/rebuild. Direct delete wiring is pending. D1 remains write authority for -all live paths; owner-facing reads may use Mailbox when the default-off -`mailbox-read-cutover` flag and parity soak pass (phase 3). D1 email rows and -existing R2 inventory deletion stay authoritative during expand. **Phase 2 -contract completion** (every user-mail D1 mutation also writes the DO on live -paths, including retention deletes) remains pending; terminal inbound + parity -cover the high-risk live surface today. +message graphs and non-USER-inbound delivery events; USER inbound lifecycle and +effect transitions are owner-bound Mailbox CAS, synchronously projected to D1. +Owner-facing graph reads may use Mailbox when the default-off +`mailbox-read-cutover` flag and parity soak pass (phase 3). Existing R2 +inventory deletion stays authoritative during expand. **Phase 2 contract +completion** (every user-mail D1 mutation also writes the DO on live paths, +including retention deletes) remains pending; terminal inbound + parity cover +the high-risk live surface today. 1. **Additive scaffold / no live mail behavior** — bind `Mailbox`, freeze `idFromName(userId)`, ship client + `mirrorMessage` / `upsertDeliveryEvent` / @@ -1098,10 +1198,13 @@ cover the high-risk live surface today. The every-5-minute `mailbox_parity` scheduled lane backfills all owner messages and delivery events, durable content-watermark replays, count compares, soak tracking on `users` (migration `0125`), and repairs delete - drift via purge/rebuild. **Still pending for phase-2 contract completion:** - direct delete wiring for explicit/retention deletes (including retention - sweeper metadata-delete mirrors). D1 remains write authority; owner-facing - read cutover is wired behind the default-off flag (phase 3). + drift via purge/rebuild. USER inbound delivery lifecycle/effect state is the + exception: Mailbox is authoritative and D1 is its synchronous compatibility + projection. **Still pending for phase-2 contract completion:** direct delete + wiring for explicit/retention deletes (including retention sweeper + metadata-delete mirrors). D1 remains write authority for the message graph + and non-USER-inbound events; owner-facing read cutover is wired behind the + default-off flag (phase 3). 3. **Owner-facing reads cut over after production soak** — app inbox/detail and MCP list/get/search/attachment/delivery-event reads move to the DO only after production parity soak is verified (`mailbox_parity_matching_since` ≥ 2h @@ -1161,42 +1264,36 @@ below. verification. This phase does not flip Mailbox write authority; contextless provider-id reverse lookups must not enumerate per-user Mailbox objects. -### Inbound durability boundary (D1-authoritative dual-write) - -Today's D1-authoritative inbound commit boundary is documented under -[R2 (`EMAIL_BLOBS`)](#r2-community_assets-email_blobs): thread prework, R2 put, -D1 message/attachment rows are pre-commit; `touchEmailThread` / `received` -delivery-event writes are post-commit best-effort; ambiguous attachment-insert -failures acknowledge rather than risk duplicates. - -**Mailbox dual-write ordering (phase-2 live, D1 still authoritative):** - -- **Pre-commit (no Mailbox):** thread prework, R2 raw-MIME put, D1 - message/attachment storage, and inbound delivery finalization to `received`. -- **Post-commit (best effort, `waitUntil`):** full message graph mirror - (`mirrorMailboxMessageGraphFromD1`) only after the durable commit + - finalization win; already-received retries repair the graph idempotently - without a second charge. Rejected **post-claim** terminals mirror the delivery - event only (`mirrorMailboxDeliveryEventFromD1`). After successful effect - dispatch, re-mirror the updated delivery event. Failures/timeouts never affect - reject, refund, retry, or charge semantics. -- **Pre-claim bounded rejections** (`recordBoundedEmailRejectionEvent` before - delivery claim/charge) write D1 audit rows only on the live path; the - `mailbox_parity` lane backfills them. **`system:email` is excluded** from all - Mailbox mirrors. -- **Ambiguity:** if attachment commit fails but message cleanup (or a residual - probe) cannot prove the pre-commit state, acknowledge the already-created - message (logged, non-retry) rather than risking a duplicate on Email Routing - retry. Empty-thread cleanup stays deferred. - -When Mailbox becomes read-authoritative (phase 3+), the same shape applies with -the DO as the metadata store: - -- **Pre-commit:** thread prework + R2 raw-MIME put + atomic Mailbox - message/attachment commit. -- **Post-commit (best effort):** thread touch and delivery-event writes — log - failures without throwing (retry would duplicate mail). -- **Ambiguity:** same acknowledge-over-retry rule as today. +### Inbound durability boundary (USER Mailbox authority) + +For USER mail, the owner-bound Mailbox ledger is the lifecycle/effect authority: + +1. Mailbox CAS selects the dedupe winner. UserMeter consumes quota for that + winner, then Mailbox inserts the charged pending snapshot. The charged + pending snapshot is synchronously projected to D1. +2. Thread prework, R2 raw-MIME put, and D1 message/attachment storage build the + message graph. D1 remains authoritative for that graph. +3. Mailbox CAS finalizes the delivery as `received` or `rejected`. A received + snapshot is synchronously projected to D1 and fence-critical projection + failures fail closed. Rejection projection is best-effort because Mailbox has + already made the SMTP rejection permanent; rejected terminal work + read-repairs D1. +4. Received terminal work repairs the D1-authoritative message graph into + Mailbox without delivery events, runs externally executed effects under + Mailbox leases, then read-repairs the Mailbox → D1 projection. + +Retries inspect Mailbox state and never restore USER delivery authority from D1. +The sole reverse bridge is a missing-row-only bootstrap of a validated, +owner/provider-matched pre-deploy D1 snapshot. Malformed or cross-owner legacy +rows are skipped. Pre-claim bounded rejection audit rows remain D1-only. + +`system:email` is the explicit exception: its inbound lifecycle, effects, and +reconciliation remain D1-authoritative and never bootstrap a Mailbox. + +If attachment commit fails but message cleanup (or a residual probe) cannot +prove the pre-commit state, the handler acknowledges the already-created message +rather than risking a duplicate on Email Routing retry. Empty-thread cleanup +stays deferred. ### Package state model diff --git a/docs/contributing/disaster-recovery.md b/docs/contributing/disaster-recovery.md index 284f4a3b20..c5dd911ebb 100644 --- a/docs/contributing/disaster-recovery.md +++ b/docs/contributing/disaster-recovery.md @@ -359,6 +359,19 @@ Disable ingress / put the app in maintenance before execute. After restore: reindex Vectorize, re-arm jobs/alarms from D1, recreate queues from Wrangler config, and expect users to reauthorize OAuth and remote connectors. +### Mailbox authority rollback repair + +Rollback from the USER inbound Mailbox-authority Worker to its predecessor has +an accepted roll-forward caveat: legacy `json_set` lifecycle writes do not bump +`updated_at`, and the roll-forward does not auto-detect newer D1 state when a +Mailbox row already exists. Before a roll-forward after such a rollback, use the +backup-gated, owner-by-owner metadata purge and full D1 parity rebuild procedure +in +[Data storage → Mailbox](./architecture/data-storage.md#durable-objects-mailbox). +That procedure is gated on the verified backup SHA-256 prefix `7787f8c9`, keeps +the rollback Worker quiesced, and requires effect/finalization verification. Do +not use normal purge as an exploratory or routine repair. + ## Schedules and freshness | When (UTC) | Who | What | diff --git a/packages/worker/migrations/0129-email-inbound-mailbox-authority-mirror.sql b/packages/worker/migrations/0129-email-inbound-mailbox-authority-mirror.sql new file mode 100644 index 0000000000..4d54753ecf --- /dev/null +++ b/packages/worker/migrations/0129-email-inbound-mailbox-authority-mirror.sql @@ -0,0 +1,125 @@ +-- USER inbound delivery authority now lives in the per-owner Mailbox Durable +-- Object. These columns are a complete compatibility projection used for +-- synchronous D1 graph-write fences and global scheduled owner discovery. +-- system:email continues to use the same D1 row as its authority. + +ALTER TABLE email_delivery_events ADD COLUMN state TEXT +CHECK ( + state IS NULL + OR state IN ( + 'pending', 'storing', 'cleaning', 'received', 'rejected', 'orphan-cleaned' + ) +); +ALTER TABLE email_delivery_events ADD COLUMN fingerprint TEXT; +ALTER TABLE email_delivery_events ADD COLUMN storage_lease TEXT; +ALTER TABLE email_delivery_events ADD COLUMN storage_lease_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN cleanup_lease TEXT; +ALTER TABLE email_delivery_events ADD COLUMN cleanup_lease_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN cleanup_retry_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN expected_attachment_count INTEGER; +ALTER TABLE email_delivery_events ADD COLUMN finalization_token TEXT; +ALTER TABLE email_delivery_events ADD COLUMN reconcile_after TEXT; +ALTER TABLE email_delivery_events ADD COLUMN dedupe_expires_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN usage_effect_suppressed_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN usage_started_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN usage_effect_retry_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN usage_effect_lease TEXT; +ALTER TABLE email_delivery_events ADD COLUMN usage_effect_lease_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN subscription_effect_state TEXT +CHECK ( + subscription_effect_state IS NULL + OR subscription_effect_state IN ('pending', 'processing', 'complete', 'dead-letter') +); +ALTER TABLE email_delivery_events ADD COLUMN subscription_effect_lease TEXT; +ALTER TABLE email_delivery_events ADD COLUMN subscription_effect_lease_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN subscription_effect_retry_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN subscription_effect_attempt_count INTEGER; +ALTER TABLE email_delivery_events ADD COLUMN subscription_effect_dead_letter_at TEXT; +ALTER TABLE email_delivery_events ADD COLUMN subscription_effect_last_error TEXT; +ALTER TABLE email_delivery_events ADD COLUMN updated_at TEXT; + +UPDATE email_delivery_events +SET + state = json_extract(detail_json, '$.state'), + fingerprint = json_extract(detail_json, '$.fingerprint'), + storage_lease = json_extract(detail_json, '$.storageLease'), + storage_lease_at = json_extract(detail_json, '$.storageLeaseAt'), + cleanup_lease = json_extract(detail_json, '$.cleanupLease'), + cleanup_lease_at = json_extract(detail_json, '$.cleanupLeaseAt'), + cleanup_retry_at = json_extract(detail_json, '$.cleanupRetryAt'), + expected_attachment_count = json_extract( + detail_json, + '$.expectedAttachmentCount' + ), + finalization_token = json_extract(detail_json, '$.finalizationToken'), + reconcile_after = json_extract(detail_json, '$.reconcileAfter'), + dedupe_expires_at = json_extract(detail_json, '$.dedupeExpiresAt'), + usage_effect_suppressed_at = json_extract( + detail_json, + '$.usageEffectSuppressedAt' + ), + usage_started_at = json_extract(detail_json, '$.usageStartedAt'), + usage_effect_retry_at = json_extract(detail_json, '$.usageEffectRetryAt'), + usage_effect_lease = json_extract(detail_json, '$.usageEffectLease'), + usage_effect_lease_at = json_extract(detail_json, '$.usageEffectLeaseAt'), + subscription_effect_state = json_extract( + detail_json, + '$.subscriptionEffectState' + ), + subscription_effect_lease = json_extract( + detail_json, + '$.subscriptionEffectLease' + ), + subscription_effect_lease_at = json_extract( + detail_json, + '$.subscriptionEffectLeaseAt' + ), + subscription_effect_retry_at = json_extract( + detail_json, + '$.subscriptionEffectRetryAt' + ), + subscription_effect_attempt_count = json_extract( + detail_json, + '$.subscriptionEffectAttemptCount' + ), + subscription_effect_dead_letter_at = json_extract( + detail_json, + '$.subscriptionEffectDeadLetterAt' + ), + subscription_effect_last_error = json_extract( + detail_json, + '$.subscriptionEffectLastError' + ), + updated_at = created_at +WHERE provider IN ( + 'cloudflare-email-routing', + 'cloudflare-email-routing-dedupe' +); + +UPDATE email_delivery_events +SET updated_at = created_at +WHERE updated_at IS NULL; + +CREATE INDEX IF NOT EXISTS idx_email_delivery_events_user_state_created +ON email_delivery_events(user_id, state, created_at, id) +WHERE provider = 'cloudflare-email-routing' AND state IS NOT NULL; + +CREATE INDEX IF NOT EXISTS idx_email_delivery_events_user_dedupe_expires +ON email_delivery_events(user_id, dedupe_expires_at, id) +WHERE provider = 'cloudflare-email-routing-dedupe' + AND dedupe_expires_at IS NOT NULL; + +-- Cross-store idempotency for the external D1 rollup effect. Mailbox owns the +-- effect state/lease; this ledger only prevents replay from incrementing the +-- aggregate twice if the Worker fails after D1 commit but before Mailbox CAS. +CREATE TABLE email_inbound_usage_effects ( + user_id TEXT NOT NULL, + delivery_id TEXT NOT NULL, + finalization_token TEXT NOT NULL, + created_at TEXT NOT NULL, + PRIMARY KEY (user_id, delivery_id, finalization_token), + FOREIGN KEY (delivery_id) REFERENCES email_delivery_events(id) ON DELETE CASCADE +); + +CREATE INDEX IF NOT EXISTS idx_email_inbound_usage_effects_delivery +ON email_inbound_usage_effects(delivery_id); diff --git a/packages/worker/src/account/data-targets.ts b/packages/worker/src/account/data-targets.ts index b7a68442ed..4663cff411 100644 --- a/packages/worker/src/account/data-targets.ts +++ b/packages/worker/src/account/data-targets.ts @@ -156,6 +156,14 @@ export const accountUserDataTargets: ReadonlyArray = [ { kind: 'user_id', table: 'workflow_runs' }, { kind: 'user_id', table: 'package_service_states' }, { kind: 'user_id', table: 'user_storage_buckets' }, + { + kind: 'user_id', + table: 'email_inbound_usage_effects', + includeInExport: false, + surface: 'email_inbound_usage_effects', + reason: + 'Internal cross-store effect idempotency ledger with no user-exportable content.', + }, { kind: 'user_id', table: 'usage_rollups' }, { kind: 'user_id', table: 'feature_flag_exposure_rollups' }, { kind: 'user_id', table: 'user_activation_milestones' }, diff --git a/packages/worker/src/email/inbound-delivery-authority.node.test.ts b/packages/worker/src/email/inbound-delivery-authority.node.test.ts new file mode 100644 index 0000000000..b43f3d91f8 --- /dev/null +++ b/packages/worker/src/email/inbound-delivery-authority.node.test.ts @@ -0,0 +1,851 @@ +import { DatabaseSync } from 'node:sqlite' +import { expect, test, vi } from 'vitest' +import { consoleWarn } from '#worker/test-support/console-spies.ts' +import { createD1FromSqlite } from '#worker/test-support/create-d1-from-sqlite.ts' +import { buildInboundDelivery } from './inbound-delivery.ts' +import { + createUserInboundDeliveryAuthority, + mirrorUserInboundDeliverySnapshotToD1, +} from './inbound-delivery-authority.ts' +import { type MailboxInboundDeliverySnapshot } from './mailbox-inbound-ledger.ts' +import { claimSystemInboundDeliveryWindow } from './system-inbound-delivery-authority.ts' +import { ensureEmailTestSchema } from './test-schema.ts' + +function namespace(stub: object) { + return { + idFromName: () => ({}) as DurableObjectId, + get: () => stub, + } as unknown as DurableObjectNamespace +} + +test('dedupe claim precedes UserMeter, and insert failure replay does not double charge', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const userId = `user-${crypto.randomUUID()}` + const now = new Date('2026-08-02T12:00:00.000Z') + const delivery = await buildInboundDelivery({ + userId, + inboxId: `inbox-${crypto.randomUUID()}`, + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'From: sender@example.com\r\n\r\nhello', + quotaDay: '2026-08-02', + now, + }) + const snapshot: MailboxInboundDeliverySnapshot = { + ...delivery, + provider: 'cloudflare-email-routing', + state: 'pending', + createdAt: now.toISOString(), + updatedAt: now.toISOString(), + } + const order: Array = [] + let consumeCalls = 0 + let insertCalls = 0 + const meter = { + async consumeInboundDelivery() { + consumeCalls += 1 + order.push(`meter-${consumeCalls}`) + return { + outcome: 'ready' as const, + count: 1, + revision: 1, + mirrorUpdatedAt: now.toISOString(), + consumed: consumeCalls === 1, + replayed: consumeCalls > 1, + day: '2026-08-02', + resource: 'email_receives_per_day' as const, + } + }, + } + const mailbox = { + getInboundDelivery: vi.fn(async () => null), + claimInboundDeliveryWindow: vi.fn(async () => { + order.push('mailbox-window') + return snapshot + }), + insertChargedPendingInboundDelivery: vi.fn(async () => { + insertCalls += 1 + order.push(`mailbox-${insertCalls}`) + if (insertCalls === 1) throw new Error('injected Mailbox insert failure') + return { status: 'inserted' as const, delivery: snapshot } + }), + } + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace(meter), + MAILBOX: namespace(mailbox), + }, + userId, + }) + + await expect( + authority.charge({ delivery, plan: 'pro', limit: 100, now }), + ).rejects.toThrow('injected Mailbox insert failure') + expect(order).toEqual(['mailbox-window', 'meter-1', 'mailbox-1']) + + const replay = await authority.charge({ + delivery, + plan: 'pro', + limit: 100, + now, + }) + expect(replay).toEqual({ delivery, charged: true }) + expect(order).toEqual([ + 'mailbox-window', + 'meter-1', + 'mailbox-1', + 'mailbox-window', + 'meter-2', + 'mailbox-2', + ]) + expect(consumeCalls).toBe(2) + expect(insertCalls).toBe(2) + expect( + await db + .prepare( + `SELECT state, fingerprint FROM email_delivery_events + WHERE id = ? AND user_id = ?`, + ) + .bind(delivery.deliveryId, userId) + .first(), + ).toEqual({ state: 'pending', fingerprint: delivery.fingerprint }) + sqlite.close() +}) + +test('dedupe boundary race charges and inserts only the claimed winner id', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const userId = `user-${crypto.randomUUID()}` + const now = new Date('2026-08-02T12:00:00.000Z') + const winner = await buildInboundDelivery({ + userId, + inboxId: `inbox-${crypto.randomUUID()}`, + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'From: sender@example.com\r\n\r\nsame message', + quotaDay: '2026-08-02', + now, + }) + const loser: typeof winner = { + ...winner, + deliveryId: `${winner.deliveryId}-boundary-loser`, + messageId: `${winner.messageId}-boundary-loser`, + rawMimeKey: `${winner.rawMimeKey}-boundary-loser`, + } + const winnerSnapshot: MailboxInboundDeliverySnapshot = { + ...winner, + state: 'pending', + createdAt: now.toISOString(), + updatedAt: now.toISOString(), + } + let stored: MailboxInboundDeliverySnapshot | null = null + const consumedDeliveryIds: Array = [] + const insertedDeliveryIds: Array = [] + const meter = { + consumeInboundDelivery: vi.fn(async (input: { deliveryId: string }) => { + consumedDeliveryIds.push(input.deliveryId) + return { + outcome: 'ready' as const, + count: 1, + revision: 1, + mirrorUpdatedAt: now.toISOString(), + consumed: true, + replayed: false, + day: '2026-08-02', + resource: 'email_receives_per_day' as const, + } + }), + } + const mailbox = { + getInboundDelivery: vi.fn(async (input: { deliveryId: string }) => + stored?.deliveryId === input.deliveryId ? stored : null, + ), + claimInboundDeliveryWindow: vi.fn(async () => winnerSnapshot), + insertChargedPendingInboundDelivery: vi.fn( + async (input: { delivery: { deliveryId: string } }) => { + insertedDeliveryIds.push(input.delivery.deliveryId) + stored = winnerSnapshot + return { status: 'inserted' as const, delivery: winnerSnapshot } + }, + ), + } + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace(meter), + MAILBOX: namespace(mailbox), + }, + userId, + }) + + const first = await authority.charge({ + delivery: winner, + plan: 'pro', + limit: 100, + now, + }) + const boundaryLoser = await authority.charge({ + delivery: loser, + plan: 'pro', + limit: 100, + now, + }) + + expect(first).toEqual({ delivery: winner, charged: true }) + expect(boundaryLoser.delivery.deliveryId).toBe(winner.deliveryId) + expect(boundaryLoser.charged).toBe(false) + expect(consumedDeliveryIds).toEqual([winner.deliveryId]) + expect(insertedDeliveryIds).toEqual([winner.deliveryId]) + expect(consumedDeliveryIds).not.toContain(loser.deliveryId) + sqlite.close() +}) + +test("uncharged prior-day dedupe winner charges the current retry's quota day", async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const userId = `user-${crypto.randomUUID()}` + const yesterday = new Date('2026-08-01T23:59:00.000Z') + const retryNow = new Date('2026-08-02T00:01:00.000Z') + const winner = await buildInboundDelivery({ + userId, + inboxId: `inbox-${crypto.randomUUID()}`, + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'From: sender@example.com\r\n\r\nmidnight retry', + quotaDay: '2026-08-01', + now: yesterday, + }) + const retry = { ...winner, quotaDay: '2026-08-02' } + const winnerSnapshot: MailboxInboundDeliverySnapshot = { + ...winner, + state: 'pending', + createdAt: yesterday.toISOString(), + updatedAt: retryNow.toISOString(), + } + const consumeInboundDelivery = vi.fn( + async (input: { deliveryId: string; day: string }) => ({ + outcome: 'ready' as const, + count: 1, + revision: 1, + mirrorUpdatedAt: retryNow.toISOString(), + consumed: true, + replayed: false, + day: input.day, + resource: 'email_receives_per_day' as const, + }), + ) + const insertChargedPendingInboundDelivery = vi.fn( + async (input: { + delivery: { + deliveryId: string + messageId: string + rawMimeKey: string + quotaDay: string + } + }) => ({ + status: 'inserted' as const, + delivery: { + ...winnerSnapshot, + quotaDay: input.delivery.quotaDay, + }, + }), + ) + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace({ consumeInboundDelivery }), + MAILBOX: namespace({ + getInboundDelivery: vi.fn(async () => null), + claimInboundDeliveryWindow: vi.fn(async () => winnerSnapshot), + insertChargedPendingInboundDelivery, + }), + }, + userId, + }) + + const result = await authority.charge({ + delivery: retry, + plan: 'pro', + limit: 100, + now: retryNow, + }) + + expect(result.delivery).toEqual(retry) + expect(result.charged).toBe(true) + expect(consumeInboundDelivery).toHaveBeenCalledWith( + expect.objectContaining({ + deliveryId: winner.deliveryId, + day: '2026-08-02', + }), + ) + expect(insertChargedPendingInboundDelivery).toHaveBeenCalledWith( + expect.objectContaining({ + delivery: expect.objectContaining({ + deliveryId: winner.deliveryId, + messageId: winner.messageId, + rawMimeKey: winner.rawMimeKey, + quotaDay: '2026-08-02', + }), + }), + ) + sqlite.close() +}) + +test('storage claim projection failure releases the authoritative Mailbox lease', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const now = new Date('2026-08-02T12:00:00.000Z') + const delivery = await buildInboundDelivery({ + userId: 'user-storage-owner', + inboxId: 'inbox-storage', + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'storage projection', + quotaDay: '2026-08-02', + now, + }) + const claimed: MailboxInboundDeliverySnapshot = { + ...delivery, + state: 'storing', + storageLease: 'storage-lease-1', + storageLeaseAt: now.toISOString(), + createdAt: now.toISOString(), + updatedAt: now.toISOString(), + } + await mirrorUserInboundDeliverySnapshotToD1({ + db, + userId: 'different-owner', + snapshot: claimed, + }) + const releaseInboundDeliveryStorage = vi.fn(async () => ({ + status: 'released' as const, + delivery: { + ...claimed, + state: 'pending' as const, + storageLease: undefined, + storageLeaseAt: undefined, + }, + })) + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace({}), + MAILBOX: namespace({ + claimInboundDeliveryStorage: vi.fn(async () => ({ + status: 'claimed' as const, + delivery: claimed, + })), + releaseInboundDeliveryStorage, + }), + }, + userId: delivery.userId, + }) + + await expect( + authority.claimStorage(delivery, 2, now.toISOString(), now), + ).rejects.toThrow('owner/provider fence') + expect(releaseInboundDeliveryStorage).toHaveBeenCalledWith({ + ownerId: delivery.userId, + deliveryId: delivery.deliveryId, + storageLease: 'storage-lease-1', + now: now.toISOString(), + }) + sqlite.close() +}) + +test('cleanup projection failure releases its lease and prevents deletion work', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const now = new Date('2026-08-02T12:00:00.000Z') + const delivery = await buildInboundDelivery({ + userId: 'user-claim-owner', + inboxId: 'inbox-claim', + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'claim projection', + quotaDay: '2026-08-02', + now, + }) + const cleanup: MailboxInboundDeliverySnapshot = { + ...delivery, + state: 'cleaning', + cleanupLease: 'cleanup-1', + cleanupLeaseAt: now.toISOString(), + createdAt: now.toISOString(), + updatedAt: now.toISOString(), + } + await mirrorUserInboundDeliverySnapshotToD1({ + db, + userId: 'different-owner', + snapshot: cleanup, + }) + const releaseInboundDeliveryCleanup = vi.fn(async () => ({ + status: 'released' as const, + delivery: { + ...cleanup, + state: 'pending' as const, + cleanupLease: undefined, + cleanupLeaseAt: undefined, + }, + })) + const deleteBlob = vi.fn(async () => {}) + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace({}), + MAILBOX: namespace({ + claimInboundDeliveryCleanup: vi.fn(async () => ({ + status: 'claimed' as const, + delivery: cleanup, + })), + releaseInboundDeliveryCleanup, + }), + }, + userId: delivery.userId, + }) + + await expect( + authority + .claimCleanup({ + deliveryId: delivery.deliveryId, + expectedState: delivery.state, + expectedUpdatedAt: cleanup.updatedAt, + staleBefore: new Date(now.getTime() - 1), + now, + }) + .then(async () => await deleteBlob()), + ).rejects.toThrow('owner/provider fence') + expect(deleteBlob).not.toHaveBeenCalled() + expect(releaseInboundDeliveryCleanup).toHaveBeenCalledWith({ + ownerId: delivery.userId, + deliveryId: delivery.deliveryId, + cleanupLease: 'cleanup-1', + now: now.toISOString(), + }) + sqlite.close() +}) + +test('cleanup projection and compensation failures surface as AggregateError', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const now = new Date('2026-08-02T12:00:00.000Z') + const delivery = await buildInboundDelivery({ + userId: 'user-cleanup-compensation-owner', + inboxId: 'inbox-cleanup-compensation', + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'cleanup compensation', + quotaDay: '2026-08-02', + now, + }) + const cleanup: MailboxInboundDeliverySnapshot = { + ...delivery, + state: 'cleaning', + cleanupLease: 'cleanup-compensation-lease', + cleanupLeaseAt: now.toISOString(), + createdAt: now.toISOString(), + updatedAt: now.toISOString(), + } + await mirrorUserInboundDeliverySnapshotToD1({ + db, + userId: 'different-owner', + snapshot: cleanup, + }) + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace({}), + MAILBOX: namespace({ + claimInboundDeliveryCleanup: vi.fn(async () => ({ + status: 'claimed' as const, + delivery: cleanup, + })), + releaseInboundDeliveryCleanup: vi.fn(async () => { + throw new Error('cleanup release unavailable') + }), + }), + }, + userId: delivery.userId, + }) + + const error = await authority + .claimCleanup({ + deliveryId: delivery.deliveryId, + expectedState: delivery.state, + expectedUpdatedAt: cleanup.updatedAt, + staleBefore: new Date(now.getTime() - 1), + now, + }) + .catch((caught: unknown) => caught) + expect(error).toBeInstanceOf(AggregateError) + expect(error).toMatchObject({ + message: + 'Inbound cleanup claim projection failed and its Mailbox lease could not be released.', + errors: [ + expect.objectContaining({ message: expect.stringContaining('fence') }), + expect.objectContaining({ message: 'cleanup release unavailable' }), + ], + }) + sqlite.close() +}) + +test('effect claim projection failures warn without stranding leases', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + consoleWarn.mockImplementation(() => {}) + const now = new Date('2026-08-02T12:00:00.000Z') + const delivery = await buildInboundDelivery({ + userId: 'user-effect-claim-owner', + inboxId: 'inbox-effect-claim', + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'effect claim projection', + quotaDay: '2026-08-02', + now, + }) + const base: MailboxInboundDeliverySnapshot = { + ...delivery, + state: 'received', + finalizationToken: 'final-1', + createdAt: now.toISOString(), + updatedAt: now.toISOString(), + } + await mirrorUserInboundDeliverySnapshotToD1({ + db, + userId: 'different-owner', + snapshot: base, + }) + const usage = { + ...base, + usageEffectLease: 'usage-1', + usageEffectLeaseAt: now.toISOString(), + } + const subscription = { + ...base, + subscriptionEffectState: 'processing' as const, + subscriptionEffectLease: 'subscription-1', + subscriptionEffectLeaseAt: now.toISOString(), + } + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace({}), + MAILBOX: namespace({ + claimInboundUsageEffect: vi.fn(async () => ({ + status: 'claimed' as const, + delivery: usage, + })), + claimInboundSubscriptionEffect: vi.fn(async () => ({ + status: 'claimed' as const, + delivery: subscription, + })), + }), + }, + userId: delivery.userId, + }) + + await expect( + authority.claimUsageEffect({ deliveryId: delivery.deliveryId, now }), + ).resolves.toMatchObject({ status: 'claimed' }) + await expect( + authority.claimSubscriptionEffect({ + deliveryId: delivery.deliveryId, + now, + }), + ).resolves.toMatchObject({ status: 'claimed' }) + expect(consoleWarn).toHaveBeenCalledWith( + 'inbound-email-usage-effect-claim-projection-failed', + delivery.userId, + delivery.deliveryId, + expect.any(Error), + ) + expect(consoleWarn).toHaveBeenCalledWith( + 'inbound-email-subscription-effect-claim-projection-failed', + delivery.userId, + delivery.deliveryId, + expect.any(Error), + ) + sqlite.close() +}) + +test('full D1 snapshot mirror cannot cross an owner or provider fence', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const now = new Date('2026-08-02T12:00:00.000Z') + const delivery = await buildInboundDelivery({ + userId: 'owner-a', + inboxId: 'inbox-a', + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'hello', + quotaDay: '2026-08-02', + now, + }) + const snapshot: MailboxInboundDeliverySnapshot = { + ...delivery, + state: 'storing', + storageLease: 'lease-a', + storageLeaseAt: now.toISOString(), + createdAt: now.toISOString(), + updatedAt: now.toISOString(), + } + await mirrorUserInboundDeliverySnapshotToD1({ + db, + userId: 'owner-a', + snapshot, + }) + await expect( + mirrorUserInboundDeliverySnapshotToD1({ + db, + userId: 'owner-b', + snapshot: { ...snapshot, storageLease: 'lease-b' }, + }), + ).rejects.toThrow('owner/provider fence') + expect( + await db + .prepare( + `SELECT user_id, state, storage_lease FROM email_delivery_events + WHERE id = ?`, + ) + .bind(delivery.deliveryId) + .first(), + ).toEqual({ + user_id: 'owner-a', + state: 'storing', + storage_lease: 'lease-a', + }) + sqlite.close() +}) + +test('stale D1 mirrors are safe no-ops without replacing a newer snapshot', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const userId = `user-${crypto.randomUUID()}` + const older = new Date('2026-08-02T12:00:00.000Z') + const newer = new Date('2026-08-02T12:01:00.000Z') + const delivery = await buildInboundDelivery({ + userId, + inboxId: `inbox-${crypto.randomUUID()}`, + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'hello', + quotaDay: '2026-08-02', + now: older, + }) + const received: MailboxInboundDeliverySnapshot = { + ...delivery, + state: 'received', + finalizationToken: 'final-token', + createdAt: older.toISOString(), + updatedAt: newer.toISOString(), + } + await mirrorUserInboundDeliverySnapshotToD1({ + db, + userId, + snapshot: received, + }) + await expect( + mirrorUserInboundDeliverySnapshotToD1({ + db, + userId, + snapshot: { + ...received, + state: 'storing', + storageLease: 'stale-lease', + storageLeaseAt: older.toISOString(), + updatedAt: older.toISOString(), + }, + }), + ).resolves.toEqual({ status: 'stale' }) + expect( + await db + .prepare( + `SELECT state, finalization_token, storage_lease + FROM email_delivery_events WHERE id = ? AND user_id = ?`, + ) + .bind(delivery.deliveryId, userId) + .first(), + ).toEqual({ + state: 'received', + finalization_token: 'final-token', + storage_lease: null, + }) + sqlite.close() +}) + +test('USER authority refuses the system email owner before bootstrap', () => { + expect(() => + createUserInboundDeliveryAuthority({ + env: {} as Parameters< + typeof createUserInboundDeliveryAuthority + >[0]['env'], + userId: 'system:email', + }), + ).toThrow('must remain in D1') +}) + +test('bootstrap skips malformed and cross-owner legacy D1 rows', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + consoleWarn.mockImplementation(() => {}) + const ownerId = `owner-${crypto.randomUUID()}` + const now = new Date('2026-08-02T12:00:00.000Z') + const crossOwner = await buildInboundDelivery({ + userId: `other-${crypto.randomUUID()}`, + inboxId: 'inbox-cross-owner', + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'cross owner', + quotaDay: '2026-08-02', + now, + }) + const malformedId = `malformed-${crypto.randomUUID()}` + for (const [id, detailJson] of [ + [crossOwner.deliveryId, JSON.stringify(crossOwner)], + [malformedId, '{'], + ] as const) { + await db + .prepare( + `INSERT INTO email_delivery_events ( + id, user_id, event_type, provider, provider_event_id, + detail_json, created_at + ) VALUES (?, ?, 'receive_started', 'cloudflare-email-routing', ?, ?, ?)`, + ) + .bind(id, ownerId, id, detailJson, now.toISOString()) + .run() + } + const upsertDeliveryEvent = vi.fn() + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace({}), + MAILBOX: namespace({ + getInboundDelivery: vi.fn(async () => null), + upsertDeliveryEvent, + }), + }, + userId: ownerId, + }) + + await expect(authority.get(crossOwner.deliveryId)).resolves.toBeNull() + await expect(authority.get(malformedId)).resolves.toBeNull() + expect(upsertDeliveryEvent).not.toHaveBeenCalled() + expect(consoleWarn).toHaveBeenCalledWith( + 'inbound-email-delivery-bootstrap-row-invalid', + ownerId, + crossOwner.deliveryId, + ) + expect(consoleWarn).toHaveBeenCalledWith( + 'inbound-email-delivery-bootstrap-row-invalid', + ownerId, + malformedId, + ) + sqlite.close() +}) + +test('dedupe prune projects only the exact Mailbox IDs with owner/provider fences', async () => { + const sqlite = new DatabaseSync(':memory:') + const db = createD1FromSqlite(sqlite) + await ensureEmailTestSchema(db) + const ownerId = 'owner-prune' + const selectedId = 'email-inbound-dedupe:selected' + const unselectedId = 'email-inbound-dedupe:unselected' + const foreignId = 'email-inbound-dedupe:foreign' + const wrongProviderId = 'email-inbound-dedupe:wrong-provider' + const now = new Date('2026-08-02T12:00:00.000Z') + for (const [id, userId, provider] of [ + [selectedId, ownerId, 'cloudflare-email-routing-dedupe'], + [unselectedId, ownerId, 'cloudflare-email-routing-dedupe'], + [foreignId, 'other-owner', 'cloudflare-email-routing-dedupe'], + [wrongProviderId, ownerId, 'cloudflare-email-routing'], + ] as const) { + await db + .prepare( + `INSERT INTO email_delivery_events ( + id, user_id, event_type, provider, created_at, updated_at + ) VALUES (?, ?, 'receive_started', ?, ?, ?)`, + ) + .bind(id, userId, provider, now.toISOString(), now.toISOString()) + .run() + } + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: db, + USER_METER: namespace({}), + MAILBOX: namespace({ + pruneExpiredInboundDedupePointers: vi.fn(async () => ({ + pruned: 3, + prunedEventIds: [selectedId, foreignId, wrongProviderId], + })), + }), + }, + userId: ownerId, + }) + + expect(await authority.pruneExpiredDedupe(now, 10)).toBe(3) + const remaining = await db + .prepare( + `SELECT id FROM email_delivery_events + ORDER BY id`, + ) + .all<{ id: string }>() + expect(remaining.results?.map((row) => row.id)).toEqual([ + foreignId, + unselectedId, + wrongProviderId, + ]) + sqlite.close() +}) + +test('empty Mailbox dedupe prune does not issue a D1 delete', async () => { + const prepare = vi.fn() + const authority = createUserInboundDeliveryAuthority({ + env: { + APP_DB: { prepare } as unknown as D1Database, + USER_METER: namespace({}), + MAILBOX: namespace({ + pruneExpiredInboundDedupePointers: vi.fn(async () => ({ + pruned: 0, + prunedEventIds: [], + })), + }), + }, + userId: 'owner-empty-prune', + }) + + expect(await authority.pruneExpiredDedupe()).toBe(0) + expect(prepare).not.toHaveBeenCalled() +}) + +test('system D1 mutation surface rejects USER deliveries before writing', async () => { + const now = new Date('2026-08-02T12:00:00.000Z') + const delivery = await buildInboundDelivery({ + userId: 'user-cannot-use-system-d1', + inboxId: 'inbox-system-fence', + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + rawMime: 'system fence', + quotaDay: '2026-08-02', + now, + }) + + await expect( + claimSystemInboundDeliveryWindow({ + db: {} as D1Database, + delivery, + now, + }), + ).rejects.toThrow('restricted to system:email') +}) diff --git a/packages/worker/src/email/inbound-delivery-authority.ts b/packages/worker/src/email/inbound-delivery-authority.ts new file mode 100644 index 0000000000..2eaddd163b --- /dev/null +++ b/packages/worker/src/email/inbound-delivery-authority.ts @@ -0,0 +1,674 @@ +import { + buildEntitlementUpgradeHint, + EntitlementLimitError, +} from '#worker/entitlements/errors.ts' +import { type PlanName } from '#worker/entitlements/plans.ts' +import { + userMeterRpc, + type UserMeterEnv, +} from '#worker/entitlements/user-meter-client.ts' +import { + InboundDeliveryLeaseLostError, + type InboundDelivery, +} from './inbound-delivery.ts' +import { mailboxRpc, type MailboxEnv } from './mailbox-client.ts' +import { + type MailboxClaimInboundSubscriptionEffectResult, + type MailboxClaimInboundUsageEffectResult, + type MailboxCompleteInboundSubscriptionEffectResult, + type MailboxCompleteInboundUsageEffectResult, + type MailboxFailInboundSubscriptionEffectResult, + type MailboxListDueInboundEffectWorkResult, +} from './mailbox-inbound-effect-ledger.ts' +import { + mailboxInboundDedupePointerId, + mailboxInboundDedupeProvider, + type MailboxClaimInboundDeliveryCleanupResult, + type MailboxDeferInboundDeliveryReconcileResult, + type MailboxInboundDeliveryInsertInput, + type MailboxInboundDeliverySnapshot, + type MailboxListDueStaleInboundDeliveriesResult, + type MailboxMarkInboundDeliveryOrphanCleanedResult, + type MailboxReleaseInboundDeliveryCleanupResult, +} from './mailbox-inbound-ledger.ts' +import { + bootstrapUserInboundDeliveryFromD1, + bootstrapUserInboundDeliveryWindowFromD1, + mirrorUserInboundDeliverySnapshotToD1, + toUserInboundDelivery, +} from './inbound-delivery-projection.ts' +import { systemEmailOwnerId } from './email-owner.ts' + +export { mirrorUserInboundDeliverySnapshotToD1 } from './inbound-delivery-projection.ts' + +export type UserInboundDeliveryAuthorityEnv = { + APP_DB: D1Database +} & MailboxEnv & + UserMeterEnv + +export type CreateUserInboundDeliveryAuthorityInput = { + env: UserInboundDeliveryAuthorityEnv + userId: string +} + +export type UserInboundDeliveryChargeInput = { + delivery: InboundDelivery + plan: PlanName + limit: number + now: Date +} + +export type UserInboundDeliveryChargeResult = { + delivery: InboundDelivery + charged: boolean +} + +export type UserInboundDeliveryReceiveInput = { + delivery: InboundDelivery + usageDurationMs: number + usageMonth: string + usageBytes: number + now?: Date +} + +export type UserInboundDeliveryStorageClaimResult = + | { claimed: true; delivery: InboundDelivery } + | { claimed: false; delivery: InboundDelivery | null } + +export type UserInboundDeliveryOrphanCleanedInput = { + deliveryId: string + cleanupLease: string + outcome: 'deleted' | 'delete-failed' + now?: Date +} + +export type UserInboundDeliveryCleanupClaimInput = { + deliveryId: string + expectedState: InboundDelivery['state'] + expectedUpdatedAt: string + staleBefore: Date + now?: Date +} + +export type UserInboundUsageEffectClaimInput = { + deliveryId: string + expectedFinalizationToken?: string + now?: Date +} + +export type UserInboundUsageEffectCompleteInput = { + deliveryId: string + usageEffectLease: string + expectedFinalizationToken: string + mode: 'recorded' | 'suppressed' + usageMonth: string + usageBytes: number + usageDurationMs: number + now?: Date +} + +export type UserInboundSubscriptionEffectClaimInput = { + deliveryId: string + expectedFinalizationToken?: string + now?: Date +} + +export type UserInboundSubscriptionEffectCompleteInput = { + deliveryId: string + subscriptionEffectLease: string + expectedFinalizationToken: string + mode: 'complete' | 'suppressed' + suppressionReason?: string + now?: Date +} + +export type UserInboundSubscriptionEffectFailInput = { + deliveryId: string + subscriptionEffectLease: string + expectedFinalizationToken: string + error: string + now?: Date +} + +export type UserInboundDeliveryAuthority = { + get: (deliveryId: string) => Promise + getWindow: (fingerprint: string, now: Date) => Promise + claimWindow: ( + delivery: InboundDelivery, + now: Date, + ) => Promise + charge: ( + input: UserInboundDeliveryChargeInput, + ) => Promise + claimStorage: ( + delivery: InboundDelivery, + expectedAttachmentCount: number, + usageStartedAt?: string, + now?: Date, + ) => Promise + releaseStorage: (delivery: InboundDelivery, now?: Date) => Promise + reject: ( + delivery: InboundDelivery, + reason: string, + now?: Date, + ) => Promise + receive: (input: UserInboundDeliveryReceiveInput) => Promise + deferReconciliation: ( + deliveryId: string, + now?: Date, + ) => Promise + listDueStale: ( + now?: Date, + limit?: number, + ) => Promise + claimCleanup: ( + input: UserInboundDeliveryCleanupClaimInput, + ) => Promise + releaseCleanup: ( + deliveryId: string, + cleanupLease: string, + now?: Date, + ) => Promise + markOrphanCleaned: ( + input: UserInboundDeliveryOrphanCleanedInput, + ) => Promise + pruneExpiredDedupe: (now?: Date, limit?: number) => Promise + claimUsageEffect: ( + input: UserInboundUsageEffectClaimInput, + ) => Promise + completeUsageEffect: ( + input: UserInboundUsageEffectCompleteInput, + ) => Promise + claimSubscriptionEffect: ( + input: UserInboundSubscriptionEffectClaimInput, + ) => Promise + completeSubscriptionEffect: ( + input: UserInboundSubscriptionEffectCompleteInput, + ) => Promise + failSubscriptionEffect: ( + input: UserInboundSubscriptionEffectFailInput, + ) => Promise + listDueEffects: ( + now?: Date, + limit?: number, + ) => Promise +} + +function toInsertInput( + delivery: InboundDelivery, +): MailboxInboundDeliveryInsertInput { + return { + fingerprint: delivery.fingerprint, + deliveryId: delivery.deliveryId, + messageId: delivery.messageId, + threadId: delivery.threadId, + rawMimeKey: delivery.rawMimeKey, + inboxId: delivery.inboxId, + recipient: delivery.recipient, + envelopeFrom: delivery.envelopeFrom, + provider: delivery.provider, + quotaDay: delivery.quotaDay, + dedupeExpiresAt: delivery.dedupeExpiresAt, + usageStartedAt: delivery.usageStartedAt, + } +} + +export function createUserInboundDeliveryAuthority( + input: CreateUserInboundDeliveryAuthorityInput, +): UserInboundDeliveryAuthority { + const { env, userId } = input + if (userId === systemEmailOwnerId) { + throw new Error( + 'system:email inbound delivery authority must remain in D1.', + ) + } + const mailbox = mailboxRpc({ env, userId }) + const mirror = async ( + snapshot: MailboxInboundDeliverySnapshot, + options?: { eventId?: string; provider?: string }, + ) => { + await mirrorUserInboundDeliverySnapshotToD1({ + db: env.APP_DB, + userId, + snapshot, + ...options, + }) + return toUserInboundDelivery(userId, snapshot) + } + const mirrorClaimBestEffort = async ( + snapshot: MailboxInboundDeliverySnapshot, + logLabel: string, + ) => { + await mirror(snapshot).catch((error: unknown) => { + console.warn(logLabel, userId, snapshot.deliveryId, error) + }) + } + /** + * Mailbox is authoritative. A successful point read synchronously repairs + * the D1 compatibility projection; only a Mailbox miss may invoke the + * one-time pre-deploy bootstrap bridge. + */ + const get = async (deliveryId: string) => { + const current = + (await mailbox.getInboundDelivery({ ownerId: userId, deliveryId })) ?? + (await bootstrapUserInboundDeliveryFromD1({ env, userId, deliveryId })) + return current ? await mirror(current) : null + } + const getWindow = async (fingerprint: string, now: Date) => { + const current = + (await mailbox.getInboundDeliveryWindow({ + ownerId: userId, + fingerprint, + now: now.toISOString(), + })) ?? + (await bootstrapUserInboundDeliveryWindowFromD1({ + env, + userId, + fingerprint, + now, + })) + return current + ? await mirror(current, { + eventId: mailboxInboundDedupePointerId(fingerprint), + provider: mailboxInboundDedupeProvider, + }) + : null + } + const claimWindow = async (delivery: InboundDelivery, now: Date) => { + const snapshot = await mailbox.claimInboundDeliveryWindow({ + ownerId: userId, + delivery: toInsertInput(delivery), + now: now.toISOString(), + }) + return await mirror(snapshot, { + eventId: mailboxInboundDedupePointerId(delivery.fingerprint), + provider: mailboxInboundDedupeProvider, + }) + } + const charge = async (chargeInput: UserInboundDeliveryChargeInput) => { + // Resolve the canonical dedupe winner before charging. Concurrent + // boundary candidates therefore consume quota only under the winner id. + const claimedDelivery = await claimWindow( + chargeInput.delivery, + chargeInput.now, + ) + const existing = await get(claimedDelivery.deliveryId) + if (existing) return { delivery: existing, charged: false } + const chargedDelivery = { + ...claimedDelivery, + quotaDay: chargeInput.delivery.quotaDay, + } + const updatedAt = chargeInput.now.toISOString() + const meter = userMeterRpc({ env, userId }) + // Accepted non-atomic cross-DO window: if Email Routing exhausts retries + // after UserMeter consume but before Mailbox insert, one daily receive unit + // may burn until reset. Replays self-heal while retries continue and cannot + // duplicate a message because both operations use the dedupe winner id. + let meterResult = await meter.consumeInboundDelivery({ + deliveryId: chargedDelivery.deliveryId, + resource: 'email_receives_per_day', + day: chargedDelivery.quotaDay, + limit: chargeInput.limit, + updatedAt, + }) + if (meterResult.outcome === 'needs_bootstrap') { + await meter.initialize({ + resource: 'email_receives_per_day', + day: chargedDelivery.quotaDay, + count: 0, + updatedAt, + }) + meterResult = await meter.consumeInboundDelivery({ + deliveryId: chargedDelivery.deliveryId, + resource: 'email_receives_per_day', + day: chargedDelivery.quotaDay, + limit: chargeInput.limit, + updatedAt, + }) + if (meterResult.outcome === 'needs_bootstrap') { + throw new Error( + 'UserMeter inbound delivery consume still needs bootstrap after initialize.', + ) + } + } + if (!meterResult.consumed && !meterResult.replayed) { + throw new EntitlementLimitError({ + resource: 'email_receives_per_day', + plan: chargeInput.plan, + limit: chargeInput.limit, + current: meterResult.count, + upgradeHint: buildEntitlementUpgradeHint('email_receives_per_day'), + }) + } + const result = await mailbox.insertChargedPendingInboundDelivery({ + ownerId: userId, + delivery: toInsertInput(chargedDelivery), + now: updatedAt, + }) + const delivery = await mirror(result.delivery) + return { + delivery, + charged: + result.status === 'inserted' && + claimedDelivery.deliveryId === chargeInput.delivery.deliveryId, + } + } + return { + get, + getWindow, + claimWindow, + charge, + async claimStorage( + delivery: InboundDelivery, + expectedAttachmentCount: number, + usageStartedAt?: string, + now = new Date(), + ) { + const result = await mailbox.claimInboundDeliveryStorage({ + ownerId: userId, + deliveryId: delivery.deliveryId, + expectedAttachmentCount, + usageStartedAt, + now: now.toISOString(), + }) + if (result.status === 'claimed') { + const storageLease = result.delivery.storageLease + if (!storageLease) { + throw new Error( + 'Mailbox returned a claimed inbound storage delivery without a lease.', + ) + } + let projected: InboundDelivery + try { + projected = await mirror(result.delivery) + } catch (projectionError) { + let compensationError: unknown + try { + await mailbox.releaseInboundDeliveryStorage({ + ownerId: userId, + deliveryId: result.delivery.deliveryId, + storageLease, + now: now.toISOString(), + }) + } catch (error) { + compensationError = error + } + if (compensationError) { + throw new AggregateError( + [projectionError, compensationError], + 'Inbound storage claim projection failed and its Mailbox lease could not be released.', + ) + } + throw projectionError + } + return { + claimed: true as const, + delivery: projected, + } + } + return { + claimed: false as const, + delivery: result.delivery ? await mirror(result.delivery) : null, + } + }, + async releaseStorage(delivery: InboundDelivery, now = new Date()) { + if (!delivery.storageLease) return + const result = await mailbox.releaseInboundDeliveryStorage({ + ownerId: userId, + deliveryId: delivery.deliveryId, + storageLease: delivery.storageLease, + now: now.toISOString(), + }) + if (result.status === 'released') await mirror(result.delivery) + }, + async reject(delivery: InboundDelivery, reason: string, now = new Date()) { + const result = await mailbox.markInboundDeliveryRejected({ + ownerId: userId, + deliveryId: delivery.deliveryId, + reason, + expectedStorageLease: delivery.storageLease, + expectedState: delivery.state, + now: now.toISOString(), + }) + if ( + result.status === 'rejected' || + result.status === 'already-rejected' + ) { + const canonical = toUserInboundDelivery(userId, result.delivery) + try { + return await mirror(result.delivery) + } catch (error) { + console.warn( + 'inbound-email-rejected-projection-failed', + userId, + result.delivery.deliveryId, + error, + ) + return canonical + } + } + if (result.status === 'already-received') { + await mirror(result.delivery) + return null + } + throw new InboundDeliveryLeaseLostError( + 'Inbound rejection lost a state race; delivery should be retried.', + ) + }, + async receive(input: UserInboundDeliveryReceiveInput) { + if (!input.delivery.storageLease) { + throw new InboundDeliveryLeaseLostError( + 'Inbound delivery finalization requires a storage lease.', + ) + } + const result = await mailbox.markInboundDeliveryReceived({ + ownerId: userId, + deliveryId: input.delivery.deliveryId, + storageLease: input.delivery.storageLease, + usageDurationMs: input.usageDurationMs, + usageMonth: input.usageMonth, + usageBytes: input.usageBytes, + now: (input.now ?? new Date()).toISOString(), + }) + if ( + result.status === 'received' || + result.status === 'already-received' + ) { + return await mirror(result.delivery) + } + throw new InboundDeliveryLeaseLostError( + 'Inbound delivery storage lease was lost before finalization.', + ) + }, + async deferReconciliation(deliveryId: string, now = new Date()) { + const result = await mailbox.deferInboundDeliveryReconciliation({ + ownerId: userId, + deliveryId, + now: now.toISOString(), + }) + if (result.status === 'deferred') await mirror(result.delivery) + return result + }, + async listDueStale(now = new Date(), limit?: number) { + return await mailbox.listDueStaleInboundDeliveries({ + ownerId: userId, + now: now.toISOString(), + limit, + }) + }, + async claimCleanup(input: UserInboundDeliveryCleanupClaimInput) { + const now = input.now ?? new Date() + const result = await mailbox.claimInboundDeliveryCleanup({ + ownerId: userId, + deliveryId: input.deliveryId, + expectedState: input.expectedState, + expectedUpdatedAt: input.expectedUpdatedAt, + staleBefore: input.staleBefore.toISOString(), + now: now.toISOString(), + }) + if (result.status === 'claimed') { + const cleanupLease = result.delivery.cleanupLease + if (!cleanupLease) { + throw new Error( + 'Mailbox returned a claimed inbound cleanup delivery without a lease.', + ) + } + try { + await mirror(result.delivery) + } catch (projectionError) { + let compensationError: unknown + try { + await mailbox.releaseInboundDeliveryCleanup({ + ownerId: userId, + deliveryId: result.delivery.deliveryId, + cleanupLease, + now: now.toISOString(), + }) + } catch (error) { + compensationError = error + } + if (compensationError) { + throw new AggregateError( + [projectionError, compensationError], + 'Inbound cleanup claim projection failed and its Mailbox lease could not be released.', + ) + } + throw projectionError + } + } else if (result.delivery) { + await mirror(result.delivery) + } + return result + }, + async releaseCleanup( + deliveryId: string, + cleanupLease: string, + now = new Date(), + ) { + const result = await mailbox.releaseInboundDeliveryCleanup({ + ownerId: userId, + deliveryId, + cleanupLease, + now: now.toISOString(), + }) + if (result.status === 'released') await mirror(result.delivery) + return result + }, + async markOrphanCleaned(input: UserInboundDeliveryOrphanCleanedInput) { + const result = await mailbox.markInboundDeliveryOrphanCleaned({ + ownerId: userId, + deliveryId: input.deliveryId, + cleanupLease: input.cleanupLease, + outcome: input.outcome, + now: (input.now ?? new Date()).toISOString(), + }) + if (result.status === 'orphan-cleaned') await mirror(result.delivery) + return result + }, + async pruneExpiredDedupe(now = new Date(), limit?: number) { + const result = await mailbox.pruneExpiredInboundDedupePointers({ + ownerId: userId, + now: now.toISOString(), + limit, + }) + if (result.prunedEventIds.length === 0) return 0 + const placeholders = result.prunedEventIds.map(() => '?').join(', ') + await env.APP_DB.prepare( + `DELETE FROM email_delivery_events + WHERE user_id = ? + AND provider = ? + AND id IN (${placeholders})`, + ) + .bind(userId, mailboxInboundDedupeProvider, ...result.prunedEventIds) + .run() + return result.pruned + }, + async claimUsageEffect(input: UserInboundUsageEffectClaimInput) { + const result = await mailbox.claimInboundUsageEffect({ + ownerId: userId, + deliveryId: input.deliveryId, + expectedFinalizationToken: input.expectedFinalizationToken, + now: (input.now ?? new Date()).toISOString(), + }) + if (result.status === 'claimed') { + await mirrorClaimBestEffort( + result.delivery, + 'inbound-email-usage-effect-claim-projection-failed', + ) + } else if (result.delivery) { + await mirror(result.delivery) + } + return result + }, + async completeUsageEffect(input: UserInboundUsageEffectCompleteInput) { + const result = await mailbox.completeInboundUsageEffect({ + ownerId: userId, + deliveryId: input.deliveryId, + usageEffectLease: input.usageEffectLease, + expectedFinalizationToken: input.expectedFinalizationToken, + mode: input.mode, + usageMonth: input.usageMonth, + usageBytes: input.usageBytes, + usageDurationMs: input.usageDurationMs, + now: (input.now ?? new Date()).toISOString(), + }) + if (result.status !== 'lease-lost') await mirror(result.delivery) + return result + }, + async claimSubscriptionEffect( + input: UserInboundSubscriptionEffectClaimInput, + ) { + const result = await mailbox.claimInboundSubscriptionEffect({ + ownerId: userId, + deliveryId: input.deliveryId, + expectedFinalizationToken: input.expectedFinalizationToken, + now: (input.now ?? new Date()).toISOString(), + }) + if (result.status === 'claimed') { + await mirrorClaimBestEffort( + result.delivery, + 'inbound-email-subscription-effect-claim-projection-failed', + ) + } else if (result.delivery) { + await mirror(result.delivery) + } + return result + }, + async completeSubscriptionEffect( + input: UserInboundSubscriptionEffectCompleteInput, + ) { + const result = await mailbox.completeInboundSubscriptionEffect({ + ownerId: userId, + deliveryId: input.deliveryId, + subscriptionEffectLease: input.subscriptionEffectLease, + expectedFinalizationToken: input.expectedFinalizationToken, + mode: input.mode, + suppressionReason: input.suppressionReason, + now: (input.now ?? new Date()).toISOString(), + }) + if (result.status !== 'lease-lost') await mirror(result.delivery) + return result + }, + async failSubscriptionEffect( + input: UserInboundSubscriptionEffectFailInput, + ) { + const result = await mailbox.failInboundSubscriptionEffect({ + ownerId: userId, + deliveryId: input.deliveryId, + subscriptionEffectLease: input.subscriptionEffectLease, + expectedFinalizationToken: input.expectedFinalizationToken, + error: input.error, + now: (input.now ?? new Date()).toISOString(), + }) + if (result.status !== 'lease-lost') await mirror(result.delivery) + return result + }, + async listDueEffects(now = new Date(), limit?: number) { + return await mailbox.listDueInboundEffectWork({ + ownerId: userId, + now: now.toISOString(), + limit, + }) + }, + } satisfies UserInboundDeliveryAuthority +} diff --git a/packages/worker/src/email/inbound-delivery-projection.ts b/packages/worker/src/email/inbound-delivery-projection.ts new file mode 100644 index 0000000000..b85f064fe7 --- /dev/null +++ b/packages/worker/src/email/inbound-delivery-projection.ts @@ -0,0 +1,401 @@ +import { + parseStrictInboundDeliveryDetailJson, + type InboundDelivery, +} from './inbound-delivery.ts' +import { mailboxRpc, type MailboxEnv } from './mailbox-client.ts' +import { + detailJsonFromMailboxInboundSnapshot, + mailboxInboundDedupePointerId, + mailboxInboundDedupeProvider, + mailboxInboundProvider, + needsMailboxEffectReconcile, + type MailboxInboundDeliverySnapshot, +} from './mailbox-inbound-ledger.ts' +import { + type MailboxDeliveryEventInput, + type MailboxInboundDeliveryState, +} from './mailbox-types.ts' +import { systemEmailOwnerId } from './email-owner.ts' + +export type UserInboundDeliveryProjectionEnv = { + APP_DB: D1Database +} & MailboxEnv + +export type UserInboundDeliveryProjectionResult = + | { status: 'mirrored' } + | { status: 'stale' } + +export function toUserInboundDelivery( + userId: string, + snapshot: MailboxInboundDeliverySnapshot, +): InboundDelivery { + const { createdAt: _createdAt, updatedAt: _updatedAt, ...delivery } = snapshot + return { ...delivery, userId } +} + +function eventTypeForState(state: MailboxInboundDeliveryState) { + switch (state) { + case 'received': + return 'received' as const + case 'rejected': + return 'rejected' as const + case 'pending': + case 'storing': + case 'cleaning': + case 'orphan-cleaned': + return 'receive_started' as const + default: { + const exhaustive: never = state + throw new Error(`Unhandled inbound delivery state: ${String(exhaustive)}`) + } + } +} + +function toMailboxBootstrapEvent(input: { + delivery: InboundDelivery + eventId: string + provider: string + createdAt: string + updatedAt: string +}): MailboxDeliveryEventInput { + const delivery = input.delivery + const isLifecycle = input.provider === mailboxInboundProvider + const cleanupRetryAt = + isLifecycle && delivery.state === 'orphan-cleaned' + ? (delivery.cleanupRetryAt ?? null) + : null + const reconcileAfter = + isLifecycle && + (delivery.state === 'pending' || + delivery.state === 'storing' || + delivery.state === 'cleaning' || + delivery.state === 'orphan-cleaned') + ? (delivery.reconcileAfter ?? null) + : null + return { + id: input.eventId, + messageId: delivery.state === 'received' ? delivery.messageId : null, + inboxId: delivery.inboxId, + eventType: eventTypeForState(delivery.state), + provider: input.provider, + providerMessageId: null, + providerEventId: input.eventId, + detailJson: JSON.stringify(delivery), + needsEffectReconcile: + delivery.state === 'received' && needsMailboxEffectReconcile(delivery), + state: delivery.state, + fingerprint: delivery.fingerprint, + storageLease: delivery.storageLease ?? null, + storageLeaseAt: delivery.storageLeaseAt ?? null, + cleanupLease: delivery.cleanupLease ?? null, + cleanupLeaseAt: delivery.cleanupLeaseAt ?? null, + cleanupRetryAt, + expectedAttachmentCount: delivery.expectedAttachmentCount ?? null, + finalizationToken: delivery.finalizationToken ?? null, + reconcileAfter, + dedupeExpiresAt: delivery.dedupeExpiresAt, + usageEffectRecordedAt: delivery.usageEffectRecordedAt ?? null, + usageEffectSuppressedAt: delivery.usageEffectSuppressedAt ?? null, + usageStartedAt: delivery.usageStartedAt ?? null, + usageMonth: delivery.usageMonth ?? null, + usageBytes: delivery.usageBytes ?? null, + usageDurationMs: delivery.usageDurationMs ?? null, + usageEffectRetryAt: delivery.usageEffectRetryAt ?? null, + usageEffectLease: delivery.usageEffectLease ?? null, + usageEffectLeaseAt: delivery.usageEffectLeaseAt ?? null, + subscriptionEffectState: delivery.subscriptionEffectState ?? null, + subscriptionEffectLease: delivery.subscriptionEffectLease ?? null, + subscriptionEffectLeaseAt: delivery.subscriptionEffectLeaseAt ?? null, + subscriptionEffectRetryAt: delivery.subscriptionEffectRetryAt ?? null, + subscriptionEffectAttemptCount: + delivery.subscriptionEffectAttemptCount ?? null, + subscriptionEffectDeadLetterAt: + delivery.subscriptionEffectDeadLetterAt ?? null, + subscriptionEffectLastError: delivery.subscriptionEffectLastError ?? null, + createdAt: input.createdAt, + updatedAt: input.updatedAt, + } +} + +async function d1EventTimestamps(input: { + db: D1Database + userId: string + eventId: string + provider: string +}) { + return await input.db + .prepare( + `SELECT detail_json, created_at, + COALESCE(updated_at, created_at) AS updated_at + FROM email_delivery_events + WHERE id = ? AND user_id = ? AND provider = ? + LIMIT 1`, + ) + .bind(input.eventId, input.userId, input.provider) + .first<{ + detail_json: string | null + created_at: string + updated_at: string + }>() +} + +function validatedBootstrapDelivery(input: { + detailJson: unknown + userId: string + deliveryId?: string + fingerprint?: string +}) { + const delivery = parseStrictInboundDeliveryDetailJson(input.detailJson) + if (!delivery) return null + if ( + delivery.userId !== input.userId || + delivery.provider !== mailboxInboundProvider || + (input.deliveryId != null && delivery.deliveryId !== input.deliveryId) || + (input.fingerprint != null && delivery.fingerprint !== input.fingerprint) + ) { + return null + } + return delivery +} + +/** + * Full Mailbox → D1 compatibility snapshot. The conflict fence prevents a + * different owner/provider from being overwritten, while updated_at rejects + * delayed mirrors from an older Mailbox CAS. + */ +export async function mirrorUserInboundDeliverySnapshotToD1(input: { + db: D1Database + userId: string + snapshot: MailboxInboundDeliverySnapshot + eventId?: string + provider?: string +}): Promise { + const snapshot = input.snapshot + const eventId = input.eventId ?? snapshot.deliveryId + const provider = input.provider ?? mailboxInboundProvider + const eventType = eventTypeForState(snapshot.state) + const detailJson = detailJsonFromMailboxInboundSnapshot(snapshot, { + userId: input.userId, + }) + const result = await input.db + .prepare( + `INSERT INTO email_delivery_events ( + id, message_id, user_id, inbox_id, event_type, provider, + provider_message_id, provider_event_id, detail_json, + needs_effect_reconcile, state, fingerprint, + storage_lease, storage_lease_at, cleanup_lease, cleanup_lease_at, + cleanup_retry_at, expected_attachment_count, finalization_token, + reconcile_after, dedupe_expires_at, usage_effect_recorded_at, + usage_effect_suppressed_at, usage_started_at, usage_month, + usage_bytes, usage_duration_ms, usage_effect_retry_at, + usage_effect_lease, usage_effect_lease_at, subscription_effect_state, + subscription_effect_lease, subscription_effect_lease_at, + subscription_effect_retry_at, subscription_effect_attempt_count, + subscription_effect_dead_letter_at, subscription_effect_last_error, + created_at, updated_at + ) VALUES ( + ?, ?, ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, + ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? + ) + ON CONFLICT(id) DO UPDATE SET + message_id = excluded.message_id, + inbox_id = excluded.inbox_id, + event_type = excluded.event_type, + provider_event_id = excluded.provider_event_id, + detail_json = excluded.detail_json, + needs_effect_reconcile = excluded.needs_effect_reconcile, + state = excluded.state, + fingerprint = excluded.fingerprint, + storage_lease = excluded.storage_lease, + storage_lease_at = excluded.storage_lease_at, + cleanup_lease = excluded.cleanup_lease, + cleanup_lease_at = excluded.cleanup_lease_at, + cleanup_retry_at = excluded.cleanup_retry_at, + expected_attachment_count = excluded.expected_attachment_count, + finalization_token = excluded.finalization_token, + reconcile_after = excluded.reconcile_after, + dedupe_expires_at = excluded.dedupe_expires_at, + usage_effect_recorded_at = excluded.usage_effect_recorded_at, + usage_effect_suppressed_at = excluded.usage_effect_suppressed_at, + usage_started_at = excluded.usage_started_at, + usage_month = excluded.usage_month, + usage_bytes = excluded.usage_bytes, + usage_duration_ms = excluded.usage_duration_ms, + usage_effect_retry_at = excluded.usage_effect_retry_at, + usage_effect_lease = excluded.usage_effect_lease, + usage_effect_lease_at = excluded.usage_effect_lease_at, + subscription_effect_state = excluded.subscription_effect_state, + subscription_effect_lease = excluded.subscription_effect_lease, + subscription_effect_lease_at = excluded.subscription_effect_lease_at, + subscription_effect_retry_at = excluded.subscription_effect_retry_at, + subscription_effect_attempt_count = excluded.subscription_effect_attempt_count, + subscription_effect_dead_letter_at = excluded.subscription_effect_dead_letter_at, + subscription_effect_last_error = excluded.subscription_effect_last_error, + updated_at = excluded.updated_at + WHERE email_delivery_events.user_id = excluded.user_id + AND email_delivery_events.provider = excluded.provider + AND excluded.updated_at >= COALESCE( + email_delivery_events.updated_at, + email_delivery_events.created_at + )`, + ) + .bind( + eventId, + snapshot.state === 'received' ? snapshot.messageId : null, + input.userId, + snapshot.inboxId, + eventType, + provider, + eventId, + detailJson, + snapshot.state === 'received' && needsMailboxEffectReconcile(snapshot) + ? 1 + : 0, + snapshot.state, + snapshot.fingerprint, + snapshot.storageLease ?? null, + snapshot.storageLeaseAt ?? null, + snapshot.cleanupLease ?? null, + snapshot.cleanupLeaseAt ?? null, + snapshot.cleanupRetryAt ?? null, + snapshot.expectedAttachmentCount ?? null, + snapshot.finalizationToken ?? null, + snapshot.reconcileAfter ?? null, + snapshot.dedupeExpiresAt, + snapshot.usageEffectRecordedAt ?? null, + snapshot.usageEffectSuppressedAt ?? null, + snapshot.usageStartedAt ?? null, + snapshot.usageMonth ?? null, + snapshot.usageBytes ?? null, + snapshot.usageDurationMs ?? null, + snapshot.usageEffectRetryAt ?? null, + snapshot.usageEffectLease ?? null, + snapshot.usageEffectLeaseAt ?? null, + snapshot.subscriptionEffectState ?? null, + snapshot.subscriptionEffectLease ?? null, + snapshot.subscriptionEffectLeaseAt ?? null, + snapshot.subscriptionEffectRetryAt ?? null, + snapshot.subscriptionEffectAttemptCount ?? null, + snapshot.subscriptionEffectDeadLetterAt ?? null, + snapshot.subscriptionEffectLastError ?? null, + snapshot.createdAt, + snapshot.updatedAt, + ) + .run() + if (Number(result.meta.changes ?? 0) > 0) return { status: 'mirrored' } + const existing = await input.db + .prepare( + `SELECT user_id, provider, COALESCE(updated_at, created_at) AS updated_at + FROM email_delivery_events + WHERE id = ? + LIMIT 1`, + ) + .bind(eventId) + .first<{ user_id: string; provider: string; updated_at: string }>() + if ( + !existing || + existing.user_id !== input.userId || + existing.provider !== provider + ) { + throw new Error( + 'Mailbox inbound snapshot failed its D1 owner/provider fence.', + ) + } + if (existing.updated_at >= snapshot.updatedAt) return { status: 'stale' } + throw new Error( + 'Mailbox inbound snapshot was not applied and D1 has no newer projection.', + ) +} + +export async function bootstrapUserInboundDeliveryFromD1(input: { + env: UserInboundDeliveryProjectionEnv + userId: string + deliveryId: string +}) { + if (input.userId === systemEmailOwnerId) return null + const timestamps = await d1EventTimestamps({ + db: input.env.APP_DB, + userId: input.userId, + eventId: input.deliveryId, + provider: mailboxInboundProvider, + }) + if (!timestamps) return null + const delivery = validatedBootstrapDelivery({ + detailJson: timestamps.detail_json, + userId: input.userId, + deliveryId: input.deliveryId, + }) + if (!delivery) { + console.warn( + 'inbound-email-delivery-bootstrap-row-invalid', + input.userId, + input.deliveryId, + ) + return null + } + const mailbox = mailboxRpc({ env: input.env, userId: input.userId }) + await mailbox.bootstrapDeliveryEvents({ + ownerId: input.userId, + events: [ + toMailboxBootstrapEvent({ + delivery, + eventId: input.deliveryId, + provider: mailboxInboundProvider, + createdAt: timestamps.created_at, + updatedAt: timestamps.updated_at, + }), + ], + }) + return await mailbox.getInboundDelivery({ + ownerId: input.userId, + deliveryId: input.deliveryId, + }) +} + +export async function bootstrapUserInboundDeliveryWindowFromD1(input: { + env: UserInboundDeliveryProjectionEnv + userId: string + fingerprint: string + now: Date +}) { + if (input.userId === systemEmailOwnerId) return null + const eventId = mailboxInboundDedupePointerId(input.fingerprint) + const timestamps = await d1EventTimestamps({ + db: input.env.APP_DB, + userId: input.userId, + eventId, + provider: mailboxInboundDedupeProvider, + }) + if (!timestamps) return null + const delivery = validatedBootstrapDelivery({ + detailJson: timestamps.detail_json, + userId: input.userId, + fingerprint: input.fingerprint, + }) + if (!delivery || delivery.dedupeExpiresAt <= input.now.toISOString()) { + console.warn( + 'inbound-email-dedupe-bootstrap-row-invalid', + input.userId, + eventId, + ) + return null + } + const mailbox = mailboxRpc({ env: input.env, userId: input.userId }) + await mailbox.bootstrapDeliveryEvents({ + ownerId: input.userId, + events: [ + toMailboxBootstrapEvent({ + delivery, + eventId, + provider: mailboxInboundDedupeProvider, + createdAt: timestamps.created_at, + updatedAt: timestamps.updated_at, + }), + ], + }) + return await mailbox.getInboundDeliveryWindow({ + ownerId: input.userId, + fingerprint: input.fingerprint, + now: input.now.toISOString(), + }) +} diff --git a/packages/worker/src/email/inbound-delivery-reconciliation-authority.ts b/packages/worker/src/email/inbound-delivery-reconciliation-authority.ts new file mode 100644 index 0000000000..4c8c812f45 --- /dev/null +++ b/packages/worker/src/email/inbound-delivery-reconciliation-authority.ts @@ -0,0 +1,311 @@ +import PostalMime from 'postal-mime' +import { emailRawMimeKey } from './blob-keys.ts' +import { + createUserInboundDeliveryAuthority, + type UserInboundDeliveryAuthorityEnv, +} from './inbound-delivery-authority.ts' +import { + InboundDeliveryLeaseLostError, + parseStrictInboundDeliveryDetailJson, +} from './inbound-delivery.ts' +import { + mailboxInboundDedupePointerId, + mailboxInboundProvider, + mailboxStaleInboundDeliveryAgeMs, +} from './mailbox-inbound-ledger.ts' +import { + insertEmailAttachments, + getEmailMessageById, + listEmailAttachmentsForMessage, +} from './repo.ts' + +const staleBatchSize = 20 + +function parsedAttachmentSize( + content: string | ArrayBuffer | Uint8Array, +) { + return typeof content === 'string' + ? new TextEncoder().encode(content).byteLength + : content.byteLength +} + +async function bootstrapPreDeployDueRows(input: { + env: UserInboundDeliveryAuthorityEnv + userId: string + now: Date +}) { + const rows = await input.env.APP_DB.prepare( + `SELECT id + FROM email_delivery_events + WHERE user_id = ? + AND provider = 'cloudflare-email-routing' + AND event_type = 'receive_started' + AND created_at < ? + ORDER BY created_at ASC, id ASC + LIMIT ?`, + ) + .bind( + input.userId, + new Date(input.now.getTime() - 48 * 60 * 60 * 1000).toISOString(), + staleBatchSize, + ) + .all<{ id: string }>() + const authority = createUserInboundDeliveryAuthority(input) + for (const row of rows.results ?? []) { + await authority.get(row.id).catch((error: unknown) => { + console.warn( + 'inbound-email-stale-bootstrap-failed', + input.userId, + row.id, + error, + ) + }) + } +} + +async function recoverCommittedDelivery(input: { + env: UserInboundDeliveryAuthorityEnv & { EMAIL_BLOBS: R2Bucket } + userId: string + deliveryId: string + now: Date +}) { + const authority = createUserInboundDeliveryAuthority(input) + const delivery = await authority.get(input.deliveryId) + if (!delivery) return false + const message = await getEmailMessageById({ + db: input.env.APP_DB, + userId: input.userId, + messageId: delivery.messageId, + }) + if (!message?.rawMimeKey) return false + if ( + message.rawMimeKey !== emailRawMimeKey(input.userId, delivery.messageId) + ) { + throw new Error( + 'Inbound message raw MIME key is outside its user namespace.', + ) + } + const object = await input.env.EMAIL_BLOBS.get(message.rawMimeKey) + if (!object) return false + const parsed = await PostalMime.parse(await object.text(), { + attachmentEncoding: 'arraybuffer', + }) + const claim = await authority.claimStorage( + delivery, + parsed.attachments.length, + delivery.usageStartedAt, + input.now, + ) + if (!claim.claimed) return claim.delivery?.state === 'received' + const storageLease = claim.delivery.storageLease + if (!storageLease) { + throw new InboundDeliveryLeaseLostError( + 'Recovered inbound delivery has no storage lease.', + ) + } + try { + await insertEmailAttachments({ + db: input.env.APP_DB, + messageId: message.id, + ignoreConflicts: true, + inboundDeliveryFence: { + deliveryId: claim.delivery.deliveryId, + userId: input.userId, + storageLease, + }, + attachments: parsed.attachments.map((attachment, index) => ({ + id: `${message.id}:attachment:${index}`, + filename: attachment.filename, + contentType: attachment.mimeType, + contentId: attachment.contentId ?? null, + disposition: attachment.disposition, + size: parsedAttachmentSize(attachment.content), + storageKind: 'raw-mime', + storageKey: null, + })), + }) + const attachments = await listEmailAttachmentsForMessage({ + db: input.env.APP_DB, + messageId: message.id, + }) + if (attachments.length !== parsed.attachments.length) { + throw new InboundDeliveryLeaseLostError( + 'Inbound attachment recovery did not commit every attachment.', + ) + } + await authority.receive({ + delivery: claim.delivery, + usageDurationMs: claim.delivery.usageStartedAt + ? input.now.getTime() - Date.parse(claim.delivery.usageStartedAt) + : 0, + usageMonth: (message.receivedAt ?? message.createdAt).slice(0, 7), + usageBytes: message.rawSize ?? 0, + now: input.now, + }) + return true + } catch (error) { + await authority + .releaseStorage(claim.delivery, input.now) + .catch(() => undefined) + throw error + } +} + +export async function reconcileUserStaleInboundDeliveries(input: { + env: UserInboundDeliveryAuthorityEnv & { EMAIL_BLOBS: R2Bucket } + userId: string + now?: Date + deadlineMs?: number +}) { + const now = input.now ?? new Date() + await bootstrapPreDeployDueRows({ + env: input.env, + userId: input.userId, + now, + }) + const authority = createUserInboundDeliveryAuthority(input) + const due = await authority.listDueStale(now, staleBatchSize) + const staleBefore = new Date(now.getTime() - mailboxStaleInboundDeliveryAgeMs) + let recovered = 0 + let cleaned = 0 + let budgetExhausted = false + for (const snapshot of due.deliveries) { + if (input.deadlineMs != null && Date.now() >= input.deadlineMs) { + budgetExhausted = true + break + } + const message = await getEmailMessageById({ + db: input.env.APP_DB, + userId: input.userId, + messageId: snapshot.messageId, + }) + if (message) { + try { + if ( + await recoverCommittedDelivery({ + env: input.env, + userId: input.userId, + deliveryId: snapshot.deliveryId, + now, + }) + ) { + recovered += 1 + } else { + await authority.deferReconciliation(snapshot.deliveryId, now) + } + } catch (error) { + console.warn( + 'inbound-email-partial-delivery-recovery-failed', + snapshot.deliveryId, + error, + ) + await authority + .deferReconciliation(snapshot.deliveryId, now) + .catch(() => undefined) + } + continue + } + + const claim = await authority.claimCleanup({ + deliveryId: snapshot.deliveryId, + expectedState: snapshot.state, + expectedUpdatedAt: snapshot.updatedAt, + staleBefore, + now, + }) + if (claim.status !== 'claimed') continue + const racedMessage = await getEmailMessageById({ + db: input.env.APP_DB, + userId: input.userId, + messageId: snapshot.messageId, + }) + if (racedMessage) { + await authority.releaseCleanup( + snapshot.deliveryId, + claim.delivery.cleanupLease!, + now, + ) + continue + } + let outcome: 'deleted' | 'delete-failed' = 'deleted' + try { + await input.env.EMAIL_BLOBS.delete(snapshot.rawMimeKey) + } catch (error) { + outcome = 'delete-failed' + console.warn( + 'inbound-email-orphan-blob-delete-failed', + snapshot.rawMimeKey, + error, + ) + } + const finalized = await authority.markOrphanCleaned({ + deliveryId: snapshot.deliveryId, + cleanupLease: claim.delivery.cleanupLease!, + outcome, + now, + }) + if (finalized.status === 'orphan-cleaned') { + if (outcome === 'deleted') cleaned += 1 + } + } + return { + recovered, + cleaned, + ...(budgetExhausted ? { budgetExhausted: true as const } : {}), + } +} + +export async function pruneUserExpiredInboundDedupePointers(input: { + env: UserInboundDeliveryAuthorityEnv + userId: string + now?: Date + limit?: number +}) { + const now = input.now ?? new Date() + const authority = createUserInboundDeliveryAuthority(input) + const bridgeRows = await input.env.APP_DB.prepare( + `SELECT id, detail_json + FROM email_delivery_events + WHERE user_id = ? + AND provider = ? + AND COALESCE( + dedupe_expires_at, + json_extract(detail_json, '$.dedupeExpiresAt') + ) <= ? + ORDER BY created_at ASC, id ASC + LIMIT ?`, + ) + .bind( + input.userId, + 'cloudflare-email-routing-dedupe', + now.toISOString(), + input.limit ?? 20, + ) + .all<{ id: string; detail_json: string | null }>() + for (const row of bridgeRows.results ?? []) { + const delivery = parseStrictInboundDeliveryDetailJson(row.detail_json) + if ( + !delivery || + delivery.userId !== input.userId || + delivery.provider !== mailboxInboundProvider || + delivery.state !== 'pending' || + row.id !== mailboxInboundDedupePointerId(delivery.fingerprint) + ) { + console.warn( + 'inbound-email-dedupe-bridge-row-invalid', + input.userId, + row.id, + ) + continue + } + await authority.claimWindow(delivery, now).catch((error: unknown) => { + console.warn( + 'inbound-email-dedupe-bridge-failed', + input.userId, + row.id, + error, + ) + }) + } + return await authority.pruneExpiredDedupe(now, input.limit) +} diff --git a/packages/worker/src/email/inbound-delivery.ts b/packages/worker/src/email/inbound-delivery.ts index 44726389c7..ffd5f98410 100644 --- a/packages/worker/src/email/inbound-delivery.ts +++ b/packages/worker/src/email/inbound-delivery.ts @@ -10,6 +10,7 @@ import { type UserMeterEnv, } from '#worker/entitlements/user-meter-client.ts' import { normalizeEmailAddress } from './address.ts' +import { systemEmailOwnerId } from './email-owner.ts' import { emailRawMimeKey, getEmailMessageById, @@ -54,7 +55,9 @@ export type InboundDelivery = { expectedAttachmentCount?: number cleanupLease?: string cleanupLeaseAt?: string + cleanupRetryAt?: string finalizationToken?: string + reconcileAfter?: string usageEffectRecordedAt?: string usageEffectSuppressedAt?: string usageStartedAt?: string @@ -149,12 +152,12 @@ export async function buildInboundDelivery(input: { } } -function parseInboundDelivery( - row: InboundDeliveryEventRow | null, +export function parseInboundDeliveryDetailJson( + detailJson: unknown, ): InboundDelivery | null { - if (!row) return null + if (typeof detailJson !== 'string') return null try { - const detail = JSON.parse(row.detail_json) as Partial + const detail = JSON.parse(detailJson) as Partial if ( typeof detail.deliveryId !== 'string' || typeof detail.fingerprint !== 'string' || @@ -216,9 +219,15 @@ function parseInboundDelivery( ...(typeof detail.cleanupLeaseAt === 'string' ? { cleanupLeaseAt: detail.cleanupLeaseAt } : {}), + ...(typeof detail.cleanupRetryAt === 'string' + ? { cleanupRetryAt: detail.cleanupRetryAt } + : {}), ...(typeof detail.finalizationToken === 'string' ? { finalizationToken: detail.finalizationToken } : {}), + ...(typeof detail.reconcileAfter === 'string' + ? { reconcileAfter: detail.reconcileAfter } + : {}), ...(typeof detail.usageEffectRecordedAt === 'string' ? { usageEffectRecordedAt: detail.usageEffectRecordedAt } : {}), @@ -282,6 +291,45 @@ function parseInboundDelivery( } } +export function parseStrictInboundDeliveryDetailJson( + detailJson: unknown, +): InboundDelivery | null { + const delivery = parseInboundDeliveryDetailJson(detailJson) + if (!delivery || typeof detailJson !== 'string') return null + try { + const raw = JSON.parse(detailJson) as unknown + if ( + typeof raw !== 'object' || + raw == null || + !('state' in raw) || + raw.state !== delivery.state + ) { + return null + } + const record = raw as Record + for (const field of ['cleanupRetryAt', 'reconcileAfter'] as const) { + const value = record[field] + if ( + field in record && + (typeof value !== 'string' || + !Number.isFinite(Date.parse(value)) || + new Date(value).toISOString() !== value) + ) { + return null + } + } + return delivery + } catch { + return null + } +} + +function parseInboundDelivery( + row: InboundDeliveryEventRow | null, +): InboundDelivery | null { + return parseInboundDeliveryDetailJson(row?.detail_json) +} + export async function getInboundDelivery(input: { db: D1Database userId: string @@ -607,6 +655,9 @@ export async function chargeSystemInboundDeliveryOnce(input: { limit: number now: Date }) { + if (input.delivery.userId !== systemEmailOwnerId) { + throw new Error('System inbound delivery charge requires system:email.') + } const existing = await resolveConcurrentDelivery(input) if (existing) return { delivery: existing, overLimit: false as const } const current = await readSystemEmailDailyCounter({ diff --git a/packages/worker/src/email/inbound-effects.ts b/packages/worker/src/email/inbound-effects.ts index 4bda1b0f46..ccf0d66943 100644 --- a/packages/worker/src/email/inbound-effects.ts +++ b/packages/worker/src/email/inbound-effects.ts @@ -1,11 +1,12 @@ import { getInboundDelivery } from './inbound-delivery.ts' import { withAccountWriteLease } from '#worker/account/deletion-state.ts' +import { createUserInboundDeliveryAuthority } from './inbound-delivery-authority.ts' import { dispatchInboundEmailSubscriptionEvents, dispatchSystemInboundEmailSubscriptionEvents, } from './package-subscriptions.ts' import { getEmailMessageById } from './repo.ts' -import { systemEmailOwnerId } from './system-email.ts' +import { systemEmailOwnerId } from './email-owner.ts' const subscriptionEffectLeaseMs = 5 * 60 * 1000 const effectRetryMs = 15 * 60 * 1000 @@ -21,7 +22,12 @@ export function resolveSubscriptionEffectFailure(attemptCount: number) { type InboundEffectsEnv = Pick< Env, - 'APP_DB' | 'BUNDLE_ARTIFACTS_KV' | 'APP_BASE_URL' | 'USAGE_EVENTS' + | 'APP_DB' + | 'BUNDLE_ARTIFACTS_KV' + | 'APP_BASE_URL' + | 'USAGE_EVENTS' + | 'MAILBOX' + | 'USER_METER' > async function recordInboundUsageEffect(input: { @@ -232,7 +238,66 @@ async function recordInboundUsageEffect(input: { .run() } -async function processInboundDeliveryEffectsWithLeaseHeld(input: { +async function recordUserInboundUsageRollup(input: { + env: InboundEffectsEnv + userId: string + deliveryId: string + finalizationToken: string + usageMonth: string + usageBytes: number + usageDurationMs: number + now: Date +}) { + if (input.env.USAGE_EVENTS) return + try { + await input.env.APP_DB.batch([ + input.env.APP_DB.prepare( + `INSERT INTO usage_rollups ( + user_id, metric, month, event_count, error_count, + total_duration_ms, total_cpu_ms, total_bytes, updated_at + ) + SELECT ?, 'email_received', ?, 1, 0, ?, 0, ?, ? + WHERE NOT EXISTS ( + SELECT 1 FROM email_inbound_usage_effects + WHERE user_id = ? AND delivery_id = ? AND finalization_token = ? + ) + ON CONFLICT (user_id, metric, month) DO UPDATE SET + event_count = event_count + 1, + total_duration_ms = total_duration_ms + excluded.total_duration_ms, + total_bytes = total_bytes + excluded.total_bytes, + updated_at = excluded.updated_at`, + ).bind( + input.userId, + input.usageMonth, + Math.round(input.usageDurationMs), + Math.round(input.usageBytes), + input.now.toISOString(), + input.userId, + input.deliveryId, + input.finalizationToken, + ), + input.env.APP_DB.prepare( + `INSERT OR IGNORE INTO email_inbound_usage_effects ( + user_id, delivery_id, finalization_token, created_at + ) VALUES (?, ?, ?, ?)`, + ).bind( + input.userId, + input.deliveryId, + input.finalizationToken, + input.now.toISOString(), + ), + ]) + } catch (error) { + if ( + !(error instanceof Error) || + !error.message.includes('no such table: usage_rollups') + ) { + throw error + } + } +} + +async function processUserInboundDeliveryEffectsWithLeaseHeld(input: { env: InboundEffectsEnv userId: string deliveryId: string @@ -241,6 +306,140 @@ async function processInboundDeliveryEffectsWithLeaseHeld(input: { now?: Date waitUntil?: (promise: Promise) => void }) { + const now = input.now ?? new Date() + const authority = createUserInboundDeliveryAuthority({ + env: input.env, + userId: input.userId, + }) + const delivery = await authority.get(input.deliveryId) + if ( + delivery?.state !== 'received' || + !delivery.finalizationToken || + (input.expectedFinalizationToken != null && + delivery.finalizationToken !== input.expectedFinalizationToken) + ) { + return { outcome: 'stale' as const } + } + const message = await getEmailMessageById({ + db: input.env.APP_DB, + userId: input.userId, + messageId: delivery.messageId, + }) + const usageMonth = + delivery.usageMonth ?? + (message + ? (message.receivedAt ?? message.createdAt).slice(0, 7) + : now.toISOString().slice(0, 7)) + const usageBytes = delivery.usageBytes ?? message?.rawSize ?? 0 + const usageDurationMs = delivery.usageDurationMs ?? input.durationMs ?? 0 + const usageClaim = await authority.claimUsageEffect({ + deliveryId: delivery.deliveryId, + expectedFinalizationToken: delivery.finalizationToken, + now, + }) + if (usageClaim.status === 'claimed') { + await recordUserInboundUsageRollup({ + env: input.env, + userId: input.userId, + deliveryId: delivery.deliveryId, + finalizationToken: delivery.finalizationToken, + usageMonth, + usageBytes, + usageDurationMs, + now, + }) + const completed = await authority.completeUsageEffect({ + deliveryId: delivery.deliveryId, + usageEffectLease: usageClaim.delivery.usageEffectLease!, + expectedFinalizationToken: delivery.finalizationToken, + mode: 'recorded', + usageMonth, + usageBytes, + usageDurationMs, + now, + }) + if (completed.status === 'lease-lost') { + return { outcome: 'stale' as const } + } + } + + const subscriptionClaim = await authority.claimSubscriptionEffect({ + deliveryId: delivery.deliveryId, + expectedFinalizationToken: delivery.finalizationToken, + now, + }) + if (subscriptionClaim.status !== 'claimed') { + return { outcome: 'usage-only' as const } + } + const effectLease = subscriptionClaim.delivery.subscriptionEffectLease! + try { + if (!message) { + await authority.completeSubscriptionEffect({ + deliveryId: delivery.deliveryId, + subscriptionEffectLease: effectLease, + expectedFinalizationToken: delivery.finalizationToken, + mode: 'suppressed', + suppressionReason: 'missing-message', + now, + }) + return { outcome: 'missing-message' as const } + } + await dispatchInboundEmailSubscriptionEvents({ + env: input.env, + userId: input.userId, + message, + waitUntil: input.waitUntil, + }) + const completed = await authority.completeSubscriptionEffect({ + deliveryId: delivery.deliveryId, + subscriptionEffectLease: effectLease, + expectedFinalizationToken: delivery.finalizationToken, + mode: 'complete', + now, + }) + if (completed.status === 'lease-lost') { + return { outcome: 'stale' as const } + } + return { outcome: 'complete' as const } + } catch (error) { + const failure = await authority.failSubscriptionEffect({ + deliveryId: delivery.deliveryId, + subscriptionEffectLease: effectLease, + expectedFinalizationToken: delivery.finalizationToken, + error: error instanceof Error ? error.message : String(error), + now, + }) + if (failure.status === 'lease-lost') { + return { outcome: 'stale' as const } + } + if (failure.status === 'dead-letter') { + console.error('inbound-email-subscription-effect-dead-lettered', { + userId: input.userId, + deliveryId: delivery.deliveryId, + attemptCount: failure.delivery.subscriptionEffectAttemptCount, + error, + }) + return { outcome: 'dead-letter' as const } + } + throw error + } +} + +export type ProcessInboundDeliveryEffectsInput = { + env: InboundEffectsEnv + userId: string + deliveryId: string + expectedFinalizationToken?: string + durationMs?: number + now?: Date + waitUntil?: (promise: Promise) => void +} + +// system:email deliberately retains the legacy D1 authority and effect policy; +// USER deliveries use the Mailbox CAS engine below. Keep the split explicit. +async function processSystemInboundDeliveryEffectsWithLeaseHeld( + input: ProcessInboundDeliveryEffectsInput, +) { const now = input.now ?? new Date() const delivery = await getInboundDelivery({ db: input.env.APP_DB, @@ -505,15 +704,16 @@ async function processInboundDeliveryEffectsWithLeaseHeld(input: { } export async function processInboundDeliveryEffects( - input: Parameters[0], + input: ProcessInboundDeliveryEffectsInput, ) { if (input.userId === systemEmailOwnerId) { - return await processInboundDeliveryEffectsWithLeaseHeld(input) + return await processSystemInboundDeliveryEffectsWithLeaseHeld(input) } return await withAccountWriteLease({ db: input.env.APP_DB, stableUserId: input.userId, - write: async () => await processInboundDeliveryEffectsWithLeaseHeld(input), + write: async () => + await processUserInboundDeliveryEffectsWithLeaseHeld(input), }) } @@ -524,6 +724,59 @@ export async function reconcileInboundDeliveryEffectsForUser(input: { limit?: number }) { const now = input.now ?? new Date() + if (input.userId !== systemEmailOwnerId) { + const authority = createUserInboundDeliveryAuthority({ + env: input.env, + userId: input.userId, + }) + const bridgeRows = await input.env.APP_DB.prepare( + `SELECT id FROM email_delivery_events + WHERE user_id = ? + AND provider = 'cloudflare-email-routing' + AND event_type = 'received' + AND needs_effect_reconcile = 1 + ORDER BY created_at ASC, id ASC + LIMIT ?`, + ) + .bind(input.userId, input.limit ?? 20) + .all<{ id: string }>() + let processed = 0 + let errors = 0 + for (const row of bridgeRows.results ?? []) { + try { + await authority.get(row.id) + } catch (error) { + errors += 1 + console.warn( + 'inbound-email-effect-bridge-failed', + input.userId, + row.id, + error, + ) + } + } + const due = await authority.listDueEffects(now, input.limit) + for (const delivery of due.deliveries) { + try { + await processInboundDeliveryEffects({ + env: input.env, + userId: input.userId, + deliveryId: delivery.deliveryId, + now: input.now, + }) + processed += 1 + } catch (error) { + errors += 1 + console.warn( + 'inbound-email-effect-reconciliation-failed', + input.userId, + delivery.deliveryId, + error, + ) + } + } + return { processed, errors } + } const leaseExpiredBefore = new Date( now.getTime() - subscriptionEffectLeaseMs, ).toISOString() diff --git a/packages/worker/src/email/inbound-mailbox-authority-mirror-migration.node.test.ts b/packages/worker/src/email/inbound-mailbox-authority-mirror-migration.node.test.ts new file mode 100644 index 0000000000..bef0e65954 --- /dev/null +++ b/packages/worker/src/email/inbound-mailbox-authority-mirror-migration.node.test.ts @@ -0,0 +1,290 @@ +import { readdirSync, readFileSync } from 'node:fs' +import { DatabaseSync } from 'node:sqlite' +import { expect, test } from 'vitest' + +const migrationsDirectory = new URL('../../migrations/', import.meta.url) +const authorityMirrorMigration = + '0129-email-inbound-mailbox-authority-mirror.sql' + +function applyMigrationsBefore(db: DatabaseSync, exclusiveUpperBound: string) { + for (const fileName of readdirSync(migrationsDirectory) + .filter((file) => file.endsWith('.sql') && file < exclusiveUpperBound) + .sort()) { + db.exec(readFileSync(new URL(fileName, migrationsDirectory), 'utf8')) + } +} + +function applyMigrationLikeD1(db: DatabaseSync, fileName: string) { + const sql = readFileSync(new URL(fileName, migrationsDirectory), 'utf8') + db.exec('BEGIN') + try { + db.exec(sql) + db.exec('COMMIT') + } catch (error) { + db.exec('ROLLBACK') + throw error + } +} + +test('0129 promotes legacy inbound authority and cascades usage idempotency', () => { + using db = new DatabaseSync(':memory:') + applyMigrationsBefore(db, authorityMirrorMigration) + expect( + db + .prepare(`PRAGMA table_info(email_delivery_events)`) + .all() + .some((column) => column.name === 'state'), + ).toBe(false) + + const createdAt = '2026-08-01T00:00:00.000Z' + const insert = db.prepare( + `INSERT INTO email_delivery_events ( + id, user_id, event_type, provider, detail_json, + needs_effect_reconcile, usage_effect_recorded_at, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ) + insert.run( + 'delivery-backfill', + 'user-backfill', + 'received', + 'cloudflare-email-routing', + JSON.stringify({ + state: 'received', + fingerprint: 'fingerprint-backfill', + storageLease: 'storage-lease', + storageLeaseAt: '2026-08-01T00:01:00.000Z', + cleanupLease: 'cleanup-lease', + cleanupLeaseAt: '2026-08-01T00:02:00.000Z', + cleanupRetryAt: '2026-08-01T00:03:00.000Z', + expectedAttachmentCount: 2, + finalizationToken: 'finalization-token', + reconcileAfter: '2026-08-01T00:04:00.000Z', + dedupeExpiresAt: '2026-08-02T00:00:00.000Z', + usageEffectSuppressedAt: '2026-08-01T00:05:00.000Z', + usageStartedAt: '2026-08-01T00:06:00.000Z', + usageEffectRetryAt: '2026-08-01T00:07:00.000Z', + usageEffectLease: 'usage-lease', + usageEffectLeaseAt: '2026-08-01T00:08:00.000Z', + subscriptionEffectState: 'processing', + subscriptionEffectLease: 'subscription-lease', + subscriptionEffectLeaseAt: '2026-08-01T00:09:00.000Z', + subscriptionEffectRetryAt: '2026-08-01T00:10:00.000Z', + subscriptionEffectAttemptCount: 3, + subscriptionEffectDeadLetterAt: '2026-08-01T00:11:00.000Z', + subscriptionEffectLastError: 'dispatch failed', + }), + 0, + null, + createdAt, + ) + insert.run( + 'usage-null-lease-at', + 'user-usage-due', + 'received', + 'cloudflare-email-routing', + JSON.stringify({ + state: 'received', + fingerprint: 'fingerprint-usage', + usageEffectLease: 'legacy-usage-lease', + subscriptionEffectState: 'complete', + }), + 1, + null, + createdAt, + ) + insert.run( + 'subscription-null-lease-at', + 'user-subscription-due', + 'received', + 'cloudflare-email-routing', + JSON.stringify({ + state: 'received', + fingerprint: 'fingerprint-subscription', + subscriptionEffectState: 'processing', + subscriptionEffectLease: 'legacy-subscription-lease', + }), + 1, + '2026-08-01T00:01:00.000Z', + createdAt, + ) + insert.run( + 'email-inbound-dedupe:fingerprint-dedupe', + 'user-dedupe', + 'receive_started', + 'cloudflare-email-routing-dedupe', + JSON.stringify({ + state: 'pending', + fingerprint: 'fingerprint-dedupe', + dedupeExpiresAt: '2026-08-01T01:00:00.000Z', + }), + 0, + null, + createdAt, + ) + insert.run( + 'delivery-system', + 'system:email', + 'received', + 'kody-system-email', + JSON.stringify({ + state: 'received', + fingerprint: 'system-fingerprint', + }), + 0, + null, + createdAt, + ) + + db.exec('PRAGMA foreign_keys = ON') + applyMigrationLikeD1(db, authorityMirrorMigration) + + expect( + db + .prepare( + `SELECT + state, fingerprint, storage_lease, storage_lease_at, + cleanup_lease, cleanup_lease_at, cleanup_retry_at, + expected_attachment_count, finalization_token, reconcile_after, + dedupe_expires_at, usage_effect_suppressed_at, usage_started_at, + usage_effect_retry_at, usage_effect_lease, usage_effect_lease_at, + subscription_effect_state, subscription_effect_lease, + subscription_effect_lease_at, subscription_effect_retry_at, + subscription_effect_attempt_count, + subscription_effect_dead_letter_at, + subscription_effect_last_error, updated_at + FROM email_delivery_events + WHERE id = 'delivery-backfill'`, + ) + .get(), + ).toEqual({ + state: 'received', + fingerprint: 'fingerprint-backfill', + storage_lease: 'storage-lease', + storage_lease_at: '2026-08-01T00:01:00.000Z', + cleanup_lease: 'cleanup-lease', + cleanup_lease_at: '2026-08-01T00:02:00.000Z', + cleanup_retry_at: '2026-08-01T00:03:00.000Z', + expected_attachment_count: 2, + finalization_token: 'finalization-token', + reconcile_after: '2026-08-01T00:04:00.000Z', + dedupe_expires_at: '2026-08-02T00:00:00.000Z', + usage_effect_suppressed_at: '2026-08-01T00:05:00.000Z', + usage_started_at: '2026-08-01T00:06:00.000Z', + usage_effect_retry_at: '2026-08-01T00:07:00.000Z', + usage_effect_lease: 'usage-lease', + usage_effect_lease_at: '2026-08-01T00:08:00.000Z', + subscription_effect_state: 'processing', + subscription_effect_lease: 'subscription-lease', + subscription_effect_lease_at: '2026-08-01T00:09:00.000Z', + subscription_effect_retry_at: '2026-08-01T00:10:00.000Z', + subscription_effect_attempt_count: 3, + subscription_effect_dead_letter_at: '2026-08-01T00:11:00.000Z', + subscription_effect_last_error: 'dispatch failed', + updated_at: createdAt, + }) + expect( + db + .prepare( + `SELECT state, fingerprint, updated_at + FROM email_delivery_events + WHERE id = 'delivery-system'`, + ) + .get(), + ).toEqual({ + state: null, + fingerprint: null, + updated_at: createdAt, + }) + expect( + db + .prepare( + `SELECT state, fingerprint, dedupe_expires_at, updated_at + FROM email_delivery_events + WHERE id = 'email-inbound-dedupe:fingerprint-dedupe'`, + ) + .get(), + ).toEqual({ + state: 'pending', + fingerprint: 'fingerprint-dedupe', + dedupe_expires_at: '2026-08-01T01:00:00.000Z', + updated_at: createdAt, + }) + + expect( + db + .prepare( + `SELECT name FROM sqlite_schema + WHERE type = 'index' + AND name IN ( + 'idx_email_delivery_events_user_state_created', + 'idx_email_delivery_events_user_dedupe_expires', + 'idx_email_inbound_usage_effects_delivery' + ) + ORDER BY name`, + ) + .all(), + ).toEqual([ + { name: 'idx_email_delivery_events_user_dedupe_expires' }, + { name: 'idx_email_delivery_events_user_state_created' }, + { name: 'idx_email_inbound_usage_effects_delivery' }, + ]) + expect( + db + .prepare( + `SELECT id + FROM email_delivery_events + WHERE provider = 'cloudflare-email-routing' + AND event_type = 'received' + AND needs_effect_reconcile = 1 + AND fingerprint IS NOT NULL + AND ( + ( + usage_effect_recorded_at IS NULL + AND usage_effect_suppressed_at IS NULL + AND ( + usage_effect_lease IS NULL + OR usage_effect_lease_at IS NULL + OR usage_effect_lease_at < ? + ) + ) + OR ( + usage_effect_recorded_at IS NOT NULL + AND subscription_effect_state = 'processing' + AND ( + subscription_effect_lease_at IS NULL + OR subscription_effect_lease_at < ? + ) + ) + ) + ORDER BY id`, + ) + .all('2026-08-01T00:30:00.000Z', '2026-08-01T00:30:00.000Z'), + ).toEqual([ + { id: 'subscription-null-lease-at' }, + { id: 'usage-null-lease-at' }, + ]) + + expect( + db.prepare(`PRAGMA foreign_key_list(email_inbound_usage_effects)`).all(), + ).toContainEqual( + expect.objectContaining({ + table: 'email_delivery_events', + from: 'delivery_id', + to: 'id', + on_delete: 'CASCADE', + }), + ) + db.prepare( + `INSERT INTO email_inbound_usage_effects ( + user_id, delivery_id, finalization_token, created_at + ) VALUES (?, ?, ?, ?)`, + ).run('user-backfill', 'delivery-backfill', 'finalization-token', createdAt) + db.prepare( + `DELETE FROM email_delivery_events WHERE id = 'delivery-backfill'`, + ).run() + expect( + db + .prepare(`SELECT COUNT(*) AS count FROM email_inbound_usage_effects`) + .get(), + ).toEqual({ count: 0 }) +}) diff --git a/packages/worker/src/email/inbound-mailbox-mirror.workers.test.ts b/packages/worker/src/email/inbound-mailbox-mirror.workers.test.ts index 9bb2f63c29..1da43d2e4d 100644 --- a/packages/worker/src/email/inbound-mailbox-mirror.workers.test.ts +++ b/packages/worker/src/email/inbound-mailbox-mirror.workers.test.ts @@ -1,6 +1,7 @@ import { env } from 'cloudflare:workers' import { expect, test, vi } from 'vitest' import { userMeterRpc } from '#worker/entitlements/user-meter-client.ts' +import { consoleWarn } from '#worker/test-support/console-spies.ts' import { silenceIncidentalRuntimeWarnings } from '#worker/test-support/incidental-runtime-warnings.ts' import { createStableUserIdFromEmail } from '#worker/user-id.ts' import { ensureUsageRollupsTestSchema } from '#worker/usage/test-schema.ts' @@ -88,9 +89,140 @@ function createFailingEmailBlobs() { }) } +function createRejectProjectionFailingDb(db: D1Database) { + let failed = false + const failingDb = new Proxy(db, { + get(target, property) { + if (property === 'prepare') { + return (query: string) => { + const statement = target.prepare(query) + if (!query.includes('INSERT INTO email_delivery_events')) { + return statement + } + return new Proxy(statement, { + get(statementTarget, statementProperty) { + if (statementProperty === 'bind') { + return (...values: Array) => { + const bound = statementTarget.bind(...values) + if (failed || values[4] !== 'rejected') return bound + return new Proxy(bound, { + get(boundTarget, boundProperty) { + if (boundProperty === 'run') { + return async () => { + failed = true + throw new Error( + 'simulated rejected projection failure', + ) + } + } + const value = Reflect.get(boundTarget, boundProperty) + return typeof value === 'function' + ? value.bind(boundTarget) + : value + }, + }) + } + } + const value = Reflect.get(statementTarget, statementProperty) + return typeof value === 'function' + ? value.bind(statementTarget) + : value + }, + }) + } + } + const value = Reflect.get(target, property) + return typeof value === 'function' ? value.bind(target) : value + }, + }) + return { db: failingDb, didFail: () => failed } +} + const inboundMailboxMirrorTimeoutMs = 30_000 -/** MAILBOX stub that fails or hangs every mirror RPC without touching the real DO. */ +function createLedgerBackedMailboxStub( + mailbox: ReturnType, +) { + return { + async getInboundDelivery( + ...args: Parameters + ) { + return await mailbox.getInboundDelivery(...args) + }, + async getInboundDeliveryWindow( + ...args: Parameters + ) { + return await mailbox.getInboundDeliveryWindow(...args) + }, + async claimInboundDeliveryWindow( + ...args: Parameters + ) { + return await mailbox.claimInboundDeliveryWindow(...args) + }, + async insertChargedPendingInboundDelivery( + ...args: Parameters + ) { + return await mailbox.insertChargedPendingInboundDelivery(...args) + }, + async claimInboundDeliveryStorage( + ...args: Parameters + ) { + return await mailbox.claimInboundDeliveryStorage(...args) + }, + async releaseInboundDeliveryStorage( + ...args: Parameters + ) { + return await mailbox.releaseInboundDeliveryStorage(...args) + }, + async markInboundDeliveryRejected( + ...args: Parameters + ) { + return await mailbox.markInboundDeliveryRejected(...args) + }, + async markInboundDeliveryReceived( + ...args: Parameters + ) { + return await mailbox.markInboundDeliveryReceived(...args) + }, + async pruneExpiredInboundDedupePointers( + ...args: Parameters + ) { + return await mailbox.pruneExpiredInboundDedupePointers(...args) + }, + async listDueStaleInboundDeliveries( + ...args: Parameters + ) { + return await mailbox.listDueStaleInboundDeliveries(...args) + }, + async claimInboundUsageEffect( + ...args: Parameters + ) { + return await mailbox.claimInboundUsageEffect(...args) + }, + async completeInboundUsageEffect( + ...args: Parameters + ) { + return await mailbox.completeInboundUsageEffect(...args) + }, + async claimInboundSubscriptionEffect( + ...args: Parameters + ) { + return await mailbox.claimInboundSubscriptionEffect(...args) + }, + async completeInboundSubscriptionEffect( + ...args: Parameters + ) { + return await mailbox.completeInboundSubscriptionEffect(...args) + }, + async failInboundSubscriptionEffect( + ...args: Parameters + ) { + return await mailbox.failInboundSubscriptionEffect(...args) + }, + } +} + +/** MAILBOX namespace that fails graph-mirror RPCs but keeps ledger CAS real. */ function createInboundMailboxStubEnv(input: { mode: 'throw' | 'hang' base?: typeof env @@ -101,23 +233,26 @@ function createInboundMailboxStubEnv(input: { throw new Error('simulated mailbox failure') } const method = input.mode === 'hang' ? hang : fail - const stub = { - mirrorMessage: method, - upsertDeliveryEvent: method, - upsertDeliveryEvents: method, - touchThread: method, - updateMessageDelivery: method, - setMessageClassification: method, - deleteMessageMetadata: method, - deleteDeliveryEvent: method, - deleteThreadIfEmpty: method, - } return { ...base, APP_BASE_URL: platformBaseUrl, MAILBOX: { idFromName: (name: string) => base.MAILBOX.idFromName(name), - get: () => stub, + get: (id: DurableObjectId) => { + const mailbox = base.MAILBOX.get(id) + return { + ...createLedgerBackedMailboxStub(mailbox), + mirrorMessage: method, + upsertDeliveryEvent: method, + upsertDeliveryEvents: method, + touchThread: method, + updateMessageDelivery: method, + setMessageClassification: method, + deleteMessageMetadata: method, + deleteDeliveryEvent: method, + deleteThreadIfEmpty: method, + } + }, } as unknown as DurableObjectNamespace, } } @@ -257,24 +392,12 @@ test( }) const order: Array = [] const delayedStub = { + ...createLedgerBackedMailboxStub(real), async mirrorMessage(...args: Parameters) { order.push('graph-message') await graphGate return await real.mirrorMessage(...args) }, - async upsertDeliveryEvents( - ...args: Parameters - ) { - order.push('graph-batch') - await graphGate - return await real.upsertDeliveryEvents(...args) - }, - async upsertDeliveryEvent( - ...args: Parameters - ) { - order.push('post-effects-event') - return await real.upsertDeliveryEvent(...args) - }, } const orderedEnv = { @@ -334,11 +457,7 @@ test( releaseGraph() await drainWaitUntil(waitUntilPromises) - expect(order).toEqual([ - 'graph-message', - 'graph-batch', - 'post-effects-event', - ]) + expect(order).toEqual(['graph-message']) const mailbox = rpcFor(userId) const mirroredEvents = await mailbox.listDeliveryEvents({ @@ -459,8 +578,8 @@ test( handleInboundEmail(first, failingEnv, failCtx.ctx), ).rejects.toBeInstanceOf(RetryableInboundStorageError) expect(first.rejectedReason).toBeNull() - // Charge may schedule UserMeter D1 mirror via waitUntil; no Mailbox - // terminal work should have produced a message below. + // The charged Mailbox delivery survives this retryable graph-storage + // failure; no terminal work should have produced a message below. await drainWaitUntil(failCtx.waitUntilPromises) expect(await readUserDailyReceiveCount(userId)).toBe(1) expect( @@ -796,3 +915,82 @@ test( }, inboundMailboxMirrorTimeoutMs, ) + +test.each([ + { label: 'without an execution context', withContext: false }, + { label: 'with an execution context', withContext: true }, +])( + 'SMTP rejection survives a rejected D1 projection failure $label', + async ({ withContext }) => { + silenceIncidentalRuntimeWarnings([ + 'inbound-email-rejected-projection-failed', + ]) + await ensureEmailTestSchema(env.APP_DB) + const username = `mbx-rej-fail-${crypto.randomUUID().slice(0, 8)}` + const accountEmail = `mbx-rej-fail-${crypto.randomUUID()}@example.com` + const userId = await createStableUserIdFromEmail(accountEmail) + const address = `${username}@${platformDomain}` + await seedVerifiedAccount({ + db: env.APP_DB, + email: accountEmail, + username, + }) + const mailbox = rpcFor(userId) + await mailbox.getMessage({ messageId: 'warmup-nonexistent' }) + const raw = [ + 'From: Sender ', + `To: ${address}`, + 'Subject: Permanent reject despite projection failure', + `Message-ID: `, + '', + 'Body', + ].join('\r\n') + const projection = createRejectProjectionFailingDb(env.APP_DB) + const inboundEnv = { + ...createInboundEnv(), + APP_DB: projection.db, + } + const captured = withContext ? createCapturedWaitUntilContext() : null + const parseSpy = vi + .spyOn(parser, 'parseForwardableEmailRawMime') + .mockRejectedValueOnce(new Error('permanent parse rejection')) + try { + const message = createForwardableEmailMessage({ + from: 'sender@example.net', + to: address, + raw, + }) + await handleInboundEmail(message, inboundEnv, captured?.ctx) + + expect(projection.didFail()).toBe(true) + expect(message.rejectedReason).toBe('permanent parse rejection') + expect(consoleWarn).toHaveBeenCalledWith( + 'inbound-email-rejected-projection-failed', + userId, + expect.any(String), + expect.any(Error), + ) + if (captured) await drainWaitUntil(captured.waitUntilPromises) + + const rejected = ( + await mailbox.listDeliveryEvents({ messageId: null, limit: 20 }) + ).find((event) => event.eventType === 'rejected') + expect(rejected).toMatchObject({ + eventType: 'rejected', + provider: 'cloudflare-email-routing', + }) + if (!rejected) throw new Error('Expected authoritative Mailbox rejection') + expect( + await env.APP_DB.prepare( + `SELECT event_type FROM email_delivery_events + WHERE id = ? AND user_id = ?`, + ) + .bind(rejected.id, userId) + .first<{ event_type: string }>(), + ).toEqual({ event_type: 'rejected' }) + } finally { + parseSpy.mockRestore() + } + }, + inboundMailboxMirrorTimeoutMs, +) diff --git a/packages/worker/src/email/inbound-mailbox.node.test.ts b/packages/worker/src/email/inbound-mailbox.node.test.ts index 925a1c524e..5bb8142742 100644 --- a/packages/worker/src/email/inbound-mailbox.node.test.ts +++ b/packages/worker/src/email/inbound-mailbox.node.test.ts @@ -9,9 +9,8 @@ const mocks = vi.hoisted(() => ({ events: [], eventsTruncated: false, })), - mirrorMailboxDeliveryEventFromD1: vi.fn(async () => ({ - status: 'mirrored' as const, - })), + authorityGet: vi.fn(async () => ({ state: 'received' as const })), + createAuthority: vi.fn(), processInboundDeliveryEffects: vi.fn(async () => ({ outcome: 'complete' as const, })), @@ -19,7 +18,10 @@ const mocks = vi.hoisted(() => ({ vi.mock('./mailbox-live-mirror.ts', () => ({ mirrorMailboxMessageGraphFromD1: mocks.mirrorMailboxMessageGraphFromD1, - mirrorMailboxDeliveryEventFromD1: mocks.mirrorMailboxDeliveryEventFromD1, +})) + +vi.mock('./inbound-delivery-authority.ts', () => ({ + createUserInboundDeliveryAuthority: mocks.createAuthority, })) vi.mock('./inbound-effects.ts', () => ({ @@ -44,7 +46,8 @@ function createCapturedWaitUntilContext() { function resetMocks() { mocks.mirrorMailboxMessageGraphFromD1.mockReset() - mocks.mirrorMailboxDeliveryEventFromD1.mockReset() + mocks.authorityGet.mockReset() + mocks.createAuthority.mockReset() mocks.processInboundDeliveryEffects.mockReset() mocks.mirrorMailboxMessageGraphFromD1.mockResolvedValue({ messageId: 'msg-1', @@ -52,15 +55,14 @@ function resetMocks() { events: [], eventsTruncated: false, }) - mocks.mirrorMailboxDeliveryEventFromD1.mockResolvedValue({ - status: 'mirrored' as const, - }) + mocks.authorityGet.mockResolvedValue({ state: 'received' as const }) + mocks.createAuthority.mockReturnValue({ get: mocks.authorityGet }) mocks.processInboundDeliveryEffects.mockResolvedValue({ outcome: 'complete' as const, }) } -test('received terminal work orders graph then effects then event mirror via waitUntil', async () => { +test('received terminal work orders graph then effects then authority snapshot via waitUntil', async () => { resetMocks() consoleError.mockImplementation(() => {}) const order: Array = [] @@ -84,9 +86,9 @@ test('received terminal work orders graph then effects then event mirror via wai order.push('effects') return { outcome: 'complete' as const } }) - mocks.mirrorMailboxDeliveryEventFromD1.mockImplementation(async () => { + mocks.authorityGet.mockImplementation(async () => { order.push('event') - return { status: 'mirrored' as const } + return { state: 'received' as const } }) const { ctx, waitUntilPromises } = createCapturedWaitUntilContext() @@ -121,12 +123,7 @@ test('received terminal work orders graph then effects then event mirror via wai durationMs: 12, waitUntil: expect.any(Function), }) - expect(mocks.mirrorMailboxDeliveryEventFromD1).toHaveBeenCalledWith({ - env, - db: env.APP_DB, - userId: 'user-aaa', - eventId: 'delivery-1', - }) + expect(mocks.authorityGet).toHaveBeenCalledWith('delivery-1') }) test('graph failure does not skip D1 effects; effects failure skips event mirror and logs once', async () => { @@ -154,7 +151,7 @@ test('graph failure does not skip D1 effects; effects failure skips event mirror }) expect(mocks.processInboundDeliveryEffects).toHaveBeenCalledTimes(1) - expect(mocks.mirrorMailboxDeliveryEventFromD1).not.toHaveBeenCalled() + expect(mocks.authorityGet).not.toHaveBeenCalled() expect(consoleError).toHaveBeenCalledWith( 'Inbound email effect dispatch failed', expect.objectContaining({ message: 'effects exploded' }), @@ -182,10 +179,10 @@ test('received and rejected coordinators skip system:email owners', async () => expect(mocks.mirrorMailboxMessageGraphFromD1).not.toHaveBeenCalled() expect(mocks.processInboundDeliveryEffects).not.toHaveBeenCalled() - expect(mocks.mirrorMailboxDeliveryEventFromD1).not.toHaveBeenCalled() + expect(mocks.authorityGet).not.toHaveBeenCalled() }) -test('rejected terminal schedules delivery-event mirror only', async () => { +test('rejected terminal schedules authority snapshot repair only', async () => { resetMocks() const { ctx, waitUntilPromises } = createCapturedWaitUntilContext() const env = { APP_DB: {} } as unknown as Parameters< @@ -203,10 +200,50 @@ test('rejected terminal schedules delivery-event mirror only', async () => { await Promise.all(waitUntilPromises) expect(mocks.mirrorMailboxMessageGraphFromD1).not.toHaveBeenCalled() expect(mocks.processInboundDeliveryEffects).not.toHaveBeenCalled() - expect(mocks.mirrorMailboxDeliveryEventFromD1).toHaveBeenCalledWith({ - env, - db: env.APP_DB, - userId: 'user-ccc', - eventId: 'delivery-3', + expect(mocks.authorityGet).toHaveBeenCalledWith('delivery-3') +}) + +test('rejected terminal contains and logs projection repair failures', async () => { + resetMocks() + consoleError.mockImplementation(() => {}) + mocks.authorityGet.mockRejectedValueOnce(new Error('projection unavailable')) + const { ctx, waitUntilPromises } = createCapturedWaitUntilContext() + + await scheduleInboundRejectedTerminalWork({ + env: { APP_DB: {} } as unknown as Parameters< + typeof scheduleInboundRejectedTerminalWork + >[0]['env'], + userId: 'user-ddd', + deliveryId: 'delivery-4', + ctx, }) + + expect(waitUntilPromises).toHaveLength(1) + await expect(Promise.all(waitUntilPromises)).resolves.toEqual([undefined]) + expect(consoleError).toHaveBeenCalledWith( + 'Inbound email rejection projection repair failed', + expect.objectContaining({ message: 'projection unavailable' }), + ) +}) + +test('rejected terminal contains synchronous authority construction failures', async () => { + resetMocks() + consoleError.mockImplementation(() => {}) + mocks.createAuthority.mockImplementation(() => { + throw new Error('MAILBOX binding unavailable') + }) + + await expect( + scheduleInboundRejectedTerminalWork({ + env: { APP_DB: {} } as unknown as Parameters< + typeof scheduleInboundRejectedTerminalWork + >[0]['env'], + userId: 'user-sync-construction', + deliveryId: 'delivery-sync-construction', + }), + ).resolves.toBeUndefined() + expect(consoleError).toHaveBeenCalledWith( + 'Inbound email rejection projection repair failed', + expect.objectContaining({ message: 'MAILBOX binding unavailable' }), + ) }) diff --git a/packages/worker/src/email/inbound-mailbox.ts b/packages/worker/src/email/inbound-mailbox.ts index 5076d63c1e..6e82151f83 100644 --- a/packages/worker/src/email/inbound-mailbox.ts +++ b/packages/worker/src/email/inbound-mailbox.ts @@ -1,7 +1,7 @@ import { isSystemEmailOwner } from './email-owner.ts' +import { createUserInboundDeliveryAuthority } from './inbound-delivery-authority.ts' import { processInboundDeliveryEffects } from './inbound-effects.ts' import { - mirrorMailboxDeliveryEventFromD1, mirrorMailboxMessageGraphFromD1, type MailboxLiveMirrorEnv, } from './mailbox-live-mirror.ts' @@ -9,7 +9,8 @@ import { type EmailReportingEnv } from './reporting-events.ts' /** * Env surface for inbound Mailbox terminal coordination. - * D1 remains authoritative; Mailbox dual-write is best-effort after commit. + * Mailbox owns USER inbound delivery state; D1 is a synchronous compatibility + * snapshot used by graph fences and scheduled owner discovery. */ export type InboundMailboxEnv = Pick< Env, @@ -17,6 +18,7 @@ export type InboundMailboxEnv = Pick< | 'BUNDLE_ARTIFACTS_KV' | 'APP_BASE_URL' | 'USAGE_EVENTS' + | 'USER_METER' | 'MAILBOX' | 'EMAIL_EVENTS' > & @@ -44,8 +46,8 @@ export type InboundRejectedTerminalWorkInput = { /** * One ordered received-terminal task: * 1) bounded full message graph mirror (never throws) - * 2) D1 inbound delivery effects - * 3) on effects success only, mirror the updated delivery event + * 2) Mailbox-authoritative inbound delivery effects + * 3) final Mailbox → D1 delivery snapshot repair * * Attach to `ctx.waitUntil` when present; otherwise await. Catch/log once so * Mailbox/effects failures never throw into Email Routing. @@ -68,6 +70,7 @@ export async function scheduleInboundReceivedTerminalWork( db: input.env.APP_DB, userId: input.userId, messageId: input.messageId, + includeDeliveryEvents: false, }) await processInboundDeliveryEffects({ env: input.env, @@ -77,12 +80,11 @@ export async function scheduleInboundReceivedTerminalWork( durationMs: input.durationMs, waitUntil: nestedWaitUntil, }) - await mirrorMailboxDeliveryEventFromD1({ + const authority = createUserInboundDeliveryAuthority({ env: input.env, - db: input.env.APP_DB, userId: input.userId, - eventId: input.deliveryId, }) + await authority.get(input.deliveryId) })().catch((error: unknown) => { console.error(input.logLabel, error) }) @@ -95,7 +97,7 @@ export async function scheduleInboundReceivedTerminalWork( } /** - * Rejected-terminal delivery-event mirror only (no message graph). + * Rejected-terminal Mailbox → D1 snapshot repair only (no message graph). * `system:email` is skipped. Never throws into Email Routing. */ export async function scheduleInboundRejectedTerminalWork( @@ -103,12 +105,17 @@ export async function scheduleInboundRejectedTerminalWork( ) { if (isSystemEmailOwner(input.userId)) return - const task = mirrorMailboxDeliveryEventFromD1({ - env: input.env, - db: input.env.APP_DB, - userId: input.userId, - eventId: input.deliveryId, - }) + const task = Promise.resolve() + .then(async () => { + const authority = createUserInboundDeliveryAuthority({ + env: input.env, + userId: input.userId, + }) + await authority.get(input.deliveryId) + }) + .catch((error: unknown) => { + console.error('Inbound email rejection projection repair failed', error) + }) if (input.ctx) { input.ctx.waitUntil(task) diff --git a/packages/worker/src/email/inbound.ts b/packages/worker/src/email/inbound.ts index adb4147f6d..8926454fef 100644 --- a/packages/worker/src/email/inbound.ts +++ b/packages/worker/src/email/inbound.ts @@ -26,22 +26,32 @@ import { } from './parser.ts' import { buildInboundDelivery, - claimInboundDeliveryWindow, - claimInboundDeliveryStorage, chargeSystemInboundDeliveryOnce, - chargeUserInboundDeliveryOnce, getInboundDeliveryWindow, getInboundDelivery, - markInboundDeliveryRejected, - pruneExpiredInboundDedupePointers, - reconcileStaleInboundDeliveries, readSystemInboundReceiveCount, readUserInboundReceiveCount, - releaseInboundDeliveryStorage, systemInboundQuotaDay, type InboundDelivery, userInboundQuotaDay, } from './inbound-delivery.ts' +import { + claimSystemInboundDeliveryStorage, + claimSystemInboundDeliveryWindow, + markSystemInboundDeliveryRejected, + pruneSystemExpiredInboundDedupePointers, + reconcileSystemStaleInboundDeliveries, + releaseSystemInboundDeliveryStorage, +} from './system-inbound-delivery-authority.ts' +import { + createUserInboundDeliveryAuthority, + type UserInboundDeliveryAuthority, + type UserInboundDeliveryAuthorityEnv, +} from './inbound-delivery-authority.ts' +import { + pruneUserExpiredInboundDedupePointers, + reconcileUserStaleInboundDeliveries, +} from './inbound-delivery-reconciliation-authority.ts' import { getPlatformEmailDomain, getSystemEmailDomain, @@ -85,12 +95,17 @@ async function rejectClaimedInboundDelivery(input: { message: ForwardableEmailMessage delivery: InboundDelivery reason: string + authority?: UserInboundDeliveryAuthority }) { - const transitioned = await markInboundDeliveryRejected({ - db: input.db, - delivery: input.delivery, - reason: input.reason, - }).catch((error: unknown) => { + const transitioned = await ( + input.authority + ? input.authority.reject(input.delivery, input.reason) + : markSystemInboundDeliveryRejected({ + db: input.db, + delivery: input.delivery, + reason: input.reason, + }) + ).catch((error: unknown) => { warnRejectionAuditWriteFailed(error) throw error }) @@ -120,6 +135,7 @@ async function parseAndStoreInboundEmail(input: { blobs: R2Bucket delivery: InboundDelivery parsed: Awaited> + authority?: UserInboundDeliveryAuthority }) { const now = new Date().toISOString() try { @@ -130,6 +146,7 @@ async function parseAndStoreInboundEmail(input: { parsed: input.parsed, subjectNormalized: normalizeSubject(input.parsed.subject), now, + authority: input.authority, }) } catch (error) { if (error instanceof RetryableInboundStorageError) throw error @@ -141,18 +158,26 @@ async function parseAndStoreInboundEmail(input: { } async function cleanupInboundDurability(input: { - db: D1Database - blobs: R2Bucket + env: UserInboundDeliveryAuthorityEnv & { EMAIL_BLOBS: R2Bucket } userId: string }) { try { - await reconcileStaleInboundDeliveries(input) - await pruneExpiredInboundDedupePointers({ - db: input.db, - userId: input.userId, - }) + if (input.userId === systemEmailOwnerId) { + await reconcileSystemStaleInboundDeliveries({ + db: input.env.APP_DB, + blobs: input.env.EMAIL_BLOBS, + userId: input.userId, + }) + await pruneSystemExpiredInboundDedupePointers({ + db: input.env.APP_DB, + userId: input.userId, + }) + } else { + await reconcileUserStaleInboundDeliveries(input) + await pruneUserExpiredInboundDedupePointers(input) + } await deleteEmptyEmailThreads({ - db: input.db, + db: input.env.APP_DB, userId: input.userId, before: new Date(Date.now() - 48 * 60 * 60 * 1000).toISOString(), limit: 20, @@ -276,6 +301,7 @@ export async function handleInboundEmail( db: env.APP_DB, stableUserId: userId, async write() { + const authority = createUserInboundDeliveryAuthority({ env, userId }) // Require email + canonical stable id together (same contract as // getUserPlan / isAccountEmailVerified) so a mismatched identity pair // cannot apply another account's plan or verification state. @@ -435,8 +461,7 @@ export async function handleInboundEmail( } await cleanupInboundDurability({ - db: env.APP_DB, - blobs: env.EMAIL_BLOBS, + env, userId, }) let rawMime: string @@ -458,18 +483,12 @@ export async function handleInboundEmail( quotaDay: userInboundQuotaDay(quotaNow), now: quotaNow, }) - const activeWindow = await getInboundDeliveryWindow({ - db: env.APP_DB, - userId, - fingerprint: candidateDelivery.fingerprint, - now: quotaNow, - }) - let delivery = activeWindow ?? candidateDelivery - let existingDelivery = await getInboundDelivery({ - db: env.APP_DB, - userId, - deliveryId: delivery.deliveryId, - }) + const activeWindow = await authority.getWindow( + candidateDelivery.fingerprint, + quotaNow, + ) + const delivery = activeWindow ?? candidateDelivery + let existingDelivery = await authority.get(delivery.deliveryId) if (!existingDelivery) { try { // New deliveries check storage bytes and stored-message caps @@ -534,26 +553,6 @@ export async function handleInboundEmail( return } } - if (!activeWindow) { - delivery = await claimInboundDeliveryWindow({ - db: env.APP_DB, - delivery: candidateDelivery, - now: quotaNow, - }) - if ( - !existingDelivery || - existingDelivery.deliveryId !== delivery.deliveryId - ) { - existingDelivery = await getInboundDelivery({ - db: env.APP_DB, - userId, - deliveryId: delivery.deliveryId, - }) - } - } - const waitUntil = ctx - ? (promise: Promise) => ctx.waitUntil(promise) - : undefined let claimedDelivery: InboundDelivery let chargedReceive = false try { @@ -564,9 +563,7 @@ export async function handleInboundEmail( ...delivery, quotaDay: userInboundQuotaDay(quotaNow), } - claimedDelivery = await chargeUserInboundDeliveryOnce({ - db: env.APP_DB, - env, + const chargeResult = await authority.charge({ delivery: chargeCandidate, plan: account.plan, limit: resolveEmailResourceLimit( @@ -574,12 +571,9 @@ export async function handleInboundEmail( 'email_receives_per_day', ), now: quotaNow, - waitUntil, }) - // The charge helper returns the candidate object only when this - // invocation won the atomic D1 charge. A concurrently committed - // delivery is returned as a separately parsed object. - chargedReceive = claimedDelivery === chargeCandidate + claimedDelivery = chargeResult.delivery + chargedReceive = chargeResult.charged } } catch (error) { if (!isEntitlementLimitError(error)) throw error @@ -647,6 +641,7 @@ export async function handleInboundEmail( message, delivery: claimedDelivery, reason, + authority, }) if (!rejected) return await recordReceiveUsage({ outcome: 'error' }) @@ -658,12 +653,11 @@ export async function handleInboundEmail( }) return } - const storageClaim = await claimInboundDeliveryStorage({ - db: env.APP_DB, - delivery: claimedDelivery, - expectedAttachmentCount: parsed.attachments.length, - usageStartedAt: new Date(receiveStartedAtMs).toISOString(), - }) + const storageClaim = await authority.claimStorage( + claimedDelivery, + parsed.attachments.length, + new Date(receiveStartedAtMs).toISOString(), + ) if (!storageClaim.claimed) { if (storageClaim.delivery?.state === 'received') { await scheduleInboundReceivedTerminalWork({ @@ -689,18 +683,18 @@ export async function handleInboundEmail( blobs: env.EMAIL_BLOBS, delivery: storageClaim.delivery, parsed, + authority, }) } catch (error) { - await releaseInboundDeliveryStorage({ - db: env.APP_DB, - delivery: storageClaim.delivery, - }).catch((releaseError) => { - console.error( - 'inbound-email-storage-lease-release-failed', - storageClaim.delivery.deliveryId, - releaseError, - ) - }) + await authority + .releaseStorage(storageClaim.delivery) + .catch((releaseError) => { + console.error( + 'inbound-email-storage-lease-release-failed', + storageClaim.delivery.deliveryId, + releaseError, + ) + }) throw error } // Mailbox dual-write only after durable D1/R2 commit + finalization win. @@ -729,6 +723,8 @@ async function handleSystemInboundEmail(input: { | 'BUNDLE_ARTIFACTS_KV' | 'APP_BASE_URL' | 'USAGE_EVENTS' + | 'MAILBOX' + | 'USER_METER' > recipient: string localPart: SystemEmailLocal @@ -798,8 +794,7 @@ async function handleSystemInboundEmail(input: { } await cleanupInboundDurability({ - db: input.env.APP_DB, - blobs: input.env.EMAIL_BLOBS, + env: input.env, userId: systemEmailOwnerId, }) let rawMime: string @@ -870,7 +865,7 @@ async function handleSystemInboundEmail(input: { } } if (!activeWindow) { - delivery = await claimInboundDeliveryWindow({ + delivery = await claimSystemInboundDeliveryWindow({ db: input.env.APP_DB, delivery: candidateDelivery, now: quotaNow, @@ -952,7 +947,7 @@ async function handleSystemInboundEmail(input: { await recordReceiveUsage({ outcome: 'error' }) return } - const storageClaim = await claimInboundDeliveryStorage({ + const storageClaim = await claimSystemInboundDeliveryStorage({ db: input.env.APP_DB, delivery: claimedDelivery, expectedAttachmentCount: parsed.attachments.length, @@ -973,7 +968,7 @@ async function handleSystemInboundEmail(input: { parsed, }) } catch (error) { - await releaseInboundDeliveryStorage({ + await releaseSystemInboundDeliveryStorage({ db: input.env.APP_DB, delivery: storageClaim.delivery, }).catch((releaseError) => { diff --git a/packages/worker/src/email/inbound.workers.test.ts b/packages/worker/src/email/inbound.workers.test.ts index e00a53a583..351ae7b9bb 100644 --- a/packages/worker/src/email/inbound.workers.test.ts +++ b/packages/worker/src/email/inbound.workers.test.ts @@ -4,10 +4,13 @@ import { utcDayKey } from '@kody-internal/shared/date-keys.ts' import { userMeterRpc } from '#worker/entitlements/user-meter-client.ts' import { handleInboundEmail } from './inbound.ts' import { processInboundDeliveryEffects } from './inbound-effects.ts' +import { createUserInboundDeliveryAuthority } from './inbound-delivery-authority.ts' +import { mailboxRpc } from './mailbox-client.ts' import { buildInboundDelivery, claimInboundDeliveryWindow, claimInboundDeliveryStorage, + getInboundDelivery, inboundDeliveryDedupeWindowMs, InboundDeliveryLeaseLostError, markInboundDeliveryRejected, @@ -1475,9 +1478,8 @@ test('identical MIME from distinct envelope senders creates separate deliveries' expect(await readUserDailyReceiveCount(userId)).toBe(2) }) -test('inbound post-commit bookkeeping failure keeps one stored row without refund or retry throw', async () => { +test('D1 fence mirror failure fails closed and replay repairs without a second charge', async () => { silenceIncidentalRuntimeWarnings() - silenceExpectedConsoleErrors(['inbound-email-post-commit-bookkeeping-failed']) await ensureEmailTestSchema(env.APP_DB) const username = `postcommit-${crypto.randomUUID().slice(0, 8)}` const accountEmail = `postcommit-${crypto.randomUUID()}@example.com` @@ -1488,24 +1490,27 @@ test('inbound post-commit bookkeeping failure keeps one stored row without refun email: accountEmail, username, }) + const raw = [ + 'From: Sender ', + `To: ${address}`, + 'Subject: Post-commit bookkeeping', + 'Message-ID: ', + '', + 'Body', + ].join('\r\n') const message = createForwardableEmailMessage({ from: 'sender@example.net', to: address, - raw: [ - 'From: Sender ', - `To: ${address}`, - 'Subject: Post-commit bookkeeping', - 'Message-ID: ', - '', - 'Body', - ].join('\r\n'), + raw, }) const failingEnv = { ...createInboundEnv(), APP_DB: createPostCommitBookkeepingFailureDb(), } as Parameters[1] - await handleInboundEmail(message, failingEnv) + await expect(handleInboundEmail(message, failingEnv)).rejects.toBeInstanceOf( + RetryableInboundStorageError, + ) expect(message.rejectedReason).toBeNull() expect(await readUserDailyReceiveCount(userId)).toBe(1) expect( @@ -1515,9 +1520,18 @@ test('inbound post-commit bookkeeping failure keeps one stored row without refun limit: 10, }), ).toHaveLength(1) + await handleInboundEmail( + createForwardableEmailMessage({ + from: 'sender@example.net', + to: address, + raw, + }), + createInboundEnv(), + ) + expect(await readUserDailyReceiveCount(userId)).toBe(1) }) -test('delivery-ledger finalization failure retries before usage or subscription success', async () => { +test('Mailbox finalization runs effects once and ignores D1 authority reset attempts', async () => { silenceIncidentalRuntimeWarnings() await ensureEmailTestSchema(env.APP_DB) await ensureUsageRollupsTestSchema(env.APP_DB) @@ -1530,33 +1544,6 @@ test('delivery-ledger finalization failure retries before usage or subscription email: accountEmail, username, }) - let finalizationFailed = false - const failingDb = new Proxy(env.APP_DB, { - get(target, property, receiver) { - if (property === 'prepare') { - return (query: string) => { - const statement = target.prepare(query) - if ( - finalizationFailed || - !query.includes('UPDATE email_delivery_events') || - !query.includes('SET message_id = ?') - ) { - return statement - } - return { - bind: () => ({ - run: async () => { - finalizationFailed = true - throw new Error('simulated ledger finalization failure') - }, - }), - } - } - } - const value = Reflect.get(target, property, receiver) - return typeof value === 'function' ? value.bind(target) : value - }, - }) as D1Database const raw = [ 'From: Sender ', `To: ${address}`, @@ -1571,20 +1558,15 @@ test('delivery-ledger finalization failure retries before usage or subscription raw, }) - await expect( - handleInboundEmail(first, { - ...createInboundEnv(), - APP_DB: failingDb, - }), - ).rejects.toBeInstanceOf(RetryableInboundStorageError) + await handleInboundEmail(first, createInboundEnv()) expect( await env.APP_DB.prepare( `SELECT event_count FROM usage_rollups WHERE user_id = ? AND metric = 'email_received'`, ) .bind(userId) - .first(), - ).toBeNull() + .first<{ event_count: number }>(), + ).toEqual({ event_count: 1 }) const retry = createForwardableEmailMessage({ from: 'sender@example.net', @@ -1641,11 +1623,13 @@ test('delivery-ledger finalization failure retries before usage or subscription ) .bind(delivery.id, userId) .run() - await processInboundDeliveryEffects({ - env: createInboundEnv(), - userId, - deliveryId: delivery.id, - }) + expect( + await processInboundDeliveryEffects({ + env: createInboundEnv(), + userId, + deliveryId: delivery.id, + }), + ).toEqual({ outcome: 'usage-only' }) expect( await env.APP_DB.prepare( `SELECT event_count, total_duration_ms FROM usage_rollups @@ -1653,7 +1637,7 @@ test('delivery-ledger finalization failure retries before usage or subscription ) .bind(userId) .first<{ event_count: number; total_duration_ms: number }>(), - ).toEqual({ event_count: 1, total_duration_ms: 4321 }) + ).toBeNull() }) test('production usage outbox records one durable D1 event without retry data points', async () => { @@ -2044,12 +2028,58 @@ test('stale rejection cannot overwrite finalized delivery usage', async () => { usageMonth: '2026-07', usageBytes: 456, }) - await processInboundDeliveryEffects({ + expect( + await getInboundDelivery({ + db: env.APP_DB, + userId, + deliveryId: pending.deliveryId, + }), + ).toMatchObject({ state: 'received', finalizationToken: expect.any(String) }) + const authorityDelivery = await createUserInboundDeliveryAuthority({ + env: createInboundEnv(), + userId, + }).get(pending.deliveryId) + const mailboxEvents = await mailboxRpc({ + env: createInboundEnv(), + userId, + }).listDeliveryEvents({ limit: 10 }) + expect(mailboxEvents[0]).toMatchObject({ + id: pending.deliveryId, + provider: 'cloudflare-email-routing', + state: 'received', + }) + expect(JSON.parse(mailboxEvents[0]!.detailJson)).toMatchObject({ + fingerprint: expect.any(String), + deliveryId: pending.deliveryId, + messageId: pending.messageId, + threadId: pending.threadId, + rawMimeKey: pending.rawMimeKey, + inboxId: pending.inboxId, + recipient: pending.recipient, + envelopeFrom: pending.envelopeFrom, + quotaDay: pending.quotaDay, + dedupeExpiresAt: pending.dedupeExpiresAt, + }) + expect( + await mailboxRpc({ + env: createInboundEnv(), + userId, + }).getInboundDelivery({ + ownerId: userId, + deliveryId: pending.deliveryId, + }), + ).not.toBeNull() + expect(authorityDelivery).toMatchObject({ + state: 'received', + finalizationToken: expect.any(String), + }) + const effectResult = await processInboundDeliveryEffects({ env: createInboundEnv(), userId, deliveryId: pending.deliveryId, now, }) + expect(effectResult).toEqual({ outcome: 'complete' }) expect( await markInboundDeliveryRejected({ @@ -2648,6 +2678,20 @@ test('scheduled sweep cleans quiet-user orphans and recovers partial commits', a ) .run() + // Match the production 0129 backfill: USER owner discovery reads promoted + // compatibility columns, while only system:email retains legacy JSON reads. + await env.APP_DB.prepare( + `UPDATE email_delivery_events + SET state = json_extract(detail_json, '$.state'), + fingerprint = json_extract(detail_json, '$.fingerprint'), + reconcile_after = json_extract(detail_json, '$.reconcileAfter'), + cleanup_retry_at = json_extract(detail_json, '$.cleanupRetryAt'), + dedupe_expires_at = json_extract(detail_json, '$.dedupeExpiresAt') + WHERE user_id IN (?, ?, ?, ?)`, + ) + .bind(orphanUserId, partialUserId, pointerUserId, deletingUserId) + .run() + expect( await sweepStaleInboundDeliveries({ env: createInboundEnv(), diff --git a/packages/worker/src/email/mailbox-delivery-event-bootstrap.ts b/packages/worker/src/email/mailbox-delivery-event-bootstrap.ts new file mode 100644 index 0000000000..68765cc88b --- /dev/null +++ b/packages/worker/src/email/mailbox-delivery-event-bootstrap.ts @@ -0,0 +1,70 @@ +import { writeMailboxDeliveryEventRow } from './mailbox-delivery-events.ts' +import { + assertUserInboundLegacyBootstrapSnapshot, + hasMalformedUserInboundBootstrapSchedule, +} from './mailbox-inbound-bootstrap.ts' +import { + assertMailboxNonEmptyString, + mailboxUpsertDeliveryEventsMax, + type MailboxBootstrapDeliveryEventsResult, + type MailboxDeliveryEventInput, +} from './mailbox-types.ts' + +/** + * Missing-only legacy USER inbound bootstrap inside the caller's transaction. + * Validation failure throws so transactionSync rolls back every prior insert. + */ +export function bootstrapMailboxDeliveryEvents( + sql: SqlStorage, + input: { + ownerId: string + events: Array + }, +): MailboxBootstrapDeliveryEventsResult { + if (!Array.isArray(input.events) || input.events.length === 0) { + throw new Error('Mailbox bootstrapDeliveryEvents events must be non-empty.') + } + if (input.events.length > mailboxUpsertDeliveryEventsMax) { + throw new Error( + `Mailbox bootstrapDeliveryEvents events exceed max of ${mailboxUpsertDeliveryEventsMax}.`, + ) + } + const result: MailboxBootstrapDeliveryEventsResult = { + inserted: 0, + existing: 0, + skipped: 0, + results: [], + } + for (const event of input.events) { + const eventId = assertMailboxNonEmptyString(event.id, 'event.id') + if (hasMalformedUserInboundBootstrapSchedule(event)) { + result.skipped += 1 + result.results.push({ eventId, status: 'skipped' }) + continue + } + assertUserInboundLegacyBootstrapSnapshot({ + ownerId: input.ownerId, + event, + }) + const existing = sql + .exec( + `SELECT id FROM email_delivery_events WHERE id = ? LIMIT 1`, + eventId, + ) + .toArray() + if (existing.length > 0) { + result.existing += 1 + result.results.push({ eventId, status: 'existing' }) + continue + } + const write = writeMailboxDeliveryEventRow(sql, event) + if (write.inserted) { + result.inserted += 1 + result.results.push({ eventId, status: 'inserted' }) + } else { + result.skipped += 1 + result.results.push({ eventId, status: 'skipped' }) + } + } + return result +} diff --git a/packages/worker/src/email/mailbox-delivery-event-upsert.ts b/packages/worker/src/email/mailbox-delivery-event-upsert.ts new file mode 100644 index 0000000000..41ed5f9431 --- /dev/null +++ b/packages/worker/src/email/mailbox-delivery-event-upsert.ts @@ -0,0 +1,33 @@ +import { writeMailboxDeliveryEventRow } from './mailbox-delivery-events.ts' +import { shouldSkipMailboxDeliveryEventWrite } from './mailbox-inbound-bootstrap.ts' +import { + assertMailboxNonEmptyString, + mailboxUpsertDeliveryEventsMax, + type MailboxDeliveryEventInput, + type MailboxUpsertDeliveryEventsResult, +} from './mailbox-types.ts' + +/** Normal bounded delivery-event upsert inside the caller's transaction. */ +export function upsertMailboxDeliveryEvents( + sql: SqlStorage, + events: Array, +): MailboxUpsertDeliveryEventsResult { + if (!Array.isArray(events) || events.length === 0) { + throw new Error('Mailbox upsertDeliveryEvents events must be non-empty.') + } + if (events.length > mailboxUpsertDeliveryEventsMax) { + throw new Error( + `Mailbox upsertDeliveryEvents events exceed max of ${mailboxUpsertDeliveryEventsMax}.`, + ) + } + const results: MailboxUpsertDeliveryEventsResult['results'] = [] + for (const event of events) { + const eventId = assertMailboxNonEmptyString(event.id, 'event.id') + if (shouldSkipMailboxDeliveryEventWrite(sql, { event })) { + results.push({ eventId, inserted: false, accepted: false }) + continue + } + results.push({ eventId, ...writeMailboxDeliveryEventRow(sql, event) }) + } + return { results } +} diff --git a/packages/worker/src/email/mailbox-do.ts b/packages/worker/src/email/mailbox-do.ts index 0bea8aec63..3d3b4fba1f 100644 --- a/packages/worker/src/email/mailbox-do.ts +++ b/packages/worker/src/email/mailbox-do.ts @@ -21,6 +21,14 @@ import { touchMailboxThread, updateMailboxMessageDelivery, } from './mailbox-mutations.ts' +import { + claimMailboxInboundDeliveryCleanup, + markMailboxInboundDeliveryOrphanCleaned, + releaseMailboxInboundDeliveryCleanup, +} from './mailbox-inbound-cleanup-ledger.ts' +import { bootstrapMailboxDeliveryEvents } from './mailbox-delivery-event-bootstrap.ts' +import { upsertMailboxDeliveryEvents } from './mailbox-delivery-event-upsert.ts' +import { shouldSkipMailboxDeliveryEventWrite } from './mailbox-inbound-bootstrap.ts' import { claimMailboxInboundDeliveryStorage, claimMailboxInboundDeliveryWindow, @@ -46,10 +54,10 @@ import { } from './mailbox-inbound-effect-ledger.ts' import { assertMailboxNonEmptyString, - mailboxUpsertDeliveryEventsMax, type MailboxAttachmentInput, type MailboxAttachmentRecord, type MailboxBlobReferencePage, + type MailboxBootstrapDeliveryEventsResult, type MailboxCountMessagesInput, type MailboxCountResult, type MailboxDeleteDeliveryEventInput, @@ -72,7 +80,6 @@ import { type MailboxThreadRecord, type MailboxTouchThreadInput, type MailboxUpdateMessageDeliveryInput, - type MailboxUpsertDeliveryEventBatchItemResult, type MailboxUpsertDeliveryEventsResult, } from './mailbox-types.ts' @@ -84,23 +91,13 @@ import { * external attachment bytes stay in `EMAIL_BLOBS`; rows retain keys. * `system:email` stays in D1 by design. * - * Dual-write / read-cutover live paths are wired elsewhere. Additive step 2a - * inbound ledger CAS RPCs are exposed but not live-wired (D1 remains - * authority for inbound ledger/effects). + * Dual-write / read-cutover live paths are wired elsewhere. USER inbound + * ledger/effect transitions are authoritative here; `system:email` remains D1. */ class MailboxBase extends DurableObject implements MailboxRpc { private readonly store: MailboxStore - /** - * In-isolate cache: once an alarm is scheduled, hot writes skip - * getAlarm/setAlarm. Cleared when retention becomes idle or a write may - * need an earlier wake. - */ private retentionAlarmArmed = false - /** - * In-isolate: no future retention work. Cleared on every data write so - * ensureRetentionAlarm re-arms for the new row's due-time. - */ private retentionIdleConfirmed = false constructor(ctx: DurableObjectState, env: Env) { @@ -108,7 +105,6 @@ class MailboxBase extends DurableObject implements MailboxRpc { this.store = new MailboxStore(ctx.storage) this.ctx.blockConcurrencyWhile(async () => { this.store.initializeSchema() - // Observe existing alarm only — never arm in the constructor. this.retentionAlarmArmed = (await this.ctx.storage.getAlarm()) != null }) } @@ -216,7 +212,6 @@ class MailboxBase extends DurableObject implements MailboxRpc { return this.runRetentionPass() } - /** Atomic mirror of thread + message + attachments for dual-write. */ async mirrorMessage(input: { ownerId: string thread?: MailboxThreadInput | null @@ -247,10 +242,6 @@ class MailboxBase extends DurableObject implements MailboxRpc { return { ok: true, accepted } } - /** - * Upsert a delivery event (idempotent on `provider_event_id`) and optionally - * apply a monotonic latest `delivery_status` update on the message. - */ async upsertDeliveryEvent(input: { ownerId: string event: MailboxDeliveryEventInput @@ -269,6 +260,13 @@ class MailboxBase extends DurableObject implements MailboxRpc { let updatedLatestStatus = false this.ctx.storage.transactionSync(() => { this.store.assertOwner(input.ownerId) + if ( + shouldSkipMailboxDeliveryEventWrite(this.ctx.storage.sql, { + event: input.event, + }) + ) { + return + } const write = this.store.writeDeliveryEventRow(input.event) inserted = write.inserted accepted = write.accepted @@ -290,40 +288,36 @@ class MailboxBase extends DurableObject implements MailboxRpc { return { inserted, accepted, updatedLatestStatus } } - /** - * Upsert a bounded batch of complete immutable delivery-event snapshots in - * one transaction / one DO RPC. Validates non-empty and - * {@link mailboxUpsertDeliveryEventsMax}. Does not patch message latest - * delivery status. Marks retention dirty once for the batch. - */ async upsertDeliveryEvents(input: { ownerId: string events: Array }): Promise { - if (!Array.isArray(input.events) || input.events.length === 0) { - throw new Error('Mailbox upsertDeliveryEvents events must be non-empty.') - } - if (input.events.length > mailboxUpsertDeliveryEventsMax) { - throw new Error( - `Mailbox upsertDeliveryEvents events exceed max of ${mailboxUpsertDeliveryEventsMax}.`, - ) - } - const results: Array = [] + let result: MailboxUpsertDeliveryEventsResult | undefined this.ctx.storage.transactionSync(() => { this.store.assertOwner(input.ownerId) - for (const event of input.events) { - const eventId = assertMailboxNonEmptyString(event.id, 'event.id') - const write = this.store.writeDeliveryEventRow(event) - results.push({ - eventId, - inserted: write.inserted, - accepted: write.accepted, - }) - } + result = upsertMailboxDeliveryEvents(this.ctx.storage.sql, input.events) }) + if (!result) throw new Error('Mailbox upsert transaction did not run.') this.markRetentionDirty() await this.ensureRetentionAlarm() - return { results } + return result + } + + async bootstrapDeliveryEvents(input: { + ownerId: string + events: Array + }): Promise { + let result: MailboxBootstrapDeliveryEventsResult | undefined + this.ctx.storage.transactionSync(() => { + this.store.assertOwner(input.ownerId) + result = bootstrapMailboxDeliveryEvents(this.ctx.storage.sql, input) + }) + if (!result) throw new Error('Mailbox bootstrap transaction did not run.') + if (result.inserted > 0) { + this.markRetentionDirty() + await this.ensureRetentionAlarm() + } + return result } /** @@ -507,11 +501,6 @@ class MailboxBase extends DurableObject implements MailboxRpc { return this.store.listBlobReferences(input) } - /** - * Additive step 2a inbound ledger CAS RPCs. Not live-wired — D1 remains - * authority. Mirror `upsertDeliveryEvent(s)` stay the compatibility path. - */ - async getInboundDelivery(input: { ownerId: string deliveryId: string @@ -688,6 +677,58 @@ class MailboxBase extends DurableObject implements MailboxRpc { return result } + async claimInboundDeliveryCleanup(input: { + ownerId: string + deliveryId: string + expectedState: MailboxInboundDeliveryState + expectedUpdatedAt: string + staleBefore: string + now?: string + }) { + let result!: Awaited> + this.ctx.storage.transactionSync(() => { + this.store.assertOwner(input.ownerId) + result = claimMailboxInboundDeliveryCleanup(this.ctx.storage.sql, input) + }) + return result + } + + async releaseInboundDeliveryCleanup(input: { + ownerId: string + deliveryId: string + cleanupLease: string + now?: string + }) { + let result!: Awaited< + ReturnType + > + this.ctx.storage.transactionSync(() => { + this.store.assertOwner(input.ownerId) + result = releaseMailboxInboundDeliveryCleanup(this.ctx.storage.sql, input) + }) + return result + } + + async markInboundDeliveryOrphanCleaned(input: { + ownerId: string + deliveryId: string + cleanupLease: string + outcome: 'deleted' | 'delete-failed' + now?: string + }) { + let result!: Awaited< + ReturnType + > + this.ctx.storage.transactionSync(() => { + this.store.assertOwner(input.ownerId) + result = markMailboxInboundDeliveryOrphanCleaned( + this.ctx.storage.sql, + input, + ) + }) + return result + } + async claimInboundUsageEffect(input: { ownerId: string deliveryId: string @@ -798,15 +839,9 @@ class MailboxBase extends DurableObject implements MailboxRpc { return listMailboxDueInboundEffectWork(this.ctx.storage.sql, input) } - /** - * Clear SQLite state only. During expand, D1 deletion remains authoritative - * for R2 objects. - */ async purge(): Promise<{ ok: true }> { await this.ctx.blockConcurrencyWhile(async () => { - await this.ctx.storage.deleteAlarm().catch(() => { - // Best effort: deleteAll below still clears persisted alarm state. - }) + await this.ctx.storage.deleteAlarm().catch(() => undefined) await this.ctx.storage.deleteAll() this.retentionAlarmArmed = false this.retentionIdleConfirmed = true diff --git a/packages/worker/src/email/mailbox-do.workers.test.ts b/packages/worker/src/email/mailbox-do.workers.test.ts index a753e9ae4b..e7e752ef87 100644 --- a/packages/worker/src/email/mailbox-do.workers.test.ts +++ b/packages/worker/src/email/mailbox-do.workers.test.ts @@ -496,55 +496,6 @@ test('Mailbox delivery status, promoted inbound fields, export paging, and curso updatedLatestStatus: true, }) - const inbound = await mailbox.upsertDeliveryEvent({ - ownerId: userId, - event: baseDeliveryEvent({ - id: 'inbound-delivery-1', - messageId: 'inbound-msg-1', - inboxId: 'inbox-1', - eventType: 'received', - provider: 'cloudflare-email-routing', - createdAt: '2026-07-02T11:00:00.000Z', - updatedAt: '2026-07-02T11:00:00.000Z', - needsEffectReconcile: true, - state: 'received', - fingerprint: 'fp-abc', - expectedAttachmentCount: 2, - finalizationToken: 'lease-token-1', - usageStartedAt: '2026-07-02T10:59:00.000Z', - usageMonth: '2026-07', - usageBytes: 2048, - usageDurationMs: 120, - subscriptionEffectState: 'pending', - subscriptionEffectRetryAt: '2026-07-02T12:00:00.000Z', - detailJson: JSON.stringify({ recipient: 'owner@example.com' }), - }), - }) - expect(inbound.inserted).toBe(true) - const inboundRow = ( - await mailbox.listDeliveryEvents({ - messageId: 'inbound-msg-1', - limit: 1, - }) - )[0] - expect(inboundRow).toMatchObject({ - id: 'inbound-delivery-1', - needsEffectReconcile: true, - state: 'received', - fingerprint: 'fp-abc', - expectedAttachmentCount: 2, - finalizationToken: 'lease-token-1', - usageStartedAt: '2026-07-02T10:59:00.000Z', - usageMonth: '2026-07', - usageBytes: 2048, - subscriptionEffectState: 'pending', - subscriptionEffectRetryAt: '2026-07-02T12:00:00.000Z', - updatedAt: '2026-07-02T11:00:00.000Z', - }) - expect(JSON.parse(inboundRow!.detailJson)).toMatchObject({ - recipient: 'owner@example.com', - }) - // Explicit needsEffectReconcile: false is stored as false. await mailbox.upsertDeliveryEvent({ ownerId: userId, diff --git a/packages/worker/src/email/mailbox-inbound-authority-guard.workers.test.ts b/packages/worker/src/email/mailbox-inbound-authority-guard.workers.test.ts new file mode 100644 index 0000000000..3527de7dde --- /dev/null +++ b/packages/worker/src/email/mailbox-inbound-authority-guard.workers.test.ts @@ -0,0 +1,311 @@ +import { runInDurableObject } from 'cloudflare:test' +import { expect, test } from 'vitest' +import { emailRawMimeKey } from './blob-keys.ts' +import { insertInput } from './mailbox-inbound-ledger-test-helpers.ts' +import { mailboxUpsertDeliveryEventsMax } from './mailbox-types.ts' +import { + assertMailboxThrows, + baseDeliveryEvent, + rpcFor, + stubFor, + uniqueUserId, +} from './mailbox-test-helpers.ts' + +function preClaimAuditEvent(input: { + inboxId: string + day: string + count?: number +}) { + const at = `${input.day}T12:00:00.000Z` + return baseDeliveryEvent({ + id: `email-rejections:${input.inboxId}:${input.day}`, + inboxId: input.inboxId, + eventType: 'rejected', + provider: 'cloudflare-email-routing', + detailJson: JSON.stringify({ + aggregate: true, + day: input.day, + count: input.count ?? 1, + last_reason: 'Recipient mailbox is over quota.', + last_phase: 'entitlement', + last_at: at, + }), + createdAt: at, + updatedAt: at, + }) +} + +function legacyPendingEvent(ownerId: string, deliveryId: string) { + const messageId = `email-inbound-message:${deliveryId}` + const fingerprint = `fingerprint-${deliveryId}` + const detail = { + fingerprint, + deliveryId, + messageId, + threadId: `email-inbound-thread:${deliveryId}`, + rawMimeKey: emailRawMimeKey(ownerId, messageId), + userId: ownerId, + inboxId: 'inbox-legacy', + recipient: 'owner@example.com', + envelopeFrom: 'sender@example.com', + provider: 'cloudflare-email-routing', + quotaDay: '2026-08-02', + dedupeExpiresAt: '2026-08-04T12:00:00.000Z', + state: 'pending' as const, + } + return baseDeliveryEvent({ + id: deliveryId, + inboxId: detail.inboxId, + eventType: 'receive_started', + provider: 'cloudflare-email-routing', + providerEventId: deliveryId, + detailJson: JSON.stringify(detail), + state: 'pending', + fingerprint, + dedupeExpiresAt: detail.dedupeExpiresAt, + createdAt: '2026-08-02T12:00:00.000Z', + updatedAt: '2026-08-02T12:00:00.000Z', + }) +} + +test('D1 mirror accepts only non-authoritative bounded inbound rejection audits', async () => { + const ownerId = uniqueUserId('audit-guard') + const mailbox = rpcFor(ownerId) + const audit = preClaimAuditEvent({ + inboxId: 'inbox-audit', + day: '2026-08-02', + }) + + await expect( + mailbox.upsertDeliveryEvent({ ownerId, event: audit }), + ).resolves.toEqual({ + inserted: true, + accepted: true, + updatedLatestStatus: false, + }) + await expect( + mailbox.upsertDeliveryEvents({ + ownerId, + events: [ + { + ...audit, + detailJson: preClaimAuditEvent({ + inboxId: 'inbox-audit', + day: '2026-08-02', + count: 2, + }).detailJson, + }, + ], + }), + ).resolves.toEqual({ + results: [{ eventId: audit.id, inserted: false, accepted: true }], + }) + + await runInDurableObject(stubFor(ownerId), async (instance) => { + await assertMailboxThrows(/missing-row bootstrap RPC/, () => + instance.upsertDeliveryEvents({ + ownerId, + events: [ + { + ...audit, + state: 'pending', + fingerprint: 'smuggled-authority-state', + }, + ], + }), + ) + await assertMailboxThrows(/missing-row bootstrap RPC/, () => + instance.upsertDeliveryEvents({ + ownerId, + events: [ + baseDeliveryEvent({ + id: 'email-inbound-delivery:lifecycle', + provider: 'cloudflare-email-routing', + eventType: 'receive_started', + state: 'pending', + fingerprint: 'lifecycle-fingerprint', + }), + ], + }), + ) + await assertMailboxThrows(/missing-row bootstrap RPC/, () => + instance.upsertDeliveryEvents({ + ownerId, + events: [ + baseDeliveryEvent({ + id: 'email-inbound-dedupe:lifecycle', + provider: 'cloudflare-email-routing-dedupe', + eventType: 'receive_started', + state: 'pending', + fingerprint: 'lifecycle-fingerprint', + }), + ], + }), + ) + }) +}) + +test('audit-shaped D1 snapshots cannot overwrite authoritative Mailbox rows', async () => { + const ownerId = uniqueUserId('audit-collision') + const mailbox = rpcFor(ownerId) + const audit = preClaimAuditEvent({ + inboxId: 'inbox-collision', + day: '2026-08-02', + }) + const authoritative = insertInput(ownerId, { + deliveryId: audit.id, + inboxId: 'inbox-collision', + fingerprint: 'authoritative-fingerprint', + }) + await mailbox.insertChargedPendingInboundDelivery({ + ownerId, + delivery: authoritative, + now: '2026-08-02T11:00:00.000Z', + }) + + await expect( + mailbox.upsertDeliveryEvents({ ownerId, events: [audit] }), + ).resolves.toEqual({ + results: [{ eventId: audit.id, inserted: false, accepted: false }], + }) + await expect( + mailbox.getInboundDelivery({ + ownerId, + deliveryId: audit.id, + }), + ).resolves.toMatchObject({ + state: 'pending', + fingerprint: 'authoritative-fingerprint', + }) +}) + +test('bootstrapDeliveryEvents is owner-bound, validated, bounded, transactional, and missing-only', async () => { + const ownerId = uniqueUserId('bootstrap-rpc') + const otherOwnerId = uniqueUserId('bootstrap-rpc-other') + const mailbox = rpcFor(ownerId) + const deliveryId = 'email-inbound-delivery:legacy-rpc' + const pending = legacyPendingEvent(ownerId, deliveryId) + + await expect( + mailbox.bootstrapDeliveryEvents({ ownerId, events: [pending] }), + ).resolves.toEqual({ + inserted: 1, + existing: 0, + skipped: 0, + results: [{ eventId: deliveryId, status: 'inserted' }], + }) + await mailbox.claimInboundDeliveryStorage({ + ownerId, + deliveryId, + expectedAttachmentCount: 3, + now: '2026-08-02T13:00:00.000Z', + }) + const newer = await mailbox.getInboundDelivery({ ownerId, deliveryId }) + expect(newer).toMatchObject({ state: 'storing', expectedAttachmentCount: 3 }) + + await expect( + mailbox.bootstrapDeliveryEvents({ ownerId, events: [pending] }), + ).resolves.toMatchObject({ inserted: 0, existing: 1, skipped: 0 }) + expect( + await mailbox.getInboundDelivery({ ownerId, deliveryId }), + ).toMatchObject({ + state: 'storing', + expectedAttachmentCount: 3, + updatedAt: newer?.updatedAt, + }) + const skippedId = 'email-inbound-delivery:provider-id-conflict' + await mailbox.upsertDeliveryEvent({ + ownerId, + event: baseDeliveryEvent({ + id: 'provider-id-conflict-holder', + provider: 'kody', + providerEventId: skippedId, + }), + }) + await expect( + mailbox.bootstrapDeliveryEvents({ + ownerId, + events: [legacyPendingEvent(ownerId, skippedId)], + }), + ).resolves.toEqual({ + inserted: 0, + existing: 0, + skipped: 1, + results: [{ eventId: skippedId, status: 'skipped' }], + }) + const malformedId = 'email-inbound-delivery:malformed-schedule' + const malformed = legacyPendingEvent(ownerId, malformedId) + await expect( + mailbox.bootstrapDeliveryEvents({ + ownerId, + events: [ + { + ...malformed, + reconcileAfter: 'not-a-timestamp', + detailJson: JSON.stringify({ + ...JSON.parse(malformed.detailJson), + reconcileAfter: 'not-a-timestamp', + }), + }, + ], + }), + ).resolves.toEqual({ + inserted: 0, + existing: 0, + skipped: 1, + results: [{ eventId: malformedId, status: 'skipped' }], + }) + + await runInDurableObject(stubFor(ownerId), async (instance) => { + await assertMailboxThrows(/ownerId mismatch/, () => + instance.bootstrapDeliveryEvents({ + ownerId: otherOwnerId, + events: [ + legacyPendingEvent(otherOwnerId, 'email-inbound-delivery:other'), + ], + }), + ) + await assertMailboxThrows( + /requires an inbound lifecycle or dedupe snapshot/, + () => + instance.bootstrapDeliveryEvents({ + ownerId, + events: [baseDeliveryEvent({ id: 'not-inbound', provider: 'kody' })], + }), + ) + await assertMailboxThrows(/valid owner-bound complete snapshot/, () => + instance.bootstrapDeliveryEvents({ + ownerId, + events: [ + legacyPendingEvent(ownerId, 'email-inbound-delivery:txn-valid'), + { + ...legacyPendingEvent( + ownerId, + 'email-inbound-delivery:txn-invalid', + ), + state: 'storing', + }, + ], + }), + ) + await assertMailboxThrows(/exceed max of 100/, () => + instance.bootstrapDeliveryEvents({ + ownerId, + events: Array.from( + { length: mailboxUpsertDeliveryEventsMax + 1 }, + (_, index) => + legacyPendingEvent( + ownerId, + `email-inbound-delivery:overflow-${index}`, + ), + ), + }), + ) + }) + expect( + await mailbox.getInboundDelivery({ + ownerId, + deliveryId: 'email-inbound-delivery:txn-valid', + }), + ).toBeNull() +}) diff --git a/packages/worker/src/email/mailbox-inbound-bootstrap.ts b/packages/worker/src/email/mailbox-inbound-bootstrap.ts new file mode 100644 index 0000000000..9d2d5ada0f --- /dev/null +++ b/packages/worker/src/email/mailbox-inbound-bootstrap.ts @@ -0,0 +1,302 @@ +import { + mailboxInboundDedupePointerId, + mailboxInboundDedupeProvider, + mailboxInboundProvider, +} from './mailbox-inbound-ledger.ts' +import { + parseInboundDeliveryDetailJson, + parseStrictInboundDeliveryDetailJson, +} from './inbound-delivery.ts' +import { mapMailboxDeliveryEventRow } from './mailbox-mappers.ts' +import { + assertMailboxNonEmptyString, + type MailboxDeliveryEventInput, + type MailboxDeliveryEventRecord, +} from './mailbox-types.ts' + +const preClaimAuditPhases = new Set([ + 'entitlement', + 'size', + 'account-verification', + 'account-suspension', + 'sender-policy', + 'system-limit', +]) +const auditDayPattern = /^\d{4}-\d{2}-\d{2}$/ + +function isCanonicalTimestamp(value: string) { + const parsed = Date.parse(value) + return Number.isFinite(parsed) && new Date(parsed).toISOString() === value +} + +function isUserInboundAuthorityEvent(event: MailboxDeliveryEventInput) { + return ( + event.provider === mailboxInboundProvider || + event.provider === mailboxInboundDedupeProvider + ) +} + +function hasNoInboundAuthorityState( + event: MailboxDeliveryEventInput | MailboxDeliveryEventRecord, +) { + return ( + event.messageId == null && + event.providerMessageId == null && + event.providerEventId == null && + event.state == null && + event.fingerprint == null && + event.storageLease == null && + event.storageLeaseAt == null && + event.cleanupLease == null && + event.cleanupLeaseAt == null && + event.cleanupRetryAt == null && + event.expectedAttachmentCount == null && + event.finalizationToken == null && + event.reconcileAfter == null && + event.dedupeExpiresAt == null && + event.usageEffectRecordedAt == null && + event.usageEffectSuppressedAt == null && + event.usageStartedAt == null && + event.usageMonth == null && + event.usageBytes == null && + event.usageDurationMs == null && + event.usageEffectRetryAt == null && + event.usageEffectLease == null && + event.usageEffectLeaseAt == null && + event.subscriptionEffectState == null && + event.subscriptionEffectLease == null && + event.subscriptionEffectLeaseAt == null && + event.subscriptionEffectRetryAt == null && + event.subscriptionEffectAttemptCount == null && + event.subscriptionEffectDeadLetterAt == null && + event.subscriptionEffectLastError == null + ) +} + +export function isPreClaimAuditSnapshot( + event: MailboxDeliveryEventInput | MailboxDeliveryEventRecord, +) { + if ( + event.provider !== mailboxInboundProvider || + event.eventType !== 'rejected' || + typeof event.inboxId !== 'string' || + event.inboxId.length === 0 || + !hasNoInboundAuthorityState(event) + ) { + return false + } + let detail: unknown + try { + detail = JSON.parse(event.detailJson) + } catch { + return false + } + if ( + typeof detail !== 'object' || + detail == null || + !('aggregate' in detail) || + detail.aggregate !== true || + !('day' in detail) || + typeof detail.day !== 'string' || + !auditDayPattern.test(detail.day) || + !('count' in detail) || + typeof detail.count !== 'number' || + !Number.isInteger(detail.count) || + detail.count < 1 || + !('last_reason' in detail) || + typeof detail.last_reason !== 'string' || + !('last_phase' in detail) || + typeof detail.last_phase !== 'string' || + !preClaimAuditPhases.has(detail.last_phase) || + !('last_at' in detail) || + typeof detail.last_at !== 'string' || + !isCanonicalTimestamp(detail.last_at) + ) { + return false + } + return event.id === `email-rejections:${event.inboxId}:${detail.day}` +} + +export function isUserInboundLegacyAuthoritySnapshot( + event: MailboxDeliveryEventInput, +) { + return isUserInboundAuthorityEvent(event) && !isPreClaimAuditSnapshot(event) +} + +export function hasMalformedUserInboundBootstrapSchedule( + event: MailboxDeliveryEventInput, +) { + if (!isUserInboundLegacyAuthoritySnapshot(event)) return false + if (!parseInboundDeliveryDetailJson(event.detailJson)) return false + try { + const detail = JSON.parse(event.detailJson) as Record + return ['cleanupRetryAt', 'reconcileAfter'].some((field) => { + if (!(field in detail)) return false + const value = detail[field] + return ( + typeof value !== 'string' || + !Number.isFinite(Date.parse(value)) || + new Date(value).toISOString() !== value + ) + }) + } catch { + return false + } +} + +function readMailboxDeliveryEvent( + sql: SqlStorage, + eventId: string, +): MailboxDeliveryEventRecord | null { + const row = sql + .exec>( + `SELECT * FROM email_delivery_events WHERE id = ? LIMIT 1`, + eventId, + ) + .toArray()[0] + return row ? mapMailboxDeliveryEventRow(row) : null +} + +/** + * Validate the only D1 → Mailbox USER inbound bridge. Callers may insert this + * snapshot only when its owner-bound row is missing. + */ +export function assertUserInboundLegacyBootstrapSnapshot(input: { + ownerId: string + event: MailboxDeliveryEventInput +}) { + if (!isUserInboundLegacyAuthoritySnapshot(input.event)) { + throw new Error( + 'USER inbound bootstrap requires an inbound lifecycle or dedupe snapshot.', + ) + } + const delivery = parseStrictInboundDeliveryDetailJson(input.event.detailJson) + const expectedEventId = + input.event.provider === mailboxInboundDedupeProvider && delivery + ? mailboxInboundDedupePointerId(delivery.fingerprint) + : delivery?.deliveryId + const expectedEventType = + delivery?.state === 'received' + ? 'received' + : delivery?.state === 'rejected' + ? 'rejected' + : 'receive_started' + const optionalMatches = (actual: unknown, expected: unknown) => + actual === (expected ?? null) + const lifecycle = input.event.provider === mailboxInboundProvider + const expectedCleanupRetryAt = + lifecycle && delivery?.state === 'orphan-cleaned' + ? delivery.cleanupRetryAt + : null + const expectedReconcileAfter = + lifecycle && + (delivery?.state === 'pending' || + delivery?.state === 'storing' || + delivery?.state === 'cleaning' || + delivery?.state === 'orphan-cleaned') + ? delivery.reconcileAfter + : null + if ( + !delivery || + delivery.userId !== input.ownerId || + delivery.provider !== mailboxInboundProvider || + input.event.id !== expectedEventId || + input.event.inboxId !== delivery.inboxId || + input.event.eventType !== expectedEventType || + input.event.providerMessageId != null || + input.event.providerEventId !== input.event.id || + input.event.messageId !== + (delivery.state === 'received' ? delivery.messageId : null) || + input.event.state !== delivery.state || + input.event.fingerprint !== delivery.fingerprint || + !optionalMatches(input.event.storageLease, delivery.storageLease) || + !optionalMatches(input.event.storageLeaseAt, delivery.storageLeaseAt) || + !optionalMatches(input.event.cleanupLease, delivery.cleanupLease) || + !optionalMatches(input.event.cleanupLeaseAt, delivery.cleanupLeaseAt) || + !optionalMatches(input.event.cleanupRetryAt, expectedCleanupRetryAt) || + !optionalMatches( + input.event.expectedAttachmentCount, + delivery.expectedAttachmentCount, + ) || + !optionalMatches( + input.event.finalizationToken, + delivery.finalizationToken, + ) || + !optionalMatches(input.event.reconcileAfter, expectedReconcileAfter) || + !optionalMatches(input.event.dedupeExpiresAt, delivery.dedupeExpiresAt) || + !optionalMatches( + input.event.usageEffectRecordedAt, + delivery.usageEffectRecordedAt, + ) || + !optionalMatches( + input.event.usageEffectSuppressedAt, + delivery.usageEffectSuppressedAt, + ) || + !optionalMatches(input.event.usageStartedAt, delivery.usageStartedAt) || + !optionalMatches(input.event.usageMonth, delivery.usageMonth) || + !optionalMatches(input.event.usageBytes, delivery.usageBytes) || + !optionalMatches(input.event.usageDurationMs, delivery.usageDurationMs) || + !optionalMatches( + input.event.usageEffectRetryAt, + delivery.usageEffectRetryAt, + ) || + !optionalMatches(input.event.usageEffectLease, delivery.usageEffectLease) || + !optionalMatches( + input.event.usageEffectLeaseAt, + delivery.usageEffectLeaseAt, + ) || + !optionalMatches( + input.event.subscriptionEffectState, + delivery.subscriptionEffectState, + ) || + !optionalMatches( + input.event.subscriptionEffectLease, + delivery.subscriptionEffectLease, + ) || + !optionalMatches( + input.event.subscriptionEffectLeaseAt, + delivery.subscriptionEffectLeaseAt, + ) || + !optionalMatches( + input.event.subscriptionEffectRetryAt, + delivery.subscriptionEffectRetryAt, + ) || + !optionalMatches( + input.event.subscriptionEffectAttemptCount, + delivery.subscriptionEffectAttemptCount, + ) || + !optionalMatches( + input.event.subscriptionEffectDeadLetterAt, + delivery.subscriptionEffectDeadLetterAt, + ) || + !optionalMatches( + input.event.subscriptionEffectLastError, + delivery.subscriptionEffectLastError, + ) + ) { + throw new Error( + 'USER inbound bootstrap requires a valid owner-bound complete snapshot.', + ) + } +} + +/** + * Reject authority-bearing lifecycle/dedupe snapshots from normal mirror + * upserts. The only inbound-provider exception is the bounded pre-claim audit. + */ +export function shouldSkipMailboxDeliveryEventWrite( + sql: SqlStorage, + input: { + event: MailboxDeliveryEventInput + }, +) { + if (!isUserInboundAuthorityEvent(input.event)) return false + if (!isPreClaimAuditSnapshot(input.event)) { + throw new Error( + 'USER inbound delivery events require the missing-row bootstrap RPC.', + ) + } + const eventId = assertMailboxNonEmptyString(input.event.id, 'event.id') + const existing = readMailboxDeliveryEvent(sql, eventId) + return existing != null && !isPreClaimAuditSnapshot(existing) +} diff --git a/packages/worker/src/email/mailbox-inbound-cleanup-ledger.ts b/packages/worker/src/email/mailbox-inbound-cleanup-ledger.ts new file mode 100644 index 0000000000..0e5d72f208 --- /dev/null +++ b/packages/worker/src/email/mailbox-inbound-cleanup-ledger.ts @@ -0,0 +1,228 @@ +import { + assertMailboxCanonicalIsoTimestamp, + assertMailboxInboundDeliveryState, + assertMailboxNonEmptyString, + type MailboxInboundDeliveryState, +} from './mailbox-types.ts' +import { + detailJsonFromMailboxInboundSnapshot, + mailboxInboundProvider, + mailboxInboundReconciliationRetryMs, + mailboxInboundStorageLeaseMs, + normalizeMailboxInboundNow, + readMailboxInboundDeliveryById, + type MailboxClaimInboundDeliveryCleanupResult, + type MailboxInboundDeliverySnapshot, + type MailboxMarkInboundDeliveryOrphanCleanedResult, + type MailboxReleaseInboundDeliveryCleanupResult, +} from './mailbox-inbound-ledger.ts' +import { mailboxInboundOrphanVerificationMs } from './mailbox-inbound-ledger-shared.ts' + +export function claimMailboxInboundDeliveryCleanup( + sql: SqlStorage, + input: { + deliveryId: string + expectedState: MailboxInboundDeliveryState + expectedUpdatedAt: string + staleBefore: string + now?: string + }, +): MailboxClaimInboundDeliveryCleanupResult { + const now = normalizeMailboxInboundNow(input.now) + const deliveryId = assertMailboxNonEmptyString(input.deliveryId, 'deliveryId') + const expectedState = assertMailboxInboundDeliveryState(input.expectedState) + const expectedUpdatedAt = assertMailboxCanonicalIsoTimestamp( + input.expectedUpdatedAt, + 'expectedUpdatedAt', + ) + const staleBefore = assertMailboxCanonicalIsoTimestamp( + input.staleBefore, + 'staleBefore', + ) + const current = readMailboxInboundDeliveryById(sql, deliveryId) + if (!current) return { status: 'not-claimed', delivery: null } + const leaseExpiredBefore = new Date( + Date.parse(now) - mailboxInboundStorageLeaseMs, + ).toISOString() + const claimable = + current.state === expectedState && + current.updatedAt === expectedUpdatedAt && + current.createdAt < staleBefore && + (current.reconcileAfter == null || current.reconcileAfter <= now) && + (current.state === 'pending' || + (current.state === 'storing' && + current.storageLeaseAt != null && + current.storageLeaseAt < leaseExpiredBefore) || + (current.state === 'cleaning' && + current.cleanupLeaseAt != null && + current.cleanupLeaseAt < leaseExpiredBefore) || + (current.state === 'orphan-cleaned' && + current.cleanupRetryAt != null && + current.cleanupRetryAt <= now)) + if (!claimable) return { status: 'not-claimed', delivery: current } + + const cleanupLease = crypto.randomUUID() + const next: MailboxInboundDeliverySnapshot = { + ...current, + state: 'cleaning', + cleanupLease, + cleanupLeaseAt: now, + updatedAt: now, + } + const cursor = sql.exec( + `UPDATE email_delivery_events + SET detail_json = ?, + state = 'cleaning', + cleanup_lease = ?, + cleanup_lease_at = ?, + updated_at = ? + WHERE id = ? + AND provider = ? + AND state = ? + AND updated_at = ? + AND created_at < ? + AND (reconcile_after IS NULL OR reconcile_after <= ?) + AND ( + state = 'pending' + OR (state = 'storing' AND storage_lease_at < ?) + OR (state = 'cleaning' AND cleanup_lease_at < ?) + OR (state = 'orphan-cleaned' AND cleanup_retry_at <= ?) + )`, + detailJsonFromMailboxInboundSnapshot(next), + cleanupLease, + now, + now, + deliveryId, + mailboxInboundProvider, + expectedState, + expectedUpdatedAt, + staleBefore, + now, + leaseExpiredBefore, + leaseExpiredBefore, + now, + ) + const after = readMailboxInboundDeliveryById(sql, deliveryId) + if ( + cursor.rowsWritten > 0 && + after?.state === 'cleaning' && + after.cleanupLease === cleanupLease + ) { + return { status: 'claimed', delivery: after } + } + return { status: 'not-claimed', delivery: after } +} + +export function releaseMailboxInboundDeliveryCleanup( + sql: SqlStorage, + input: { + deliveryId: string + cleanupLease: string + now?: string + }, +): MailboxReleaseInboundDeliveryCleanupResult { + const now = normalizeMailboxInboundNow(input.now) + const deliveryId = assertMailboxNonEmptyString(input.deliveryId, 'deliveryId') + const cleanupLease = assertMailboxNonEmptyString( + input.cleanupLease, + 'cleanupLease', + ) + const current = readMailboxInboundDeliveryById(sql, deliveryId) + if ( + !current || + current.state !== 'cleaning' || + current.cleanupLease !== cleanupLease + ) { + return { status: 'not-held' } + } + const next: MailboxInboundDeliverySnapshot = { + ...current, + state: 'pending', + updatedAt: now, + } + delete next.cleanupLease + delete next.cleanupLeaseAt + const cursor = sql.exec( + `UPDATE email_delivery_events + SET detail_json = ?, + state = 'pending', + cleanup_lease = NULL, + cleanup_lease_at = NULL, + updated_at = ? + WHERE id = ? + AND provider = ? + AND state = 'cleaning' + AND cleanup_lease = ?`, + detailJsonFromMailboxInboundSnapshot(next), + now, + deliveryId, + mailboxInboundProvider, + cleanupLease, + ) + if (cursor.rowsWritten < 1) return { status: 'not-held' } + const after = readMailboxInboundDeliveryById(sql, deliveryId) + return after + ? { status: 'released', delivery: after } + : { status: 'not-held' } +} + +export function markMailboxInboundDeliveryOrphanCleaned( + sql: SqlStorage, + input: { + deliveryId: string + cleanupLease: string + outcome: 'deleted' | 'delete-failed' + now?: string + }, +): MailboxMarkInboundDeliveryOrphanCleanedResult { + const now = normalizeMailboxInboundNow(input.now) + const deliveryId = assertMailboxNonEmptyString(input.deliveryId, 'deliveryId') + const cleanupLease = assertMailboxNonEmptyString( + input.cleanupLease, + 'cleanupLease', + ) + const current = readMailboxInboundDeliveryById(sql, deliveryId) + if ( + !current || + current.state !== 'cleaning' || + current.cleanupLease !== cleanupLease + ) { + return { status: 'lease-lost' } + } + const retryDelayMs = + input.outcome === 'deleted' + ? mailboxInboundOrphanVerificationMs + : mailboxInboundReconciliationRetryMs + const next: MailboxInboundDeliverySnapshot = { + ...current, + state: 'orphan-cleaned', + cleanupRetryAt: new Date(Date.parse(now) + retryDelayMs).toISOString(), + updatedAt: now, + } + delete next.cleanupLease + delete next.cleanupLeaseAt + const cursor = sql.exec( + `UPDATE email_delivery_events + SET detail_json = ?, + state = 'orphan-cleaned', + cleanup_lease = NULL, + cleanup_lease_at = NULL, + cleanup_retry_at = ?, + updated_at = ? + WHERE id = ? + AND provider = ? + AND state = 'cleaning' + AND cleanup_lease = ?`, + detailJsonFromMailboxInboundSnapshot(next), + next.cleanupRetryAt, + now, + deliveryId, + mailboxInboundProvider, + cleanupLease, + ) + if (cursor.rowsWritten < 1) return { status: 'lease-lost' } + const after = readMailboxInboundDeliveryById(sql, deliveryId) + return after + ? { status: 'orphan-cleaned', delivery: after } + : { status: 'lease-lost' } +} diff --git a/packages/worker/src/email/mailbox-inbound-effect-ledger.ts b/packages/worker/src/email/mailbox-inbound-effect-ledger.ts index 913940b231..5f582cf63e 100644 --- a/packages/worker/src/email/mailbox-inbound-effect-ledger.ts +++ b/packages/worker/src/email/mailbox-inbound-effect-ledger.ts @@ -1,5 +1,5 @@ /** - * Additive Mailbox USER inbound effect CAS primitives (step 2a). + * Mailbox USER inbound effect authority CAS primitives. * * Lease claim / complete / fail for usage and subscription effects. Does not * perform external usage recording or subscription dispatch — callers do that diff --git a/packages/worker/src/email/mailbox-inbound-ledger-shared.ts b/packages/worker/src/email/mailbox-inbound-ledger-shared.ts index 3a87b0de59..85ce3e455b 100644 --- a/packages/worker/src/email/mailbox-inbound-ledger-shared.ts +++ b/packages/worker/src/email/mailbox-inbound-ledger-shared.ts @@ -1,6 +1,5 @@ /** - * Shared snapshot / detail helpers and constants for Mailbox inbound ledger - * CAS (step 2a). + * Shared snapshot / detail helpers and constants for Mailbox inbound authority. */ import { mapMailboxDeliveryEventRow } from './mailbox-mappers.ts' @@ -16,6 +15,7 @@ export const mailboxInboundProvider = 'cloudflare-email-routing' export const mailboxInboundDedupeProvider = 'cloudflare-email-routing-dedupe' export const mailboxInboundStorageLeaseMs = 5 * 60 * 1000 export const mailboxInboundReconciliationRetryMs = 15 * 60 * 1000 +export const mailboxInboundOrphanVerificationMs = 60 * 60 * 1000 export const mailboxInboundDeliveryDedupeWindowMs = 48 * 60 * 60 * 1000 export const mailboxStaleInboundDeliveryAgeMs = 48 * 60 * 60 * 1000 export const mailboxUsageEffectLeaseMs = 5 * 60 * 1000 @@ -112,7 +112,10 @@ export function snapshotFromMailboxDeliveryEventRow( const inboxId = optionalDetailString(detail['inboxId']) ?? row.inboxId ?? undefined const recipient = optionalDetailString(detail['recipient']) - const envelopeFrom = optionalDetailString(detail['envelopeFrom']) + const envelopeFrom = + typeof detail['envelopeFrom'] === 'string' + ? detail['envelopeFrom'] + : undefined const quotaDay = optionalDetailString(detail['quotaDay']) const dedupeExpiresAt = row.dedupeExpiresAt ?? optionalDetailString(detail['dedupeExpiresAt']) @@ -123,7 +126,7 @@ export function snapshotFromMailboxDeliveryEventRow( !rawMimeKey || !inboxId || !recipient || - !envelopeFrom || + envelopeFrom === undefined || !quotaDay || !dedupeExpiresAt ) { diff --git a/packages/worker/src/email/mailbox-inbound-ledger.ts b/packages/worker/src/email/mailbox-inbound-ledger.ts index c607316ea3..44393939b2 100644 --- a/packages/worker/src/email/mailbox-inbound-ledger.ts +++ b/packages/worker/src/email/mailbox-inbound-ledger.ts @@ -1,11 +1,11 @@ /** - * Additive Mailbox USER inbound ledger CAS primitives (step 2a). + * Mailbox USER inbound ledger authority CAS primitives. * - * Owner-bound SQLite helpers matching D1 USER transitions in - * `inbound-delivery.ts`. Effect lease CAS lives in + * Owner-bound SQLite helpers implementing authoritative USER transitions. + * The legacy D1 implementation is system-only. Effect lease CAS lives in * `mailbox-inbound-effect-ledger.ts`. No external usage/subscription side - * effects. Mirror upsert RPCs remain the compatibility write path; these CAS - * RPCs are not live-wired (D1 stays authority). `system:email` stays on D1. + * effects. D1 receives synchronous compatibility snapshots after CAS; + * `system:email` stays on D1. */ import { emailRawMimeKey } from './blob-keys.ts' @@ -105,7 +105,22 @@ export type MailboxDeferInboundDeliveryReconcileResult = | { status: 'missing' } | { status: 'not-applicable' } -export type MailboxPruneExpiredInboundDedupeResult = { pruned: number } +export type MailboxClaimInboundDeliveryCleanupResult = + | { status: 'claimed'; delivery: MailboxInboundDeliverySnapshot } + | { status: 'not-claimed'; delivery: MailboxInboundDeliverySnapshot | null } + +export type MailboxReleaseInboundDeliveryCleanupResult = + | { status: 'released'; delivery: MailboxInboundDeliverySnapshot } + | { status: 'not-held' } + +export type MailboxMarkInboundDeliveryOrphanCleanedResult = + | { status: 'orphan-cleaned'; delivery: MailboxInboundDeliverySnapshot } + | { status: 'lease-lost' } + +export type MailboxPruneExpiredInboundDedupeResult = { + pruned: number + prunedEventIds: Array +} export type MailboxListDueStaleInboundDeliveriesResult = { deliveries: Array @@ -171,6 +186,27 @@ export type MailboxInboundDeliveryLedgerRpc = { deliveryId: string now?: string }) => Promise + claimInboundDeliveryCleanup: (input: { + ownerId: string + deliveryId: string + expectedState: MailboxInboundDeliveryState + expectedUpdatedAt: string + staleBefore: string + now?: string + }) => Promise + releaseInboundDeliveryCleanup: (input: { + ownerId: string + deliveryId: string + cleanupLease: string + now?: string + }) => Promise + markInboundDeliveryOrphanCleaned: (input: { + ownerId: string + deliveryId: string + cleanupLease: string + outcome: 'deleted' | 'delete-failed' + now?: string + }) => Promise listDueStaleInboundDeliveries: (input: { ownerId: string now?: string @@ -225,7 +261,9 @@ function assertInsertInput( ) assertMailboxNonEmptyString(delivery.inboxId, 'delivery.inboxId') assertMailboxNonEmptyString(delivery.recipient, 'delivery.recipient') - assertMailboxNonEmptyString(delivery.envelopeFrom, 'delivery.envelopeFrom') + if (typeof delivery.envelopeFrom !== 'string') { + throw new Error('Mailbox delivery.envelopeFrom must be a string.') + } assertMailboxNonEmptyString(delivery.quotaDay, 'delivery.quotaDay') return { deliveryId, messageId, fingerprint, threadId, rawMimeKey } } @@ -322,11 +360,11 @@ export function claimMailboxInboundDeliveryWindow( fingerprint = excluded.fingerprint, dedupe_expires_at = excluded.dedupe_expires_at, usage_started_at = excluded.usage_started_at, - provider = excluded.provider, provider_event_id = excluded.provider_event_id, created_at = excluded.created_at, updated_at = excluded.updated_at - WHERE email_delivery_events.dedupe_expires_at <= ?`, + WHERE email_delivery_events.provider = ? + AND email_delivery_events.dedupe_expires_at <= ?`, pointerId, input.delivery.inboxId, mailboxInboundDedupeProvider, @@ -337,9 +375,13 @@ export function claimMailboxInboundDeliveryWindow( input.delivery.usageStartedAt ?? null, now, now, + mailboxInboundDedupeProvider, now, ) const row = readMailboxDeliveryEventRow(sql, pointerId) + if (row?.provider !== mailboxInboundDedupeProvider) { + throw new Error('Mailbox inbound dedupe pointer failed its provider fence.') + } const snapshot = row ? snapshotFromMailboxDeliveryEventRow(row) : null if (!snapshot) { throw new Error('Failed to resolve the inbound delivery dedupe window.') @@ -748,7 +790,7 @@ export function pruneMailboxExpiredInboundDedupePointers( ) .toArray() .map((row) => String(row.id)) - let pruned = 0 + const prunedEventIds: Array = [] for (const id of ids) { const cursor = sql.exec( `DELETE FROM email_delivery_events @@ -757,9 +799,9 @@ export function pruneMailboxExpiredInboundDedupePointers( mailboxInboundDedupeProvider, now, ) - pruned += cursor.rowsWritten + if (cursor.rowsWritten > 0) prunedEventIds.push(id) } - return { pruned } + return { pruned: prunedEventIds.length, prunedEventIds } } export function deferMailboxInboundDeliveryReconciliation( diff --git a/packages/worker/src/email/mailbox-inbound-ledger.workers.test.ts b/packages/worker/src/email/mailbox-inbound-ledger.workers.test.ts index ebc255abcd..83e623d6e7 100644 --- a/packages/worker/src/email/mailbox-inbound-ledger.workers.test.ts +++ b/packages/worker/src/email/mailbox-inbound-ledger.workers.test.ts @@ -5,10 +5,13 @@ import { emailRawMimeKey } from './blob-keys.ts' import { Mailbox } from './mailbox-do.ts' import { insertInput } from './mailbox-inbound-ledger-test-helpers.ts' import { + mailboxInboundDedupePointerId, mailboxInboundDedupeProvider, mailboxInboundProvider, + mailboxInboundReconciliationRetryMs, mailboxInboundStorageLeaseMs, } from './mailbox-inbound-ledger.ts' +import { mailboxInboundOrphanVerificationMs } from './mailbox-inbound-ledger-shared.ts' import { initializeMailboxSchema } from './mailbox-schema.ts' import { mailboxMetaSchemaVersionKey, @@ -21,7 +24,7 @@ import { uniqueUserId, } from './mailbox-test-helpers.ts' -test('Mailbox inbound ledger CAS covers USER transition matrix without live flip', async () => { +test('Mailbox inbound ledger CAS covers USER authority transition matrix', async () => { silenceIncidentalRuntimeWarnings() const ownerA = uniqueUserId('ledger-a') const ownerB = uniqueUserId('ledger-b') @@ -252,6 +255,76 @@ test('Mailbox inbound ledger CAS covers USER transition matrix without live flip expect( dueStale.deliveries.some((d) => d.deliveryId === stalePending.deliveryId), ).toBe(true) + const racedPending = insertInput(ownerA, { + fingerprint: 'fp-stale-race', + deliveryId: 'email-inbound-delivery:stale-race', + messageId: 'email-inbound-message:stale-race', + threadId: 'email-inbound-thread:stale-race', + }) + await mailboxA.insertChargedPendingInboundDelivery({ + ownerId: ownerA, + delivery: racedPending, + now: '2026-07-19T00:00:00.000Z', + }) + const racedDue = await mailboxA.listDueStaleInboundDeliveries({ + ownerId: ownerA, + now, + limit: 50, + }) + const racedSnapshot = racedDue.deliveries.find( + (delivery) => delivery.deliveryId === racedPending.deliveryId, + ) + if (!racedSnapshot) throw new Error('expected stale race snapshot') + const racedStorage = await mailboxA.claimInboundDeliveryStorage({ + ownerId: ownerA, + deliveryId: racedPending.deliveryId, + expectedAttachmentCount: 0, + now: '2026-07-22T00:00:01.000Z', + }) + if ( + racedStorage.status !== 'claimed' || + !racedStorage.delivery.storageLease + ) { + throw new Error('expected raced storage claim') + } + await mailboxA.releaseInboundDeliveryStorage({ + ownerId: ownerA, + deliveryId: racedPending.deliveryId, + storageLease: racedStorage.delivery.storageLease, + now: '2026-07-22T00:00:02.000Z', + }) + expect( + await mailboxA.claimInboundDeliveryCleanup({ + ownerId: ownerA, + deliveryId: racedPending.deliveryId, + expectedState: racedSnapshot.state, + expectedUpdatedAt: racedSnapshot.updatedAt, + staleBefore: '2026-07-21T00:00:00.000Z', + now: '2026-07-22T00:00:03.000Z', + }), + ).toMatchObject({ status: 'not-claimed' }) + const cleanupClaim = await mailboxA.claimInboundDeliveryCleanup({ + ownerId: ownerA, + deliveryId: stalePending.deliveryId, + expectedState: 'pending', + expectedUpdatedAt: '2026-07-19T00:00:00.000Z', + staleBefore: '2026-07-21T00:00:00.000Z', + now, + }) + expect(cleanupClaim.status).toBe('claimed') + if ( + cleanupClaim.status !== 'claimed' || + !cleanupClaim.delivery.cleanupLease + ) { + throw new Error('expected cleanup claim') + } + const cleanupRelease = await mailboxA.releaseInboundDeliveryCleanup({ + ownerId: ownerA, + deliveryId: stalePending.deliveryId, + cleanupLease: cleanupClaim.delivery.cleanupLease, + now, + }) + expect(cleanupRelease.status).toBe('released') const foreign = insertInput(ownerB, { fingerprint: 'fp-b', @@ -259,6 +332,11 @@ test('Mailbox inbound ledger CAS covers USER transition matrix without live flip messageId: 'email-inbound-message:owner-b', threadId: 'email-inbound-thread:owner-b', }) + await mailboxB.claimInboundDeliveryWindow({ + ownerId: ownerB, + delivery: foreign, + now, + }) await mailboxB.insertChargedPendingInboundDelivery({ ownerId: ownerB, delivery: foreign, @@ -297,6 +375,17 @@ test('Mailbox inbound ledger CAS covers USER transition matrix without live flip limit: 50, }) expect(pruned.pruned).toBeGreaterThan(0) + expect(pruned.prunedEventIds).toHaveLength(pruned.pruned) + expect(pruned.prunedEventIds).not.toContain( + mailboxInboundDedupePointerId(foreign.fingerprint), + ) + expect( + await mailboxB.getInboundDeliveryWindow({ + ownerId: ownerB, + fingerprint: foreign.fingerprint, + now, + }), + ).toMatchObject({ deliveryId: foreign.deliveryId }) // Defer reconcile. const deferred = await mailboxA.deferInboundDeliveryReconciliation({ @@ -305,6 +394,74 @@ test('Mailbox inbound ledger CAS covers USER transition matrix without live flip now, }) expect(deferred.status).toBe('deferred') + const orphanNow = '2026-07-22T00:30:00.000Z' + const orphanClaim = await mailboxA.claimInboundDeliveryCleanup({ + ownerId: ownerA, + deliveryId: stalePending.deliveryId, + expectedState: 'pending', + expectedUpdatedAt: now, + staleBefore: '2026-07-21T00:00:00.000Z', + now: orphanNow, + }) + if (orphanClaim.status !== 'claimed' || !orphanClaim.delivery.cleanupLease) { + throw new Error('expected orphan cleanup claim') + } + expect( + await mailboxA.markInboundDeliveryOrphanCleaned({ + ownerId: ownerA, + deliveryId: stalePending.deliveryId, + cleanupLease: 'stale-cleanup-lease', + outcome: 'deleted', + now: orphanNow, + }), + ).toEqual({ status: 'lease-lost' }) + const orphaned = await mailboxA.markInboundDeliveryOrphanCleaned({ + ownerId: ownerA, + deliveryId: stalePending.deliveryId, + cleanupLease: orphanClaim.delivery.cleanupLease, + outcome: 'delete-failed', + now: orphanNow, + }) + expect(orphaned.status).toBe('orphan-cleaned') + if (orphaned.status !== 'orphan-cleaned') { + throw new Error('expected orphan-cleaned result') + } + expect(orphaned.delivery.cleanupRetryAt).toBe( + new Date( + Date.parse(orphanNow) + mailboxInboundReconciliationRetryMs, + ).toISOString(), + ) + const deletedNow = '2026-07-22T01:00:00.000Z' + const deletedClaim = await mailboxA.claimInboundDeliveryCleanup({ + ownerId: ownerA, + deliveryId: stalePending.deliveryId, + expectedState: 'orphan-cleaned', + expectedUpdatedAt: orphanNow, + staleBefore: '2026-07-21T00:00:00.000Z', + now: deletedNow, + }) + if ( + deletedClaim.status !== 'claimed' || + !deletedClaim.delivery.cleanupLease + ) { + throw new Error('expected deleted verification cleanup claim') + } + const deleted = await mailboxA.markInboundDeliveryOrphanCleaned({ + ownerId: ownerA, + deliveryId: stalePending.deliveryId, + cleanupLease: deletedClaim.delivery.cleanupLease, + outcome: 'deleted', + now: deletedNow, + }) + expect(deleted.status).toBe('orphan-cleaned') + if (deleted.status !== 'orphan-cleaned') { + throw new Error('expected deleted orphan-cleaned result') + } + expect(deleted.delivery.cleanupRetryAt).toBe( + new Date( + Date.parse(deletedNow) + mailboxInboundOrphanVerificationMs, + ).toISOString(), + ) // Mirror upsert compatibility still works alongside ledger rows. const mirror = await mailboxA.upsertDeliveryEvent({ diff --git a/packages/worker/src/email/mailbox-live-mirror.ts b/packages/worker/src/email/mailbox-live-mirror.ts index 493a6c0e66..83f609621c 100644 --- a/packages/worker/src/email/mailbox-live-mirror.ts +++ b/packages/worker/src/email/mailbox-live-mirror.ts @@ -17,13 +17,19 @@ import { listEmailAttachmentsForMessage, } from './repo.ts' import { type EmailThreadRecord } from './types.ts' +import { + mailboxInboundDedupeProvider, + mailboxInboundProvider, +} from './mailbox-inbound-ledger.ts' /** * High-level D1 → Mailbox live-mirror orchestration. * - * D1 remains authoritative. These helpers load cohesive snapshots from D1 and - * call best-effort mirror RPCs; they never throw into caller paths and do not - * perform deletes or inbound-lifecycle special cases. + * D1 remains authoritative for the message graph and non-USER-inbound events. + * These helpers load cohesive snapshots from D1 and call best-effort mirror + * RPCs; they never throw into caller paths and do not perform deletes or + * inbound-lifecycle special cases. USER inbound terminal work explicitly omits + * delivery events because those snapshots flow Mailbox → D1. * * Callers that just created a thread may pass it via `thread` to avoid a * round-trip; otherwise the graph helper loads it with `getEmailThreadById`. @@ -44,6 +50,13 @@ export const mailboxLiveMirrorMaxAnalyticsWrites = 2 export type MailboxLiveMirrorEnv = MailboxMirrorEnv +function isUserInboundAuthorityProvider(provider: string | null) { + return ( + provider === mailboxInboundProvider || + provider === mailboxInboundDedupeProvider + ) +} + export type MailboxLiveMirrorEventResult = { eventId: string result: MailboxMirrorResult @@ -94,6 +107,9 @@ export async function mirrorMailboxDeliveryEventFromD1(input: { eventId: input.eventId, }) if (!projection) return { status: 'missing' } + if (isUserInboundAuthorityProvider(projection.provider)) { + return { status: 'skipped', reason: 'user-inbound-authority' } + } const sourceMutationAt = input.sourceMutationAt != null && input.sourceMutationAt.length > 0 ? input.sourceMutationAt @@ -132,6 +148,8 @@ export async function mirrorMailboxMessageGraphFromD1(input: { userId: string messageId: string thread?: EmailThreadRecord | null + /** USER inbound terminal work keeps its DO-authoritative event untouched. */ + includeDeliveryEvents?: boolean }): Promise { try { const message = await getEmailMessageById({ @@ -168,6 +186,9 @@ export async function mirrorMailboxMessageGraphFromD1(input: { message, attachments, }) + if (input.includeDeliveryEvents === false) { + return emptyGraphSummary(input.messageId, messageResult) + } // Newest max+1 from D1 (chrono-restored). When truncated, keep the // trailing newest max — drop the oldest overflow row at index 0. @@ -186,9 +207,11 @@ export async function mirrorMailboxMessageGraphFromD1(input: { max: mailboxLiveMirrorMaxEvents, }) } - const eventInputs = eventsTruncated - ? loadedEvents.slice(-mailboxLiveMirrorMaxEvents) - : loadedEvents + const eventInputs = ( + eventsTruncated + ? loadedEvents.slice(-mailboxLiveMirrorMaxEvents) + : loadedEvents + ).filter((event) => !isUserInboundAuthorityProvider(event.provider)) const events = eventInputs.length === 0 diff --git a/packages/worker/src/email/mailbox-mirror.node.test.ts b/packages/worker/src/email/mailbox-mirror.node.test.ts index 23ae74b51b..b12b9671e2 100644 --- a/packages/worker/src/email/mailbox-mirror.node.test.ts +++ b/packages/worker/src/email/mailbox-mirror.node.test.ts @@ -550,6 +550,87 @@ test('mailbox mirror batch helper uses one RPC, one telemetry outcome, and maps expect(upsertDeliveryEvents).toHaveBeenCalledTimes(1) }) +test('mailbox mirror batch partitions legacy authority from audits and counts existing/skipped without errors', async () => { + const legacyExisting = baseDeliveryEventInput({ + id: 'email-inbound-delivery:existing', + provider: 'cloudflare-email-routing', + eventType: 'receive_started', + state: 'pending', + fingerprint: 'existing-fingerprint', + }) + const legacySkipped = baseDeliveryEventInput({ + id: 'email-inbound-dedupe:skipped-fingerprint', + provider: 'cloudflare-email-routing-dedupe', + eventType: 'receive_started', + state: 'pending', + fingerprint: 'skipped-fingerprint', + }) + const audit = baseDeliveryEventInput({ + id: 'email-rejections:inbox-audit:2026-07-02', + messageId: null, + inboxId: 'inbox-audit', + provider: 'cloudflare-email-routing', + eventType: 'rejected', + detailJson: JSON.stringify({ + aggregate: true, + day: '2026-07-02', + count: 1, + last_reason: 'Recipient mailbox is over quota.', + last_phase: 'entitlement', + last_at: '2026-07-02T10:00:00.000Z', + }), + }) + const upsertDeliveryEvents = vi.fn(async () => ({ + results: [{ eventId: audit.id, inserted: true, accepted: true }], + })) + const bootstrapDeliveryEvents = vi.fn(async () => ({ + inserted: 0, + existing: 1, + skipped: 1, + results: [ + { eventId: legacyExisting.id, status: 'existing' as const }, + { eventId: legacySkipped.id, status: 'skipped' as const }, + ], + })) + const { env, writeDataPoint } = fakeMailboxEnv({ + upsertDeliveryEvents, + bootstrapDeliveryEvents, + }) + + await expect( + mirrorMailboxDeliveryEventSnapshots({ + env, + ownerId: 'user-aaa', + events: [legacyExisting, audit, legacySkipped], + }), + ).resolves.toEqual([ + { eventId: legacyExisting.id, result: { status: 'stale' } }, + { eventId: audit.id, result: { status: 'mirrored' } }, + { + eventId: legacySkipped.id, + result: { status: 'skipped', reason: 'user-inbound-authority' }, + }, + ]) + expect(upsertDeliveryEvents).toHaveBeenCalledWith({ + ownerId: 'user-aaa', + events: [audit], + }) + expect(bootstrapDeliveryEvents).toHaveBeenCalledWith({ + ownerId: 'user-aaa', + events: [legacyExisting, legacySkipped], + }) + expect(consoleWarn).not.toHaveBeenCalled() + expect(writeDataPoint).toHaveBeenCalledWith( + expect.objectContaining({ + blobs: [ + 'mailbox_mirror:upsert_delivery_event_batch', + 'skipped', + expect.any(String), + ], + }), + ) +}) + test('mailbox mirror batch helper maps timeout and error to every event', async () => { vi.useFakeTimers() consoleWarn.mockImplementation(() => {}) diff --git a/packages/worker/src/email/mailbox-mirror.ts b/packages/worker/src/email/mailbox-mirror.ts index a5dbd44afa..200b904ace 100644 --- a/packages/worker/src/email/mailbox-mirror.ts +++ b/packages/worker/src/email/mailbox-mirror.ts @@ -15,6 +15,7 @@ import { toMailboxMessageInput, toMailboxThreadInput, } from './mailbox-snapshots.ts' +import { isUserInboundLegacyAuthoritySnapshot } from './mailbox-inbound-bootstrap.ts' import { type EmailAttachmentRecord, type EmailDeliveryStatus, @@ -31,7 +32,6 @@ import { type MailboxSetMessageClassificationInput, type MailboxTouchThreadInput, type MailboxUpdateMessageDeliveryInput, - type MailboxUpsertDeliveryEventsResult, } from './mailbox-types.ts' /** @@ -62,6 +62,7 @@ export type MailboxMirrorSkipReason = | 'system-email' | 'mailbox-unconfigured' | 'missing-owner' + | 'user-inbound-authority' export type MailboxMirrorResult = | { status: 'mirrored' } @@ -80,7 +81,10 @@ function resolveMirrorOwner( ownerId: string | null | undefined, ): { ok: true; ownerId: string } | { ok: false; result: MailboxMirrorResult } { if (ownerId == null || ownerId.length === 0) { - return { ok: false, result: { status: 'skipped', reason: 'missing-owner' } } + return { + ok: false, + result: { status: 'skipped', reason: 'missing-owner' }, + } } if (isSystemEmailOwner(ownerId)) { return { @@ -177,12 +181,20 @@ function outcomeFromAccepted(accepted: boolean): MailboxMirrorResult { return accepted ? { status: 'mirrored' } : { status: 'stale' } } -function outcomeFromBatchAccepted( - results: MailboxUpsertDeliveryEventsResult['results'], +function aggregateBatchOutcome( + results: Array, ): MailboxMirrorResult { - return results.every((item) => item.accepted) - ? { status: 'mirrored' } - : { status: 'stale' } + for (const status of [ + 'error', + 'timeout', + 'skipped', + 'missing', + 'stale', + ] as const) { + const result = results.find((item) => item.result.status === status)?.result + if (result) return result + } + return { status: 'mirrored' } } /** @@ -324,10 +336,11 @@ export async function mirrorMailboxDeliveryEventSnapshot(input: { /** * Best-effort batch delivery-event snapshot mirror. * - * One timeout (default {@link mailboxMirrorRpcTimeoutMs}) and one - * `upsert_delivery_event_batch` telemetry outcome. Timeout/error/skip apply - * uniformly to each per-event summary entry. Empty `events` returns `[]` - * without RPC or telemetry. + * Partitions legacy USER inbound authority snapshots to missing-only bootstrap + * and all other rows to normal upsert. Each non-empty subset gets the timeout + * bound (default {@link mailboxMirrorRpcTimeoutMs}); one aggregate + * `upsert_delivery_event_batch` telemetry outcome covers the input page. Empty + * `events` returns `[]` without RPC or telemetry. */ export async function mirrorMailboxDeliveryEventSnapshots(input: { env: MailboxMirrorEnv @@ -350,46 +363,121 @@ export async function mirrorMailboxDeliveryEventSnapshots(input: { return mapUniformBatchResults(input.events, skippedMailbox) } - try { - const raced = await awaitMailboxMirrorRpc( - mailboxRpc({ - env: input.env, - userId: owner.ownerId, - }).upsertDeliveryEvents({ - ownerId: owner.ownerId, - events: input.events, - }), - input.timeoutMs ?? mailboxMirrorRpcTimeoutMs, - ) - if (!raced.ok) { - const result = { status: 'timeout' as const } - recordMirrorOutcome(input.env, owner.ownerId, operation, result) - return mapUniformBatchResults(input.events, result) + const timeoutMs = input.timeoutMs ?? mailboxMirrorRpcTimeoutMs + const rpc = mailboxRpc({ + env: input.env, + userId: owner.ownerId, + }) + const bootstrapEvents = input.events.filter( + isUserInboundLegacyAuthoritySnapshot, + ) + const normalEvents = input.events.filter( + (event) => !isUserInboundLegacyAuthoritySnapshot(event), + ) + const resultsByEventId = new Map() + + if (normalEvents.length > 0) { + try { + const raced = await awaitMailboxMirrorRpc( + rpc.upsertDeliveryEvents({ + ownerId: owner.ownerId, + events: normalEvents, + }), + timeoutMs, + ) + if (!raced.ok) { + for (const event of normalEvents) { + resultsByEventId.set(event.id, { status: 'timeout' }) + } + } else { + const byEventId = new Map( + raced.value.results.map((item) => [item.eventId, item] as const), + ) + for (const event of normalEvents) { + const item = byEventId.get(event.id) + resultsByEventId.set( + event.id, + item ? outcomeFromAccepted(item.accepted) : { status: 'missing' }, + ) + } + } + } catch (error) { + console.warn('mailbox-mirror-delivery-event-batch-failed', error) + for (const event of normalEvents) { + resultsByEventId.set(event.id, { status: 'error', error }) + } } - recordMirrorOutcome( - input.env, - owner.ownerId, - operation, - outcomeFromBatchAccepted(raced.value.results), - ) - const byEventId = new Map( - raced.value.results.map((item) => [item.eventId, item] as const), - ) - return input.events.map((event) => { - const item = byEventId.get(event.id) - return { - eventId: event.id, - result: item - ? outcomeFromAccepted(item.accepted) - : { status: 'missing' as const }, + } + + if (bootstrapEvents.length > 0) { + try { + const raced = await awaitMailboxMirrorRpc( + rpc.bootstrapDeliveryEvents({ + ownerId: owner.ownerId, + events: bootstrapEvents, + }), + timeoutMs, + ) + if (!raced.ok) { + for (const event of bootstrapEvents) { + resultsByEventId.set(event.id, { status: 'timeout' }) + } + } else { + const byEventId = new Map( + raced.value.results.map((item) => [item.eventId, item] as const), + ) + for (const event of bootstrapEvents) { + const item = byEventId.get(event.id) + const status = item?.status + let result: MailboxMirrorResult + switch (status) { + case 'inserted': + result = { status: 'mirrored' } + break + case 'existing': + result = { status: 'stale' } + break + case 'skipped': + result = { + status: 'skipped', + reason: 'user-inbound-authority', + } + break + case undefined: + result = { status: 'missing' } + break + default: { + const exhaustive: never = status + throw new Error( + `Unhandled bootstrap delivery-event status: ${String(exhaustive)}`, + ) + } + } + resultsByEventId.set(event.id, result) + } } - }) - } catch (error) { - console.warn('mailbox-mirror-delivery-event-batch-failed', error) - const result = { status: 'error' as const, error } - recordMirrorOutcome(input.env, owner.ownerId, operation, result) - return mapUniformBatchResults(input.events, result) + } catch (error) { + console.warn( + 'mailbox-mirror-delivery-event-bootstrap-batch-failed', + error, + ) + for (const event of bootstrapEvents) { + resultsByEventId.set(event.id, { status: 'error', error }) + } + } } + + const results = input.events.map((event) => ({ + eventId: event.id, + result: resultsByEventId.get(event.id) ?? ({ status: 'missing' } as const), + })) + recordMirrorOutcome( + input.env, + owner.ownerId, + operation, + aggregateBatchOutcome(results), + ) + return results } /** Best-effort thread touch mirror. */ diff --git a/packages/worker/src/email/mailbox-parity-phases.ts b/packages/worker/src/email/mailbox-parity-phases.ts index fc68c25383..10c390a050 100644 --- a/packages/worker/src/email/mailbox-parity-phases.ts +++ b/packages/worker/src/email/mailbox-parity-phases.ts @@ -19,7 +19,7 @@ import { mailboxUpsertDeliveryEventsMax } from './mailbox-types.ts' export const mailboxParityMessagePageSize = 10 /** - * Events per backfill page / single `upsertDeliveryEvents` RPC. + * Events per backfill page, partitioned across normal/bootstrap batch RPCs. * Must stay ≤ {@link mailboxUpsertDeliveryEventsMax}. */ export const mailboxParityEventPageSize = 25 @@ -133,9 +133,10 @@ export function eventMirrorAllowsCursorAdvance( case 'stale': case 'missing': return true + case 'skipped': + return result.reason === 'user-inbound-authority' case 'timeout': case 'error': - case 'skipped': return false default: { const exhaustive: never = result @@ -238,8 +239,8 @@ export async function backfillMessagesForUser(input: { } /** - * Keyset-page ready delivery-event snapshots and mirror each page with one - * `upsertDeliveryEvents` batch RPC ({@link mailboxParityEventMirrorTimeoutMs}). + * Keyset-page ready delivery-event snapshots and mirror each page with bounded + * normal/bootstrap batch RPCs ({@link mailboxParityEventMirrorTimeoutMs}). * * Cursor advances through per-event mirrored/stale/missing results in page * order (including equal `createdAt` by id). Uniform timeout/error/skip keep diff --git a/packages/worker/src/email/mailbox-reconcile.workers.test.ts b/packages/worker/src/email/mailbox-reconcile.workers.test.ts index fd0350d209..fb5670ca8b 100644 --- a/packages/worker/src/email/mailbox-reconcile.workers.test.ts +++ b/packages/worker/src/email/mailbox-reconcile.workers.test.ts @@ -3,6 +3,7 @@ import { expect, test, vi } from 'vitest' import { silenceIncidentalRuntimeWarnings } from '#worker/test-support/incidental-runtime-warnings.ts' import { createStableUserIdFromEmail } from '#worker/user-id.ts' import { ensureUsersTestSchema } from '#worker/users-test-schema.ts' +import { emailRawMimeKey } from './blob-keys.ts' import { mailboxLiveMirrorMaxEvents } from './mailbox-live-mirror.ts' import { countD1MailboxParity, @@ -100,18 +101,145 @@ async function seedMessageGraph(input: { } } -async function seedOrphanEvent(input: { +async function seedPreClaimAuditEvent(input: { userId: string - eventId: string + inboxId: string createdAt: string }) { + const day = input.createdAt.slice(0, 10) + const eventId = `email-rejections:${input.inboxId}:${day}` await env.APP_DB.prepare( `INSERT INTO email_delivery_events ( - id, message_id, user_id, event_type, provider, detail_json, created_at - ) VALUES (?, NULL, ?, 'receive_started', 'cloudflare-email-routing', '{}', ?)`, + id, message_id, user_id, inbox_id, event_type, provider, + detail_json, created_at + ) VALUES (?, NULL, ?, ?, 'rejected', 'cloudflare-email-routing', ?, ?)`, ) - .bind(input.eventId, input.userId, input.createdAt) + .bind( + eventId, + input.userId, + input.inboxId, + JSON.stringify({ + aggregate: true, + day, + count: 2, + last_reason: 'Recipient mailbox is over quota.', + last_phase: 'entitlement', + last_at: input.createdAt, + }), + input.createdAt, + ) + .run() + return eventId +} + +async function seedLegacyInboundEvent(input: { + userId: string + deliveryId: string + state: 'pending' | 'storing' | 'received' | 'orphan-cleaned' + createdAt: string + messageId?: string + fingerprint?: string + provider?: 'cloudflare-email-routing' | 'cloudflare-email-routing-dedupe' + eventId?: string + cleanupRetryAt?: string + reconcileAfter?: string +}) { + const provider = input.provider ?? 'cloudflare-email-routing' + const fingerprint = input.fingerprint ?? `fingerprint-${input.deliveryId}` + const messageId = + input.messageId ?? `email-inbound-message:${input.deliveryId}` + const eventId = + input.eventId ?? + (provider === 'cloudflare-email-routing-dedupe' + ? `email-inbound-dedupe:${fingerprint}` + : input.deliveryId) + const detail = { + fingerprint, + deliveryId: input.deliveryId, + messageId, + threadId: `email-inbound-thread:${input.deliveryId}`, + rawMimeKey: emailRawMimeKey(input.userId, messageId), + userId: input.userId, + inboxId: 'legacy-inbox', + recipient: 'owner@example.test', + envelopeFrom: 'sender@example.test', + provider: 'cloudflare-email-routing', + quotaDay: '2026-07-01', + dedupeExpiresAt: '2026-09-01T00:00:00.000Z', + state: input.state, + ...(input.cleanupRetryAt == null + ? {} + : { cleanupRetryAt: input.cleanupRetryAt }), + ...(input.reconcileAfter == null + ? {} + : { reconcileAfter: input.reconcileAfter }), + ...(input.state === 'storing' + ? { + storageLease: 'legacy-storage-lease', + storageLeaseAt: '2026-07-01T12:01:30.000Z', + expectedAttachmentCount: 2, + } + : {}), + ...(input.state === 'received' + ? { + expectedAttachmentCount: 1, + finalizationToken: 'legacy-finalization-token', + usageEffectRecordedAt: '2026-07-01T12:04:00.000Z', + usageStartedAt: '2026-07-01T12:02:30.000Z', + usageMonth: '2026-07', + usageBytes: 4096, + usageDurationMs: 900, + subscriptionEffectState: 'complete', + } + : {}), + } + await env.APP_DB.prepare( + `INSERT INTO email_delivery_events ( + id, message_id, user_id, inbox_id, event_type, provider, + provider_event_id, detail_json, needs_effect_reconcile, + state, fingerprint, storage_lease, storage_lease_at, + cleanup_retry_at, expected_attachment_count, finalization_token, + reconcile_after, dedupe_expires_at, + usage_effect_recorded_at, usage_started_at, usage_month, usage_bytes, + usage_duration_ms, subscription_effect_state, created_at, updated_at + ) VALUES ( + ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? + )`, + ) + .bind( + eventId, + input.state === 'received' ? messageId : null, + input.userId, + detail.inboxId, + input.state === 'received' ? 'received' : 'receive_started', + provider, + eventId, + JSON.stringify(detail), + 0, + input.state, + fingerprint, + 'storageLease' in detail ? detail.storageLease : null, + 'storageLeaseAt' in detail ? detail.storageLeaseAt : null, + 'cleanupRetryAt' in detail ? detail.cleanupRetryAt : null, + 'expectedAttachmentCount' in detail + ? detail.expectedAttachmentCount + : null, + 'finalizationToken' in detail ? detail.finalizationToken : null, + 'reconcileAfter' in detail ? detail.reconcileAfter : null, + detail.dedupeExpiresAt, + 'usageEffectRecordedAt' in detail ? detail.usageEffectRecordedAt : null, + 'usageStartedAt' in detail ? detail.usageStartedAt : null, + 'usageMonth' in detail ? detail.usageMonth : null, + 'usageBytes' in detail ? detail.usageBytes : null, + 'usageDurationMs' in detail ? detail.usageDurationMs : null, + 'subscriptionEffectState' in detail + ? detail.subscriptionEffectState + : null, + input.createdAt, + input.createdAt, + ) .run() + return { eventId, fingerprint } } async function readParityState(userId: string) { @@ -189,9 +317,9 @@ test('reconcileMailboxParity mismatch full reset and soak', async () => { threadId: 'parity-thread-2', createdAt: '2026-07-01T12:01:00.000Z', }) - await seedOrphanEvent({ + const auditEventId = await seedPreClaimAuditEvent({ userId, - eventId: 'parity-orphan-1', + inboxId: 'parity-inbox-1', createdAt: '2026-07-01T12:02:00.000Z', }) await seedMessageGraph({ @@ -206,7 +334,7 @@ test('reconcileMailboxParity mismatch full reset and soak', async () => { const firstNow = new Date('2026-08-01T10:00:00.000Z') vi.setSystemTime(firstNow) - // 2 messages + 2 bound events + 1 orphan + // 2 messages + 2 bound events + 1 bounded pre-claim audit event await expect( reconcileMailboxParity({ env, now: firstNow, batchSize: 1 }), ).resolves.toEqual({ @@ -232,7 +360,7 @@ test('reconcileMailboxParity mismatch full reset and soak', async () => { messagesCompletedAt: expect.any(String), eventsCompletedAt: expect.any(String), messageCursorId: 'parity-msg-2', - eventCursorId: 'parity-orphan-1', + eventCursorId: auditEventId, lastError: null, }) @@ -505,3 +633,264 @@ test('reconcileMailboxParity eventually repairs >100 bound delivery events', asy .deliveryEvents, ).toBe(heavyEventCount) }, 60_000) + +test('reconcileMailboxParity bootstraps production legacy USER inbound shapes without overwriting authority', async () => { + silenceIncidentalRuntimeWarnings() + await ensureUsersTestSchema({ db: env.APP_DB }) + await ensureEmailTestSchema(env.APP_DB) + await env.APP_DB.prepare( + `UPDATE users + SET mailbox_parity_checked_at = '9999-12-31T23:59:59.999Z'`, + ).run() + + const userId = await seedParityUser({ + email: `legacy-inbound-${crypto.randomUUID()}@example.test`, + checkedAt: null, + }) + const receivedMessageId = 'legacy-received-message' + await seedMessageGraph({ + userId, + messageId: receivedMessageId, + threadId: 'legacy-received-thread', + createdAt: '2026-07-01T12:02:00.000Z', + eventCount: 0, + }) + const pendingId = 'email-inbound-delivery:legacy-pending' + const storingId = 'email-inbound-delivery:legacy-storing' + const receivedId = 'email-inbound-delivery:legacy-received' + const orphanId = 'email-inbound-delivery:legacy-orphan-cleaned' + const reconcileAfter = '2026-08-03T12:00:00.000Z' + const cleanupRetryAt = '2026-08-03T13:00:00.000Z' + const pending = await seedLegacyInboundEvent({ + userId, + deliveryId: pendingId, + state: 'pending', + createdAt: '2026-07-01T12:00:00.000Z', + reconcileAfter, + }) + await seedLegacyInboundEvent({ + userId, + deliveryId: storingId, + state: 'storing', + createdAt: '2026-07-01T12:01:00.000Z', + reconcileAfter, + }) + await seedLegacyInboundEvent({ + userId, + deliveryId: receivedId, + messageId: receivedMessageId, + state: 'received', + createdAt: '2026-07-01T12:03:00.000Z', + }) + await seedLegacyInboundEvent({ + userId, + deliveryId: orphanId, + state: 'orphan-cleaned', + createdAt: '2026-07-01T12:04:00.000Z', + cleanupRetryAt, + reconcileAfter, + }) + await seedLegacyInboundEvent({ + userId, + deliveryId: pendingId, + fingerprint: pending.fingerprint, + state: 'pending', + provider: 'cloudflare-email-routing-dedupe', + createdAt: '2026-07-01T12:00:01.000Z', + }) + const auditEventId = await seedPreClaimAuditEvent({ + userId, + inboxId: 'legacy-inbox', + createdAt: '2026-07-01T12:05:00.000Z', + }) + + const mailbox = rpcFor(userId) + expect((await mailbox.countMailbox()).deliveryEvents).toBe(0) + const firstNow = new Date('2026-08-02T15:00:00.000Z') + await expect( + reconcileMailboxParity({ env, now: firstNow, batchSize: 1 }), + ).resolves.toMatchObject({ + matched: 1, + mismatched: 0, + failed: 0, + }) + expect(await mailbox.countMailbox()).toEqual( + await countD1MailboxParity({ db: env.APP_DB, userId }), + ) + expect( + await mailbox.getInboundDelivery({ + ownerId: userId, + deliveryId: pendingId, + }), + ).toMatchObject({ state: 'pending', reconcileAfter }) + expect( + await mailbox.getInboundDelivery({ + ownerId: userId, + deliveryId: storingId, + }), + ).toMatchObject({ + state: 'storing', + storageLease: 'legacy-storage-lease', + expectedAttachmentCount: 2, + reconcileAfter, + }) + expect( + await mailbox.getInboundDelivery({ + ownerId: userId, + deliveryId: receivedId, + }), + ).toMatchObject({ + state: 'received', + finalizationToken: 'legacy-finalization-token', + usageEffectRecordedAt: '2026-07-01T12:04:00.000Z', + subscriptionEffectState: 'complete', + }) + expect( + await mailbox.getInboundDelivery({ + ownerId: userId, + deliveryId: orphanId, + }), + ).toMatchObject({ + state: 'orphan-cleaned', + cleanupRetryAt, + reconcileAfter, + }) + const unboundEvents = await mailbox.listDeliveryEvents({ + messageId: null, + limit: 20, + }) + expect( + JSON.parse( + unboundEvents.find((event) => event.id === orphanId)!.detailJson, + ), + ).toMatchObject({ cleanupRetryAt, reconcileAfter }) + expect( + await mailbox.getInboundDeliveryWindow({ + ownerId: userId, + fingerprint: pending.fingerprint, + now: '2026-08-02T15:00:00.000Z', + }), + ).toMatchObject({ deliveryId: pendingId }) + expect(unboundEvents.some((event) => event.id === auditEventId)).toBe(true) + await expect( + mailbox.listDueStaleInboundDeliveries({ + ownerId: userId, + now: '2026-08-02T16:00:00.000Z', + limit: 20, + }), + ).resolves.toEqual({ deliveries: [] }) + const dueAfterSchedules = await mailbox.listDueStaleInboundDeliveries({ + ownerId: userId, + now: '2026-08-04T16:00:00.000Z', + limit: 20, + }) + expect( + new Set(dueAfterSchedules.deliveries.map((row) => row.deliveryId)), + ).toEqual(new Set([pendingId, storingId, orphanId])) + + const replayEventBackfill = () => + env.APP_DB.prepare( + `UPDATE users + SET mailbox_parity_checked_at = NULL, + mailbox_parity_event_backfill_cursor_created_at = NULL, + mailbox_parity_event_backfill_cursor_id = NULL, + mailbox_parity_event_backfill_completed_at = NULL + WHERE stable_user_id = ?`, + ) + .bind(userId) + .run() + const countAfterFirst = await mailbox.countMailbox() + await replayEventBackfill() + const secondNow = new Date('2026-08-02T15:30:00.000Z') + await expect( + reconcileMailboxParity({ env, now: secondNow, batchSize: 1 }), + ).resolves.toMatchObject({ + matched: 1, + mismatched: 0, + failed: 0, + }) + expect(await mailbox.countMailbox()).toEqual(countAfterFirst) + expect( + await mailbox.getInboundDelivery({ + ownerId: userId, + deliveryId: orphanId, + }), + ).toMatchObject({ cleanupRetryAt, reconcileAfter }) + + await mailbox.claimInboundDeliveryStorage({ + ownerId: userId, + deliveryId: pendingId, + expectedAttachmentCount: 7, + now: '2026-08-02T15:01:00.000Z', + }) + const newerAuthority = await mailbox.getInboundDelivery({ + ownerId: userId, + deliveryId: pendingId, + }) + expect(newerAuthority).toMatchObject({ + state: 'storing', + expectedAttachmentCount: 7, + }) + await replayEventBackfill() + + const thirdNow = new Date('2026-08-02T16:00:00.000Z') + await expect( + reconcileMailboxParity({ env, now: thirdNow, batchSize: 1 }), + ).resolves.toMatchObject({ + matched: 1, + mismatched: 0, + failed: 0, + }) + expect(await mailbox.countMailbox()).toEqual( + await countD1MailboxParity({ db: env.APP_DB, userId }), + ) + expect( + await mailbox.getInboundDelivery({ + ownerId: userId, + deliveryId: pendingId, + }), + ).toMatchObject({ + state: 'storing', + expectedAttachmentCount: 7, + updatedAt: newerAuthority?.updatedAt, + }) + + const malformedUserId = await seedParityUser({ + email: `legacy-malformed-${crypto.randomUUID()}@example.test`, + checkedAt: null, + }) + await seedLegacyInboundEvent({ + userId: malformedUserId, + deliveryId: 'email-inbound-delivery:malformed-schedule', + state: 'pending', + createdAt: '2026-07-01T13:00:00.000Z', + reconcileAfter: 'not-a-timestamp', + }) + await seedPreClaimAuditEvent({ + userId: malformedUserId, + inboxId: 'malformed-schedule-inbox', + createdAt: '2026-07-01T13:01:00.000Z', + }) + const malformedMailbox = rpcFor(malformedUserId) + await expect( + reconcileMailboxParity({ + env, + now: new Date('2026-08-02T17:00:00.000Z'), + batchSize: 1, + }), + ).resolves.toMatchObject({ + compared: 1, + matched: 0, + mismatched: 1, + failed: 0, + }) + expect( + (await countD1MailboxParity({ db: env.APP_DB, userId: malformedUserId })) + .deliveryEvents, + ).toBe(2) + expect((await malformedMailbox.countMailbox()).deliveryEvents).toBe(0) + expect(await readParityState(malformedUserId)).toMatchObject({ + mismatchCount: 1, + lastError: null, + }) +}, 30_000) diff --git a/packages/worker/src/email/mailbox-snapshot-repo.ts b/packages/worker/src/email/mailbox-snapshot-repo.ts index cd7ea51392..30f2823b20 100644 --- a/packages/worker/src/email/mailbox-snapshot-repo.ts +++ b/packages/worker/src/email/mailbox-snapshot-repo.ts @@ -105,9 +105,8 @@ export async function getEmailDeliveryEventMirrorProjection(input: { /** * Load and convert one ready Mailbox delivery-event snapshot for dual-write. * - * @param sourceMutationAt - Canonical mirror `updatedAt`. D1 delivery events - * lack `updated_at`. Phase-2 high-risk lifecycle mutations must pass the - * same timestamp used for the D1 mutation; inserts use `created_at`. + * @param sourceMutationAt - Canonical mirror `updatedAt` for D1-authoritative + * message-graph and non-USER-inbound events; inserts use `created_at`. */ export async function getMailboxDeliveryEventMirrorInput(input: { db: D1Database diff --git a/packages/worker/src/email/mailbox-snapshots.ts b/packages/worker/src/email/mailbox-snapshots.ts index 20d2c5a6c3..16769acda2 100644 --- a/packages/worker/src/email/mailbox-snapshots.ts +++ b/packages/worker/src/email/mailbox-snapshots.ts @@ -33,9 +33,8 @@ import { */ /** - * Complete D1 delivery-event mirror projection: base row fields plus - * authoritative promoted columns. `detail_json` still owns inbound/effect - * lease fields that have not been promoted. + * Complete D1 delivery-event mirror projection for D1-authoritative events: + * base row fields plus promoted compatibility columns. * * Authoritative D1 columns (never read from `detail_json`): * - `needs_effect_reconcile` @@ -214,9 +213,7 @@ export function toMailboxAttachmentInput( * Convert a complete D1 delivery-event projection into a Mailbox input. * Promoted columns win; remaining inbound/effect fields come from JSON. * - * `sourceMutationAt` becomes Mailbox `updatedAt`. D1 delivery events lack - * `updated_at`: phase-2 high-risk lifecycle mutations must pass the same - * canonical timestamp used for the D1 mutation; inserts use `created_at`. + * `sourceMutationAt` becomes Mailbox `updatedAt`; inserts use `created_at`. */ export function toMailboxDeliveryEventInput(input: { projection: EmailDeliveryEventMirrorProjection diff --git a/packages/worker/src/email/mailbox-store.ts b/packages/worker/src/email/mailbox-store.ts index 62964e5f46..6d989e07b7 100644 --- a/packages/worker/src/email/mailbox-store.ts +++ b/packages/worker/src/email/mailbox-store.ts @@ -106,10 +106,10 @@ function buildMailboxMessageFilterClauses(input: { /** * SQLite write/query helpers for one Mailbox DO. No alarm / R2 side effects. * - * Additive step 2a USER inbound ledger/effect CAS helpers live in - * `mailbox-inbound-ledger.ts` / `mailbox-inbound-effect-ledger.ts` (wired as - * owner-bound DO RPCs; not live-called — D1 remains authority). Mirror upsert - * paths below stay the compatibility write API. + * USER inbound ledger/effect CAS helpers live in + * `mailbox-inbound-ledger.ts` / `mailbox-inbound-effect-ledger.ts` and are + * authoritative. Generic mirror upserts are fenced from USER inbound providers; + * only the missing-row bootstrap bridge may insert those snapshots. */ export class MailboxStore { private readonly storage: DurableObjectStorage diff --git a/packages/worker/src/email/mailbox-types.ts b/packages/worker/src/email/mailbox-types.ts index 4a8a7f1cf8..8ab2c351f8 100644 --- a/packages/worker/src/email/mailbox-types.ts +++ b/packages/worker/src/email/mailbox-types.ts @@ -478,10 +478,21 @@ export type MailboxUpsertDeliveryEventsResult = { results: Array } +export type MailboxBootstrapDeliveryEventItemResult = { + eventId: string + status: 'inserted' | 'existing' | 'skipped' +} + +export type MailboxBootstrapDeliveryEventsResult = { + inserted: number + existing: number + skipped: number + results: Array +} + /** - * Mirror / read / retention / purge surface. Inbound ledger CAS RPCs - * (additive step 2a) are intersected below — not live-wired; D1 remains - * authority. + * Mirror / read / retention / purge surface. Authoritative USER inbound ledger + * CAS RPCs are intersected below; `system:email` remains D1-only. */ type MailboxCoreRpc = { mirrorMessage: (input: { @@ -500,8 +511,8 @@ type MailboxCoreRpc = { attachments?: Array }) => Promise<{ ok: true; accepted: boolean }> /** - * Compatibility mirror upsert for complete delivery-event snapshots. - * Remains available during step 2a; CAS ledger RPCs are additive. + * Complete delivery-event snapshot upsert. Rejects USER inbound + * lifecycle/dedupe authority snapshots; use `bootstrapDeliveryEvents`. */ upsertDeliveryEvent: (input: { ownerId: string @@ -525,6 +536,14 @@ type MailboxCoreRpc = { ownerId: string events: Array }) => Promise + /** + * Missing-row-only deployment bridge for validated legacy USER inbound + * lifecycle/dedupe snapshots. Existing rows are never updated. + */ + bootstrapDeliveryEvents: (input: { + ownerId: string + events: Array + }) => Promise touchThread: ( input: MailboxTouchThreadInput, ) => Promise diff --git a/packages/worker/src/email/reconcile-inbound-deliveries.ts b/packages/worker/src/email/reconcile-inbound-deliveries.ts index fdbb018a31..557f171e99 100644 --- a/packages/worker/src/email/reconcile-inbound-deliveries.ts +++ b/packages/worker/src/email/reconcile-inbound-deliveries.ts @@ -1,10 +1,14 @@ -import { - pruneExpiredInboundDedupePointers, - reconcileStaleInboundDeliveries, - staleInboundDeliveryAgeMs, -} from './inbound-delivery.ts' +import { staleInboundDeliveryAgeMs } from './inbound-delivery.ts' import { reconcileInboundDeliveryEffectsForUser } from './inbound-effects.ts' -import { systemEmailOwnerId } from './system-email.ts' +import { + pruneUserExpiredInboundDedupePointers, + reconcileUserStaleInboundDeliveries, +} from './inbound-delivery-reconciliation-authority.ts' +import { systemEmailOwnerId } from './email-owner.ts' +import { + pruneSystemExpiredInboundDedupePointers, + reconcileSystemStaleInboundDeliveries, +} from './system-inbound-delivery-authority.ts' import { withAccountWriteLease } from '#worker/account/deletion-state.ts' const reconciliationUserBatchSize = 25 @@ -19,6 +23,8 @@ export async function sweepStaleInboundDeliveries(input: { | 'BUNDLE_ARTIFACTS_KV' | 'APP_BASE_URL' | 'USAGE_EVENTS' + | 'MAILBOX' + | 'USER_METER' > now?: Date }) { @@ -42,64 +48,124 @@ export async function sweepStaleInboundDeliveries(input: { // still performed under one explicit userId. Deferring failed attempts and // verification tombstones moves them behind older untouched users. const rows = await input.env.APP_DB.prepare( - `WITH due_users AS ( - SELECT user_id, created_at AS due_at + `WITH authority(system_owner_id) AS (VALUES (?)), + projected_events AS ( + SELECT email_delivery_events.*, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.reconcileAfter') + ELSE reconcile_after + END AS authority_reconcile_after, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.state') + ELSE state + END AS authority_state, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.cleanupRetryAt') + ELSE cleanup_retry_at + END AS authority_cleanup_retry_at, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.dedupeExpiresAt') + ELSE dedupe_expires_at + END AS authority_dedupe_expires_at, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.fingerprint') + ELSE fingerprint + END AS authority_fingerprint, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.usageEffectRecordedAt') + ELSE usage_effect_recorded_at + END AS authority_usage_recorded_at, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.usageEffectSuppressedAt') + ELSE usage_effect_suppressed_at + END AS authority_usage_suppressed_at, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.usageEffectRetryAt') + ELSE usage_effect_retry_at + END AS authority_usage_retry_at, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.usageEffectLease') + ELSE usage_effect_lease + END AS authority_usage_lease, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.usageEffectLeaseAt') + ELSE usage_effect_lease_at + END AS authority_usage_lease_at, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.subscriptionEffectState') + ELSE subscription_effect_state + END AS authority_subscription_state, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.subscriptionEffectRetryAt') + ELSE subscription_effect_retry_at + END AS authority_subscription_retry_at, + CASE WHEN user_id = system_owner_id + THEN json_extract(detail_json, '$.subscriptionEffectLeaseAt') + ELSE subscription_effect_lease_at + END AS authority_subscription_lease_at FROM email_delivery_events + CROSS JOIN authority + ), + due_users AS ( + SELECT user_id, created_at AS due_at + FROM projected_events WHERE provider = 'cloudflare-email-routing' AND event_type = 'receive_started' AND created_at < ? AND ( - json_extract(detail_json, '$.reconcileAfter') IS NULL - OR json_extract(detail_json, '$.reconcileAfter') <= ? + authority_reconcile_after IS NULL + OR authority_reconcile_after <= ? ) AND ( - json_extract(detail_json, '$.state') != 'orphan-cleaned' - OR json_extract(detail_json, '$.cleanupRetryAt') <= ? + authority_state != 'orphan-cleaned' + OR authority_cleanup_retry_at <= ? ) UNION ALL - SELECT user_id, json_extract(detail_json, '$.dedupeExpiresAt') - FROM email_delivery_events + SELECT user_id, authority_dedupe_expires_at + FROM projected_events WHERE provider = 'cloudflare-email-routing-dedupe' - AND json_extract(detail_json, '$.dedupeExpiresAt') <= ? + AND authority_dedupe_expires_at <= ? UNION ALL SELECT user_id, COALESCE( - json_extract(detail_json, '$.usageEffectRetryAt'), - json_extract(detail_json, '$.subscriptionEffectRetryAt'), + authority_usage_retry_at, + authority_subscription_retry_at, created_at ) - FROM email_delivery_events + FROM projected_events WHERE provider = 'cloudflare-email-routing' AND event_type = 'received' AND needs_effect_reconcile = 1 - AND json_extract(detail_json, '$.fingerprint') IS NOT NULL + AND authority_fingerprint IS NOT NULL AND ( ( - json_extract(detail_json, '$.usageEffectRecordedAt') IS NULL - AND json_extract(detail_json, '$.usageEffectSuppressedAt') IS NULL + authority_usage_recorded_at IS NULL + AND authority_usage_suppressed_at IS NULL AND ( - json_extract(detail_json, '$.usageEffectRetryAt') IS NULL - OR json_extract(detail_json, '$.usageEffectRetryAt') <= ? + authority_usage_retry_at IS NULL + OR authority_usage_retry_at <= ? ) AND ( - json_extract(detail_json, '$.usageEffectLease') IS NULL - OR json_extract(detail_json, '$.usageEffectLeaseAt') < ? + authority_usage_lease IS NULL + OR authority_usage_lease_at IS NULL + OR authority_usage_lease_at < ? ) ) OR ( ( - json_extract(detail_json, '$.subscriptionEffectState') IS NULL - OR json_extract( - detail_json, - '$.subscriptionEffectState' - ) NOT IN ('complete', 'dead-letter') + authority_subscription_state IS NULL + OR authority_subscription_state NOT IN ( + 'complete', + 'dead-letter' + ) ) AND ( - json_extract(detail_json, '$.subscriptionEffectRetryAt') IS NULL - OR json_extract(detail_json, '$.subscriptionEffectRetryAt') <= ? + authority_subscription_retry_at IS NULL + OR authority_subscription_retry_at <= ? ) AND ( - json_extract(detail_json, '$.subscriptionEffectState') != 'processing' - OR json_extract(detail_json, '$.subscriptionEffectLeaseAt') < ? + authority_subscription_state != 'processing' + OR authority_subscription_lease_at IS NULL + OR authority_subscription_lease_at < ? ) ) ) @@ -111,6 +177,7 @@ export async function sweepStaleInboundDeliveries(input: { LIMIT ?`, ) .bind( + systemEmailOwnerId, cutoff, now.toISOString(), now.toISOString(), @@ -126,18 +193,32 @@ export async function sweepStaleInboundDeliveries(input: { if (Date.now() >= deadlineMs) break try { const reconcileUser = async () => { - const result = await reconcileStaleInboundDeliveries({ - db: input.env.APP_DB, - blobs: input.env.EMAIL_BLOBS, - userId, - now, - deadlineMs, - }) - const pruned = await pruneExpiredInboundDedupePointers({ - db: input.env.APP_DB, - userId, - now, - }) + const systemOwner = userId === systemEmailOwnerId + const result = systemOwner + ? await reconcileSystemStaleInboundDeliveries({ + db: input.env.APP_DB, + blobs: input.env.EMAIL_BLOBS, + userId, + now, + deadlineMs, + }) + : await reconcileUserStaleInboundDeliveries({ + env: input.env, + userId, + now, + deadlineMs, + }) + const pruned = systemOwner + ? await pruneSystemExpiredInboundDedupePointers({ + db: input.env.APP_DB, + userId, + now, + }) + : await pruneUserExpiredInboundDedupePointers({ + env: input.env, + userId, + now, + }) const effectResult = await reconcileInboundDeliveryEffectsForUser({ env: input.env, userId, diff --git a/packages/worker/src/email/service.ts b/packages/worker/src/email/service.ts index 19fea67aa7..1b38941c5b 100644 --- a/packages/worker/src/email/service.ts +++ b/packages/worker/src/email/service.ts @@ -4,11 +4,9 @@ import PostalMime from 'postal-mime' import { withAccountWriteLease } from '#worker/account/deletion-state.ts' import { normalizeEmailAddress } from './address.ts' import { resolveInboundEmailAuthVerdict } from './auth-verdict.ts' -import { - getInboundDelivery, - markInboundDeliveryReceived, - type InboundDelivery, -} from './inbound-delivery.ts' +import { getInboundDelivery, type InboundDelivery } from './inbound-delivery.ts' +import { type UserInboundDeliveryAuthority } from './inbound-delivery-authority.ts' +import { markSystemInboundDeliveryReceived } from './system-inbound-delivery-authority.ts' import { mirrorMailboxMessageGraphFromD1, type MailboxLiveMirrorEnv, @@ -353,6 +351,7 @@ export async function storeIdempotentInboundEmail(input: { parsed: ParsedInboundEmail subjectNormalized: string now: string + authority?: Pick }) { const { delivery, parsed } = input if (!delivery.storageLease) { @@ -496,21 +495,30 @@ export async function storeIdempotentInboundEmail(input: { let finalizedDelivery: InboundDelivery try { - finalizedDelivery = await markInboundDeliveryReceived({ - db: input.db, + const finalization = { delivery, usageDurationMs: delivery.usageStartedAt ? Date.now() - Date.parse(delivery.usageStartedAt) : 0, usageMonth: (stored.receivedAt ?? stored.createdAt).slice(0, 7), usageBytes: stored.rawSize ?? 0, - }) + } + finalizedDelivery = input.authority + ? await input.authority.receive(finalization) + : await markSystemInboundDeliveryReceived({ + db: input.db, + ...finalization, + }) } catch (error) { - const committed = await getInboundDelivery({ - db: input.db, - userId: delivery.userId, - deliveryId: delivery.deliveryId, - }).catch(() => null) + const committed = await ( + input.authority + ? input.authority.get(delivery.deliveryId) + : getInboundDelivery({ + db: input.db, + userId: delivery.userId, + deliveryId: delivery.deliveryId, + }) + ).catch(() => null) if (committed?.state !== 'received') { throw new RetryableInboundStorageError( 'Failed to finalize the inbound delivery ledger; the stable delivery will be retried.', @@ -689,8 +697,9 @@ export async function recordBoundedEmailRejectionEvent(input: { const row = await input.db .prepare( `INSERT INTO email_delivery_events ( - id, user_id, inbox_id, event_type, provider, detail_json, created_at - ) VALUES (?, ?, ?, 'rejected', 'cloudflare-email-routing', ?, ?) + id, user_id, inbox_id, event_type, provider, detail_json, + needs_effect_reconcile, created_at + ) VALUES (?, ?, ?, 'rejected', 'cloudflare-email-routing', ?, 0, ?) ON CONFLICT(id) DO UPDATE SET detail_json = json_set( email_delivery_events.detail_json, '$.count', COALESCE(json_extract(email_delivery_events.detail_json, '$.count'), 0) + 1, diff --git a/packages/worker/src/email/system-inbound-delivery-authority.ts b/packages/worker/src/email/system-inbound-delivery-authority.ts new file mode 100644 index 0000000000..f833acd3cb --- /dev/null +++ b/packages/worker/src/email/system-inbound-delivery-authority.ts @@ -0,0 +1,72 @@ +import { + claimInboundDeliveryStorage, + claimInboundDeliveryWindow, + markInboundDeliveryReceived, + markInboundDeliveryRejected, + pruneExpiredInboundDedupePointers, + reconcileStaleInboundDeliveries, + releaseInboundDeliveryStorage, + type InboundDelivery, +} from './inbound-delivery.ts' +import { systemEmailOwnerId } from './email-owner.ts' + +function assertSystemInboundOwner(userId: string) { + if (userId !== systemEmailOwnerId) { + throw new Error( + 'Legacy D1 inbound delivery mutations are restricted to system:email.', + ) + } +} + +function assertSystemInboundDelivery(delivery: InboundDelivery) { + assertSystemInboundOwner(delivery.userId) +} + +export async function claimSystemInboundDeliveryWindow( + input: Parameters[0], +) { + assertSystemInboundDelivery(input.delivery) + return await claimInboundDeliveryWindow(input) +} + +export async function claimSystemInboundDeliveryStorage( + input: Parameters[0], +) { + assertSystemInboundDelivery(input.delivery) + return await claimInboundDeliveryStorage(input) +} + +export async function releaseSystemInboundDeliveryStorage( + input: Parameters[0], +) { + assertSystemInboundDelivery(input.delivery) + return await releaseInboundDeliveryStorage(input) +} + +export async function markSystemInboundDeliveryRejected( + input: Parameters[0], +) { + assertSystemInboundDelivery(input.delivery) + return await markInboundDeliveryRejected(input) +} + +export async function markSystemInboundDeliveryReceived( + input: Parameters[0], +) { + assertSystemInboundDelivery(input.delivery) + return await markInboundDeliveryReceived(input) +} + +export async function pruneSystemExpiredInboundDedupePointers( + input: Parameters[0], +) { + assertSystemInboundOwner(input.userId) + return await pruneExpiredInboundDedupePointers(input) +} + +export async function reconcileSystemStaleInboundDeliveries( + input: Parameters[0], +) { + assertSystemInboundOwner(input.userId) + return await reconcileStaleInboundDeliveries(input) +} diff --git a/packages/worker/src/email/test-schema.ts b/packages/worker/src/email/test-schema.ts index 6d35927cf4..787df7fc15 100644 --- a/packages/worker/src/email/test-schema.ts +++ b/packages/worker/src/email/test-schema.ts @@ -8,6 +8,7 @@ export async function ensureEmailTestSchema(db: D1Database) { `DROP TABLE IF EXISTS system_email_daily_counters;`, `DROP TABLE IF EXISTS email_sender_policies;`, `DROP TABLE IF EXISTS email_sender_rules;`, + `DROP TABLE IF EXISTS email_inbound_usage_effects;`, `DROP TABLE IF EXISTS email_delivery_events;`, `DROP TABLE IF EXISTS email_attachments;`, `DROP TABLE IF EXISTS email_outbound_provider_index;`, @@ -154,13 +155,62 @@ ON email_sender_rules(user_id, kind, value);`, usage_month TEXT, usage_bytes INTEGER, usage_duration_ms INTEGER, - created_at TEXT NOT NULL + state TEXT CHECK ( + state IS NULL OR state IN ( + 'pending', 'storing', 'cleaning', 'received', 'rejected', 'orphan-cleaned' + ) + ), + fingerprint TEXT, + storage_lease TEXT, + storage_lease_at TEXT, + cleanup_lease TEXT, + cleanup_lease_at TEXT, + cleanup_retry_at TEXT, + expected_attachment_count INTEGER, + finalization_token TEXT, + reconcile_after TEXT, + dedupe_expires_at TEXT, + usage_effect_suppressed_at TEXT, + usage_started_at TEXT, + usage_effect_retry_at TEXT, + usage_effect_lease TEXT, + usage_effect_lease_at TEXT, + subscription_effect_state TEXT CHECK ( + subscription_effect_state IS NULL OR subscription_effect_state IN ( + 'pending', 'processing', 'complete', 'dead-letter' + ) + ), + subscription_effect_lease TEXT, + subscription_effect_lease_at TEXT, + subscription_effect_retry_at TEXT, + subscription_effect_attempt_count INTEGER, + subscription_effect_dead_letter_at TEXT, + subscription_effect_last_error TEXT, + created_at TEXT NOT NULL, + updated_at TEXT );`, `CREATE INDEX IF NOT EXISTS idx_email_delivery_events_pending_effects ON email_delivery_events(user_id, created_at) WHERE provider = 'cloudflare-email-routing' AND event_type = 'received' AND needs_effect_reconcile = 1;`, + `CREATE INDEX IF NOT EXISTS idx_email_delivery_events_user_state_created +ON email_delivery_events(user_id, state, created_at, id) +WHERE provider = 'cloudflare-email-routing' AND state IS NOT NULL;`, + `CREATE INDEX IF NOT EXISTS idx_email_delivery_events_user_dedupe_expires +ON email_delivery_events(user_id, dedupe_expires_at, id) +WHERE provider = 'cloudflare-email-routing-dedupe' + AND dedupe_expires_at IS NOT NULL;`, + `CREATE TABLE IF NOT EXISTS email_inbound_usage_effects ( + user_id TEXT NOT NULL, + delivery_id TEXT NOT NULL, + finalization_token TEXT NOT NULL, + created_at TEXT NOT NULL, + PRIMARY KEY (user_id, delivery_id, finalization_token), + FOREIGN KEY (delivery_id) REFERENCES email_delivery_events(id) ON DELETE CASCADE +);`, + `CREATE INDEX IF NOT EXISTS idx_email_inbound_usage_effects_delivery +ON email_inbound_usage_effects(delivery_id);`, `CREATE INDEX IF NOT EXISTS idx_email_delivery_events_recorded_usage_month ON email_delivery_events(usage_month, user_id) WHERE provider = 'cloudflare-email-routing' diff --git a/tools/migration-ledger.json b/tools/migration-ledger.json index 9cf767641c..487c16e6cd 100644 --- a/tools/migration-ledger.json +++ b/tools/migration-ledger.json @@ -527,6 +527,10 @@ { "filename": "0128-email-outbound-provider-index.sql", "sha256": "b51c4f9cc4416cd81c1c1648311e0267e28a208d386d4729295b1d3dad54ed5a" + }, + { + "filename": "0129-email-inbound-mailbox-authority-mirror.sql", + "sha256": "2b5d01a5b5f2adf7ab145a17d7a2cc58b06398cc08850838ffba537f23a31b59" } ] }