diff --git a/docs/contributing/architecture/authentication.md b/docs/contributing/architecture/authentication.md index ce24d6b6c4..630feb2a86 100644 --- a/docs/contributing/architecture/authentication.md +++ b/docs/contributing/architecture/authentication.md @@ -195,6 +195,15 @@ assistant features: `packages/worker/src/email/inbound.ts` rejects routed mail for unverified accounts right after username routing (`setReject` plus a bounded `rejected` email delivery event); nothing is stored. + +Platform suspension (`users.suspended_at`, set by admins from `/admin/users`) +follows the same chokepoint pattern and also fails closed: browser session +resolution treats a suspended session as signed out (`readAuthenticatedAppUser` +/ `loadSessionInfo`), `handleMcpRequest` rejects with a `403 account_suspended` +JSON response after the verification gate, and both email directions reject +(inbound with a bounded `account-suspension` rejection event, outbound with an +error). See the "Abuse controls" section of [`security.md`](../security.md). + - **Email capabilities**: every capability in the MCP `email` domain calls `requireVerifiedEmailAccountUser` (`packages/worker/src/mcp/capabilities/email/require-verified-user.ts`) as diff --git a/docs/contributing/architecture/entitlements.md b/docs/contributing/architecture/entitlements.md index 04c77be3f8..099dfbcf8b 100644 --- a/docs/contributing/architecture/entitlements.md +++ b/docs/contributing/architecture/entitlements.md @@ -82,6 +82,30 @@ those caps. All other resources use the ordinary `planLimits.max` numbers. | `repo_sessions` | 2,000 | | `secrets` | 10,000 | | `storage_bytes` | 100 GiB | +| `execute_calls_per_day` | 500,000 | +| `outbound_fetches_per_day` | 2,000,000 | + +## Compute rate limits + +`execute_calls_per_day` and `outbound_fetches_per_day` are daily-counter +resources (same mechanism as `email_sends_per_day`, consumed atomically with +`consumeDailyEntitlement`). They close the metering → enforcement loop for the +two compute surfaces `usage-metering.md` already observes: + +- **Execute calls** are consumed at the top of the MCP `execute` tool handler + (`packages/worker/src/mcp/tools/execute.ts`) before any bundling or sandbox + work, so over-limit calls cost nothing. The `EntitlementLimitError` propagates + as a structured MCP error. +- **Outbound fetches** are consumed at the top of `executeGatewayFetch` + (`packages/worker/src/mcp/fetch-gateway.ts`), which every sandbox fetch passes + through, before secret expansion. `FetchGatewayProps.email` carries the acting + user's account email for plan lookup; when a caller cannot carry one (OpenAPI + provider requests, package runtime), the gateway reverse-resolves the account + via `findUserAccountByStableUserId` so the caller's real plan binds. Genuinely + accountless synthetic contexts resolve to `max`, whose limit is still finite. + +Both consume only when the context has a `userId`, matching the usage-metering +rule that events without an owning user are skipped. ## Schema history diff --git a/docs/contributing/security.md b/docs/contributing/security.md index 22dd119692..53c648deff 100644 --- a/docs/contributing/security.md +++ b/docs/contributing/security.md @@ -154,6 +154,35 @@ guarded by a bearer secret comparison. `Cf-Access-Jwt-Assertion` verification; production restore is a graduated prepare → typed confirmation → Workflow path, never a single click. +## Abuse controls (suspension, email pause, compute quotas) + +One bad actor can poison shared platform identity — every user sends mail from +one platform domain through one Cloudflare Email Sending account, and every +sandbox fetch leaves through the same Worker egress. Three controls bound that +blast radius: + +- **Platform suspension (`users.suspended_at`).** An admin-set kill switch, + distinct from a community ban (which only blocks community-surface actions). + Enforced fail-closed at every chokepoint: browser session resolution + (`readAuthenticatedAppUser` / `loadSessionInfo` treat a suspended session as + signed out), MCP bearer auth (`handleMcpRequest` returns a 403 + `account_suspended` response, mirroring the email-verification gate), and both + email directions (inbound storage rejects with a bounded `account-suspension` + rejection event; outbound send throws). Set and cleared through the audited + `suspend_user` / `unsuspend_user` actions on `POST /admin/users.json`. +- **Automatic outbound-email pause (`users.email_outbound_paused_at`).** The + delivery queue evaluates provider delivery events + (`packages/worker/src/email/outbound-abuse.ts`): one spam complaint, or five + or more bounced sends within a UTC day, pauses that account's outbound email + and notifies admin accounts through the transactional sender. The pause write + is idempotent (only transitions NULL), the send path rejects while paused, and + the audited `resume_email_outbound` admin action clears it after review. The + `/admin/insights` "Email delivery health" chart shows platform-wide outcome + trends so reputation trouble is visible before providers act on it. +- **Compute quotas.** `execute_calls_per_day` and `outbound_fetches_per_day` + entitlements bound sandbox compute and egress volume per user per day (see + [`architecture/entitlements.md`](./architecture/entitlements.md)). + ## Accepted residual risks and out-of-scope items These were reviewed and intentionally left as-is for this project. Document any diff --git a/docs/use/email-primitives.md b/docs/use/email-primitives.md index 6e45b29701..2844da98ae 100644 --- a/docs/use/email-primitives.md +++ b/docs/use/email-primitives.md @@ -116,6 +116,12 @@ Inbound storage is quota-gated per user: events independently populate `delivery_status` with `delivered`, `deferred`, `bounced`, `failed`, `rejected`, or `complained`; use `email_delivery_event_list` for the event history and SMTP details. +- Delivery events feed an automatic abuse pause: one spam complaint, or five or + more bounced sends within a UTC day, pauses outbound sending for the account + (receiving is unaffected). Every user sends from the same platform domain, so + one account's complaints damage delivery for everyone. A paused send fails + with a clear error; contact the operator to have the pause reviewed and + cleared. - System inbox mail is not gated by a user plan or account-verification state. It has fixed platform caps and retention: messages are pruned after 90 days and the stored system inbox is capped so arbitrary sender traffic cannot grow diff --git a/packages/worker/client/routes/admin-insights.tsx b/packages/worker/client/routes/admin-insights.tsx index 5d2d6cb39b..96a3eae680 100644 --- a/packages/worker/client/routes/admin-insights.tsx +++ b/packages/worker/client/routes/admin-insights.tsx @@ -483,6 +483,64 @@ function renderDashboard(data: AdminInsightsLoaderData) { /> + + } + > + day.delivered), + }, + { + label: 'Deferred', + color: chartColor.cyan, + values: data.emailDeliveryByDay.map((day) => day.deferred), + }, + { + label: 'Bounced', + color: chartColor.amber, + values: data.emailDeliveryByDay.map((day) => day.bounced), + }, + { + label: 'Failed', + color: chartColor.fuchsia, + values: data.emailDeliveryByDay.map((day) => day.failed), + }, + { + label: 'Rejected', + color: chartColor.violet, + values: data.emailDeliveryByDay.map((day) => day.rejected), + }, + { + label: 'Complained', + color: chartColor.rose, + values: data.emailDeliveryByDay.map((day) => day.complained), + }, + ]} + xLabels={data.emailDeliveryByDay.map((day) => + formatDayLabel(day.day), + )} + /> + + (response) + if (!response.ok || !payload?.ok) { + throw new Error(payload?.error || 'Unable to update the account.') + } + applyMutationPayload(payload, href) + lastLoadedDataKey = getDataKey(href) + message = + action === 'suspend_user' + ? 'Account suspended.' + : action === 'unsuspend_user' + ? 'Account suspension cleared.' + : 'Outbound email resumed.' + status = 'ready' + actionState = 'idle' + handle.update() + } catch (error) { + actionState = 'idle' + message = + error instanceof Error ? error.message : 'Unable to update the account.' + handle.update() + } + } + const primaryButtonCss = getPrimaryButtonCss() const secondaryButtonCss = getSecondaryButtonCss() @@ -827,6 +885,18 @@ export function AdminUsersRoute(handle: Handle) { label: 'Plan', value: selectedUser.plan ?? 'free', }, + { + label: 'Suspended', + value: selectedUser.suspended_at + ? formatTimestamp(selectedUser.suspended_at) + : 'No', + }, + { + label: 'Outbound email', + value: selectedUser.email_outbound_paused_at + ? `Paused ${formatTimestamp(selectedUser.email_outbound_paused_at)}` + : 'Active', + }, { label: 'Created', value: formatTimestamp(selectedUser.created_at), @@ -944,6 +1014,72 @@ export function AdminUsersRoute(handle: Handle) { + +
+ {selectedUser.suspended_at ? ( + + ) : ( + + )} + {selectedUser.email_outbound_paused_at ? ( + + ) : null} +
+
{ + const stableUserId = normalizeStableUserId(input.stableUserId) + if (!stableUserId) return null + const row = await input.db + .prepare( + `SELECT suspended_at, email_outbound_paused_at FROM users + WHERE stable_user_id = ?`, + ) + .bind(stableUserId) + .first<{ + suspended_at: string | null + email_outbound_paused_at: string | null + }>() + if (!row) return null + return { + suspendedAt: row.suspended_at, + emailOutboundPausedAt: row.email_outbound_paused_at, + } +} + +/** + * Whether the account is suspended, resolved the same way as + * `isAccountEmailVerified`: when both the email and the stable id are + * present (typical MCP grant props) the pair must match one row, so a + * stale grant email owned by another account can never be consulted. + */ +export async function isAccountSuspended(input: { + db: D1Database + email?: string | null + stableUserId?: string | null +}): Promise { + const normalizedEmail = + typeof input.email === 'string' ? normalizeEmail(input.email) : '' + const stableUserId = input.stableUserId?.trim() ?? '' + + if (normalizedEmail && stableUserId) { + const row = await input.db + .prepare( + `SELECT suspended_at FROM users + WHERE email = ? AND stable_user_id = ?`, + ) + .bind(normalizedEmail, stableUserId) + .first<{ suspended_at: string | null }>() + return Boolean(row?.suspended_at) + } + + if (normalizedEmail) { + const row = await input.db + .prepare(`SELECT suspended_at FROM users WHERE email = ?`) + .bind(normalizedEmail) + .first<{ suspended_at: string | null }>() + return Boolean(row?.suspended_at) + } + + if (!stableUserId) return false + const row = await input.db + .prepare(`SELECT suspended_at FROM users WHERE stable_user_id = ?`) + .bind(stableUserId) + .first<{ suspended_at: string | null }>() + return Boolean(row?.suspended_at) +} diff --git a/packages/worker/src/app/admin-insights-data.node.test.ts b/packages/worker/src/app/admin-insights-data.node.test.ts index 1320ec227b..64b9560851 100644 --- a/packages/worker/src/app/admin-insights-data.node.test.ts +++ b/packages/worker/src/app/admin-insights-data.node.test.ts @@ -2,6 +2,7 @@ import { expect, test } from 'vitest' import { buildAuthDays, buildEmailDays, + buildEmailDeliveryDays, buildHeatmapCells, buildSignupWeeks, buildUsageMonths, @@ -79,6 +80,37 @@ test('admin insights date helpers bucket, zero-fill, and fold correctly', () => { day: '2026-07-08', sends: 0, receives: 0 }, ]) + const emailDeliveryDays = buildEmailDeliveryDays( + [ + { day: '2026-07-07', event_type: 'delivered', n: 4 }, + { day: '2026-07-07', event_type: 'bounced', n: 2 }, + { day: '2026-07-08', event_type: 'complained', n: 1 }, + { day: '2026-07-08', event_type: 'not-an-outcome', n: 9 }, + ], + now, + 2, + ) + expect(emailDeliveryDays).toEqual([ + { + day: '2026-07-07', + delivered: 4, + deferred: 0, + bounced: 2, + failed: 0, + rejected: 0, + complained: 0, + }, + { + day: '2026-07-08', + delivered: 0, + deferred: 0, + bounced: 0, + failed: 0, + rejected: 0, + complained: 1, + }, + ]) + const authDays = buildAuthDays( [ { day: '2026-07-08', result: 'success', n: 9 }, @@ -192,6 +224,14 @@ function createInsightsTestDb() { ] as Array, } } + if (normalizedQuery.includes('from email_delivery_events')) { + return { + results: [ + { day: '2026-07-08', event_type: 'delivered', n: 5 }, + { day: '2026-07-08', event_type: 'bounced', n: 1 }, + ] as Array, + } + } if (normalizedQuery.includes("coalesce(plan, 'none')")) { return { results: [ @@ -281,6 +321,16 @@ test('loadAdminInsightsData assembles the dashboard payload', async () => { sends: 3, receives: 0, }) + expect(data.emailDeliveryByDay).toHaveLength(28) + expect(data.emailDeliveryByDay.at(-1)).toEqual({ + day: '2026-07-08', + delivered: 5, + deferred: 0, + bounced: 1, + failed: 0, + rejected: 0, + complained: 0, + }) expect(data.authByDay.at(-1)?.success).toBe(4) expect(data.authByCategory).toEqual([{ category: 'auth', count: 4 }]) expect(data.authHeatmap).toEqual([{ weekday: 3, hour: 9, count: 4 }]) diff --git a/packages/worker/src/app/admin-insights-data.ts b/packages/worker/src/app/admin-insights-data.ts index 4d679f101d..bd4c66bdf6 100644 --- a/packages/worker/src/app/admin-insights-data.ts +++ b/packages/worker/src/app/admin-insights-data.ts @@ -6,6 +6,7 @@ import { type AdminInsightsAuthCategory, type AdminInsightsAuthDay, type AdminInsightsEmailDay, + type AdminInsightsEmailDeliveryDay, type AdminInsightsHeatmapCell, type AdminInsightsJobHealth, type AdminInsightsLoaderData, @@ -39,6 +40,7 @@ type UsageMonthRow = { errors: number } type EmailDayRow = { day: string; resource: string; n: number } +type EmailDeliveryDayRow = { day: string; event_type: string; n: number } type PlanRow = { plan: string; n: number } type AuthDayRow = { day: string; result: string; n: number } type AuthCategoryRow = { category: string; n: number } @@ -62,7 +64,7 @@ export async function loadAdminInsightsData( : null if (!cache) return await queryAdminInsights(env.APP_DB, now) return await cachified({ - key: 'admin-insights:v1', + key: 'admin-insights:v2', cache, ttl: insightsCacheTtlMs, getFreshValue: () => queryAdminInsights(env.APP_DB, now), @@ -94,6 +96,7 @@ async function queryAdminInsights( usersBeforeWindow, usageRows, emailRows, + emailDeliveryRows, planRows, authDayRows, authCategoryRows, @@ -141,6 +144,22 @@ async function queryAdminInsights( ) .bind(dayCutoff) .all(), + db + .prepare( + // Provider ('cloudflare-email') events only: outbound + // delivery outcomes, excluding inbound routing rejections. + // Deliberately unscoped like every other query in this + // admin-only platform-wide read model: the aggregation + // returns per-day outcome counts with no user identifiers + // or message content, monitoring the shared sending + // domain's reputation. + `SELECT substr(created_at, 1, 10) AS day, event_type, COUNT(*) AS n + FROM email_delivery_events + WHERE provider = 'cloudflare-email' AND created_at >= ? + GROUP BY day, event_type`, + ) + .bind(dayCutoff) + .all(), db .prepare( `SELECT COALESCE(plan, 'none') AS plan, COUNT(*) AS n @@ -218,6 +237,11 @@ async function queryAdminInsights( now, adminInsightsActivityDays, ), + emailDeliveryByDay: buildEmailDeliveryDays( + emailDeliveryRows.results ?? [], + now, + adminInsightsActivityDays, + ), plans: (planRows.results ?? []).map( (row): AdminInsightsPlanSlice => ({ plan: row.plan, @@ -397,6 +421,47 @@ export function buildEmailDays( return Array.from(byDay.values()) } +export function buildEmailDeliveryDays( + rows: Array, + now: Date, + days: number, +): Array { + const byDay = new Map() + for (const day of listUtcDayKeys(now, days)) { + byDay.set(day, { + day, + delivered: 0, + deferred: 0, + bounced: 0, + failed: 0, + rejected: 0, + complained: 0, + }) + } + for (const row of rows) { + const entry = byDay.get(row.day) + if (!entry) continue + if (!isEmailDeliveryOutcome(row.event_type)) continue + entry[row.event_type] += Number(row.n) + } + return Array.from(byDay.values()) +} + +const emailDeliveryOutcomes = [ + 'delivered', + 'deferred', + 'bounced', + 'failed', + 'rejected', + 'complained', +] as const + +function isEmailDeliveryOutcome( + value: string, +): value is (typeof emailDeliveryOutcomes)[number] { + return (emailDeliveryOutcomes as ReadonlyArray).includes(value) +} + export function buildAuthDays( rows: Array, now: Date, diff --git a/packages/worker/src/app/admin-user-usage-data.ts b/packages/worker/src/app/admin-user-usage-data.ts index a0230fe595..48710928fc 100644 --- a/packages/worker/src/app/admin-user-usage-data.ts +++ b/packages/worker/src/app/admin-user-usage-data.ts @@ -41,6 +41,8 @@ const adminUserUsageEntitlementResources = [ 'stored_email_messages', 'secrets', 'concurrent_workflows', + 'execute_calls_per_day', + 'outbound_fetches_per_day', ] as const satisfies ReadonlyArray const warningThreshold = 0.8 diff --git a/packages/worker/src/app/admin-users-data.ts b/packages/worker/src/app/admin-users-data.ts index 45c072015e..f5bba22f1c 100644 --- a/packages/worker/src/app/admin-users-data.ts +++ b/packages/worker/src/app/admin-users-data.ts @@ -20,6 +20,8 @@ export const adminUserListItemFieldNames = [ 'email_verified', 'email_verified_at', 'plan', + 'suspended_at', + 'email_outbound_paused_at', 'created_at', 'updated_at', 'roles', @@ -35,6 +37,8 @@ export type AdminUserListItem = Record & { email_verified: boolean email_verified_at: string | null plan: PlanName + suspended_at: string | null + email_outbound_paused_at: string | null created_at: string updated_at: string roles: Array @@ -152,7 +156,8 @@ export async function loadAdminUsersData( .bind(...params) .first<{ total: number }>(), env.APP_DB.prepare( - `SELECT id, username, email, email_verified_at, plan, created_at, updated_at + `SELECT id, username, email, email_verified_at, plan, suspended_at, + email_outbound_paused_at, created_at, updated_at FROM users ${whereClause} ORDER BY id ASC @@ -191,7 +196,8 @@ export async function loadAdminUserByIdOrEmail( const userRow = input.id ? await db .prepare( - `SELECT id, username, email, email_verified_at, plan, created_at, updated_at + `SELECT id, username, email, email_verified_at, plan, suspended_at, + email_outbound_paused_at, created_at, updated_at FROM users WHERE id = ?`, ) @@ -200,7 +206,8 @@ export async function loadAdminUserByIdOrEmail( : email ? await db .prepare( - `SELECT id, username, email, email_verified_at, plan, created_at, updated_at + `SELECT id, username, email, email_verified_at, plan, suspended_at, + email_outbound_paused_at, created_at, updated_at FROM users WHERE email = ? COLLATE NOCASE`, ) @@ -233,6 +240,50 @@ export async function updateAdminUserPlan( return loadAdminUserByIdOrEmail(db, { id: existing.id }) } +/** + * Set or clear the platform suspension on one user account. Suspension is + * fail-closed at the browser-session, MCP, and email chokepoints; clearing + * it restores normal access on the next request. Returns the updated + * account metadata record, or null when no user matches `id`. + */ +export async function updateAdminUserSuspension( + db: D1Database, + input: { id: number; suspended: boolean }, +): Promise { + const existing = await loadAdminUserByIdOrEmail(db, { id: input.id }) + if (!existing) return null + + const now = utcSqliteTimestamp() + await db + .prepare(`UPDATE users SET suspended_at = ?, updated_at = ? WHERE id = ?`) + .bind(input.suspended ? now : null, now, existing.id) + .run() + + return loadAdminUserByIdOrEmail(db, { id: existing.id }) +} + +/** + * Clear an automatic outbound-email pause (set by the delivery-event abuse + * monitor) after operator review. Returns the updated account metadata + * record, or null when no user matches `id`. + */ +export async function clearAdminUserEmailOutboundPause( + db: D1Database, + input: { id: number }, +): Promise { + const existing = await loadAdminUserByIdOrEmail(db, { id: input.id }) + if (!existing) return null + + await db + .prepare( + `UPDATE users SET email_outbound_paused_at = NULL, updated_at = ? WHERE id = ?`, + ) + .bind(utcSqliteTimestamp(), existing.id) + .run() + + return loadAdminUserByIdOrEmail(db, { id: existing.id }) +} + export async function loadRolesByUserIds( db: D1Database, userIds: Array, @@ -274,6 +325,8 @@ type AdminUserRow = { email: string email_verified_at: string | null plan: string + suspended_at: string | null + email_outbound_paused_at: string | null created_at: string updated_at: string } @@ -289,6 +342,8 @@ function toAdminUserListItem( email_verified: Boolean(row.email_verified_at), email_verified_at: row.email_verified_at, plan: parseStoredPlanName(row.plan), + suspended_at: row.suspended_at, + email_outbound_paused_at: row.email_outbound_paused_at, created_at: row.created_at, updated_at: row.updated_at, roles, diff --git a/packages/worker/src/app/authenticated-user.ts b/packages/worker/src/app/authenticated-user.ts index 6b48ce355d..e523b559fa 100644 --- a/packages/worker/src/app/authenticated-user.ts +++ b/packages/worker/src/app/authenticated-user.ts @@ -25,6 +25,9 @@ async function readAuthenticatedAppUserInternal( const resolved = await loadResolvedRequestAuth(request, env) if (!resolved.user || !resolved.sessionUserId) return null if (resolved.user.accountDeleting && !allowDeleting) return null + // Suspension is a platform kill switch: no surface honors a suspended + // session, so every consumer of this helper fails closed. + if (resolved.user.accountSuspended) return null return { sessionUserId: resolved.sessionUserId, diff --git a/packages/worker/src/app/handlers/admin-users.node.test.ts b/packages/worker/src/app/handlers/admin-users.node.test.ts index 620a17a775..324713b9cf 100644 --- a/packages/worker/src/app/handlers/admin-users.node.test.ts +++ b/packages/worker/src/app/handlers/admin-users.node.test.ts @@ -36,6 +36,8 @@ type UserRow = { email: string email_verified_at?: string | null plan?: string | null + suspended_at?: string | null + email_outbound_paused_at?: string | null created_at: string updated_at: string } @@ -76,6 +78,8 @@ function createAdminTestEnv(input: { // Normal fixtures default to free; unknown/null stay // explicit so the dedicated stored-plan coercion test can warn. plan: user.plan === undefined ? 'free' : user.plan, + suspended_at: user.suspended_at ?? null, + email_outbound_paused_at: user.email_outbound_paused_at ?? null, }, ]), ) @@ -199,7 +203,7 @@ function createAdminTestEnv(input: { } if ( normalizedQuery.includes( - 'select id, username, email, email_verified_at, plan, created_at, updated_at from users where id =', + 'select id, username, email, email_verified_at, plan, suspended_at, email_outbound_paused_at, created_at, updated_at from users where id =', ) ) { const user = users.get(Number(params[0])) @@ -266,6 +270,29 @@ function createAdminTestEnv(input: { user.updated_at = String(params[1]) return { meta: { changes: 1 } } } + if ( + normalizedQuery.includes( + 'update users set suspended_at = ?, updated_at = ? where id =', + ) + ) { + const user = users.get(Number(params[2])) + if (!user) return { meta: { changes: 0 } } + user.suspended_at = + params[0] === null ? null : String(params[0]) + user.updated_at = String(params[1]) + return { meta: { changes: 1 } } + } + if ( + normalizedQuery.includes( + 'update users set email_outbound_paused_at = null, updated_at = ? where id =', + ) + ) { + const user = users.get(Number(params[1])) + if (!user) return { meta: { changes: 0 } } + user.email_outbound_paused_at = null + user.updated_at = String(params[0]) + return { meta: { changes: 1 } } + } return { meta: { changes: 0 } } }, } @@ -732,6 +759,122 @@ test('update plan action sets, maps null to free, validates, and scopes plan cha expect(missingUserResponse.status).toBe(404) }) +test('suspend, unsuspend, and resume email actions update flags and log audit events', async () => { + logAuditEventSpy.mockClear() + mockModule.readAuthenticatedAppUser.mockResolvedValue( + createAdminActor(['admin']), + ) + const env = createAdminTestEnv({ + users: [ + { + id: 2, + username: 'member', + email: 'member@example.com', + email_outbound_paused_at: '2026-07-20T00:00:00.000Z', + created_at: '2026-01-03 00:00:00', + updated_at: '2026-01-04 00:00:00', + }, + ], + userRoles: [{ user_id: 2, role_name: 'user' }], + }) + const handler = createAdminUsersApiHandler(env as unknown as Env) + const postAction = (body: Record) => + handler.handler({ + request: new Request('https://example.com/admin/users.json', { + method: 'POST', + headers: { + Accept: 'application/json', + 'Content-Type': 'application/json', + }, + body: JSON.stringify(body), + }), + params: {}, + url: new URL('https://example.com/admin/users.json'), + } as never) + + const suspendResponse = await postAction({ + action: 'suspend_user', + userId: 2, + }) + expect(suspendResponse.status).toBe(200) + const suspended = await suspendResponse.json() + expect(suspended.users[0].suspended_at).toBeTruthy() + expect(logAuditEventSpy).toHaveBeenCalledWith( + expect.objectContaining({ + category: 'admin', + action: 'suspend_user', + result: 'success', + reason: 'target_user_id=2', + }), + ) + + const unsuspendResponse = await postAction({ + action: 'unsuspend_user', + userId: 2, + }) + expect(unsuspendResponse.status).toBe(200) + expect((await unsuspendResponse.json()).users[0].suspended_at).toBeNull() + expect(logAuditEventSpy).toHaveBeenCalledWith( + expect.objectContaining({ + category: 'admin', + action: 'unsuspend_user', + result: 'success', + reason: 'target_user_id=2', + }), + ) + + const resumeResponse = await postAction({ + action: 'resume_email_outbound', + userId: 2, + }) + expect(resumeResponse.status).toBe(200) + expect( + (await resumeResponse.json()).users[0].email_outbound_paused_at, + ).toBeNull() + expect(logAuditEventSpy).toHaveBeenCalledWith( + expect.objectContaining({ + category: 'admin', + action: 'resume_email_outbound', + result: 'success', + reason: 'target_user_id=2', + }), + ) + + // Admins cannot suspend their own account (actor id is 1). + const selfSuspendEnv = createAdminTestEnv({ + users: [ + { + id: 1, + username: 'admin-user', + email: 'admin@example.com', + created_at: '2026-01-01 00:00:00', + updated_at: '2026-01-01 00:00:00', + }, + ], + userRoles: [{ user_id: 1, role_name: 'admin' }], + }) + const selfHandler = createAdminUsersApiHandler( + selfSuspendEnv as unknown as Env, + ) + const selfResponse = await selfHandler.handler({ + request: new Request('https://example.com/admin/users.json', { + method: 'POST', + headers: { + Accept: 'application/json', + 'Content-Type': 'application/json', + }, + body: JSON.stringify({ action: 'suspend_user', userId: 1 }), + }), + params: {}, + url: new URL('https://example.com/admin/users.json'), + } as never) + expect(selfResponse.status).toBe(400) + + expect( + (await postAction({ action: 'suspend_user', userId: 42 })).status, + ).toBe(404) +}) + test('admin users API returns 403 without read:user:any permission', async () => { mockModule.readAuthenticatedAppUser.mockResolvedValue( createAdminActor(['user']), diff --git a/packages/worker/src/app/handlers/admin-users.ts b/packages/worker/src/app/handlers/admin-users.ts index c9b6d31ff4..1a62982b2c 100644 --- a/packages/worker/src/app/handlers/admin-users.ts +++ b/packages/worker/src/app/handlers/admin-users.ts @@ -4,11 +4,13 @@ import { type Action } from 'remix/router' import { getRequestIp, logAuditEvent } from '#app/audit-log.ts' import { loadAdminUserUsageData } from '#app/admin-user-usage-data.ts' import { + clearAdminUserEmailOutboundPause, loadAdminUserByIdOrEmail, loadAdminUsersData, loadRolesByUserIds, adminUserListItemFieldNames, updateAdminUserPlan, + updateAdminUserSuspension, type AdminUserListItem, } from '#app/admin-users-data.ts' import { @@ -136,6 +138,25 @@ export function createAdminUsersApiHandler(env: Env) { body, }) } + if (action === 'suspend_user' || action === 'unsuspend_user') { + return handleSuspensionAction({ + env, + request, + url, + actor, + body, + suspended: action === 'suspend_user', + }) + } + if (action === 'resume_email_outbound') { + return handleResumeEmailOutboundAction({ + env, + request, + url, + actor, + body, + }) + } return jsonResponse({ ok: false, error: 'Invalid action.' }, 400) } catch (error) { @@ -377,6 +398,95 @@ async function handleUpdatePlanAction(input: { return buildMutationResponse(input.env, input.request.url, targetUserId) } +/** + * Set or clear the platform suspension. Unlike a community ban (community + * surfaces only), suspension is fail-closed at the browser-session, MCP, + * and email chokepoints. + */ +async function handleSuspensionAction(input: { + env: Env + request: Request + url: URL + actor: Awaited> + body: object + suspended: boolean +}) { + const targetUserId = readPositiveInt( + readNonEmptyTrimmedStringOrNumber(input.body, 'userId'), + 0, + ) + if (!targetUserId) { + return jsonResponse({ ok: false, error: 'User id is required.' }, 400) + } + if (input.suspended && targetUserId === input.actor.userId) { + return jsonResponse( + { ok: false, error: 'You cannot suspend your own account.' }, + 400, + ) + } + + const updatedUser = await updateAdminUserSuspension(input.env.APP_DB, { + id: targetUserId, + suspended: input.suspended, + }) + if (!updatedUser) { + return jsonResponse({ ok: false, error: 'User not found.' }, 404) + } + + const requestIp = getRequestIp(input.request) ?? undefined + void logAuditEvent({ + category: 'admin', + action: input.suspended ? 'suspend_user' : 'unsuspend_user', + result: 'success', + email: input.actor.email, + ip: requestIp, + path: input.url.pathname, + reason: `target_user_id=${targetUserId}`, + }) + + return buildMutationResponse(input.env, input.request.url, targetUserId) +} + +/** + * Clear an automatic outbound-email pause (set by the delivery-event + * abuse monitor) after reviewing the account's delivery history. + */ +async function handleResumeEmailOutboundAction(input: { + env: Env + request: Request + url: URL + actor: Awaited> + body: object +}) { + const targetUserId = readPositiveInt( + readNonEmptyTrimmedStringOrNumber(input.body, 'userId'), + 0, + ) + if (!targetUserId) { + return jsonResponse({ ok: false, error: 'User id is required.' }, 400) + } + + const updatedUser = await clearAdminUserEmailOutboundPause(input.env.APP_DB, { + id: targetUserId, + }) + if (!updatedUser) { + return jsonResponse({ ok: false, error: 'User not found.' }, 404) + } + + const requestIp = getRequestIp(input.request) ?? undefined + void logAuditEvent({ + category: 'admin', + action: 'resume_email_outbound', + result: 'success', + email: input.actor.email, + ip: requestIp, + path: input.url.pathname, + reason: `target_user_id=${targetUserId}`, + }) + + return buildMutationResponse(input.env, input.request.url, targetUserId) +} + /** * Read the requested plan value: a known plan name sets it; explicit null * (and empty string) map to `free` (the normal default). Writers never diff --git a/packages/worker/src/app/loader-data.ts b/packages/worker/src/app/loader-data.ts index d51ece3c40..27c0d11fd3 100644 --- a/packages/worker/src/app/loader-data.ts +++ b/packages/worker/src/app/loader-data.ts @@ -122,6 +122,8 @@ export type AdminUserListItem = { email_verified: boolean email_verified_at: string | null plan: AdminPlanName + suspended_at: string | null + email_outbound_paused_at: string | null created_at: string updated_at: string roles: Array @@ -227,6 +229,8 @@ export type AdminUsageEntitlementResource = | 'secrets' | 'concurrent_workflows' | 'storage_bytes' + | 'execute_calls_per_day' + | 'outbound_fetches_per_day' export type AdminPlanName = 'free' | 'partner' | 'pro' | 'max' @@ -306,6 +310,21 @@ export type AdminInsightsEmailDay = { receives: number } +/** + * Platform-wide outbound delivery outcomes from provider delivery events. + * Bounce and complaint volume is the early-warning signal for shared + * sender-domain reputation trouble (all users send from one domain). + */ +export type AdminInsightsEmailDeliveryDay = { + day: string + delivered: number + deferred: number + bounced: number + failed: number + rejected: number + complained: number +} + export type AdminInsightsAuthDay = { day: string success: number @@ -350,6 +369,7 @@ export type AdminInsightsLoaderData = { signupsByWeek: Array usageByMonth: Array emailByDay: Array + emailDeliveryByDay: Array plans: Array authByDay: Array authByCategory: Array diff --git a/packages/worker/src/app/request-auth-cache.ts b/packages/worker/src/app/request-auth-cache.ts index e17b3f865f..263c923835 100644 --- a/packages/worker/src/app/request-auth-cache.ts +++ b/packages/worker/src/app/request-auth-cache.ts @@ -31,6 +31,7 @@ type ResolvedAuthUser = { username: string displayName: string accountDeleting: boolean + accountSuspended: boolean roles: Array permissions: Array mcpUser: McpUserContext @@ -115,6 +116,7 @@ async function resolveRequestAuth( username: userRecord.username, displayName, accountDeleting: Boolean(userRecord.deleting_at), + accountSuspended: Boolean(userRecord.suspended_at), roles, permissions, mcpUser: { diff --git a/packages/worker/src/app/session-info.ts b/packages/worker/src/app/session-info.ts index 89876e55b4..ac5b0fcb95 100644 --- a/packages/worker/src/app/session-info.ts +++ b/packages/worker/src/app/session-info.ts @@ -24,7 +24,9 @@ export async function loadSessionInfo( env: Env, ): Promise { const resolved = await loadResolvedRequestAuth(request, env) - if (!resolved.user) { + // A suspended account's session is treated as signed out everywhere, + // including the SSR shell (matching readAuthenticatedAppUser). + if (!resolved.user || resolved.user.accountSuspended) { return { session: null, setCookie: resolved.setCookie, diff --git a/packages/worker/src/community/community-flow-test-schema.ts b/packages/worker/src/community/community-flow-test-schema.ts index e10f5979fd..847cab8c3a 100644 --- a/packages/worker/src/community/community-flow-test-schema.ts +++ b/packages/worker/src/community/community-flow-test-schema.ts @@ -20,6 +20,8 @@ export async function ensureCommunityFlowSchema(db: D1Database) { stripe_plan TEXT, stripe_plan_refreshed_at TEXT, deleting_at TEXT, + suspended_at TEXT, + email_outbound_paused_at TEXT, active_write_count INTEGER NOT NULL DEFAULT 0, active_write_expires_at TEXT, created_at TEXT NOT NULL DEFAULT (CURRENT_TIMESTAMP), @@ -243,6 +245,8 @@ export async function ensureCommunityFlowSchema(db: D1Database) { `ALTER TABLE users ADD COLUMN profile_visibility TEXT NOT NULL DEFAULT 'public'`, `ALTER TABLE users ADD COLUMN plan TEXT NOT NULL DEFAULT 'free'`, `ALTER TABLE users ADD COLUMN deleting_at TEXT`, + `ALTER TABLE users ADD COLUMN suspended_at TEXT`, + `ALTER TABLE users ADD COLUMN email_outbound_paused_at TEXT`, `ALTER TABLE users ADD COLUMN active_write_count INTEGER NOT NULL DEFAULT 0`, `ALTER TABLE users ADD COLUMN active_write_expires_at TEXT`, `ALTER TABLE saved_packages ADD COLUMN is_private INTEGER NOT NULL DEFAULT 1`, diff --git a/packages/worker/src/db.ts b/packages/worker/src/db.ts index 61a7ce8952..b885fd4038 100644 --- a/packages/worker/src/db.ts +++ b/packages/worker/src/db.ts @@ -20,6 +20,8 @@ export const usersTable = table({ stripe_plan: c.text(), stripe_plan_refreshed_at: c.text(), deleting_at: c.text(), + suspended_at: c.text(), + email_outbound_paused_at: c.text(), active_write_count: c.integer(), active_write_expires_at: c.text(), created_at: c.text(), diff --git a/packages/worker/src/email/delivery-queue.node.test.ts b/packages/worker/src/email/delivery-queue.node.test.ts index 56c5547378..1fc041619f 100644 --- a/packages/worker/src/email/delivery-queue.node.test.ts +++ b/packages/worker/src/email/delivery-queue.node.test.ts @@ -43,8 +43,14 @@ test('email delivery Queue acknowledges permanent outcomes and retries unmatched const dispatchFailure = createQueueMessage('queue-dispatch-failure', { kind: 'dispatch-failure', }) + // A `delivered` status keeps the outbound abuse monitor on its no-op + // path, so this test stays focused on queue mechanics. const providerEvent = { - payload: { eventId: 'event-1', messageId: 'provider-1' }, + payload: { + eventId: 'event-1', + messageId: 'provider-1', + delivery: { status: 'delivered' }, + }, } const storedMessage = { id: 'message-1', userId: 'user-1' } mocks.processCloudflareEmailDeliveryEvent diff --git a/packages/worker/src/email/delivery-queue.ts b/packages/worker/src/email/delivery-queue.ts index c53c07686b..225be6fc23 100644 --- a/packages/worker/src/email/delivery-queue.ts +++ b/packages/worker/src/email/delivery-queue.ts @@ -1,4 +1,5 @@ import { processCloudflareEmailDeliveryEvent } from './delivery-events.ts' +import { applyOutboundEmailAbusePause } from './outbound-abuse.ts' import { dispatchEmailDeliverySubscriptionEvents } from './package-subscriptions.ts' const unmatchedRetryDelaySeconds = 30 @@ -27,10 +28,31 @@ export async function handleEmailDeliveryQueue( queueMessage.retry({ delaySeconds: unmatchedRetryDelaySeconds }) break case 'stale': + // Out-of-order events still count toward the abuse + // thresholds when they persisted: the bounce/complaint + // happened even when a newer status already superseded + // it. Conflicting duplicates that were not inserted only + // pause when persisted events back them. + await applyOutboundEmailAbusePause({ + env, + userId: result.message.userId, + deliveryStatus: result.event.payload.delivery.status, + eventRecorded: false, + }) queueMessage.ack() break case 'duplicate': case 'recorded': { + // Abuse evaluation runs before subscription dispatch (and + // also for replayed duplicates) so a crash or dispatch + // failure can never skip the pause; the pause write itself + // is idempotent. + await applyOutboundEmailAbusePause({ + env, + userId: result.message.userId, + deliveryStatus: result.event.payload.delivery.status, + eventRecorded: result.outcome === 'recorded', + }) await dispatchEmailDeliverySubscriptionEvents({ env, message: result.message, diff --git a/packages/worker/src/email/inbound.ts b/packages/worker/src/email/inbound.ts index 9088933da6..ce6ebd17c9 100644 --- a/packages/worker/src/email/inbound.ts +++ b/packages/worker/src/email/inbound.ts @@ -261,7 +261,7 @@ export async function handleInboundEmail( // getUserPlan / isAccountEmailVerified) so a mismatched identity pair // cannot apply another account's plan or verification state. const accountRow = await env.APP_DB.prepare( - `SELECT plan, stripe_plan, email_verified_at FROM users + `SELECT plan, stripe_plan, email_verified_at, suspended_at FROM users WHERE email = ? AND stable_user_id = ?`, ) .bind(identity.email, userId) @@ -269,6 +269,7 @@ export async function handleInboundEmail( plan: string stripe_plan: string | null email_verified_at: string | null + suspended_at: string | null }>() const account = { email: identity.email, @@ -278,6 +279,7 @@ export async function handleInboundEmail( accountRow?.stripe_plan ?? null, ), emailVerified: Boolean(accountRow?.email_verified_at), + suspended: Boolean(accountRow?.suspended_at), } const provisioned = await ensureDefaultEmailInbox({ @@ -337,6 +339,24 @@ export async function handleInboundEmail( return } + // A platform-suspended account cannot receive mail either; the + // rejection goes through the same bounded recorder so suspended + // aliases cannot be used to grow storage. + if (account.suspended) { + const reason = 'Account is suspended.' + message.setReject(reason) + await recordBoundedEmailRejectionEvent({ + db: env.APP_DB, + userId, + inboxId: inbox.id, + recipient, + reason, + phase: 'account-suspension', + }).catch(warnRejectionAuditWriteFailed) + await recordReceiveUsage({ outcome: 'error' }) + return + } + try { // Size first: an oversize message is rejected without consuming any // of the owner's daily receive quota (griefing resistance) and diff --git a/packages/worker/src/email/inbound.workers.test.ts b/packages/worker/src/email/inbound.workers.test.ts index 6d54405308..7b8dcfe416 100644 --- a/packages/worker/src/email/inbound.workers.test.ts +++ b/packages/worker/src/email/inbound.workers.test.ts @@ -600,6 +600,64 @@ test('inbound email handler rejects mail for unverified accounts', async () => { expect(rollup).toMatchObject({ event_count: 2, error_count: 2 }) }) +test('inbound email handler rejects mail for suspended accounts', async () => { + await ensureEmailTestSchema(env.APP_DB) + await ensureUsageRollupsTestSchema(env.APP_DB) + const username = `suspended-${crypto.randomUUID().slice(0, 8)}` + const accountEmail = `email-suspended-${crypto.randomUUID()}@example.com` + const userId = await createStableUserIdFromEmail(accountEmail) + const address = `${username}@${platformDomain}` + await seedAccount({ + db: env.APP_DB, + email: accountEmail, + username, + }) + await env.APP_DB.prepare( + `UPDATE users SET suspended_at = ? WHERE stable_user_id = ?`, + ) + .bind(new Date().toISOString(), userId) + .run() + + const message = createForwardableEmailMessage({ + from: 'stranger@example.net', + to: address, + raw: [ + 'From: Stranger ', + `To: ${address}`, + 'Subject: Should be rejected', + 'Message-ID: ', + '', + 'Please help.', + ].join('\r\n'), + }) + await handleInboundEmail(message, createInboundEnv()) + expect(message.rejectedReason).toBe('Account is suspended.') + + expect( + await listEmailMessages({ + db: env.APP_DB, + userId, + limit: 10, + }), + ).toEqual([]) + const events = await env.APP_DB.prepare( + `SELECT event_type, detail_json FROM email_delivery_events WHERE user_id = ?`, + ) + .bind(userId) + .all<{ event_type: string; detail_json: string }>() + const details = (events.results ?? []).map((row) => ({ + eventType: row.event_type, + detail: JSON.parse(row.detail_json) as Record, + })) + expect(details.every((row) => row.eventType === 'rejected')).toBe(true) + expect( + details.find((row) => row.detail['aggregate'] !== true)?.detail, + ).toMatchObject({ + reason: 'Account is suspended.', + phase: 'account-suspension', + }) +}) + test('getEmailAttachmentById reconstructs unnamed attachments from raw MIME', async () => { await ensureEmailTestSchema(env.APP_DB) const userId = `email-attachment-user-${crypto.randomUUID()}` diff --git a/packages/worker/src/email/outbound-abuse.ts b/packages/worker/src/email/outbound-abuse.ts new file mode 100644 index 0000000000..6b7ade6c2f --- /dev/null +++ b/packages/worker/src/email/outbound-abuse.ts @@ -0,0 +1,202 @@ +/** + * Automatic outbound-email abuse pause. + * + * All user mail sends from one shared platform domain through one + * Cloudflare Email Sending account, so a single account generating spam + * complaints or sustained bounces burns sender reputation for every user. + * This monitor reacts to provider delivery events: any spam complaint, or + * repeated hard bounces within a UTC day, sets + * `users.email_outbound_paused_at`, which blocks further outbound sends + * (see `sendOutboundEmail`) until an admin reviews and clears it from the + * admin users page. + */ + +import { utcDayKey } from '@kody-internal/shared/date-keys.ts' +import { sendCloudflareEmail } from '#app/email/cloudflare-email.ts' +import { getSystemEmailDomain } from './platform-address.ts' +import { type EmailDeliveryStatus } from './types.ts' + +/** + * Bounced sends tolerated per user per UTC day before outbound email is + * paused. Complaints pause immediately: one recipient marking mail as spam + * is already a reputation signal the shared domain cannot absorb quietly. + */ +export const outboundEmailBouncePauseThresholdPerDay = 5 + +export const emailOutboundPausedMessage = + 'Outbound email is paused for this account after spam complaints or repeated bounces. Contact the operator of this Kody deployment to have it reviewed and re-enabled.' + +type OutboundAbuseEnv = Pick< + Env, + | 'APP_DB' + | 'APP_BASE_URL' + | 'CLOUDFLARE_ACCOUNT_ID' + | 'CLOUDFLARE_API_BASE_URL' + | 'CLOUDFLARE_API_TOKEN' +> + +/** + * Evaluate one provider delivery event against the abuse thresholds and + * pause the sending account when they are exceeded. Idempotent: the pause + * write only transitions a NULL `email_outbound_paused_at`, so replayed + * queue messages and duplicate provider events can never re-pause or + * re-notify. Returns whether this call performed the pause transition. + */ +export async function applyOutboundEmailAbusePause(input: { + env: OutboundAbuseEnv + /** Stable MCP userId of the account that sent the message. */ + userId: string + deliveryStatus: EmailDeliveryStatus + /** + * Whether this delivery event was persisted by the current processing + * run (queue outcome `recorded`). Replayed and conflicting-duplicate + * signals are deduped by provider_event_id at insert time, so the same + * provider event can only ever count once: non-recorded signals only + * pause when a matching persisted event backs them. + */ + eventRecorded: boolean + now?: Date +}): Promise<{ paused: boolean }> { + const now = input.now ?? new Date() + switch (input.deliveryStatus) { + case 'complained': { + // A freshly recorded complaint always pauses. A replayed or + // conflicting-duplicate complaint signal only pauses when a + // real persisted complaint event backs it (crash recovery + // between recording and pausing), never on its own. + if (!input.eventRecorded) { + const complaintsToday = await countProviderDeliveryEventsToday({ + db: input.env.APP_DB, + userId: input.userId, + eventType: 'complained', + now, + }) + if (complaintsToday < 1) return { paused: false } + } + break + } + case 'bounced': { + const bouncesToday = await countProviderDeliveryEventsToday({ + db: input.env.APP_DB, + userId: input.userId, + eventType: 'bounced', + now, + }) + if (bouncesToday < outboundEmailBouncePauseThresholdPerDay) { + return { paused: false } + } + break + } + case 'delivered': + case 'deferred': + case 'failed': + case 'rejected': + return { paused: false } + default: { + const exhaustive: never = input.deliveryStatus + throw new Error(`Unsupported delivery status: ${String(exhaustive)}`) + } + } + + const result = await input.env.APP_DB.prepare( + `UPDATE users + SET email_outbound_paused_at = ?, updated_at = ? + WHERE stable_user_id = ? AND email_outbound_paused_at IS NULL`, + ) + .bind(now.toISOString(), now.toISOString(), input.userId) + .run() + const paused = Number(result.meta.changes ?? 0) > 0 + if (paused) { + console.warn('email-outbound-paused', { + deliveryStatus: input.deliveryStatus, + }) + await notifyAdminsOfOutboundEmailPause({ + env: input.env, + userId: input.userId, + deliveryStatus: input.deliveryStatus, + }) + } + return { paused } +} + +async function countProviderDeliveryEventsToday(input: { + db: D1Database + userId: string + eventType: EmailDeliveryStatus + now: Date +}) { + const dayStart = `${utcDayKey(input.now)}T00:00:00` + const row = await input.db + .prepare( + `SELECT COUNT(*) AS count FROM email_delivery_events + WHERE user_id = ? + AND event_type = ? + AND provider = 'cloudflare-email' + AND created_at >= ?`, + ) + .bind(input.userId, input.eventType, dayStart) + .first<{ count: number }>() + return Number(row?.count ?? 0) +} + +/** + * Best-effort operator notification through the transactional system + * sender (`kody@`), addressed to every admin account email. A send + * failure must never fail delivery-event processing — the pause itself is + * already committed and the admin users page shows the paused state. + */ +async function notifyAdminsOfOutboundEmailPause(input: { + env: OutboundAbuseEnv + userId: string + deliveryStatus: EmailDeliveryStatus +}) { + try { + const systemDomain = getSystemEmailDomain(input.env) + if (!systemDomain) return + // The admin roster is operator-controlled and small; no LIMIT so + // every admin is notified. + const admins = await input.env.APP_DB.prepare( + `SELECT u.email, u.username FROM users u + INNER JOIN user_roles ur ON ur.user_id = u.id + INNER JOIN roles r ON r.id = ur.role_id + WHERE r.name = 'admin' + ORDER BY u.id ASC`, + ).all<{ email: string; username: string }>() + const recipients = (admins.results ?? []).map((row) => row.email) + if (recipients.length === 0) return + const pausedUser = await input.env.APP_DB.prepare( + `SELECT username FROM users WHERE stable_user_id = ?`, + ) + .bind(input.userId) + .first<{ username: string }>() + const username = pausedUser?.username ?? 'unknown' + const reason = + input.deliveryStatus === 'complained' + ? 'a spam complaint' + : `${outboundEmailBouncePauseThresholdPerDay}+ bounced sends today` + const text = [ + `Outbound email was automatically paused for user "${username}" after ${reason}.`, + 'The account can still receive mail and use every other capability; only outbound sending is blocked.', + `Review the delivery history and clear the pause from ${input.env.APP_BASE_URL?.trim() || `https://${systemDomain}`}/admin/users if this was a false positive.`, + ].join('\n\n') + await sendCloudflareEmail( + { + accountId: input.env.CLOUDFLARE_ACCOUNT_ID, + apiBaseUrl: input.env.CLOUDFLARE_API_BASE_URL, + apiToken: input.env.CLOUDFLARE_API_TOKEN, + }, + { + to: recipients.length === 1 ? recipients[0]! : recipients, + from: `kody@${systemDomain}`, + subject: `Outbound email paused for ${username}`, + text, + html: `${text + .split('\n\n') + .map((paragraph) => `

${paragraph}

`) + .join('')}`, + }, + ) + } catch (error) { + console.warn('email-outbound-pause-notification-failed', error) + } +} diff --git a/packages/worker/src/email/outbound-abuse.workers.test.ts b/packages/worker/src/email/outbound-abuse.workers.test.ts new file mode 100644 index 0000000000..b24f863625 --- /dev/null +++ b/packages/worker/src/email/outbound-abuse.workers.test.ts @@ -0,0 +1,225 @@ +import { env } from 'cloudflare:workers' +import { expect, test } from 'vitest' +import { createStableUserIdFromEmail } from '#worker/user-id.ts' +import { consoleWarn } from '#worker/test-support/console-spies.ts' +import { processCloudflareEmailDeliveryEvent } from './delivery-events.ts' +import { + applyOutboundEmailAbusePause, + emailOutboundPausedMessage, + outboundEmailBouncePauseThresholdPerDay, +} from './outbound-abuse.ts' +import { sendOutboundEmail } from './outbound.ts' +import { insertEmailMessage } from './repo.ts' +import { ensureEmailTestSchema } from './test-schema.ts' + +const platformBaseUrl = 'https://kody.example.com' + +async function seedVerifiedAccount(input: { email: string }) { + const username = `sender-${crypto.randomUUID().slice(0, 8)}` + const stableUserId = await createStableUserIdFromEmail(input.email) + await env.APP_DB.prepare( + `INSERT INTO users (username, email, password_hash, email_verified_at, plan, stable_user_id) + VALUES (?, ?, ?, ?, 'max', ?)`, + ) + .bind( + username, + input.email, + 'test-password-hash', + new Date().toISOString(), + stableUserId, + ) + .run() + return { username, stableUserId } +} + +async function recordProviderEvent(input: { + providerMessageId: string + eventId: string + status: 'delivered' | 'bounced' | 'complained' + eventTimestamp: string +}) { + const result = await processCloudflareEmailDeliveryEvent({ + db: env.APP_DB, + body: { + type: `cf.email.sending.message.${input.status}`, + source: { + type: 'email.sending', + zoneId: 'zone-1', + domain: 'inbox.kody.example.com', + }, + payload: { + eventId: input.eventId, + messageId: input.providerMessageId, + sender: 'user@inbox.kody.example.com', + recipient: 'recipient@example.net', + terminal: true, + delivery: { status: input.status }, + }, + metadata: { + accountId: 'account-1', + eventSubscriptionId: 'subscription-1', + eventSchemaVersion: 1, + eventTimestamp: input.eventTimestamp, + }, + }, + }) + expect(result.outcome).not.toBe('invalid') + return result +} + +async function readPauseTimestamp(stableUserId: string) { + const row = await env.APP_DB.prepare( + `SELECT email_outbound_paused_at FROM users WHERE stable_user_id = ?`, + ) + .bind(stableUserId) + .first<{ email_outbound_paused_at: string | null }>() + return row?.email_outbound_paused_at ?? null +} + +test('a spam complaint pauses outbound email once and blocks further sends', async () => { + consoleWarn.mockImplementation(() => {}) + await ensureEmailTestSchema(env.APP_DB) + const accountEmail = `complainer-${crypto.randomUUID()}@example.com` + const { stableUserId } = await seedVerifiedAccount({ email: accountEmail }) + const providerMessageId = `provider-${crypto.randomUUID()}` + await insertEmailMessage({ + db: env.APP_DB, + message: { + direction: 'outbound', + userId: stableUserId, + fromAddress: 'user@inbox.kody.example.com', + toAddresses: ['recipient@example.net'], + subject: 'Complaint pause', + processingStatus: 'sent', + providerMessageId, + sentAt: '2026-07-17T20:00:00.000Z', + }, + }) + await recordProviderEvent({ + providerMessageId, + eventId: 'event-complained', + status: 'complained', + eventTimestamp: new Date().toISOString(), + }) + + const first = await applyOutboundEmailAbusePause({ + env, + userId: stableUserId, + deliveryStatus: 'complained', + eventRecorded: true, + }) + expect(first.paused).toBe(true) + const pausedAt = await readPauseTimestamp(stableUserId) + expect(pausedAt).not.toBeNull() + expect( + consoleWarn.mock.calls.some((call) => call[0] === 'email-outbound-paused'), + ).toBe(true) + + // Replayed queue messages must not re-pause or move the timestamp. + const second = await applyOutboundEmailAbusePause({ + env, + userId: stableUserId, + deliveryStatus: 'complained', + eventRecorded: false, + }) + expect(second.paused).toBe(false) + expect(await readPauseTimestamp(stableUserId)).toBe(pausedAt) + + await expect( + sendOutboundEmail({ + env: { + ...env, + APP_BASE_URL: platformBaseUrl, + EMAIL: { + async send() { + throw new Error('Paused accounts must not reach the provider.') + }, + }, + }, + userId: stableUserId, + accountEmail, + recipientPolicy: 'self', + subject: 'Blocked while paused', + text: 'Body', + }), + ).rejects.toThrow(emailOutboundPausedMessage) +}) + +test('bounces below the daily threshold do not pause; reaching it does', async () => { + consoleWarn.mockImplementation(() => {}) + await ensureEmailTestSchema(env.APP_DB) + const accountEmail = `bouncer-${crypto.randomUUID()}@example.com` + const { stableUserId } = await seedVerifiedAccount({ email: accountEmail }) + const now = new Date() + + for ( + let attempt = 1; + attempt <= outboundEmailBouncePauseThresholdPerDay; + attempt += 1 + ) { + const providerMessageId = `provider-${crypto.randomUUID()}` + await insertEmailMessage({ + db: env.APP_DB, + message: { + direction: 'outbound', + userId: stableUserId, + fromAddress: 'user@inbox.kody.example.com', + toAddresses: ['recipient@example.net'], + subject: `Bounce ${attempt}`, + processingStatus: 'sent', + providerMessageId, + sentAt: now.toISOString(), + }, + }) + await recordProviderEvent({ + providerMessageId, + eventId: `event-bounced-${attempt}`, + status: 'bounced', + eventTimestamp: now.toISOString(), + }) + const result = await applyOutboundEmailAbusePause({ + env, + userId: stableUserId, + deliveryStatus: 'bounced', + eventRecorded: true, + now, + }) + if (attempt < outboundEmailBouncePauseThresholdPerDay) { + expect(result.paused).toBe(false) + expect(await readPauseTimestamp(stableUserId)).toBeNull() + } else { + expect(result.paused).toBe(true) + expect(await readPauseTimestamp(stableUserId)).not.toBeNull() + } + } +}) + +test('delivered events never pause outbound email', async () => { + await ensureEmailTestSchema(env.APP_DB) + const accountEmail = `deliverer-${crypto.randomUUID()}@example.com` + const { stableUserId } = await seedVerifiedAccount({ email: accountEmail }) + const result = await applyOutboundEmailAbusePause({ + env, + userId: stableUserId, + deliveryStatus: 'delivered', + eventRecorded: true, + }) + expect(result.paused).toBe(false) + expect(await readPauseTimestamp(stableUserId)).toBeNull() +}) + +test('a complaint signal that was never persisted does not pause on its own', async () => { + await ensureEmailTestSchema(env.APP_DB) + const accountEmail = `phantom-${crypto.randomUUID()}@example.com` + const { stableUserId } = await seedVerifiedAccount({ email: accountEmail }) + // Conflicting-duplicate queue outcome: the complained event's insert + // was deduped by provider_event_id, so no persisted complaint backs it. + const result = await applyOutboundEmailAbusePause({ + env, + userId: stableUserId, + deliveryStatus: 'complained', + eventRecorded: false, + }) + expect(result.paused).toBe(false) + expect(await readPauseTimestamp(stableUserId)).toBeNull() +}) diff --git a/packages/worker/src/email/outbound.ts b/packages/worker/src/email/outbound.ts index 7b7e6d3ef8..03372d4357 100644 --- a/packages/worker/src/email/outbound.ts +++ b/packages/worker/src/email/outbound.ts @@ -1,5 +1,9 @@ import { base64ToBytes } from '@kody-internal/shared/base64.ts' import { getErrorMessage } from '@kody-internal/shared/error-message.ts' +import { + accountSuspendedMessage, + getAccountRestrictionsByStableUserId, +} from '#app/account-suspension.ts' import { sendCloudflareEmail } from '#app/email/cloudflare-email.ts' import { isAccountEmailVerified } from '#app/email-verification.ts' import { normalizeEmail } from '#app/normalize-email.ts' @@ -12,6 +16,7 @@ import { } from '#worker/entitlements/service.ts' import { recordUsage } from '#worker/usage/record-usage.ts' import { normalizeEmailAddress } from './address.ts' +import { emailOutboundPausedMessage } from './outbound-abuse.ts' import { resolveUserPlatformSender } from './platform-address.ts' import { createEmailThread, @@ -433,6 +438,21 @@ export async function sendOutboundEmail( if (!accountEmailVerified) { throw new Error('Account email must be verified before sending email.') } + // Abuse gates: a suspended account cannot send at all, and an + // account paused by the delivery-event abuse monitor (spam + // complaints / repeated bounces) stays blocked until an admin + // clears the pause. Both protect the shared platform sending + // domain's reputation. + const restrictions = await getAccountRestrictionsByStableUserId({ + db: input.env.APP_DB, + stableUserId: input.userId, + }) + if (restrictions?.suspendedAt) { + throw new Error(accountSuspendedMessage) + } + if (restrictions?.emailOutboundPausedAt) { + throw new Error(emailOutboundPausedMessage) + } // Sends reference the platform-provisioned sender identity. It is // normally created alongside the default inbox at signup; ensuring it // here also covers accounts provisioned before the identity existed. diff --git a/packages/worker/src/email/outbound.workers.test.ts b/packages/worker/src/email/outbound.workers.test.ts index ee7001ae40..8ad4c18a7d 100644 --- a/packages/worker/src/email/outbound.workers.test.ts +++ b/packages/worker/src/email/outbound.workers.test.ts @@ -188,6 +188,30 @@ test('sendOutboundEmail sends from the platform-assigned username address to the expect(listed.map((message) => message.id)).toContain(result.message.id) }) +test('sendOutboundEmail rejects suspended accounts', async () => { + silenceIncidentalRuntimeWarnings() + await ensureEmailTestSchema(env.APP_DB) + const accountEmail = `suspended-${crypto.randomUUID()}@example.com` + const userId = await createStableUserIdFromEmail(accountEmail) + await seedVerifiedAccount({ email: accountEmail }) + await env.APP_DB.prepare( + `UPDATE users SET suspended_at = ? WHERE stable_user_id = ?`, + ) + .bind(new Date().toISOString(), userId) + .run() + + await expect( + sendOutboundEmail({ + env: createBindingSendEnv(), + userId, + accountEmail, + recipientPolicy: 'self', + subject: 'Blocked while suspended', + text: 'Body', + }), + ).rejects.toThrow('This account is suspended') +}) + test('sendOutboundEmail rejects non-self recipients under the self policy', async () => { // Usage recording degrades with a warn when the usage_rollups table is // not part of this test's schema; that is incidental to the policy. diff --git a/packages/worker/src/email/service.ts b/packages/worker/src/email/service.ts index 3403603359..9bd36bd25b 100644 --- a/packages/worker/src/email/service.ts +++ b/packages/worker/src/email/service.ts @@ -564,7 +564,12 @@ export async function recordBoundedEmailRejectionEvent(input: { inboxId: string recipient: string reason: string - phase: 'entitlement' | 'size' | 'account-verification' | 'system-limit' + phase: + | 'entitlement' + | 'size' + | 'account-verification' + | 'account-suspension' + | 'system-limit' now?: Date }) { const now = input.now ?? new Date() diff --git a/packages/worker/src/entitlements/plans.ts b/packages/worker/src/entitlements/plans.ts index 0e115e60e6..43789f2c44 100644 --- a/packages/worker/src/entitlements/plans.ts +++ b/packages/worker/src/entitlements/plans.ts @@ -153,6 +153,15 @@ export type PlanLimits = { maxStorageBytes: number /** Maximum concurrently active workflow runs. */ maxConcurrentWorkflows: number + /** Maximum MCP execute-tool runs per UTC day. */ + maxExecuteCallsPerDay: number + /** + * Maximum sandbox outbound fetches (through the fetch gateway) per UTC + * day. Bounds cost abuse and third-party hammering from user code; the + * shared Worker egress identity means one user's fetch flood can burn + * reputation for the whole deployment. + */ + maxOutboundFetchesPerDay: number } export const entitlementResources = [ @@ -168,6 +177,8 @@ export const entitlementResources = [ 'secrets', 'storage_bytes', 'concurrent_workflows', + 'execute_calls_per_day', + 'outbound_fetches_per_day', ] as const export type EntitlementResource = (typeof entitlementResources)[number] @@ -186,6 +197,8 @@ export const entitlementResourceLabels: Record = { secrets: 'secrets', storage_bytes: 'storage bytes', concurrent_workflows: 'concurrent workflows', + execute_calls_per_day: 'execute calls per day', + outbound_fetches_per_day: 'outbound fetches per day', } /** @@ -232,6 +245,8 @@ export const planLimits: Record = { maxSecrets: 5, maxStorageBytes: 64 * 1024 * 1024, maxConcurrentWorkflows: 3, + maxExecuteCallsPerDay: 500, + maxOutboundFetchesPerDay: 2_000, }, pro: { maxSavedPackages: 100, @@ -246,6 +261,8 @@ export const planLimits: Record = { maxSecrets: 100, maxStorageBytes: 1024 * 1024 * 1024, maxConcurrentWorkflows: 50, + maxExecuteCallsPerDay: 5_000, + maxOutboundFetchesPerDay: 20_000, }, partner: { maxSavedPackages: 200, @@ -260,6 +277,8 @@ export const planLimits: Record = { maxSecrets: 200, maxStorageBytes: 5 * 1024 * 1024 * 1024, maxConcurrentWorkflows: 100, + maxExecuteCallsPerDay: 10_000, + maxOutboundFetchesPerDay: 40_000, }, max: { // 100× pro (100) → 10_000. @@ -283,6 +302,10 @@ export const planLimits: Record = { maxStorageBytes: 100 * 1024 * 1024 * 1024, // 100× pro (50) → 5_000 (explicit product choice). maxConcurrentWorkflows: 5_000, + // 100× pro (5_000) → 500_000. + maxExecuteCallsPerDay: 500_000, + // 100× pro (20_000) → 2_000_000. + maxOutboundFetchesPerDay: 2_000_000, }, } @@ -339,6 +362,10 @@ export function resolvePlanLimit( return limits.maxStorageBytes case 'concurrent_workflows': return limits.maxConcurrentWorkflows + case 'execute_calls_per_day': + return limits.maxExecuteCallsPerDay + case 'outbound_fetches_per_day': + return limits.maxOutboundFetchesPerDay default: { const exhaustive: never = resource throw new Error(`Unknown entitlement resource: ${String(exhaustive)}`) diff --git a/packages/worker/src/entitlements/service.ts b/packages/worker/src/entitlements/service.ts index 0f1fd0beee..88e8acd61f 100644 --- a/packages/worker/src/entitlements/service.ts +++ b/packages/worker/src/entitlements/service.ts @@ -557,6 +557,8 @@ export async function readEntitlementResourceUsage(input: { ) case 'email_sends_per_day': case 'email_receives_per_day': + case 'execute_calls_per_day': + case 'outbound_fetches_per_day': return await readDailyEntitlementCounter({ db, userId, diff --git a/packages/worker/src/entitlements/test-schema.ts b/packages/worker/src/entitlements/test-schema.ts index bbe4beb79c..c61a40927e 100644 --- a/packages/worker/src/entitlements/test-schema.ts +++ b/packages/worker/src/entitlements/test-schema.ts @@ -29,6 +29,8 @@ export async function ensureEntitlementTestSchema(db: D1Database) { stripe_plan TEXT, stripe_plan_refreshed_at TEXT, deleting_at TEXT, + suspended_at TEXT, + email_outbound_paused_at TEXT, active_write_count INTEGER NOT NULL DEFAULT 0, active_write_expires_at TEXT, created_at TEXT NOT NULL DEFAULT (CURRENT_TIMESTAMP), @@ -46,6 +48,8 @@ export async function ensureEntitlementTestSchema(db: D1Database) { 'stripe_plan TEXT', 'stripe_plan_refreshed_at TEXT', 'deleting_at TEXT', + 'suspended_at TEXT', + 'email_outbound_paused_at TEXT', 'active_write_count INTEGER NOT NULL DEFAULT 0', 'active_write_expires_at TEXT', ]) { diff --git a/packages/worker/src/execute-maintenance.node.test.ts b/packages/worker/src/execute-maintenance.node.test.ts index 08087d3a66..788e6a1037 100644 --- a/packages/worker/src/execute-maintenance.node.test.ts +++ b/packages/worker/src/execute-maintenance.node.test.ts @@ -32,6 +32,7 @@ test('runExecuteSmokeCheck executes trivial code through the dynamic worker path gatewayProps: { baseUrl: 'https://example.com', userId: null, + email: null, storageContext: null, }, }), diff --git a/packages/worker/src/execute-maintenance.ts b/packages/worker/src/execute-maintenance.ts index bf0a67d606..90405717d4 100644 --- a/packages/worker/src/execute-maintenance.ts +++ b/packages/worker/src/execute-maintenance.ts @@ -31,6 +31,7 @@ export async function runExecuteSmokeCheck(env: Env) { gatewayProps: { baseUrl: origin, userId: null, + email: null, storageContext: null, }, timeoutMs: 10_000, diff --git a/packages/worker/src/mcp-auth.ts b/packages/worker/src/mcp-auth.ts index fbb17ccc06..8ff79a5e14 100644 --- a/packages/worker/src/mcp-auth.ts +++ b/packages/worker/src/mcp-auth.ts @@ -2,6 +2,7 @@ import { type OAuthHelpers, type TokenSummary, } from '@cloudflare/workers-oauth-provider' +import { isAccountSuspended } from '#app/account-suspension.ts' import { getAppBaseUrl } from '#app/app-base-url.ts' import { isAccountEmailVerified } from '#app/email-verification.ts' import { buildMcpUserContextFromGrantProps } from './mcp-auth-user-context.ts' @@ -79,6 +80,18 @@ export function createEmailVerificationRequiredResponse(origin: string) { ) } +export function createAccountSuspendedResponse() { + return Response.json( + { + error: 'account_suspended', + error_description: + 'This account is suspended, so MCP access is disabled. ' + + 'Contact the operator of this Kody deployment to appeal.', + }, + { status: 403 }, + ) +} + function audienceMatches( audience: string | Array | undefined, origin: string, @@ -146,6 +159,19 @@ export async function handleMcpRequest({ return createEmailVerificationRequiredResponse(origin) } + // Fail-closed suspension gate, mirroring email verification: a + // suspended account keeps its OAuth grants (stateless tokens cannot be + // revoked individually) but every MCP request is rejected until an + // admin clears the suspension. + const suspended = await isAccountSuspended({ + db: env.APP_DB, + email: mcpUser.email, + stableUserId: mcpUser.userId, + }) + if (suspended) { + return createAccountSuspendedResponse() + } + const remoteConnectors = await listAttachedRemoteConnectorRefs({ env, userId: mcpUser.userId, diff --git a/packages/worker/src/mcp-auth.workers.test.ts b/packages/worker/src/mcp-auth.workers.test.ts index 17b226962a..515e040e57 100644 --- a/packages/worker/src/mcp-auth.workers.test.ts +++ b/packages/worker/src/mcp-auth.workers.test.ts @@ -79,6 +79,8 @@ type VerificationLookupKind = type MockDbOptions = { // Row returned for the `email_verified_at` lookup keyed by account email. emailVerifiedAt?: string | null + // Row returned for the `suspended_at` lookup keyed by account identity. + suspendedAt?: string | null // Row returned for the indexed `stable_user_id` verification lookup. stableUserVerifiedAt?: string | null // Expected bind values for the default verified fixture identity. @@ -160,6 +162,20 @@ function createMockDb(options: MockDbOptions = {}) { email_verified_at: verifiedAt, } satisfies MockAccountRow } + if (normalized.includes('select suspended_at from users')) { + // Validate the bound identity like the adjacent + // verification branches so a widened isAccountSuspended + // query scope fails this mock. + const email = + typeof boundParams[0] === 'string' ? boundParams[0] : null + if (email !== defaultEmail) return null + if (normalized.includes('stable_user_id')) { + const stableUserId = + typeof boundParams[1] === 'string' ? boundParams[1] : null + if (stableUserId !== defaultStableUserId) return null + } + return { suspended_at: options.suspendedAt ?? null } + } if (normalized.includes('email = ? and stable_user_id')) { recordVerificationLookup('email_and_stable_user_id') const email = @@ -528,6 +544,29 @@ test('mcp request rejects unverified and unidentifiable accounts fail-closed', a expect(noUserResponse.status).toBe(403) expect(fetchMcpCalled).toBe(false) + // Verified but suspended accounts are rejected with a dedicated error. + const suspendedResponse = await handleMcpRequest({ + request, + env: createEnv( + createHelpers({ + unwrapToken: async () => + createToken({ userId: 'user', email: 'user@example.com' }), + }), + {}, + { + emailVerifiedAt: new Date(0).toISOString(), + suspendedAt: new Date(0).toISOString(), + }, + ), + ctx: createContext(), + fetchMcp, + }) + expect(suspendedResponse.status).toBe(403) + expect(await suspendedResponse.json()).toMatchObject({ + error: 'account_suspended', + }) + expect(fetchMcpCalled).toBe(false) + // Indexed stable-user-id lookup verifies accounts when grant props lack email. const fallbackEmail = 'fallback@example.com' const stableUserId = await createStableUserIdFromEmail(fallbackEmail) diff --git a/packages/worker/src/mcp/capabilities/admin/admin-capabilities.node.test.ts b/packages/worker/src/mcp/capabilities/admin/admin-capabilities.node.test.ts index 3658638ff8..dd7442d8be 100644 --- a/packages/worker/src/mcp/capabilities/admin/admin-capabilities.node.test.ts +++ b/packages/worker/src/mcp/capabilities/admin/admin-capabilities.node.test.ts @@ -20,6 +20,8 @@ type UserRow = { email: string stable_user_id: string plan?: string + suspended_at?: string | null + email_outbound_paused_at?: string | null created_at: string updated_at: string password_hash?: string @@ -32,6 +34,8 @@ function adminTestUser( return { ...input, plan: input.plan ?? 'free', + suspended_at: input.suspended_at ?? null, + email_outbound_paused_at: input.email_outbound_paused_at ?? null, stable_user_id: input.stable_user_id ?? testStableUserIdFromEmail(input.email), } @@ -176,7 +180,7 @@ function createAdminCapabilityTestDb(input: { } if ( normalizedQuery.includes( - 'select id, username, email, email_verified_at, plan, created_at, updated_at from users where id = ?', + 'select id, username, email, email_verified_at, plan, suspended_at, email_outbound_paused_at, created_at, updated_at from users where id = ?', ) ) { return (users.find((user) => user.id === params[0]) ?? @@ -203,7 +207,7 @@ function createAdminCapabilityTestDb(input: { } if ( normalizedQuery.includes( - 'select id, username, email, email_verified_at, plan, created_at, updated_at from users where email = ? collate nocase', + 'select id, username, email, email_verified_at, plan, suspended_at, email_outbound_paused_at, created_at, updated_at from users where email = ? collate nocase', ) ) { const email = String(params[0]).toLowerCase() @@ -279,7 +283,7 @@ function createAdminCapabilityTestDb(input: { async all() { if ( normalizedQuery.includes( - 'select id, username, email, email_verified_at, plan, created_at, updated_at from users order by id asc limit ? offset ?', + 'select id, username, email, email_verified_at, plan, suspended_at, email_outbound_paused_at, created_at, updated_at from users order by id asc limit ? offset ?', ) ) { const pageSize = Number(params[0]) diff --git a/packages/worker/src/mcp/capabilities/admin/admin-shared.ts b/packages/worker/src/mcp/capabilities/admin/admin-shared.ts index 21b46d6366..eb822085e3 100644 --- a/packages/worker/src/mcp/capabilities/admin/admin-shared.ts +++ b/packages/worker/src/mcp/capabilities/admin/admin-shared.ts @@ -36,6 +36,18 @@ export const adminUserMetadataSchema = z.object({ plan: planNameSchema.describe( 'Entitlement plan name. Unknown or unexpected NULL stored values resolve to max.', ), + suspended_at: z + .string() + .nullable() + .describe( + 'Set when the account is platform-suspended (blocked at session, MCP, and email chokepoints).', + ), + email_outbound_paused_at: z + .string() + .nullable() + .describe( + 'Set when outbound email was automatically paused after spam complaints or repeated bounces.', + ), created_at: z.string(), updated_at: z.string(), roles: z.array(roleNameSchema), diff --git a/packages/worker/src/mcp/capabilities/admin/admin-user-usage.ts b/packages/worker/src/mcp/capabilities/admin/admin-user-usage.ts index 9656efff55..7e86da7833 100644 --- a/packages/worker/src/mcp/capabilities/admin/admin-user-usage.ts +++ b/packages/worker/src/mcp/capabilities/admin/admin-user-usage.ts @@ -32,6 +32,8 @@ const entitlementResourceSchema = z.enum([ 'secrets', 'concurrent_workflows', 'storage_bytes', + 'execute_calls_per_day', + 'outbound_fetches_per_day', ]) const planSchema = z.enum(planNames) diff --git a/packages/worker/src/mcp/capabilities/openapi-provider/operation-request.ts b/packages/worker/src/mcp/capabilities/openapi-provider/operation-request.ts index a9eb07d9a2..788c8894ab 100644 --- a/packages/worker/src/mcp/capabilities/openapi-provider/operation-request.ts +++ b/packages/worker/src/mcp/capabilities/openapi-provider/operation-request.ts @@ -96,6 +96,7 @@ export async function executeOpenApiOperationRequest(input: { props: { baseUrl: input.baseUrl, userId: input.userId, + email: null, storageContext: input.storageContext, }, request, @@ -123,6 +124,7 @@ export async function executeOpenApiOperationRequest(input: { props: { baseUrl: input.baseUrl, userId: input.userId, + email: null, storageContext: input.storageContext, }, request: new Request(url.toString(), requestInit), @@ -418,6 +420,7 @@ async function tryRefreshIntegrationAccessToken(input: { props: { baseUrl: input.baseUrl, userId: input.userId, + email: null, storageContext: input.storageContext, }, request: new Request(input.integration.tokenUrl, { diff --git a/packages/worker/src/mcp/fetch-gateway.node.test.ts b/packages/worker/src/mcp/fetch-gateway.node.test.ts index ecc2ad0606..cb6467eab9 100644 --- a/packages/worker/src/mcp/fetch-gateway.node.test.ts +++ b/packages/worker/src/mcp/fetch-gateway.node.test.ts @@ -14,8 +14,49 @@ import * as secretService from '#mcp/secrets/service.ts' import * as communityRepo from '#worker/community/repo.ts' import * as packageRepo from '#worker/package-registry/repo.ts' +/** + * Minimal D1 stub for the entitlement reads/writes executeGatewayFetch + * performs (account reverse-resolution `first()` and the conditional + * counter upsert `run()`); secret resolution itself is spied at the + * service layer. Both statements must bind the acting userId — the stub + * fails loudly when a query drops the per-user scoping. + */ const env = { - APP_DB: {} as D1Database, + APP_DB: { + prepare(query: string) { + const normalizedQuery = query.replace(/\s+/g, ' ').trim().toLowerCase() + return { + bind(...params: Array) { + return { + async run() { + if ( + normalizedQuery.includes( + 'insert into entitlement_daily_counters', + ) && + params[0] !== 'user-123' + ) { + throw new Error( + 'Entitlement counter upsert must bind the acting userId.', + ) + } + return { meta: { changes: 1 } } + }, + async first() { + if ( + normalizedQuery.includes('where stable_user_id') && + params[0] !== 'user-123' + ) { + throw new Error( + 'Account reverse-resolution must bind the acting userId.', + ) + } + return null + }, + } + }, + } + }, + } as unknown as D1Database, COOKIE_SECRET: 'test-cookie-secret', SECRET_STORE_KEY: 'test-secret-store-key-32-chars-minimum', } @@ -23,6 +64,7 @@ const env = { const props = { baseUrl: 'https://example.com', userId: 'user-123', + email: null, storageContext: null, } diff --git a/packages/worker/src/mcp/fetch-gateway.ts b/packages/worker/src/mcp/fetch-gateway.ts index fe8d32c601..d646e17315 100644 --- a/packages/worker/src/mcp/fetch-gateway.ts +++ b/packages/worker/src/mcp/fetch-gateway.ts @@ -22,11 +22,22 @@ import { normalizeHost } from '#mcp/secrets/allowed-hosts.ts' import { resolveSecret, type ResolvedSecret } from '#mcp/secrets/service.ts' import { assertPackageCanAccessResolvedSecret } from '#mcp/secrets/package-access.ts' import { type StorageContext } from '#mcp/storage.ts' +import { + consumeDailyEntitlement, + findUserAccountByStableUserId, +} from '#worker/entitlements/service.ts' import { recordUsage, type UsageEnv } from '#worker/usage/record-usage.ts' type FetchGatewayProps = { baseUrl: string userId: string | null + /** + * Acting user's account email when the caller context carries one. + * Backs the entitlement plan lookup for the outbound-fetch quota; + * when absent, the gateway reverse-resolves the account from the + * stable userId so the caller's real plan still binds. + */ + email: string | null storageContext: StorageContext | null } export type { FetchGatewayProps } @@ -69,6 +80,33 @@ export async function executeGatewayFetch(input: { let meteredEntityId = readMeteredRequestHostname(input.request.url, null) try { + // Daily outbound-fetch quota: every sandbox fetch leaves through + // this gateway, so the atomic counter here bounds cost abuse and + // third-party hammering from user code. Consumed before secret + // expansion so over-limit requests never resolve secrets. + if (input.props.userId) { + // Plan lookup requires the account email. Callers that cannot + // carry one (OpenAPI provider requests, package runtime) get it + // reverse-resolved from the stable userId so authenticated + // fetches count against the caller's real plan rather than + // failing open to `max` (whose limit is still finite for + // genuinely accountless synthetic contexts). + const email = + input.props.email ?? + ( + await findUserAccountByStableUserId( + input.env.APP_DB, + input.props.userId, + ) + )?.email ?? + null + await consumeDailyEntitlement({ + db: input.env.APP_DB, + userId: input.props.userId, + email, + resource: 'outbound_fetches_per_day', + }) + } const transformed = await expandSecretPlaceholders({ request: input.request, props: input.props, diff --git a/packages/worker/src/mcp/fetch-gateway.workers.test.ts b/packages/worker/src/mcp/fetch-gateway.workers.test.ts new file mode 100644 index 0000000000..70a8678c0d --- /dev/null +++ b/packages/worker/src/mcp/fetch-gateway.workers.test.ts @@ -0,0 +1,85 @@ +import { env } from 'cloudflare:workers' +import { expect, test } from 'vitest' +import { utcDayKey } from '@kody-internal/shared/date-keys.ts' +import { executeGatewayFetch } from '#mcp/fetch-gateway.ts' +import { ensureEntitlementTestSchema } from '#worker/entitlements/test-schema.ts' +import { isEntitlementLimitError } from '#worker/entitlements/errors.ts' +import { planLimits } from '#worker/entitlements/plans.ts' +import { incrementDailyEntitlementCounter } from '#worker/entitlements/service.ts' +import { silenceIncidentalRuntimeWarnings } from '#worker/test-support/incidental-runtime-warnings.ts' +import { seedAccount } from '#worker/test-support/workers-seed.ts' +import { createStableUserIdFromEmail } from '#worker/user-id.ts' + +test('gateway fetches consume the daily outbound-fetch entitlement and deny over the plan limit', async () => { + silenceIncidentalRuntimeWarnings() + await ensureEntitlementTestSchema(env.APP_DB) + const email = `fetcher-${crypto.randomUUID()}@example.com` + const userId = await createStableUserIdFromEmail(email) + await seedAccount({ + db: env.APP_DB, + email, + username: `fetcher-${crypto.randomUUID().slice(0, 8)}`, + plan: 'free', + stableUserId: userId, + }) + const props = { + baseUrl: 'https://kody.example.com', + userId, + email, + storageContext: null, + } + const globalFetch = (async () => + new Response('ok')) as unknown as typeof fetch + + const allowed = await executeGatewayFetch({ + env, + props, + request: new Request('https://api.example.net/data'), + globalFetch, + }) + expect(await allowed.text()).toBe('ok') + const counter = await env.APP_DB.prepare( + `SELECT count FROM entitlement_daily_counters + WHERE user_id = ? AND resource = 'outbound_fetches_per_day' AND day = ?`, + ) + .bind(userId, utcDayKey()) + .first<{ count: number }>() + expect(Number(counter?.count)).toBe(1) + + // Fill the remaining free-plan quota, then expect denial. + await incrementDailyEntitlementCounter({ + db: env.APP_DB, + userId, + resource: 'outbound_fetches_per_day', + amount: planLimits.free.maxOutboundFetchesPerDay - 1, + }) + const denied = await executeGatewayFetch({ + env, + props, + request: new Request('https://api.example.net/data'), + globalFetch, + }).catch((error: unknown) => error) + expect(isEntitlementLimitError(denied)).toBe(true) + + // Callers that carry no email (OpenAPI provider requests, package + // runtime) still bind to the caller's real plan: the gateway + // reverse-resolves the account from the stable userId instead of + // failing open to the `max` quota. + const deniedWithoutEmail = await executeGatewayFetch({ + env, + props: { ...props, email: null }, + request: new Request('https://api.example.net/data'), + globalFetch, + }).catch((error: unknown) => error) + expect(isEntitlementLimitError(deniedWithoutEmail)).toBe(true) + + // Contextless fetches (no userId) are not metered against a user and + // pass through without consuming any counter. + const contextless = await executeGatewayFetch({ + env, + props: { ...props, userId: null, email: null }, + request: new Request('https://api.example.net/data'), + globalFetch, + }) + expect(await contextless.text()).toBe('ok') +}) diff --git a/packages/worker/src/mcp/run-kody-registry.ts b/packages/worker/src/mcp/run-kody-registry.ts index 5980c935e4..27786f8ab4 100644 --- a/packages/worker/src/mcp/run-kody-registry.ts +++ b/packages/worker/src/mcp/run-kody-registry.ts @@ -823,6 +823,7 @@ export async function runBundledModuleWithRegistry( gatewayProps: { baseUrl: callerContext.baseUrl, userId: callerContext.user?.userId ?? null, + email: callerContext.user?.email ?? null, storageContext: normalizedStorageContext, }, modules: hydratedModules, diff --git a/packages/worker/src/mcp/tools/execute.node.test.ts b/packages/worker/src/mcp/tools/execute.node.test.ts index 114d272fc5..1c65376cd4 100644 --- a/packages/worker/src/mcp/tools/execute.node.test.ts +++ b/packages/worker/src/mcp/tools/execute.node.test.ts @@ -40,6 +40,31 @@ vi.mock('#worker/openapi/binding-service.ts', () => ({ const { registerExecuteTool } = await import('./execute.ts') +/** + * Minimal env stub: the daily execute entitlement consumed at the top of + * the tool handler issues one conditional upsert (allowed when + * meta.changes > 0). Plan lookup never touches D1 because these caller + * contexts carry no account email (resolves to `max`). + */ +const stubEnv = { + APP_DB: { + prepare() { + return { + bind() { + return { + async run() { + return { meta: { changes: 1 } } + }, + async first() { + return null + }, + } + }, + } + }, + }, +} + const mockPerformanceNow = vi.spyOn(performance, 'now') function mockPerformanceSequence(...values: Array) { @@ -75,7 +100,7 @@ async function getExecuteRegistration( server: { registerTool, } as never, - getEnv: vi.fn(() => ({})), + getEnv: vi.fn(() => stubEnv), getCallerContext: vi.fn(() => callerContext), requireDomain: vi.fn(), getLoopbackExports: vi.fn(), @@ -365,7 +390,7 @@ test('execute tool serializes successes and errors, binds storage, passes packag }) expect(mockModule.createExecutePackageInvokeTools).toHaveBeenCalledWith({ - env: {}, + env: stubEnv, baseUrl: 'https://example.com', callerContext: expect.objectContaining(callerContext), conversationId: 'conv-packages', diff --git a/packages/worker/src/mcp/tools/execute.ts b/packages/worker/src/mcp/tools/execute.ts index 4344f0bff3..fab14336d6 100644 --- a/packages/worker/src/mcp/tools/execute.ts +++ b/packages/worker/src/mcp/tools/execute.ts @@ -49,6 +49,7 @@ import { } from '#mcp/raw-fetch-host-nudge.ts' import { listOpenApiBindings } from '#worker/openapi/binding-service.ts' import { normalizeHost } from '#mcp/secrets/allowed-hosts.ts' +import { consumeDailyEntitlement } from '#worker/entitlements/service.ts' const executeTool = { name: 'execute', @@ -185,6 +186,18 @@ export async function registerExecuteTool(agent: McpRegistrationAgent) { } async function runExecuteTool() { + // Daily execute quota, consumed before any bundling or sandbox + // work so over-limit calls cost nothing. The thrown + // EntitlementLimitError propagates through the outer catch as + // a structured MCP error. + if (callerContext.user?.userId) { + await consumeDailyEntitlement({ + db: env.APP_DB, + userId: callerContext.user.userId, + email: callerContext.user.email, + resource: 'execute_calls_per_day', + }) + } const { getCapabilityRegistryForContext } = await import('#mcp/capabilities/registry.ts') const registry = await getCapabilityRegistryForContext({ diff --git a/packages/worker/src/package-registry/test-schema.ts b/packages/worker/src/package-registry/test-schema.ts index 56d62559bc..9a9fcd3725 100644 --- a/packages/worker/src/package-registry/test-schema.ts +++ b/packages/worker/src/package-registry/test-schema.ts @@ -19,6 +19,8 @@ export async function ensurePackageScopeGrantsTestSchema(db: D1Database) { account_type TEXT NOT NULL DEFAULT 'person' CHECK (account_type IN ('person', 'platform')), plan TEXT NOT NULL DEFAULT 'free', deleting_at TEXT, + suspended_at TEXT, + email_outbound_paused_at TEXT, active_write_count INTEGER NOT NULL DEFAULT 0, active_write_expires_at TEXT, created_at TEXT NOT NULL DEFAULT (CURRENT_TIMESTAMP), @@ -32,6 +34,8 @@ export async function ensurePackageScopeGrantsTestSchema(db: D1Database) { `account_type TEXT NOT NULL DEFAULT 'person'`, `plan TEXT NOT NULL DEFAULT 'free'`, 'deleting_at TEXT', + 'suspended_at TEXT', + 'email_outbound_paused_at TEXT', 'active_write_count INTEGER NOT NULL DEFAULT 0', 'active_write_expires_at TEXT', ]) { diff --git a/packages/worker/src/package-runtime/package-app.ts b/packages/worker/src/package-runtime/package-app.ts index 6c9b95bd71..4e7c8b63de 100644 --- a/packages/worker/src/package-runtime/package-app.ts +++ b/packages/worker/src/package-runtime/package-app.ts @@ -1498,6 +1498,7 @@ async function buildPackageAppWorkerOptionsUncached(input: { props: { baseUrl: input.baseUrl, userId: input.userId, + email: input.runtime.callerContext.user?.email ?? null, storageContext: { sessionId: null, appId: input.savedPackage.id, diff --git a/tools/check-migrations.node.test.ts b/tools/check-migrations.node.test.ts index 974d3e2950..7cc4e1a861 100644 --- a/tools/check-migrations.node.test.ts +++ b/tools/check-migrations.node.test.ts @@ -139,8 +139,8 @@ test('migration ledger rejects historical mutation and lower-prefix additions wh ).toMatchObject({ ok: true, errors: [], - nextPrefix: '0091', - maxPrefix: 90, + nextPrefix: '0092', + maxPrefix: 91, }) const modifiedFiles = baselineFiles.map((entry) => @@ -199,7 +199,7 @@ test('migration ledger rejects historical mutation and lower-prefix additions wh ]) const monotonicAddition: MigrationLedgerEntry = { - filename: '0091-normal-addition.sql', + filename: '0092-normal-addition.sql', sha256: '2'.repeat(64), } const ledgerWithMonotonicAddition = structuredClone(ledger) @@ -213,8 +213,8 @@ test('migration ledger rejects historical mutation and lower-prefix additions wh ).toMatchObject({ ok: true, errors: [], - nextPrefix: '0092', - maxPrefix: 91, + nextPrefix: '0093', + maxPrefix: 92, }) }) @@ -222,11 +222,11 @@ test('trusted history rejects a migration and ledger digest co-edit', async () = const ledger = await readMigrationLedger() const bootstrapFiles = ledger.migrations.map((entry) => ({ ...entry })) const historicalEntry = { - filename: '0091-historical.sql', + filename: '0092-historical.sql', sha256: hashMigrationContent('SELECT 1;\n'), } const trustedHistory: TrustedMigrationHistory = { - ref: 'trusted-base-with-0091', + ref: 'trusted-base-with-0092', files: [...bootstrapFiles, historicalEntry], ledgerEntries: [...bootstrapFiles, historicalEntry], } @@ -244,7 +244,7 @@ test('trusted history rejects a migration and ledger digest co-edit', async () = ) expect(result.ok).toBe(false) expectErrorsMention(result.errors, [ - '0091-historical.sql', + '0092-historical.sql', 'Historical ledger entries cannot be edited', 'differs from trusted history', 'Migration and ledger digests cannot be changed together', diff --git a/tools/migration-ledger.json b/tools/migration-ledger.json index 344f743a90..e3eff671fd 100644 --- a/tools/migration-ledger.json +++ b/tools/migration-ledger.json @@ -387,6 +387,10 @@ { "filename": "0090-webhook-endpoints.sql", "sha256": "59e680a93e93300b520ddd87ff6d86da24a97a33d651fe10ff7a842b66cfe355" + }, + { + "filename": "0091-user-abuse-controls.sql", + "sha256": "a831d51f11e539c57283947fd78917f719cc5ca33b023af2f8beeec84bed53a3" } ] }