Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 7 additions & 13 deletions docs/contributing/architecture/data-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -709,11 +709,13 @@ on write unless a migration backfills existing rows.
(`0028-published-bundle-artifacts-and-archived-jobs.sql`) stores package
dependency pointers queried with SQLite JSON functions in
`packages/worker/src/repo/published-bundle-artifacts-repo.ts`.
- `package_invocations.package_ids_json`,
`package_invocations.package_kody_ids_json`,
`package_invocations.export_names_json`, `package_invocations.sources_json`,
and `package_invocations.response_json` (`0029-package-invocations.sql`) store
invocation routing and cached response projections.
- `package_invocation_tokens.package_ids_json`,
`package_invocation_tokens.package_kody_ids_json`,
`package_invocation_tokens.export_names_json`, and
`package_invocation_tokens.sources_json` (`0029-package-invocations.sql`)
store invocation-token scope projections. The sibling `package_invocations`
table from that migration was dropped (`0112-drop-package-invocations.sql`);
its replay cache lives in the RunLog Durable Object ledger now.
- `email_messages.*_addresses_json`, `email_messages.references_json`,
`email_messages.headers_json`, and `email_delivery_events.detail_json`
(`0030-email-primitives.sql`, `0031-unified-email-receipt.sql`,
Expand Down Expand Up @@ -923,14 +925,6 @@ documented exemption.

Current retention policies:

- `package_invocations`: **legacy dual-read window.** Keyed invocations now
claim and store replay responses in the per-user `RunLog` Durable Object
ledger (`package_invocation_ledger` table inside the DO), which self-enforces
the same 90-day terminal-row window; the keyed path only reads this D1 table
as a fallback for pre-migration keys and never writes it. The sweep keeps
draining pre-migration terminal rows (rows with `status = 'in_progress'` are
never pruned) until a follow-up drops the table, the fallback read, and this
policy entry.
- `mcp_memory_conversation_suppressions`: keep active suppressions and prune
expired rows only after they have not been seen for 90 days. The existing
request-time memory prune may remove expired rows sooner.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,8 @@ call, not about what user code does inside the call.
lives in the per-user `RunLog` Durable Object (see
[Run records](./run-records.md)), so the durability cost is **one awaited DO
call for claim + run-record begin and one for terminal response + run-record
finish** — not D1 round trips. During the dual-read window the keyed path also
pays one awaited D1 **read** for pre-migration keys; a follow-up removes it
with the legacy `package_invocations` table.
finish** — no D1 round trips at all (the legacy `package_invocations` table
and its dual-read fallback are gone).

## Watch the percentiles

Expand Down
13 changes: 7 additions & 6 deletions docs/contributing/architecture/run-records.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,12 +221,13 @@ means a replayed delivery for that key re-executes instead of replaying.
overwritten.
- **Ledger retention is DO-local**: terminal rows keep replay responses for 90
days (`packageInvocationLedgerRetentionDays`), pruned by the same retention
passes and alarm as run rows; `in_progress` rows are never pruned. The old D1
retention sweep remains only to drain pre-migration rows.
- **Dual-read window**: the keyed path reads the DO first and falls back to a
read-only D1 `package_invocations` lookup for keys claimed before the
migration. Writes go only to the DO. A follow-up removes the fallback, the D1
table, its sweep, and the legacy row shapes.
passes and alarm as run rows; `in_progress` rows are never pruned. There is no
D1 sweep — the legacy table is gone.
- **The DO is the only store**: the legacy D1 `package_invocations` table was
dropped (`0112-drop-package-invocations.sql`) after the dual-read window was
deliberately waived, and the keyed path performs no D1 ledger reads or writes.
Keys claimed before the DO migration no longer replay; a redelivery for such a
key executes fresh, exactly like a new key.
- Account export pages ledger rows through the same `run_records` section cursor
(runs first, then ledger rows); account deletion purges them with `clearAll`.
Disaster recovery deliberately does not stage the DO — losing it risks
Expand Down
22 changes: 11 additions & 11 deletions docs/contributing/disaster-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,17 +193,17 @@ Restore rebuilds these; do not treat them as recovery media:

**Known risk — keyed invocation idempotency ledger:** the same `RunLog` DO
also holds the keyed package-invocation idempotency ledger (claims + 90-day
terminal replay responses; migrated off the D1 `package_invocations` table).
Losing the DO therefore loses idempotency/replay state, not just
observability: webhook providers and other keyed callers that retry deliveries
would **re-execute** invocations whose keys were already terminal, and
in-flight replays would return fresh executions instead of stored responses.
This is accepted with eyes open: lost rows are gone — later traffic
re-establishes claims only for keys used after the loss, it does not restore
protection for keys used before it. The blast radius is duplicate side effects
(a loss of correctness state) bounded by the 90-day retention window; no
stored user content is lost. Restores from a D1 backup restore the legacy
table only; the DO ledger is not part of DR media.
terminal replay responses; the legacy D1 `package_invocations` table was
dropped, so the DO is the only store). Losing the DO therefore loses
idempotency/replay state, not just observability: webhook providers and other
keyed callers that retry deliveries would **re-execute** invocations whose
keys were already terminal, and in-flight replays would return fresh
executions instead of stored responses. This is accepted with eyes open: lost
rows are gone — later traffic re-establishes claims only for keys used after
the loss, it does not restore protection for keys used before it. The blast
radius is duplicate side effects (a loss of correctness state) bounded by the
90-day retention window; no stored user content is lost. The DO ledger is not
part of DR media, and D1 backups no longer carry any invocation ledger at all.

### Honest gaps

Expand Down
8 changes: 8 additions & 0 deletions packages/worker/migrations/0112-drop-package-invocations.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-- Retire the legacy keyed package-invocation idempotency ledger. The ledger
-- moved into the per-user RunLog Durable Object (see
-- docs/contributing/architecture/run-records.md); writes stopped with that
-- migration and the dual-read fallback was removed together with this drop.
-- Pre-migration keys stop replaying and simply execute fresh on redelivery
-- (explicitly accepted when the dual-read window was waived).
-- package_invocation_tokens is unrelated auth state and stays.
DROP TABLE IF EXISTS package_invocations;
1 change: 0 additions & 1 deletion packages/worker/src/account/data-targets.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,6 @@ export function getAccountExportExcludedD1Surfaces(): Array<{
* tables) are not represented.
*/
export const accountUserDataTargets: ReadonlyArray<UserScopedDataTarget> = [
{ kind: 'user_id', table: 'package_invocations' },
{ kind: 'user_id', table: 'package_invocation_tokens' },
{ kind: 'user_id', table: 'workflow_runs' },
{ kind: 'user_id', table: 'package_service_states' },
Expand Down
1 change: 0 additions & 1 deletion packages/worker/src/app/account-deletion.node.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -801,7 +801,6 @@ test('deleteUserAccount cascades user-scoped rows for the requested user', async
],
mcp_user_server_instructions: [{ user_id: userAaa }],
package_invocation_tokens: [{ id: 'pit-1', user_id: userAaa }],
package_invocations: [{ id: 'pi-1', user_id: userAaa }],
agent_package_conversation_uses: [
{
user_id: userAaa,
Expand Down
1 change: 0 additions & 1 deletion packages/worker/src/app/account-retention-dispositions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ export type AccountRetentionDisposition =

export const accountRetentionDispositions: ReadonlyArray<AccountRetentionDisposition> =
[
{ table: 'package_invocations', kind: 'scheduled_policy' },
{
table: 'mcp_memory_conversation_suppressions',
kind: 'scheduled_policy',
Expand Down
Loading
Loading