Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 8 additions & 11 deletions docs/contributing/architecture/data-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -568,17 +568,14 @@ Storage split:
workspaces for the check-only `storage_bytes` baseline. Rows use `kind` to
dispatch `getEstimatedBytes` to the correct Durable Object, cache the latest
`databaseSize` in `estimated_bytes`, and feed the same bounded
`storage_bucket_estimate_backfill` lane. Leftover retired `kind = 'service'`
rows stay inventoried until that lane clears the matching StorageRunner Durable
Objects and then deletes the rows, so account export, deletion, and DR keep a
user→storage_id map. Estimate reads treat those leftover rows as StorageRunner
(`unknown`). Index-backed repo-session reconciliation pages
`repo_session_due_owners` with a per-tick owner budget and a platform-owned
`repo_session_storage_bucket_cursor` so a sweep cannot walk the whole fleet.
Repo sessions register on open, opportunistically refresh after workspace
mutations, and remove the inventory row on discard, purge, scheduled cleanup,
source deletion, or account deletion. This estimate component is composed with
the authoritative UserMeter D1 payload bytes; it is not reserved into UserMeter.
`storage_bucket_estimate_backfill` lane. Index-backed repo-session
reconciliation pages `repo_session_due_owners` with a per-tick owner budget and
a platform-owned `repo_session_storage_bucket_cursor` so a sweep cannot walk the
whole fleet. Repo sessions register on open, opportunistically refresh after
workspace mutations, and remove the inventory row on discard, purge, scheduled
cleanup, source deletion, or account deletion. This estimate component is
composed with the authoritative UserMeter D1 payload bytes; it is not reserved
into UserMeter.

## Durable Objects (`UserMeter`)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -174,11 +174,14 @@ rejects a same-deploy `deleted_classes` migration while that binding still
exists (error 10061). Existing objects also require the script to keep exporting
the class until `deleted_classes` runs (error 10064). Export a stub, drop the
binding, then add the `deleted_classes` migration and its
`tools/ci/do-deletion-allowlist.json` entry on a later deploy.

`PackageServiceInstance` uses that sequence: production `kody-runtime` applies
tag `v2` after a stub-export deploy dropped the transferred binding. Preview
applies `v1` `new_sqlite_classes` then `v2` `deleted_classes` on first deploy.
`tools/ci/do-deletion-allowlist.json` entry on a later deploy. Preview
`deleted_classes` requires a previous script version that exported the class
(error 10074). A first preview deploy can apply the create and delete tags
together so they elide.

`PackageServiceInstance` is gone from production `kody-runtime` (tag `v2`; no
stub export). Preview applies `v1` `new_sqlite_classes` then `v2`
`deleted_classes` on first deploy so create and delete elide.

## Rollback

Expand Down
32 changes: 17 additions & 15 deletions docs/contributing/decisions/0025-no-package-services-primitive.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,18 +30,20 @@ subscriptions, and package-owned storage.
Integrations that require a continuously connected daemon need an external
deployment and communicate with Kody through supported integration surfaces.

Leftover `kind = 'service'` `user_storage_buckets` rows stay until the
`storage_bucket_estimate_backfill` lane clears the matching StorageRunner
Durable Objects and then deletes the inventory. Account export and deletion keep
discovering those storage ids until that purge succeeds.

Deleting `PackageServiceInstance` on production `kody-runtime` is two deploys.
The class arrived through a `transferred_classes` migration, so Cloudflare still
has a remote binding and existing objects after the source binding is gone. A
same-deploy `deleted_classes` migration fails with error 10061. Dropping the
binding without exporting the class fails with error 10064. Export a stub, drop
the remote binding, then apply runtime-worker tag `v2` `deleted_classes` and
remove the stub. #1559 completed the stub-and-drop-binding deploy. The
`tools/ci/do-deletion-allowlist.json` entry for tag `v2` is the contract for the
class-delete follow-up. Preview `deleted_classes` requires a previous script
version that exported the class (error 10074).
Leftover `kind = 'service'` `user_storage_buckets` rows stayed until the
`storage_bucket_estimate_backfill` lane cleared the matching StorageRunner
Durable Objects and then deleted the inventory. Production `APP_DB` had zero
leftover rows on 2026-08-19. #1565 removes the purge lane and tightens the live
CHECK so `'service'` cannot return.

Deleting `PackageServiceInstance` on production `kody-runtime` required two
deploys after #1552. The class arrived through a `transferred_classes`
migration, so Cloudflare still had a remote binding and existing objects after
the source binding was gone. A same-deploy `deleted_classes` migration failed
with error 10061 (#1552 deploy). Dropping the binding without exporting the
class failed with error 10064 (#1558). #1559 exported a stub and dropped the
remote binding. #1560 applied runtime-worker tag `v2` `deleted_classes` and
removed the stub. Preview `deleted_classes` requires a previous script version
that exported the class (error 10074); a first preview deploy applies `v1`
`new_sqlite_classes` and `v2` `deleted_classes` together so create and delete
elide.
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
-- Production APP_DB has zero leftover kind='service' inventory (confirmed
-- 2026-08-19). Tighten the CHECK so the retired kind cannot return. Fail
-- closed if a leftover row appears: the INSERT into the next table rejects
-- 'service'.

CREATE TABLE user_storage_buckets_next (
user_id TEXT NOT NULL,
storage_id TEXT NOT NULL,
kind TEXT NOT NULL CHECK (
kind IN ('job', 'package', 'execute', 'repo_session', 'unknown')
),
created_at TEXT NOT NULL,
last_seen_at TEXT NOT NULL,
estimated_bytes INTEGER,
estimated_bytes_updated_at TEXT,
PRIMARY KEY (user_id, storage_id)
);

INSERT INTO user_storage_buckets_next (
user_id,
storage_id,
kind,
created_at,
last_seen_at,
estimated_bytes,
estimated_bytes_updated_at
)
SELECT
user_id,
storage_id,
kind,
created_at,
last_seen_at,
estimated_bytes,
estimated_bytes_updated_at
FROM user_storage_buckets;

DROP TABLE user_storage_buckets;

ALTER TABLE user_storage_buckets_next RENAME TO user_storage_buckets;

CREATE INDEX idx_user_storage_buckets_user
ON user_storage_buckets(user_id);
28 changes: 0 additions & 28 deletions packages/worker/src/storage-buckets/estimate-backfill.node.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,13 @@ const mockModule = vi.hoisted(() => ({
listStorageBucketsMissingEstimates: vi.fn(),
registerMissingRepoSessionStorageBuckets: vi.fn(async () => 0),
updateStorageBucketEstimate: vi.fn(async () => true),
purgeRetiredServiceStorageBuckets: vi.fn(async () => ({
scanned: 0,
purged: 0,
failed: 0,
})),
}))

vi.mock('#worker/storage-runner.ts', () => ({
readInventoriedStorageBucketEstimatedBytes: (...args: Array<unknown>) =>
mockModule.readInventoriedStorageBucketEstimatedBytes(...args),
}))

vi.mock('./purge-retired-service-storage.ts', () => ({
purgeRetiredServiceStorageBuckets: (...args: Array<unknown>) =>
mockModule.purgeRetiredServiceStorageBuckets(...args),
}))

vi.mock('./service.ts', () => ({
listStorageBucketsMissingEstimates: (...args: Array<unknown>) =>
mockModule.listStorageBucketsMissingEstimates(...args),
Expand Down Expand Up @@ -61,10 +51,6 @@ test('backfill tolerates per-bucket probe failures and keeps sweeping peers', as
updated: 2,
failed: 1,
})
expect(mockModule.purgeRetiredServiceStorageBuckets).toHaveBeenCalledWith({
env,
limit: 24,
})

// The failing bucket is logged and left NULL for a later sweep; the two
// healthy buckets are persisted.
Expand All @@ -88,18 +74,4 @@ test('backfill tolerates per-bucket probe failures and keeps sweeping peers', as
estimatedBytes: 2048,
}),
)

mockModule.purgeRetiredServiceStorageBuckets.mockRejectedValueOnce(
new Error('purge boom'),
)
mockModule.listStorageBucketsMissingEstimates.mockResolvedValue([])
await expect(backfillStorageBucketEstimates({ env })).resolves.toEqual({
scanned: 0,
updated: 0,
failed: 0,
})
expect(consoleWarn).toHaveBeenCalledWith(
'retired-service-storage-purge-failed',
expect.any(Error),
)
})
14 changes: 0 additions & 14 deletions packages/worker/src/storage-buckets/estimate-backfill.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import { purgeRetiredServiceStorageBuckets } from './purge-retired-service-storage.ts'
import {
listStorageBucketsMissingEstimates,
registerMissingRepoSessionStorageBuckets,
Expand All @@ -18,10 +17,6 @@ import { readInventoriedStorageBucketEstimatedBytes } from '#worker/storage-runn
* retried on a later sweep; it never blocks the rest of the batch. The lane
* uses a short retry policy (the write path carries the strong one) because
* sweep retries are free.
*
* The same tick also clears leftover retired `kind = 'service'` StorageRunner
* Durable Objects and then deletes those inventory rows. D1 cannot purge
* Durable Objects, so migration 0016 keeps the rows until this succeeds.
*/

export const storageBucketEstimateBackfillBatchSize = 24
Expand All @@ -34,15 +29,6 @@ export async function backfillStorageBucketEstimates(input: {
batchSize?: number
}): Promise<{ scanned: number; updated: number; failed: number }> {
const batchSize = input.batchSize ?? storageBucketEstimateBackfillBatchSize
// Leftover kind='service' inventory is the only map to retired
// StorageRunner DOs. Clear those objects, then drop the rows. Failures
// stay inventoried so export and account deletion still find them.
await purgeRetiredServiceStorageBuckets({
env: input.env,
limit: batchSize,
}).catch((error: unknown) => {
console.warn('retired-service-storage-purge-failed', error)
})
// Recover active repo sessions whose awaited open-time registration lost
// its D1 write (registration is deliberately non-blocking for the
// session). Race-safe: the statement filters on active status, so it can
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@ import { repoSessionRpc } from '#worker/repo/repo-session-rpc.ts'
import { backfillStorageBucketEstimates } from './estimate-backfill.ts'
import {
clearStorageBucketRegistrationDedupeForTests,
flushStorageBucketRegistrationsForTests,
listUserStorageBucketEstimates,
listUserStorageBucketIds,
registerStorageBucket,
} from './service.ts'
import { ensureUserStorageBucketsTestSchema } from './test-schema.ts'
Expand Down Expand Up @@ -125,74 +123,3 @@ test(
})
},
)

test(
'backfill clears leftover service StorageRunner objects before dropping inventory',
{ timeout: testTimeout },
async () => {
await ensureUserStorageBucketsTestSchema(env.APP_DB)
clearStorageBucketRegistrationDedupeForTests()
const userId = `usb-service-purge-${crypto.randomUUID()}`
const liveBucket = `package:${crypto.randomUUID()}`
const leftoverBucket = `service:${crypto.randomUUID()}`
const now = new Date().toISOString()
await env.APP_DB.prepare(
`INSERT INTO user_storage_buckets (
user_id, storage_id, kind, created_at, last_seen_at,
estimated_bytes
) VALUES (?, ?, 'service', ?, ?, NULL), (?, ?, 'package', ?, ?, ?)`,
)
.bind(
userId,
leftoverBucket,
now,
now,
userId,
liveBucket,
now,
now,
emptyStorageRunnerEstimatedBytes,
)
.run()
await storageRunnerRpc({
env,
userId,
storageId: leftoverBucket,
}).setValue({ key: 'leftover', value: { kept: true } })
await flushStorageBucketRegistrationsForTests()
await expect(
storageRunnerRpc({
env,
userId,
storageId: leftoverBucket,
}).getValue({ key: 'leftover' }),
).resolves.toEqual({
key: 'leftover',
value: { kept: true },
})
await expect(listUserStorageBucketIds({ env, userId })).resolves.toEqual(
[liveBucket, leftoverBucket].sort(),
)

await backfillStorageBucketEstimates({ env })
await expect(listUserStorageBucketIds({ env, userId })).resolves.toEqual([
liveBucket,
])
await expect(
storageRunnerRpc({
env,
userId,
storageId: leftoverBucket,
}).getValue({ key: 'leftover' }),
).resolves.toEqual({ key: 'leftover', value: null })
await expect(
listUserStorageBucketEstimates({ env, userId }),
).resolves.toEqual([
{
storageId: liveBucket,
kind: 'package',
estimatedBytes: emptyStorageRunnerEstimatedBytes,
},
])
},
)
Loading
Loading