Skip to content

Remove the package services primitive #1552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kody-bot merged 8 commits into from
Aug 19, 2026
Merged

Remove the package services primitive #1552

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
922b71f
Remove the package services primitive.
cursoragent Aug 19, 2026
e07a0e6
Reject leftover kody.services and scrub remaining copy.
cursoragent Aug 19, 2026
629fd48
Merge main into the package-services removal branch.
cursoragent Aug 19, 2026
ef12b89
Purge leftover service StorageRunner inventory instead of dropping it.
cursoragent Aug 19, 2026
42ace7d
Update the 0016 migration ledger digest after keeping service inventory.
cursoragent Aug 19, 2026
b1b8137
Format leftover service storage purge docs and tests.
cursoragent Aug 19, 2026
ddcddf9
Retrigger CI after leftover service storage purge.
cursoragent Aug 19, 2026
033c99e
Merge main into the package-services removal branch.
cursoragent Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
153 changes: 57 additions & 96 deletions docs/contributing/architecture/data-storage.md
View file
Open in desktop

Large diffs are not rendered by default.

160 changes: 58 additions & 102 deletions docs/contributing/architecture/entitlements.md
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions docs/contributing/architecture/invocation-overhead-guardrails.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,11 @@ Analytics Engine dataset with per-surface `eventType` values. The mapping that
matters here: keyless `packages.invoke` target runs are saved-package bundled
runs, so they land under the `package_export` event type (`entityId` = package
id), while the calling surface reports its own event type (`execute`, `job_run`,
`workflow_run`, `service_runtime`). When touching an invocation path, watch the
`durationMs` percentiles (p50/p95/p99) of the affected event types before and
after the change rather than reasoning from a single local timing. A regression
in the `package_export` or `execute` percentiles is a release blocker for the
change that caused it, not a follow-up.
`workflow_run`). When touching an invocation path, watch the `durationMs`
percentiles (p50/p95/p99) of the affected event types before and after the
change rather than reasoning from a single local timing. A regression in the
`package_export` or `execute` percentiles is a release blocker for the change
that caused it, not a follow-up.

## New awaited D1 writes need a budget justification

Expand Down
15 changes: 3 additions & 12 deletions docs/contributing/architecture/primitives.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,15 +267,6 @@ primitives:
docs:
- docs/use/packages.md

- id: package-services
group: assistant
name: Package services
summary: Long-lived headless package-owned runtimes.
code:
- packages/worker/src/package-runtime/package-service.ts
docs:
- docs/contributing/architecture/request-lifecycle.md

- id: jobs
group: assistant
name: Scheduled jobs
Expand Down Expand Up @@ -544,9 +535,9 @@ primitives:
name: User meter
summary:
Per-user UserMeter DO SQLite — authoritative for daily entitlement
counters, storage bytes, inbound delivery idempotency, package-service
running counts, and account-deletion write leases; D1 keeps the deleting
point gate and lease-repair audit log.
counters, storage bytes, inbound delivery idempotency, and
account-deletion write leases; D1 keeps the deleting point gate and
lease-repair audit log.
code:
- packages/worker/src/entitlements/user-meter-do.ts
- packages/worker/src/entitlements/user-meter-client.ts
Expand Down
22 changes: 0 additions & 22 deletions docs/contributing/architecture/request-lifecycle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,28 +97,6 @@ Requests are handled in this order:
those documents and serves markdown when `Accept` prefers `text/markdown`.
DNS-AID (`_mcp._agents.<apex>` SVCB/HTTPS) is zone DNS, not a Worker route.

## Package service runtime

Saved packages may also declare long-lived package services under
`package.json#kody.services`.

- Package services are **not** routed through a public HTTP path the way package
apps are.
- Instead, the Worker hosts them via the `PackageServiceInstance` Durable Object
binding and controls them through package runtime bridges and MCP
capabilities.
- Package services share package identity with apps/jobs and can publish into
package app realtime sessions, but they own their own durable storage bucket
and lifecycle.
- A running service is kept resident with a short incoming alarm (about 15s).
Background `waitUntil` work does not keep the Durable Object alive by itself;
without that wake, Cloudflare evicts the isolate and orphans the sandbox,
including any outbound WebSocket.
- `mode: 'persistent'` means stay up until stopped. Eviction is treated as a
platform interrupt: the host immediately reschedules the same service, even
when `autoStart` is false. Persist resume cursors (`session_id`, sequence) in
`packageStorage()` so the next run can continue.

## Workflow runtime

All server-side Kody runtime contexts expose `workflows` from `kody:runtime`.
Expand Down
42 changes: 17 additions & 25 deletions docs/contributing/architecture/run-records.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ via `run_update`). The account UI is `/account/activity`.
| `subscription` | Package subscription handler dispatch |
| `app_fetch` | Package app HTTP fetch handler |
| `app_realtime` | Package app realtime websocket session |
| `service` | Package service run (bounded or persistent) |
| `job` | Scheduled or manually triggered job execution |
| `workflow` | Cloudflare Workflow run (`DynamicCallableWorkflow`) |
| `retriever` | Package retriever evaluation |
Expand Down Expand Up @@ -250,27 +249,21 @@ of replaying.

## Invariant: state vs history

Entity rows hold **current state**. Run records hold **history**. Never derive
live state (is this service running? how many?) by querying run records.

Entitlement concurrency reads the per-user UserMeter `package_service_states`,
the sole liveness authority. D1 `package_service_states` is only an enumeration
index for discovery, account export/deletion, and disaster recovery; run history
is never a liveness signal. Jobs keep schedule metadata and `last_run_at` /
`last_run_status` on the D1 `jobs` row for the hourly retention sweeper;
last-run error, duration, and counters for observability live in RunLog
`job_run_observability` and survive run-history pruning. Package activation
counters and milestones live in the same DO (`package_run_successes`,
`activation_milestones`).
Entity rows hold **current state**. Run records hold **history**. Jobs keep
schedule metadata and `last_run_at` / `last_run_status` on the D1 `jobs` row for
the hourly retention sweeper; last-run error, duration, and counters for
observability live in RunLog `job_run_observability` and survive run-history
pruning. Package activation counters and milestones live in the same DO
(`package_run_successes`, `activation_milestones`).

## Recipe: instrumenting a new surface

Modelled on the
[usage-metering chokepoint recipe](./usage-metering.md#recipe-instrumenting-a-new-chokepoint).

1. **Pick the semantic unit.** One run record per user-visible attempt (one job
execution, one webhook delivery, one service wake). Nested layers may each
record their own surface; do not double-write the same surface for one unit.
execution or one webhook delivery). Nested layers may each record their own
surface; do not double-write the same surface for one unit.
2. **Add the `RunSurface` member** to `runSurfaceValues` in
`packages/worker/src/run-records/types.ts` if it does not exist. Choose
persistence in `runPersistenceForSurface` (`eager` unless the caller already
Expand Down Expand Up @@ -332,13 +325,13 @@ Modelled on the

## Neighboring systems

| System | Answers | Store |
| ------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| **Run records** (this doc) | What failed, with logs, for one user’s runs | Per-user `RunLog` DO SQLite |
| **Usage metering** | How many / how long / aggregate cost pressure | Analytics Engine + D1 `usage_rollups` |
| **Sentry** | Platform defects operators should fix | Sentry project |
| **Entity state columns/tables** | Current status (job last-run error/duration/counters, service liveness) | RunLog `job_run_observability`; D1 `jobs` schedule/retention anchors; `package_service_states` |
| **Package subscriptions** | Same-user reaction to terminal errors | Best-effort dispatch after `finishRun` |
| System | Answers | Store |
| ------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------ |
| **Run records** (this doc) | What failed, with logs, for one user’s runs | Per-user `RunLog` DO SQLite |
| **Usage metering** | How many / how long / aggregate cost pressure | Analytics Engine + D1 `usage_rollups` |
| **Sentry** | Platform defects operators should fix | Sentry project |
| **Entity state columns/tables** | Current job status and counters | RunLog `job_run_observability`; D1 `jobs` schedule and retention anchors |
| **Package subscriptions** | Same-user reaction to terminal errors | Best-effort dispatch after `finishRun` |

After a successful terminal `finishRun` with `status: 'error'`,
`finishRunRecord` best-effort dispatches `run.error.recorded` to the owning
Expand Down Expand Up @@ -366,9 +359,8 @@ user.
**Entity state** stays on the entity (or its dedicated projection table). Job
last-run error, duration, and counters live in RunLog `job_run_observability`;
D1 `jobs` keeps schedule fields and `last_run_at` / `last_run_status` as
retention anchors only. Package services heartbeat `package_service_states`.
History browsers (`/account/activity`, `run_list` / `run_get` / `run_summary`)
read `RunLog`.
retention anchors only. History browsers (`/account/activity`, `run_list` /
`run_get` / `run_summary`) read `RunLog`.

## Soft error triage

Expand Down
31 changes: 15 additions & 16 deletions docs/contributing/architecture/runtime-worker-migration-runbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,25 @@ How to move the package runtime lane from the main `kody` Worker into the
`kody-runtime` Worker (`packages/runtime-worker/`) in production, per
[ADR 0016](../decisions/0016-mono-worker-extraction.md). The genuinely risky
step is the one-time Durable Object **script migration**: the storage of
`StorageRunner`, `RunLog`, `PackageRealtimeSession`, and
`PackageServiceInstance` moves from the `kody` script to the `kody-runtime`
script via a Wrangler `transferred_classes` migration.
`StorageRunner`, `RunLog`, and `PackageRealtimeSession` moves from the `kody`
script to the `kody-runtime` script via a Wrangler `transferred_classes`
migration.

> **Warning:** the implementation session that authored this document must NOT
> execute these production steps. The runbook is executed by an operator (the
> parent session) after preview verification and PR merge.

## Ownership after the split

| Concern | Owner |
| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| Package-app origin (`PACKAGE_APP_BASE_URL`, `kody.run`) zone routes | `kody-runtime` |
| Inline package-app serving (`/apps/...` on the app origin) | `kody-runtime` (forwarded by main via the `RUNTIME_WORKER` service binding) |
| Package invocation API | `kody-runtime` (forwarded by main) |
| `DynamicCallableWorkflow` (Cloudflare Workflow) | `kody-runtime` (main binds it cross-script) |
| `StorageRunner`, `RunLog`, `PackageRealtimeSession`, `PackageServiceInstance` | `kody-runtime` (main binds them cross-script) |
| `UserMeter` and every other Durable Object, app, MCP, OAuth, email, jobs | `kody` (runtime binds what it needs cross-script) |
| `APP_DB` / `AUDIT_DB` / KV / R2 / queues / Vectorize / AI | Shared resources; each worker binds directly (no RPC proxying) |
| Concern | Owner |
| ------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| Package-app origin (`PACKAGE_APP_BASE_URL`, `kody.run`) zone routes | `kody-runtime` |
| Inline package-app serving (`/apps/...` on the app origin) | `kody-runtime` (forwarded by main via the `RUNTIME_WORKER` service binding) |
| Package invocation API | `kody-runtime` (forwarded by main) |
| `DynamicCallableWorkflow` (Cloudflare Workflow) | `kody-runtime` (main binds it cross-script) |
| `StorageRunner`, `RunLog`, `PackageRealtimeSession` | `kody-runtime` (main binds them cross-script) |
| `UserMeter` and every other Durable Object, app, MCP, OAuth, email | `kody` (runtime binds what it needs cross-script) |
| `APP_DB` / `AUDIT_DB` / KV / R2 / queues / Vectorize / AI | Shared resources; each worker binds directly (no RPC proxying) |

## Migration configuration

Expand All @@ -40,8 +40,7 @@ top-level name) at deploy time:
"transferred_classes": [
{ "from": "StorageRunner", "from_script": "kody", "to": "StorageRunner" },
{ "from": "RunLog", "from_script": "kody", "to": "RunLog" },
{ "from": "PackageRealtimeSession", "from_script": "kody", "to": "PackageRealtimeSession" },
{ "from": "PackageServiceInstance", "from_script": "kody", "to": "PackageServiceInstance" }
{ "from": "PackageRealtimeSession", "from_script": "kody", "to": "PackageRealtimeSession" }
]
}
]
Expand Down Expand Up @@ -103,15 +102,15 @@ hand.
2. syncs secrets to both workers;
3. applies D1 migrations (shared databases, applied once);
4. **deploys `kody-runtime` first** — this applies the `v1`
`transferred_classes` migration, moving the four classes' storage out of
`transferred_classes` migration, moving the active classes' storage out of
`kody`. From this moment the still-running old `kody` deployment serves
runtime traffic against namespaces that have moved; the window until step
5 completes must be short and is why the two deploys are consecutive steps
in one job. There is deliberately no maintenance-mode traffic gate: the
accepted risk (per ADR 0016) is that runtime requests hitting the old
`kody` deployment during this seconds-long window error and surface as
failed runs/invocations, which is why the deploy runs at a quiet time;
5. **deploys `kody` second** with the config that binds the four classes
5. **deploys `kody` second** with the config that binds the three classes
cross-script (`script_name: "kody-runtime"`) plus the `RUNTIME_WORKER`
service binding, and stops routing runtime requests in-process;
6. healthchecks `kody` (`/health`) and `kody-runtime` (`/__runtime/health`),
Expand Down
15 changes: 7 additions & 8 deletions docs/contributing/architecture/usage-metering.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ events for the admin on/off cohort readout.
| `package_static_call` | one call of a statically imported package export (function-valued, incl. default) | sandbox-side wrapper stamped by the bundler; validated and recorded host-side by `packages/worker/src/usage/package-static-call-usage.ts` (wired in `run-kody-registry.ts`) | callee package id |
| `job_run` | one job execution | `packages/worker/src/jobs/service.ts` (`executeJobOnce`) | job id |
| `workflow_run` | one Cloudflare Workflow run | `packages/worker/src/package-runtime/package-workflows.ts` (`DynamicCallableWorkflow.run`) | workflow instance id |
| `service_runtime` | one package service run (bounded or persistent) | `packages/worker/src/package-runtime/package-service.ts` (run finalization) | `{packageId}:{serviceName}` |
| `realtime_session` | one realtime websocket session | reserved — not yet instrumented | session id |
| `outbound_fetch` | one outbound fetch through the gateway | `packages/worker/src/mcp/fetch-gateway.ts` (`KodyFetchGateway.fetch`) | request host |
| `email_send` | one outbound email send attempt | `packages/worker/src/email/outbound.ts` (`sendOutboundEmail`) | email message id |
Expand Down Expand Up @@ -322,17 +321,17 @@ Guarantees and rules:
- **Fleet visibility** (`/admin/insights`, loader in
`packages/worker/src/admin/fleet-usage-insights.ts`): bounded SQL over
`usage_rollups` for the current UTC month — top-10 combined runtime duration
(execute + job_run + workflow_run + service_runtime), top-10 event counts,
per-metric duration leaders, and an entitlement-pressure panel that reuses
(execute + job_run + workflow_run), top-10 event counts, per-metric duration
leaders, and an entitlement-pressure panel that reuses
`readAdminEntitlementConsumption` for the top ~15 users by event count.
Queries are `LIMIT`-bounded; entitlement reads run with modest concurrency.
- **Proactive alerts** (`usage_entitlement_alert` scheduled lane in
`packages/worker/src/app/usage-entitlement-alerts.ts`): hourly sweep of the
same ~15-user bound. Emails admin-role users when any swept account is above
80% of a plan-limit resource, or when a non-admin account's combined execute,
job_run, and workflow_run duration for the month exceeds
`fleetRuntimeDurationAlertThresholdMs` (24h). Persistent `service_runtime` and
admin-role dogfooding stay on `/admin/insights` rankings but do not page. The
KV cooldown key `ops-alert:usage-entitlement-pressure:v1` suppresses repeat
pages. Admin email links are built with `joinAppUrl` so a trailing slash on
`APP_BASE_URL` cannot produce `https://host//admin/…`.
`fleetRuntimeDurationAlertThresholdMs` (24h). Admin-role dogfooding stays on
`/admin/insights` rankings but does not page. The KV cooldown key
`ops-alert:usage-entitlement-pressure:v1` suppresses repeat pages. Admin email
links are built with `joinAppUrl` so a trailing slash on `APP_BASE_URL` cannot
produce `https://host//admin/…`.
Loading
Loading