ADR 0016: extract package runtime and jobs lanes into separate workers - #1383
Conversation
Co-Authored-By: Kent C. Dodds <me@kentcdodds.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
🔎 Preview deployed: https://kody-pr-1383.kody-a99.workers.dev Worker: Mocks:
|
|
bugbot review |
|
@coderabbitai review |
Bugbot couldn't run — GitHub account mismatchThe GitHub account linked to your Cursor account does not match the PR author. Please ensure you're using the correct GitHub account, or run Bugbot from a team that covers this repository. |
✅ Action performedReview finished.
|
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe change adds ADR 0016. It documents extracting package runtime and jobs processing into separate workers, including Durable Objects, database ownership, service bindings, deployment safeguards, migration procedures, and operational consequences. ChangesWorker extraction architecture
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/contributing/decisions/0016-mono-worker-extraction.md`:
- Around line 34-46: Expand the decision around the runtime/jobs split to
explicitly preserve per-user isolation: require authenticated user context and
authorization on every cross-worker binding call, owner predicates on all job
and artifact reads/writes, and stable user/package-derived identities for moved
Durable Objects. Add migration validation proving records remain associated with
the same user, and state that resources retained by the main worker—including
packages, secrets, values, memories, connectors, inboxes, and durable
storage—retain their existing isolation guarantees.
- Around line 43-46: Update the mono-worker extraction decision around the
export/import and “flip reads” steps to define a lossless cutover: quiesce
scheduled and retry writers or capture and replay changes, switch read and write
ownership atomically from APP_DB to kody-jobs, reconcile row counts, checksums,
and archived artifacts, and document rollback and validation before dropping the
APP_DB tables.
- Around line 53-63: Expand the deployment guidance near the Cloudflare
service-binding contracts to define backward-compatible, additive
contract/versioning rules, provider-first rollout order, and cross-worker
contract healthchecks. Clarify how callers and targets remain compatible during
mixed revisions, since the existing per-worker SHA healthchecks do not validate
contract compatibility.
- Around line 34-37: Expand the Durable Object migration section around the
runtime-worker extraction to document the transfer protocol: define the target
expecting-transfer export, source transferred tombstone, deployment order,
binding consumers, and script_name routing. State that existing per-user IDs,
including RUN_LOG.idFromName(userId), must remain unchanged, and require preview
reads from existing objects before production rollout.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 83e69ab9-1a63-42d0-a6df-474219271693
📒 Files selected for processing (2)
docs/contributing/decisions/0016-mono-worker-extraction.mddocs/contributing/decisions/index.md
| 1. **Package runtime worker** (`packages/runtime-worker/`): package apps, | ||
| package invocation, and run execution. The `PackageServiceInstance`, | ||
| `PackageRealtimeSession`, `StorageRunner`, and `RunLog` Durable Object | ||
| classes move to this script (dashboards tolerate slower/cached RunLog reads; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 5 \
'expecting-transfer|transferred|script_name|exports|migrations' \
--glob '*.jsonc' .Repository: kentcdodds/kody
Length of output: 5890
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- ADR ---'
sed -n '1,110p' docs/contributing/decisions/0016-mono-worker-extraction.md
printf '%s\n' '--- named classes and related bindings ---'
rg -n -C 4 \
'PackageServiceInstance|PackageRealtimeSession|StorageRunner|RunLog|Durable Object|durable_objects|script_name|service binding|services' \
--glob '!node_modules' --glob '!dist' --glob '!build' .
printf '%s\n' '--- relevant configuration files ---'
git ls-files '*wrangler*.json*' '*config*.json*' | sortRepository: kentcdodds/kody
Length of output: 50372
🌐 Web query:
Cloudflare Durable Objects migrations cross-Worker transfer expecting-transfer transferred tombstone current documentation
💡 Result:
Cloudflare has transitioned to a declarative approach for managing Durable Object (DO) class lifecycles using an exports map in your Wrangler configuration (e.g., wrangler.toml or wrangler.json), which replaces the older, imperative migrations array [1][2]. The cross-Worker transfer process uses two specific "tombstone" states to facilitate the move: 1. Target Worker Configuration: The target Worker must declare an expecting-transfer entry in its exports map. This entry requires the storage type and the name of the source Worker (transfer_from) [3][4]. 2. Source Worker Configuration: The source Worker must declare a transferred tombstone in its exports map, specifying the name of the target Worker (transferred_to) [3][4]. The actual handoff of the namespace commits when the source Worker's deployment is successfully landed [3][5]. It is recommended to keep the source class in the code during this process to ensure zero-downtime [1][2]. If the source Worker still needs to access the DO after the transfer, it should be updated to point at the target Worker using a durable_objects binding that specifies the script_name [5]. To cancel a pending transfer, you can simply remove the expecting-transfer entry on the target Worker (or replace it with a standard live entry), which causes Cloudflare to delete the pending record [3][4]. Declarative exports and the legacy migrations array are mutually exclusive within a single Worker [1][2].
Citations:
- 1: https://developers.cloudflare.com/changelog/post/2026-06-30-declarative-do-class-exports/
- 2: https://developers.cloudflare.com/changelog/product/durable-objects/
- 3: https://developers.cloudflare.com/durable-objects/reference/durable-objects-migrations/
- 4: https://developers.cloudflare.com/durable-objects/reference/durable-objects-migrations/index.md
- 5: https://github.com/cloudflare/cloudflare-docs/blob/production/src/content/docs/durable-objects/reference/durable-objects-migrations.mdx
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- ADR lines 1-90 ---'
sed -n '1,90p' docs/contributing/decisions/0016-mono-worker-extraction.md
printf '%s\n' '--- Durable Object configuration declarations ---'
rg -n -C 3 \
'"durable_objects"|"bindings"|"class_name"|"script_name"|"exports"|"expecting-transfer"|"transferred"' \
packages/*/wrangler.jsonc
printf '%s\n' '--- named user-owned surfaces ---'
sed -n '80,145p' packages/worker/src/account/user-owned-surfaces.ts
printf '%s\n' '--- transfer-related repository references ---'
rg -n \
'expecting-transfer|transferred_to|transferred_from|transfer_from|transfer_to|script_name' \
--glob '!node_modules' --glob '!dist' --glob '!build' \
docs packages --glob '*.md' --glob '*.jsonc' --glob '*.ts' --glob '*.tsx' \
| head -n 200Repository: kentcdodds/kody
Length of output: 21231
🌐 Web query:
site:developers.cloudflare.com/durable-objects/reference/durable-objects-migrations expecting-transfer transferred deployment order script_name exports map
💡 Result:
The terms "expecting-transfer", "transferred", "script_name", "exports", and "map" relate to the configuration and migration of Cloudflare Durable Objects [1][2]. These elements are used to define how stored Durable Object namespaces are moved between different Worker scripts during a deployment migration [1][3]. The lifecycle states and configuration fields function as follows: Lifecycle States These are used within the configuration to track the movement of a Durable Object class [1][2]: - expecting-transfer: A state declared by the target (destination) Worker, indicating it is prepared to receive a Durable Object namespace from a specified source Worker [1][2]. - transferred: A state declared by the source Worker (as a tombstone), indicating that the namespace has been moved to a specified target Worker [1][2]. Configuration Structure - exports: In the context of Durable Objects migrations, the exports field is a map (typically keyed by the Durable Object class name) used to define these lifecycle states and associated metadata [1][2]. - script_name: When a source Worker still needs to access a Durable Object after a transfer has committed, its durable_objects.bindings entry is updated to point to the target Worker using the script_name property [1][2]. Deployment Order Migrations are applied at deployment based on the migration list specified in the wrangler configuration file (wrangler.toml or wrangler.json) [3][4]. The handoff process requires careful sequencing: 1. Target Worker: Declares an expecting-transfer entry naming the source Worker. It should not yet add a durable_objects.bindings entry for the class to avoid routing conflicts [1][2]. 2. Source Worker: Commits a deployment that changes the Durable Object definition to a transferred tombstone, which names the target Worker [1][2]. The actual handoff occurs when this source Worker deployment lands [1][2].
Citations:
- 1: https://developers.cloudflare.com/durable-objects/reference/durable-objects-migrations/
- 2: https://developers.cloudflare.com/durable-objects/reference/durable-objects-migrations/index.md
- 3: https://97468409.preview.developers.cloudflare.com/durable-objects/reference/durable-objects-migrations/
- 4: https://37731a98.preview.developers.cloudflare.com/durable-objects/reference/durable-objects-migrations/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
files = [Path("packages/worker/wrangler.jsonc")]
classes = {
"PackageServiceInstance",
"PackageRealtimeSession",
"StorageRunner",
"RunLog",
}
bindings = {
"PACKAGE_SERVICE_INSTANCE",
"PACKAGE_REALTIME_SESSION",
"STORAGE_RUNNER",
"RUN_LOG",
}
text = "\n".join(path.read_text() for path in files)
print("binding declarations:")
for binding in sorted(bindings):
matches = list(re.finditer(
rf'"class_name"\s*:\s*"([^"]+)"\s*,\s*"name"\s*:\s*"{binding}"'
rf'|"name"\s*:\s*"{binding}"\s*,\s*"class_name"\s*:\s*"([^"]+)"',
text,
))
print(binding, [next(group for group in match.groups() if group) for match in matches])
print("class declarations:")
for cls in sorted(classes):
print(cls, len(re.findall(rf'"class_name"\s*:\s*"{re.escape(cls)}"', text)))
print("transfer declarations:")
for term in ("expecting-transfer", "transferred", "script_name", "exports"):
print(term, len(re.findall(re.escape(term), text)))
print("per-user identity references:")
for path in sorted(Path("packages").rglob("*.ts")):
source = path.read_text()
if "idFromName(userId)" in source or "idFromName(input.userId)" in source:
print(path)
PYRepository: kentcdodds/kody
Length of output: 1338
Document the Durable Object transfer protocol.
At lines 60-63, define the target expecting-transfer export, source transferred tombstone, deployment order, binding consumers, script_name routing, and preservation of existing per-user IDs such as RUN_LOG.idFromName(userId). Require preview reads from existing objects before production.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/contributing/decisions/0016-mono-worker-extraction.md` around lines 34 -
37, Expand the Durable Object migration section around the runtime-worker
extraction to document the transfer protocol: define the target
expecting-transfer export, source transferred tombstone, deployment order,
binding consumers, and script_name routing. State that existing per-user IDs,
including RUN_LOG.idFromName(userId), must remain unchanged, and require preview
reads from existing objects before production rollout.
There was a problem hiding this comment.
Addressed in 44258b3: the ADR now specifies DO moves use Wrangler's cross-script transfer mechanism (receiving worker declares the incoming transfer; the source worker's deploy commits the handoff) rather than implying the plain migrations array.
There was a problem hiding this comment.
Skipped: comment is from another GitHub bot.
| 1. **Package runtime worker** (`packages/runtime-worker/`): package apps, | ||
| package invocation, and run execution. The `PackageServiceInstance`, | ||
| `PackageRealtimeSession`, `StorageRunner`, and `RunLog` Durable Object | ||
| classes move to this script (dashboards tolerate slower/cached RunLog reads; | ||
| run writes originate here). It keeps direct `APP_DB` access — the security | ||
| boundary that matters is the untrusted user code, which already runs in | ||
| zero-binding Worker Loader isolates; the runtime host code is the same trust | ||
| level as main. | ||
| 2. **Jobs worker** (`packages/jobs-worker/`): `JobManager`, scheduled lanes, and | ||
| job retention. It owns a **dedicated D1 database** (`kody-jobs`) holding | ||
| `jobs` and `archived_job_artifacts`, migrated from `APP_DB` by a bounded | ||
| manual copy (export → import → flip reads → later drop). Background write | ||
| churn leaves `APP_DB` entirely. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
Preserve the per-user isolation boundary in the split.
The decision introduces a shared kody-jobs database, direct APP_DB access from the runtime worker, cross-worker bindings, and moved Durable Object classes. It does not state how requests carry authenticated user identity, how job and artifact rows enforce ownership, or how per-user Durable Object identities remain stable after the move.
Add explicit requirements for authorization on every binding call, owner predicates on every database read and write, user/package-derived Durable Object identities, and migration checks that prove data remains attached to the same user. State that resources retained in the main worker keep the same isolation guarantees.
As per coding guidelines, maintain complete per-user isolation: each signed-in user must have an independent assistant with separate packages, jobs, secrets, values, memories, remote connectors, email inboxes, and durable storage.
Also applies to: 53-58
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/contributing/decisions/0016-mono-worker-extraction.md` around lines 34 -
46, Expand the decision around the runtime/jobs split to explicitly preserve
per-user isolation: require authenticated user context and authorization on
every cross-worker binding call, owner predicates on all job and artifact
reads/writes, and stable user/package-derived identities for moved Durable
Objects. Add migration validation proving records remain associated with the
same user, and state that resources retained by the main worker—including
packages, secrets, values, memories, connectors, inboxes, and durable
storage—retain their existing isolation guarantees.
Source: Coding guidelines
There was a problem hiding this comment.
Addressed in 44258b3: added an explicit isolation-preservation paragraph — the split moves code between trust-equivalent hosts without changing any guarantee; moved surfaces keep authenticated user context and owner-scoped queries, moved DOs keep name/id-derivation, migration verification confirms records stay associated with the same user, and main-resident resources are untouched.
There was a problem hiding this comment.
Skipped: comment is from another GitHub bot.
| job retention. It owns a **dedicated D1 database** (`kody-jobs`) holding | ||
| `jobs` and `archived_job_artifacts`, migrated from `APP_DB` by a bounded | ||
| manual copy (export → import → flip reads → later drop). Background write | ||
| churn leaves `APP_DB` entirely. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Define a lossless jobs cutover.
If scheduled lanes or retries write during the export/import window, the new kody-jobs database can miss those changes. The flip reads step also leaves write ownership undefined.
Specify a cutover that quiesces writers or captures changes, switches reads and writes together, reconciles row counts, checksums, and archived artifacts, and defines rollback before dropping the APP_DB tables. The absence of cross-database transactions makes this requirement essential.
Also applies to: 71-76
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/contributing/decisions/0016-mono-worker-extraction.md` around lines 43 -
46, Update the mono-worker extraction decision around the export/import and
“flip reads” steps to define a lossless cutover: quiesce scheduled and retry
writers or capture and replay changes, switch read and write ownership
atomically from APP_DB to kody-jobs, reconcile row counts, checksums, and
archived artifacts, and document rollback and validation before dropping the
APP_DB tables.
There was a problem hiding this comment.
Addressed in 44258b3: the cutover is now defined as lossless — quiesce scheduled lanes and job mutation paths, export/import, verify row counts and representative records (incl. archived artifacts), switch reads and writes together in one deploy (never split across databases), and keep the APP_DB tables as the rollback path until verification passes.
There was a problem hiding this comment.
Skipped: comment is from another GitHub bot.
| Cross-worker calls go over Cloudflare service bindings with small, | ||
| coarse-grained, explicitly typed contracts in `packages/shared` — no chatty | ||
| per-row RPC. Table ownership is documented: post-split, `jobs` and | ||
| `archived_job_artifacts` belong to the jobs worker; a cross-worker write to | ||
| another worker's tables is a review smell. `UserMeter` stays in main and is | ||
| reachable from the extracted workers as a DO binding. | ||
|
|
||
| Deploys are fully independent per worker (nx-affected-narrowed CI, per-worker | ||
| SHA guards and healthchecks), following the status-worker pattern. DO class | ||
| moves between scripts use Wrangler script migrations executed in a documented | ||
| runbook order, verified on preview deploys before production. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
sed -n '1,110p' docs/contributing/decisions/0016-mono-worker-extraction.md
printf '\n--- service-binding references ---\n'
rg -n -i "service binding|service-binding|provider.first|rollout|contract version|healthcheck" docs packages .github 2>/dev/null | head -200Repository: kentcdodds/kody
Length of output: 24714
🌐 Web query:
Cloudflare Workers service bindings deploy target worker first compatible change caller
💡 Result:
When using Cloudflare Workers service bindings, deployment order and update strategies are critical for maintaining compatibility [1][2]. Deployment Order When setting up service bindings for the first time, you must deploy the target Worker (the service being called) before the caller Worker (the Worker that declares the binding) [1][2]. If you attempt to deploy the caller Worker first, the deployment will fail because the specified target Worker does not yet exist on your account [1][3]. Updating Strategies (Compatible Changes) When making changes to Workers that use service bindings, you should follow a pattern that ensures the caller does not break while the target is being updated [1][2]: 1. Deploy changes to the target Worker (Worker B) first, ensuring they are backward-compatible with the existing caller (Worker A). For example, if you are adding new functionality, ensure the existing methods still work as expected. 2. Deploy changes to the caller Worker (Worker A) next, so it can begin utilizing the new functionality or updated interface provided by the target. 3. Finally, remove any unused code from the target Worker (Worker B) that is no longer needed by the updated caller. This approach ensures that at no point in the deployment sequence does the caller attempt to invoke a missing or incompatible method on the target [1][4].
Citations:
- 1: https://developers.cloudflare.com/workers/runtime-apis/bindings/service-bindings/index.md
- 2: https://developers.cloudflare.com/workers/runtime-apis/bindings/service-bindings/
- 3: https://github.com/cloudflare/cloudflare-docs/blob/production/src/content/docs/workers/runtime-apis/bindings/service-bindings/index.mdx
- 4: https://cloudflare-docs.justalittlebyte.ovh/workers/runtime-apis/bindings/service-bindings/
🏁 Script executed:
sed -n '220,320p' .github/workflows/deploy.yml
sed -n '420,490p' .github/workflows/deploy.yml
sed -n '285,360p' .github/workflows/preview.yml
printf '\n--- shared contract files ---\n'
fd -t f . packages/shared packages/status packages/backup-control-plane | head -120
rg -n "export (type|interface)|contract|binding|fetch\\(" packages/shared packages/status packages/backup-control-plane 2>/dev/null | head -160Repository: kentcdodds/kody
Length of output: 26456
Specify backward-compatible service-binding rollouts.
Define additive contract rules or versioning, provider-first deployment, and cross-worker contract healthchecks. Per-worker SHA healthchecks do not detect incompatible caller-target revisions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/contributing/decisions/0016-mono-worker-extraction.md` around lines 53 -
63, Expand the deployment guidance near the Cloudflare service-binding contracts
to define backward-compatible, additive contract/versioning rules,
provider-first rollout order, and cross-worker contract healthchecks. Clarify
how callers and targets remain compatible during mixed revisions, since the
existing per-worker SHA healthchecks do not validate contract compatibility.
There was a problem hiding this comment.
Addressed in 44258b3: added the standard service-binding ordering rule (deploy the callee with backward-compatible changes before the caller) to the independent-deploy section.
There was a problem hiding this comment.
Skipped: comment is from another GitHub bot.
…s, deploy ordering, DO transfer mechanics Co-Authored-By: Kent C. Dodds <me@kentcdodds.com>
1 participant
Intent
Record the agreed mono-worker decomposition before implementing it: extract the package runtime lane and the jobs/scheduled lane into independently deployed workers, and stop there.
Summary
0016-mono-worker-extraction.md:packages/runtime-worker/gets package apps/invocation/run execution plus thePackageServiceInstance,PackageRealtimeSession,StorageRunner, andRunLogDO classes; keeps directAPP_DBaccess (untrusted code is already isolated in zero-binding Worker Loader isolates — the runtime host is main-trust-level).packages/jobs-worker/getsJobManager+ scheduled lanes and a dedicatedkody-jobsD1 owningjobs+archived_job_artifacts, migrated by a bounded manual copy.packages/shared; documented table ownership; fully independent per-worker deploys; DO script-migration runbook + preview verification.APP_DBtables, ruling out RPC-only data access.Implementation lands separately (runtime-worker and jobs-worker PRs, in progress).
Testing
Docs-only;
npm run typecheck+ migration check passed via hooks.System changes
None (decision record only).
Link to Devin session: https://app.devin.ai/sessions/b5bf26745b254f289a40e061348af106
Requested by: @kentcdodds
Summary by CodeRabbit