Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/contributing/architecture/request-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,14 @@ Saved packages may also declare long-lived package services under
- Package services share package identity with apps/jobs and can publish into
package app realtime sessions, but they own their own durable storage bucket
and lifecycle.
- A running service is kept resident with a short incoming alarm (about 15s).
Background `waitUntil` work does not keep the Durable Object alive by itself;
without that wake, Cloudflare evicts the isolate and orphans the sandbox,
including any outbound WebSocket.
- `mode: 'persistent'` means stay up until stopped. Eviction is treated as a
platform interrupt: the host immediately reschedules the same service, even
when `autoStart` is false. Persist resume cursors (`session_id`, sequence)
in `packageStorage()` so the next run can continue.

## Workflow runtime

Expand Down
14 changes: 10 additions & 4 deletions docs/guides/package-service-pattern.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,15 +187,21 @@ service status through the existing package runtime bridge.
## Prefer bounded reconnect loops over one immortal run

Cloudflare Durable Objects can open outbound WebSockets, but outgoing WebSockets
do **not** hibernate. Favor a design where the service can:
do **not** hibernate. The host keeps a running service isolate awake with a
short incoming alarm; still treat disconnect and eviction as normal:

- reconnect intentionally
- reschedule itself with alarms
- persist resumable state early
- persist resumable state early (`session_id`, sequence, cursor)
- stop cooperatively

Use `timeoutMs` to give the service enough room to run, but do not rely on
indefinite execution as the only lifecycle mechanism.
Use `mode: 'persistent'` for daemon-like supervisors (Discord Gateway, other
outbound sockets). Persistent services resume immediately after Durable Object
eviction even when `autoStart` is false. Bounded services still use crash-loop
backoff when `autoStart` is true.

Use `timeoutMs` to give a bounded service enough room to run, but do not rely
on indefinite execution as the only lifecycle mechanism.

## What to persist

Expand Down
3 changes: 3 additions & 0 deletions docs/use/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,9 @@ Use the package service model when the package needs:
- package-owned daemon-like logic
- package state that is separate from browser sessions
- a service that should publish updates into a package app
- an outbound WebSocket or reconnect loop (`mode: 'persistent'`, persist
resume cursors in `packageStorage()`; the host keeps the isolate awake and
immediately resumes after eviction)

Treat package services like package-owned runtime modules:

Expand Down
141 changes: 137 additions & 4 deletions packages/worker/src/package-runtime/package-service.node.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ const {
buildPackageServiceStorageId,
buildServiceRuntimeUsageEvent,
computePackageServiceRetryDelayMs,
packageServiceKeepaliveMs,
packageServiceStateHeartbeatMs,
} = await import('./package-service.ts')

const serviceBinding = {
Expand Down Expand Up @@ -374,7 +376,9 @@ test('package service start and stop project liveness into package_service_state
'2026-07-05T12:00:00.000Z',
'2026-07-05T12:00:00.000Z',
])
expect(created.getAlarmAt()).toBe(Date.parse('2026-07-05T13:00:00.000Z'))
expect(created.getAlarmAt()).toBe(
Date.parse('2026-07-05T12:00:00.000Z') + packageServiceKeepaliveMs,
)

await flushWaitUntilTasks(created.waitUntilTasks)
expect(mockModule.runBundledModuleWithRegistry.mock.calls[0]?.[1]).toEqual(
Expand Down Expand Up @@ -608,6 +612,102 @@ test('durable object restore meters an eviction-orphaned in-flight run without a
}
})

test('durable object restore immediately resumes an evicted persistent service', async () => {
resetMocks()
const usageSpy = vi
.spyOn(usageModule, 'recordUsage')
.mockResolvedValue(undefined)
vi.useFakeTimers()
try {
const restoredAt = new Date('2026-07-05T13:00:00.000Z')
vi.setSystemTime(restoredAt)
setupSavedPackage('persistent')
mockModule.runBundledModuleWithRegistry.mockResolvedValue({
result: { resumed: true },
error: null,
})
const restored = createPackageServiceState()
restored.persistedEntries.set('package-service-state', {
binding: serviceBinding,
autoStart: false,
mode: 'persistent',
timeoutMs: null,
stopRequested: false,
currentRunId: 'run-evicted',
nextAlarmAt: null,
nextAlarmSource: null,
lastStartedAt: '2026-07-05T12:55:00.000Z',
lastStoppedAt: null,
status: 'running',
lastError: null,
lastResult: null,
lastRunFinishedAt: null,
consecutiveFailureCount: 0,
})
const meter = createInMemoryUserMeterEnv()
const instance = new PackageServiceInstance(restored.state, {
APP_DB: {},
USER_METER: meter.env.USER_METER,
} as Env)
await waitForRestoreState(restored.state)
await flushWaitUntilTasks(restored.waitUntilTasks)

expect(restored.getAlarmAt()).toBe(restoredAt.valueOf())
const persisted = restored.persistedEntries.get(
'package-service-state',
) as { status: string; currentRunId: string | null }
expect(persisted.status).toBe('stopped')
expect(persisted.currentRunId).toBeNull()

await instance.alarm()
await flushWaitUntilTasks(restored.waitUntilTasks)
expect(mockModule.runBundledModuleWithRegistry).toHaveBeenCalledTimes(1)
} finally {
usageSpy.mockRestore()
vi.useRealTimers()
}
})

test('running keepalive alarms do not start a second service run', async () => {
resetMocks()
vi.useFakeTimers()
vi.setSystemTime(new Date('2026-07-05T12:00:00.000Z'))
try {
setupSavedPackage('persistent')
mockModule.runBundledModuleWithRegistry.mockImplementation(
() => new Promise(() => {}),
)
const created = await createPackageServiceInstance({
APP_DB: {},
} as Env)
const start = await created.instance.fetch(
new Request('https://package-service.invalid/service/start', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ binding: serviceBinding }),
}),
)
expect(start.status).toBe(200)
await drainSettledWaitUntilTasks(created.waitUntilTasks)
expect(mockModule.runBundledModuleWithRegistry).toHaveBeenCalledTimes(1)
expect(created.getAlarmAt()).toBe(
Date.parse('2026-07-05T12:00:00.000Z') + packageServiceKeepaliveMs,
)

vi.setSystemTime(
new Date(Date.parse('2026-07-05T12:00:00.000Z') + packageServiceKeepaliveMs),
)
await created.instance.alarm()
await drainSettledWaitUntilTasks(created.waitUntilTasks)
expect(mockModule.runBundledModuleWithRegistry).toHaveBeenCalledTimes(1)
expect(created.getAlarmAt()).toBe(
Date.parse('2026-07-05T12:00:00.000Z') + packageServiceKeepaliveMs * 2,
)
} finally {
vi.useRealTimers()
}
})

test('computePackageServiceRetryDelayMs doubles from the base delay to a 15-minute cap with jitter', () => {
const atJitterMax = () => 1
const atJitterMin = () => 0
Expand Down Expand Up @@ -951,9 +1051,40 @@ test('package service start/heartbeat/stop/purge shadow UserMeter transitions',
sourceUpdatedAt: '2026-07-05T12:00:00.000Z',
revision: 1,
})
expect(created.getAlarmAt()).toBe(Date.parse('2026-07-05T13:00:00.000Z'))
expect(created.getAlarmAt()).toBe(
Date.parse('2026-07-05T12:00:00.000Z') + packageServiceKeepaliveMs,
)

vi.setSystemTime(new Date('2026-07-05T13:00:00.000Z'))
vi.setSystemTime(
new Date(Date.parse('2026-07-05T12:00:00.000Z') + packageServiceKeepaliveMs),
)
const afterKeepaliveWaitUntilCount = created.waitUntilTasks.length
await created.instance.alarm()
await drainSettledWaitUntilTasks(
created.waitUntilTasks,
afterKeepaliveWaitUntilCount,
)
expect(
packageServiceShadowRow(
meter,
'user-123',
'package-1',
'realtime-supervisor',
),
).toMatchObject({
status: 'running',
sourceUpdatedAt: '2026-07-05T12:00:00.000Z',
revision: 1,
})
expect(created.getAlarmAt()).toBe(
Date.parse('2026-07-05T12:00:00.000Z') + packageServiceKeepaliveMs * 2,
)

vi.setSystemTime(
new Date(
Date.parse('2026-07-05T12:00:00.000Z') + packageServiceStateHeartbeatMs,
),
)
const afterStartWaitUntilCount = created.waitUntilTasks.length
await created.instance.alarm()
await drainSettledWaitUntilTasks(
Expand All @@ -972,7 +1103,9 @@ test('package service start/heartbeat/stop/purge shadow UserMeter transitions',
sourceUpdatedAt: '2026-07-05T13:00:00.000Z',
revision: 2,
})
expect(created.getAlarmAt()).toBe(Date.parse('2026-07-05T14:00:00.000Z'))
expect(created.getAlarmAt()).toBe(
Date.parse('2026-07-05T13:00:00.000Z') + packageServiceKeepaliveMs,
)

const afterHeartbeatWaitUntilCount = created.waitUntilTasks.length
await postPackageService(created.instance, 'stop')
Expand Down
Loading
Loading