refactor(chat): run recovery continuations on Tasks

.changeset/chat-recovery-tasks.md

@@ -0,0 +1,11 @@
+---
+"agents": patch
+"@cloudflare/ai-chat": patch
+"@cloudflare/think": patch
+---
+
+Run root-agent chat recovery continuations as chained Tasks instead of schedule rows. Initial recovery attempts deduplicate by incident, delayed retries use durable Task sleeps, and platform failures replay through Task claims. AI Chat and Think share one reserved recovery definition and preserve their existing bounded callback handoff behavior: a failure before handoff stays with the current queue execution, while a detached post-handoff platform failure enqueues exactly one replacement.
+
+Tasks now propagate condemned-isolate failures out of journaled steps and apply alarm memory-limit backoff and sealing to the run whose wake struck — claim stripped and deadline pushed, so startup reconciliation cannot resurrect it and the reclaim still sees an interrupted attempt. Task wake jobs are pushed with a single dispatch attempt so a platform failure rejects the alarm instead of being retried into a silent reschedule of the still-claimed run. Lifecycle gains `trackAlarmWork()`: work a job hands off at a bounded return stays inside that alarm's memory-limit breaker domain after the alarm returns, so other jobs stay live while a memory reset from the handoff still records a strike — one strike per reset however many flows observe it — and strikes clear only once no handed-off work is outstanding and the last of it settled clean. `retain: false` now removes failed and cancelled runs as well as completed runs, releasing journals and idempotency keys after every terminal outcome. Routed dynamic agents temporarily retain the root-owned schedule transport until Tasks supports routed child wakes.
+
+AI Chat and Think require `agents >=0.23.0`, the pending release batch containing the shared recovery Task definition and internal enqueue support.

.changeset/tidy-breaker-domains.md

@@ -13,26 +13,27 @@ the breaker on a strike and purged when it seals at the strike budget,
 without disturbing unrelated rows — a recovery schedule can no longer
 silently escape the breaker. The public `ScheduleOptions` vocabulary is
 unchanged: schedules only shape future work, and chat recovery reaches the
-flag through internal scaffolding (`RecoveryLoopScheduleOptions`) that will
-be deleted when recovery migrates onto the Tasks capability, where
-OOM-prone loops belong.
+flag through internal scaffolding (`RecoveryLoopScheduleOptions`) retained
+only for legacy rows and routed dynamic agents. Root recovery moves to Tasks;
+the scaffolding can be deleted when Tasks supports routed child wakes.
 Capabilities can react to a strike through the new optional `onMemoryLimit`
 hook, hosts through `onAlarmMemoryLimit`, and the context identifies the job
 that was executing when one exists. The strike budget is real Lifecycle
 configuration (`Lifecycle.install(host, { maxAlarmMemoryLimitStrikes })`)
-rather than a composition-root side channel. Until chat recovery moves to
-Tasks, a sealed routed recovery schedule also forwards the seal to its owning
+rather than a composition-root side channel. Until Tasks supports routed
+child wakes, a sealed recovery schedule also forwards the seal to its owning
 dynamic agent so a chat child under a plain Agent root persists its exhausted
 incident and terminal notification.
 
 Removed accordingly: `Agent.onAlarmMemoryLimit`'s policy relay, the
 `_cf_recoveryAlarmCallbacks` template hook, `Scheduler.applyMemoryLimitPolicy`,
 and
 `setLifecycleAlarmMemoryLimitStrikes`. `AIChatAgent` and `Think` flag their
-recovery schedules via `chatRecoverySchedulePolicy` and seal in-flight
+routed recovery fallback via `chatRecoverySchedulePolicy` and seal in-flight
 incidents from their own protected `onAlarmMemoryLimit` hooks; both now
-require `agents >= 0.23.0` (they consume new `agents/chat` exports and no
-longer implement the old template-method breaker hooks). Agent retains a
+require `agents >= 0.23.0` from the pending release batch (they consume its
+new `agents/chat` recovery exports and no longer implement the old
+template-method breaker hooks). Agent retains a
 sealed-only call to `_cf_sealMemoryLimitedRecovery` so already-published chat
 packages whose peer ranges accept agents 0.23 keep terminal notifications;
 that fallback carries no callback-name or queue policy.

design/alarm-coordination.md

@@ -25,11 +25,10 @@ capabilities push jobs instead of contributing wake times.
 - **Deadman pre-alarm** — armed before the event loop drives any due job, so
   an isolate death mid-drive still wakes the object to resume its queue.
 - **Tasks** — the capability for durable replayable execution. Every
-  non-terminal run's authoritative `next_at` deadline (acceptance, sleeps,
-  retries, and claim backstops all write it) is mirrored as one queue job
-  per run (`id = "task:" + the run id`, so a retime is a same-id push); the run's
-  wake dispatches through `onJob` and settles or reschedules via the drive
-  result.
+  non-terminal run's authoritative `next_at` deadline is mirrored as one queue
+  job per run. Task wakes use one job-dispatch attempt because ReplayStep owns
+  their durable retry budget; a propagated platform failure must reach a fresh
+  alarm invocation instead of entering JobDriver's generic retry loop.
 
 ## How the alarm is derived
 
@@ -50,14 +49,40 @@ coalesced and applied after startup completes.
 `Lifecycle.alarm()` runs the event loop: arm the deadman, drive due jobs in
 due order (single-flight skip/hung-recovery, per-job retry, platform-failure
 deferral, `onJobError` for terminal application failures), run host
-`onAlarm()`, clear the memory-limit strike counter, re-arm from queue state.
-The loop stops if teardown disabled alarms mid-phase.
+`onAlarm()`, clear the memory-limit strike counter when no handed-off work is
+outstanding, and re-arm from queue state. A job that must keep working past
+its bounded return registers that work with `trackAlarmWork`; the alarm still
+returns promptly, so other jobs stay live, and the handoff is classified when
+it settles. The loop stops if teardown disabled alarms mid-phase.
 
-The alarm memory-limit circuit breaker (#1825) lives at this boundary: a
-memory-limit reset is intercepted (everything else re-throws so platform
-alarm-retry semantics hold), a durable strike counter backs off the executing
-job, and at the strike budget the executing job is purged and the host's
-`onAlarmMemoryLimit()` hook applies domain policy (chat recovery sealing).
+The alarm memory-limit circuit breaker (#1825) lives at this boundary. A
+memory-limit reset is intercepted while every other error keeps platform alarm
+retry semantics; a reset reported by handed-off work enters the same breaker
+against the job that handed it off. One reset is one strike even when several
+flows observe it (an in-alarm job plus handed-off work, or several handoffs
+awaiting the same condemned storage): the counter moves once per event, each
+observing job is backed off or purged, and the policy hooks run once per
+observing job. Under the strike budget, the executing job and flagged
+recovery-loop jobs move to a backoff wake. At the budget they are purged.
+Lifecycle then supplies the executing-job identity to capability and host
+policy hooks. Strikes clear only when an alarm ends, or the last outstanding
+handoff settles, with no memory reset and nothing still running — a clean
+sibling cannot hide a slower handoff's reset.
+
+Tasks uses that hook to update the struck run's authoritative row — claim
+stripped, deadline pushed, state kept so the reclaim still sees an interrupted
+attempt — preventing startup reconciliation from undoing queue backoff. At its
+five-second job handoff it registers the still-running attempt with Lifecycle.
+AI Chat and Think register the post-handoff model dispatch when the callback
+runs under the alarm owner's dispatch — root Task runs and legacy root-owned
+Scheduler rows. Root chat recovery uses the reserved
+`__cf_internal_chat_recovery` definition. Routed dynamic-agent recovery
+temporarily keeps its root-owned Scheduler rows; its callback executes on the
+facet's own Lifecycle, outside any alarm, so `trackAlarmWork` declines there
+and the facet's post-handoff turn is bounded by the incident's own memory-reset
+budget rather than the breaker until Tasks can mirror child wakes to the alarm
+owner. A sealing strike still reaches the facet through the routed
+compatibility bridge.
 
 ## Agent integration
 

design/chat-shared-layer.md

@@ -48,6 +48,7 @@ packages/agents/src/chat/          ← shared foundation
   recovery.ts                      chat-fiber snapshot codec
   recovery-incident.ts             incident budget math + storage helpers
   recovery-engine.ts               ChatRecoveryEngine + adapter / wake-hook seams
+  recovery-task.ts                 reserved chained Task definition for continuations
   recovery-codec.ts                ChatRecoveryCodec (AISDKRecoveryCodec)
   resume-handshake.ts              ResumeHandshake stream-resume driver
   stall-watchdog.ts                iterateWithStallWatchdog
@@ -154,7 +155,7 @@ Pure functions for aligning client messages with server state during persistence
 
 ### recovery-engine.ts
 
-**`ChatRecoveryEngine`** owns the **shared durable chat-recovery orchestration** — the sequence both `AIChatAgent` and `Think` run when a Durable Object wakes and finds an interrupted chat turn (a `runFiber` that died mid-stream from hibernation, process death, or deploy churn). This state machine was previously duplicated across both packages, and the duplication was already drifting (better fixes landing in one but not the other).
+**`ChatRecoveryEngine`** owns the **shared durable chat-recovery orchestration** — the sequence both `AIChatAgent` and `Think` run when a Durable Object wakes and finds an interrupted chat turn (a Task run for a root agent, or `runFiber` for a routed facet, that died mid-stream from hibernation, process death, or deploy churn). This state machine was previously duplicated across both packages, and the duplication was already drifting (better fixes landing in one but not the other).
 
 Durable chat recovery is an invariant in both hosts: every chat entry path runs inside a recovery fiber. `ChatRecoveryConfig` accepts `true` or a tuning object, not `false`; previously compiled JavaScript that still supplies `false` is resolved to the default configuration. This guarantees that agent-tool child inspection can use durable recovery state after a restart instead of relying on instance-local abort controllers or stream managers.
 
@@ -176,6 +177,25 @@ Durable chat recovery is an invariant in both hosts: every chat entry path runs
 
 (b) is the one legitimately per-package step: ai-chat reads the stored stream `message_id` (#1691) because a flat `UIMessage[]` can't express parent/child (`AIChatAgent._resolveOrphanTargetId`); Think resolves it structurally from its Session tree. (c) `reconcileOrphanPartial` keeps an existing in-place tool result that lives only in storage — ai-chat's early tool-approval persist — rather than letting a replayed chunk re-advance it; Think has no early persist, so its whole-message replace is already dedup-safe and it doesn't use the helper. (d) is recognizably the same shape on both: ai-chat does `findIndex` → map-replace / append over its flat array; `Think._upsertMessageInHistory` does `session.getMessage` → `updateMessage` / `appendMessage` over a Session tree.
 
+Root-agent continuation attempts run as chained `__cf_internal_chat_recovery`
+Task runs. Initial detection joins by incident identity; stable-state and OOM
+retries enqueue a separate run and express their delay with `step.sleep`. The
+Task calls the existing bounded `_chatRecoveryContinue` / `_chatRecoveryRetry`
+entry point; both hosts run it through the shared
+`dispatchChatRecoveryToHandoff`, which returns at model handoff so a long turn
+never blocks the Lifecycle job loop and registers the turn as tracked alarm
+work (on the root; on a facet `trackAlarmWork` declines, and the detached turn
+stays outside the breaker until Tasks supports routed child wakes). A platform
+failure before handoff propagates through the
+current Task or compatibility schedule; after handoff that execution has
+settled, so the detached continuation enqueues exactly one replacement attempt.
+The incident record remains the recovery state machine.
+
+Facets do not yet have routed Task wakes. Dynamic-agent recovery therefore
+keeps the root-owned routed Scheduler transport as a compatibility path until
+Tasks can mirror a child run's wake to its alarm owner. Legacy recovery schedule
+rows use the same callback entry points and drain without migration.
+
 Full design + point-in-time decision record: [rfc-chat-recovery-foundation.md](./rfc-chat-recovery-foundation.md).
 
 ## Key decisions

design/lifecycle-work-queue.md

@@ -67,6 +67,9 @@ type LifecycleJobs = {
   list(): LifecycleJob[]; // sync SQL read
   rearm(): Promise<void>; // recover a lost alarm for existing jobs
 };
+
+// Available on Lifecycle and every capability's LifecycleServices:
+trackAlarmWork(work: Promise<unknown>): boolean; // current alarm only
 ```
 
 Every queue mutation re-arms the physical alarm automatically (deferred and
@@ -105,7 +108,9 @@ Four named rules define what a job owner can and cannot rely on:
    durable evidence, return) rather than awaiting it in the hook. The
    driver cannot safely abandon owner code, so the rule is enforced by
    visibility: a dispatch that outlives the job's `hungTimeoutSeconds`
-   (default 30s) logs a warning and emits `job:slow_dispatch`.
+   (default 30s) logs a warning and emits `job:slow_dispatch`. Work that must
+   continue after a bounded handoff registers its promise with
+   `trackAlarmWork()` so it remains part of the current alarm's breaker domain.
 3. **Newer pushes win over drive results.** Every dispatched job carries
    a durable in-flight marker; a same-id `push()` or `reschedule()` made
    while the job executes clears it, and `applyOutcome` only applies a
@@ -156,12 +161,19 @@ and `Lifecycle` wires them to the host and capabilities through the narrow
      complete);
    - the drive result is applied: delete, retime, or leave due.
 4. Run host `onAlarm()`.
-5. Re-arm the physical alarm from queue state.
+5. Clear the memory-limit strike counter, unless work registered through
+   `trackAlarmWork()` is still outstanding.
+6. Re-arm the physical alarm from queue state.
 
-Startup and steps 3–4 run inside the alarm memory-limit circuit breaker
+Startup and steps 3–5 run inside the alarm memory-limit circuit breaker
 (initialization included because a severe reset can be thrown during boot
 hydration, before any job runs — the original #1825 case), moved here from
-`Agent.alarm()`: the durable strike counter (`cf_agents:oom_alarm_strikes`)
+`Agent.alarm()`. Registered work extends that boundary past the alarm's
+return: a memory reset it reports records a strike against the job that
+registered it (one strike per reset, however many flows observe it), and the
+strike counter clears only once no registered work is outstanding and the last
+of it settled clean. The durable strike counter
+(`cf_agents:oom_alarm_strikes`)
 tolerates `maxAlarmMemoryLimitStrikes` consecutive resets (composition-root
 aperture; default 3), backs off the executing job, then seals — purging the
 executing job and invoking the host's `onAlarmMemoryLimit()` hook. After a

design/rfc-fibers.md

@@ -113,13 +113,20 @@ modules.
 ### Agent integration and the chat replatform
 
 `Agent` installs the capability as experimental `this.tasks`. Subclass
-definitions go on the overridable `taskDefinitions` field; framework
-definitions attach through a composition-root resolver
-(`setTaskDefinitionResolver`) so they never occupy the user's map, and
-reserved `__cf`-prefixed names cannot be started through the public
-`run()`. An internal aperture (`runAttached`) accepts a run durably and
-executes it inline while the isolate lives — the shape a request-driven
-chat turn needs.
+definitions go on the overridable `taskDefinitions` field, bridged by a
+composition-root resolver (`setTaskDefinitionResolver`) that stays lazy
+because a further-downstream subclass's field initializer only runs after
+every constructor up the chain returns. Framework definitions — chat
+turns, chat recovery, messenger replies — never touch that field at all:
+each host subclass (`AIChatAgent`, `Think`) registers them eagerly from
+its own constructor through `Tasks#register`, a capability-owned
+aperture that lives on `Tasks` itself rather than on `Agent`, so it
+composes correctly no matter how many subclass layers sit between the
+framework and the end user or what any of them do with their own
+`taskDefinitions` field. Reserved `__cf`-prefixed names cannot be started
+through the public `run()`. An internal aperture (`runAttached`) accepts
+a run durably and executes it inline while the isolate lives — the shape
+a request-driven chat turn needs.
 
 Think and AIChatAgent run their chat turns and messenger replies on this
 capability. The shared turn definition lives once in `agents/chat`

docs/agents/tasks.md

@@ -104,7 +104,7 @@ const receipt = await this.tasks.run("build-report@v1", input, {
 completion. The same `idempotencyKey` (or a caller-selected `runId`) joins
 the existing run instead of creating a second one; `accepted: false` on the
 receipt marks that join. Pass `metadata` to retain JSON alongside the run and
-`retain: false` to remove the record after successful completion.
+`retain: false` to remove the record after terminal settlement.
 
 `run()` and `handle()` type the definition name and its input against the
 declared map. A handle is a typed lens scoped to one definition — its `run`,