feat(lifecycle): Lifecycle owns a durable job queue driven as an alarm event loop - #2175
Merged
mattzcarey merged 15 commits intoAug 28, 2026
Merged
Conversation
…m event loop Replaces pull-based alarm contributions (getNextAlarm/onAlarm) with a timestamp-ordered job queue owned by Lifecycle. Capabilities and the host push jobs (capability + fn + time + payload); Lifecycle drives due jobs, owns retry/deferral policy and the alarm memory-limit circuit breaker, arms a deadman pre-alarm before driving, and derives the physical alarm from queue state. Scheduler becomes vocabulary over the queue and migrates cf_agents_schedules into cf_agents_jobs.
🦋 Changeset detectedLatest commit: e2dcae2 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…ueue # Conflicts: # docs/agents/lifecycle.md # packages/agents/src/lifecycle/capability-runner.ts
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
🔍 Devin Review: 2 flags
Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
Code-quality pass on the job-queue change: - durable-object-lifecycle.ts had grown 714 -> 1093 lines; the drive loop, deadman, backlog warning, and memory-limit breaker move to lifecycle/job-driver.ts behind a narrow JobDriverOptions contract, bringing Lifecycle back to 789 lines. - One row->job converter: jobFromRow is exported from job-queue and the Lifecycle-side duplicate is gone. - isPlatformFailure(error) in retries.ts replaces the three copies of the platform-class failure triple (which was also internally redundant: isPlatformTransientError already covers code-update resets). - JobDispatch is properly typed, removing both LifecycleJobOutcome casts; the drive settle path is collapsed to one applyOutcome site. No behavior change; full agents (1912), think (887), chat, and node suites pass.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
…ry isolate reset
ctx.abort() now accepts { retryAlarm: false } (workerd AbortOptions;
ignored by older runtimes and outside alarm handlers). destroy() uses it
so a completed teardown's alarm is never retried into a fresh constructor
that recreates the deleted schema, and the breaker schedules the same
reset after a strike's writes sync — reclaiming the memory footprint and
handing the next wake to the backoff alarm. Pinned workers-types predate
the option, so a typed helper carries the cast.
Devin review catch: schedules created without a per-call retry stored no retry options, so the Lifecycle driver fell back to its own defaults and the Scheduler's configured retry (wired from Agent's static retry option) never applied to local dispatch — while routed facet dispatch still honored it. Jobs now carry fully resolved retry options; the raw user override moves into the payload so the public Schedule.retry projection still reports undefined when none was given.
…nd OOMs Deployed proof for the job queue: a self-rescheduling tick job that survives ctx.abort() restarts, a simulated memory-limit job that walks the breaker through strike/backoff/seal while unrelated ticks keep running, and a real allocate-until-death job that the platform retries across fresh isolates until it completes. /status exposes isolate id, strikes, jobs, and a durable wake log.
Audit pass: drop the host-level onJobError hook (nothing implements it — Agent and Think re-derive host jobs from durable state, so a terminal failure completing the job is the right default; the capability hook stays, Scheduler uses it) and unexport the queue-private hung-timeout default.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Second quality pass: _rearmAlarm survived the queue migration as a misnamed one-line delegate whose tracing span applied to only some sync paths. One method now owns the sync and the span; call sites, comments, and the cast-based test harnesses follow.
Devin review catch: moving the breaker into the job driver left Lifecycle.alarm() initializing BEFORE runAlarm(), so a memory-limit reset thrown during boot hydration — the original cloudflare#1825 crash-loop case — escaped interception and would re-throw to the platform's unbounded alarm retry. Initialization now runs inside the breaker, restoring the old Agent.alarm() coverage; regression test evicts the object, forces onStart to throw the reset cold, and asserts the alarm resolves with a durable strike recorded.
mattzcarey
added a commit
that referenced
this pull request
Aug 28, 2026
Merges the Lifecycle work-queue rework (#2175) and the WebSockets capability extraction (#2169), porting this branch's capabilities to the new pattern: - Tasks no longer implements the removed alarm-contribution surface (getNextAlarm/onAlarm/alarms.rearm). Every non-terminal run's authoritative next_at is mirrored as one Lifecycle queue job (id = run id, so a retime is a same-id push); wakes dispatch through onJob, whose outcome is derived from the run row after execution — the single source of truth that supersedes any same-id push made mid-drive. Startup reconcile mirrors every non-terminal run, which also covers seeded rows. - The maxRunsPerAlarm batching option, the batch harness, and its test are gone: dispatch pacing (due ordering, backlog warnings, the memory-limit breaker) is the queue driver's job now. - The boot-recovery aperture (__DO_NOT_USE_WILL_BREAK__dispatchDueRuns) and Agent's startup call are gone: interrupted runs' mirror jobs are overdue after a crash and re-fire on the post-startup alarm derivation. - Streams needs no port: it has no time-based behavior (appends happen in the producer's invocation; readers wake by append or reconnect), pushes zero jobs, and keeps working on facets. - Test seed helpers mirror seeded runs into cf_agents_jobs and backdate both rows, matching what acceptance does. - Agent installs tasks alongside main's _webSockets in the capability chain; docs and changesets now describe the queue model.
This was referenced Aug 28, 2026
mattzcarey
added a commit
that referenced
this pull request
Sep 2, 2026
cf_agents_jobs (and its recovery_loop column) has never shipped in a release — the whole table was introduced in this same unreleased branch (#2175, then extended with recovery_loop in #2190/#2194). No deployed Durable Object can have this table without the column already present: CREATE TABLE IF NOT EXISTS already includes it in the same statement. The pragma_table_info probe + ALTER TABLE fallback was defending against a schema history that cannot exist yet. Removes the dead branch, its regression test (which only proved the migration path itself, not anything a real caller depends on), and the two doc/changeset sentences describing it. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB
mattzcarey
added a commit
that referenced
this pull request
Sep 2, 2026
* fix(tasks): contain memory-limit loops with the alarm breaker Two confirmed gaps (pinned red-first in memory-limit.test.ts): 1. The driver's in-process dispatch retry converted a memory-limit reset into a silent success — the retry found the half-claimed run not due, returned void, and the wake was deleted without the breaker ever engaging. Memory-limit resets now defer to the alarm boundary like code-update resets: the isolate is condemned either way. 2. The run row outlives the breaker's queue-row policy: startup reconciliation re-derives due-now wakes from it, resurrecting a doomed run through backoff and past sealing. MemoryLimitContext now carries the striking job's identity, and Tasks applies the breaker to the run itself — demoted to the backoff wake on a strike (claim stripped so reconcile honors the deadline), terminally failed (task:failed, TaskMemoryLimitSealed) when the breaker seals. Generic for every definition: the striking run is the one that exhausted memory. Claude-Session: https://claude.ai/code/session_011QZUJztM1rMTsHEC7mbcbz * refactor(chat): run recovery continuations on Tasks Replace root-agent recovery schedule rows with chained runs of the reserved __cf_internal_chat_recovery Task definition shared by AI Chat and Think. Initial attempts deduplicate by incident, delayed retries use step.sleep, and the existing bounded callback methods still detach at model handoff. Legacy and routed dynamic-agent schedules remain as compatibility shims. Make Tasks breaker-safe for recovery definitions: condemned-isolate failures escape journal retries, flagged framework definitions carry queue membership, and onMemoryLimit aligns or seals their authoritative run rows so startup cannot resurrect purged work. Think's submission sweep now inspects Tasks rather than Scheduler rows. AI Chat and Think require agents >=0.23.1, the release containing the shared definition and internal enqueue aperture. * fix(tasks): release non-retained terminal runs Apply retain: false through one terminal cleanup path shared by completed, failed, and cancelled runs. A memory-limit seal now emits task:failed, removes the run and journal, cancels its wake, and releases its idempotency key. Make the breaker regression tolerate only the intentional workerd isolate reset and use per-attempt Durable Object names so Vitest retries cannot collide with durable rows from the failed attempt. * chore: restack recovery tasks on main Preserve #2192's exact atomic helper shape while checking the Task transport through a separate same-RPC method. Align chat peer ranges with the pending changesets release batch, which publishes this work in agents 0.23.0. * fix(recovery): preserve queue attempt ownership Upgrade existing Lifecycle job tables before reading the recovery-loop flag. Let Task wakes defer after one JobDriver attempt because ReplayStep owns their durable retry budget, including reconciliation of older wake rows. Keep pre-handoff chat failures on the current Task or schedule, and enqueue a replacement only after the bounded callback has handed off to the model turn. * fix(tasks): carry late memory resets to breaker When bounded Task dispatch returns before an attempt, preserve a durable late-memory-limit marker if that detached attempt later OOMs. The next alarm rethrows the canonical signal inside JobDriver so existing strike, backoff, and sealing policy remains authoritative. Mark detached job outcomes so their alarm is not treated as a clean breaker cycle, preserve marker wakes through startup reconciliation, and cover queued and warm attempts plus marker cleanup. * fix(tasks): clear stale strikes after detached settlement When work previously detached from an alarm settles without a memory reset, enqueue one no-op Task wake. That wake restores any authoritative run deadline and gives the existing JobDriver a clean alarm boundary to clear stale strikes. Late OOMs keep using the existing late-memory-limit marker. * refactor(lifecycle): own bounded alarm work Keep Lifecycle's deadman alarm armed while it joins promises registered at bounded job handoffs. Classify the whole dynamically growing alarm batch once: all clean work clears prior strikes, while any memory reset enters the existing breaker once with its captured executing job. Tasks now only registers attempts at its five-second handoff, and AI Chat and Think register post-handoff model work so Task and Scheduler transports share the same breaker. Remove the Task-specific late marker jobs and their state. * refactor(lifecycle,tasks): track handed-off alarm work without holding the alarm Lifecycle: `trackAlarmWork` no longer holds the alarm invocation open or coalesces physical re-arms. Handed-off work is classified when it settles: a memory reset records a strike against the job that handed it off, one strike per reset however many flows observe it, and strikes clear only once no handed-off work is outstanding and the last of it settled clean. Attribution follows the dispatch's async context and alarms in flight are counted, so overlapping invocations (tests drive alarm() by hand while the pool auto-fires the physical alarm) stay correct. Tasks: drop the per-definition recoveryLoop grouping and its composition root plumbing (setTaskRecoveryLoopDefinitionResolver, Agent._recoveryLoopTaskDefinitions, the recoveryLoop option on _registerInternalTaskDefinition). onMemoryLimit acts only on the run whose wake struck, stripping the claim and pushing the deadline so the reclaim still sees an interrupted attempt. onJob hands off one canonical promise per attempt, #syncWake skips same-values pushes, and runAttached actually attaches instead of racing the warm start. Chat: both hosts run the bounded recovery callbacks through the shared dispatchChatRecoveryToHandoff and register the recovery Task with hooks keyed by callback name. Design docs, changeset, stale Scheduler/recovery-engine comments and tests updated; new tests pin one-strike-per-reset and the startup re-arm. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * fix(lifecycle): attribute strikes correctly under overlapping alarms Two correctness fixes from Devin Review, both against the previous commit's job-driver.ts: - Overlapping alarm() invocations attributed a memory-limit strike to whichever job a shared instance field (#executingRow) last pointed at, not the job that actually struck — the struck job could escape backoff/sealing while an unrelated one absorbed it. Replaced with an AttributedPlatformFailure carrying the row out of the throw itself, correct by construction regardless of overlap. - The quiescence-clear check for the strike counter read stale snapshots of #alarmsInFlight/#outstandingAlarmWork depending on which of two transitions (an alarm ending, a handoff settling) ran first, so a strike recorded in the gap between checks could never clear. Fixed by having each transition re-check both counters fresh, with no yield point between the mutation and the check on either side. Fixing the second issue introduced a regression caught by the full suite: the quiescence check started running unconditionally in runAlarm's finally, including right after a strike was just recorded, immediately clearing what had just been set. Restored the placement-based guard (only a clean pass triggers the check) that the prior design relied on. New regression test isolates the attribution fix from Tasks' own active-attempt tracking, which independently and correctly re-tracks a Task run an overlapping alarm re-dispatches and was masking the bug in two earlier attempts at this test. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * fix(lifecycle): preserve a strike through a later clean sibling settling Third correctness fix from Devin Review against job-driver.ts, following the same shape as the two already fixed: a strike could be fully recorded (durable counter written, isolate reset scheduled via setTimeout(0)), and then a slower, unrelated clean handoff still outstanding at that moment could settle in the gap before the reset actually lands, find the alarm domain quiescent, and clear what was just recorded. Fixed with #strikeRecordedThisIsolate: once any flow records a strike in this isolate, no later settlement may clear the durable counter for the rest of this isolate's life. The isolate always resets shortly after a recorded strike, so the next genuinely clean cycle runs in a fresh isolate with a fresh JobDriver and the flag back at its default — there is no cross-isolate state to reset. New regression test (oomBeforeCleanSibling) orders a run's own strike distinctly before a separately tracked clean sibling; verified against the pre-fix code that it exercises the intended scenario and passes correctly with the fix. I could not force this specific narrow race window (bounded by workerd's own near-instant setTimeout(0) teardown scheduling) to reproducibly fail pre-fix within this test harness after several timing attempts, so this is a positive regression guard rather than a failing-then-passing proof like the other two fixes in this PR. The fix itself is verified by direct code tracing, matching Devin's own independent diagnosis of the same race. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * refactor(tasks): add Tasks#register for framework-reserved definitions Replaces the imperative _registerInternalTaskDefinition/_internalTaskDefinitions mechanism on Agent (a Map populated via a protected method, five near-identical one-line wrapper methods across AIChatAgent and Think just to call it) with a plain `register(name, definition)` method directly on Tasks. Framework code calls this.tasks.register(...) once per reserved name from its own constructor; it throws if the name lacks the `__cf` prefix or is already registered, so registration composes correctly no matter how many subclass layers exist or what any of them do with their own taskDefinitions field — it no longer depends on Agent's own field-override machinery at all. Agent's constructor keeps setTaskDefinitionResolver, but its only remaining job is bridging the end user's own overridable taskDefinitions field, which genuinely cannot be read at Tasks-construction time (a further-downstream subclass's field initializer runs only after every constructor up the chain returns) — register() has no such problem since it's called eagerly from each host's own constructor, after this.tasks already exists. design/rfc-fibers.md updated to describe the new mechanism (an adversarial review caught the stale reference to the old resolver-based path). New tests cover register()'s own validation branches directly (missing __cf prefix, empty name, duplicate registration, collision with a constructor-declared definition, and that a registered definition is reachable only through the internal aperture, never the public run()) — the only real definitions that previously exercised it (chat turn, chat recovery, messenger reply) never hit any of its failure paths. No public API surface changes: docs/agents/tasks.md and the existing changeset need no edits, matching every other __DO_NOT_USE_WILL_BREAK__- style internal aperture on Tasks. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * chore(lifecycle): remove dead ALTER TABLE migration for cf_agents_jobs cf_agents_jobs (and its recovery_loop column) has never shipped in a release — the whole table was introduced in this same unreleased branch (#2175, then extended with recovery_loop in #2190/#2194). No deployed Durable Object can have this table without the column already present: CREATE TABLE IF NOT EXISTS already includes it in the same statement. The pragma_table_info probe + ALTER TABLE fallback was defending against a schema history that cannot exist yet. Removes the dead branch, its regression test (which only proved the migration path itself, not anything a real caller depends on), and the two doc/changeset sentences describing it. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * feat(tasks): support routed wakes so facet chat recovery drops the Scheduler bridge Tasks now mirrors a routed sub-agent's run deadline to the root's job queue while the run row and step journal stay put: only the root owns the physical alarm, so only the wake needs to cross that boundary. onRoute handles syncWake/dispatch/memoryLimit; a routed strike forwards to the owning facet's own onAlarmMemoryLimit hook, the same bridge Scheduler already used, since the facet's own Lifecycle never observes the root's alarm directly. AIChatAgent and Think's _enqueueChatRecovery now always uses Tasks, removing the parentPath-gated Scheduler fallback and the now-dead chatRecoverySchedulePolicy/chatRecoveryRedeferPolicy helpers. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * fix(ai-chat): import AgentContext for AIChatAgentToolChild's constructor Fixes a typecheck failure in packages/ai-chat/src/tests/tsconfig.json CI caught: the test-only OOM Task definition added in the routed-wake work needed the child's own constructor to register it, which used AgentContext without importing it. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * fix(tasks): fix three Devin-flagged routed-dispatch bugs - onRoute's "dispatch" case awaited #executeRun directly, bypassing onJob's active-run claim refresh and its five-second dispatch budget. A routed run already active on the facet lost its wake when the root's mirror job settled with no reschedule, and a long-running routed handler blocked the root's whole alarm cycle. Extracted the shared bounded-dispatch logic into #dispatchRun, used by both onJob locally and onRoute's "dispatch" case, whose wake outcome now flows back as the RPC's own return value instead of being discarded. - onMemoryLimit only forwarded a routed run's SEALED strike to its owning facet. A non-sealed strike backed off the root's mirror job but left the facet's own claim (generation, next_at) untouched; any facet startup before the backoff elapsed read that claim as an interrupted attempt and reconciled it due again now, resurrecting the run through the breaker. Forward every strike, sealed or not. Verified the backoff fix is real: reverting to sealed-only forwarding reproduces the failure (generation stays set) in the new "backs off a routed facet's own claim" test. The active-run and dispatch-budget fixes reuse #dispatchRun verbatim from the local path, already covered by the existing local dispatch tests; a dedicated routed-side regression test for the concurrent-active and >5s-budget cases was not added given the cost of reliably constructing those races across a real DO-RPC boundary in this harness. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * fix(tasks): track routed dispatch on the root alarm, not the facet; cancel stale wakes on facet deletion Two more Devin-flagged bugs in the routed-Tasks work: - Routed tasks bypassed the memory breaker. A routed dispatch that exceeded the five-second budget called trackAlarmWork from inside onRoute on the facet, but that call only attaches to a live alarm invocation's AsyncLocalStorage scope — onRoute runs in response to an incoming RPC, not the facet's own alarm, so the call silently did nothing. Every redispatch of a routed run goes through this same path (root owns the physical alarm), so a routed run that regularly overran the budget could OOM repeatedly without the breaker ever seeing it. Moved the budget race to the root side instead: root now races its own await of the routed RPC call, and on budget keeps the still-pending call itself tracked via trackAlarmWork, which works because onJob runs inside root's own live alarm scope. The facet's onRoute dispatch handler no longer needs its own budget or tracking at all — it just fully awaits, since the call keeps running on the facet regardless of whether root is still waiting on it. Verified this is actually safe with a deployed repro (not simulated via ctx.abort(), which is deliberately deferred and wouldn't reject an in-flight caller): a callee DO whose isolate is killed by the platform's real memory-limit enforcement while mid-flight on an RPC call reliably rejects the caller's pending promise with the platform's own "exceeded its memory limit" text — exactly what the breaker already matches on. Confirmed across 3 trials against a real deployment, then deleted the repro worker. Added a regression test whose failure lands after root's own budget elapses (a real >5s delay, not a simulated abort) and confirmed it fails when reverted to the old (silently-inert) tracking call. - Deleting a facet left its routed Task wake mirrored on the root forever, since only Scheduler's routed rows were cleaned up on facet-subtree deletion. Added the same __DO_NOT_USE_WILL_BREAK__cleanupRoutePrefix aperture to Tasks, mirroring Scheduler's, and wired it into the same cleanupPrefix call site. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * fix(tasks,chat): mirror a routed run's claim deadline before dispatch; retry a failed post-handoff redefer Two more Devin-flagged bugs from the latest routed-Tasks review pass: - Once root started racing its own await of a routed dispatch (the RPC-tracking fix earlier today), a budget win now returns undefined as root's own job outcome. JobDriver deletes the mirror job on an undefined outcome — but only when the row is still marked "running" from that same dispatch. #executeRun now pushes the claim deadline via #syncWake immediately after claiming, before the handler runs: that push clears the row's in-flight marker (job-queue's own "newer durable intent wins" guard), so root's later stale undefined outcome no-ops against it instead of deleting a still-live claim. A hung or interrupted routed attempt keeps its alarm either way now, independent of whether this specific attempt happens to settle cleanly, fail, or never resolve at all. - dispatchChatRecoveryToHandoff swallowed a failed post-handoff redefer entirely. The Task that dispatched it has already settled by the time redefer runs, so nothing else owned that incident — a transient failure to enqueue the replacement abandoned recovery silently. Wrapped it in a bounded retry (tryN, 3 attempts) and surface the final failure through the same onDetachedError channel an unowned detached failure already uses, rather than a bare swallowed catch. Both verified non-vacuous: reverting each fix reproduces the exact failure Devin described (the mirror job actually gets deleted; the redefer failure is actually silent) in new regression tests, then passes again restored. The claim-mirroring test uses a real 6.5s delayed failure, not a simulated one. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * fix(chat): dedupe a retried post-handoff redefer instead of creating duplicate runs The redefer retry added last commit (tryN, 3 attempts) has its own bug: Tasks acceptance can throw after already durably inserting the run row — most likely on the wake-mirror push, not the insert itself — so a rejected enqueue does not prove nothing was created. Retrying the same unkeyed "redefer" enqueue (intentionally unkeyed for a genuinely new attempt, per chatRecoveryTaskRunOptions) could create up to three replacement runs for one incident instead of joining the first. dispatchChatRecoveryToHandoff now generates one dedupe key per failure and passes it to every retried redefer call; chatRecoveryTaskRunOptions keys the run by it (runId) when supplied, so a retry after a partial success joins that same row instead of duplicating it. Verified non-vacuous: reverting chatRecoveryTaskRunOptions's use of the key reproduces exactly the gap in the new "keys the run by dedupeKey" test. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * fix(tasks): repair a missing wake mirror when acceptance joins an existing run The dedupe fix from the last commit prevents a retried enqueue from creating a duplicate run, but joining the existing row alone doesn't help if that row's wake was never durably pushed in the first place. #accept can throw after already inserting the run — most likely on the wake-mirror #syncWake call itself, not the insert — so a retry that finds the existing row and returns accepted:false was reporting success against a run nothing would ever wake again. The join branch now calls #syncWake before returning, for every caller (runId or idempotencyKey match), not just chat recovery's retried redefer — any Tasks caller retrying acceptance after a partial failure benefits the same way. Cheap in the common case (#syncWake already no-ops when the mirror already matches). Verified non-vacuous: reverting the added #syncWake call and rerunning the new "repairs a missing wake mirror" capability test reproduces the gap exactly — the mirror job stays missing after the retry joins the run. Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB * refactor(scheduler): remove the retired routed chat-recovery memory-limit bridge Scheduler's routed memory-limit compatibility bridge existed to deliver a sealed strike to a dynamic agent whose chat-recovery schedule row was purged, back when routed chat recovery ran on Scheduler. Chat recovery now always uses Tasks (this PR), which has its own routed memory-limit bridge (setTaskRoutedMemoryLimitHandler), so this path carries no live traffic. Removed setSchedulerRoutedMemoryLimitHandler, the WeakMap backing it, the "memoryLimit" SchedulerRouteMessage variant, and Scheduler's onMemoryLimit/onRoute handling of it, plus Agent's wiring. Deliberately left in place: LEGACY_RECOVERY_LOOP_CALLBACKS' migration of pre-existing legacy schedule rows (still needed regardless of routing, so the local alarm breaker still counts them), and MemoryLimitContext.purgedRecoveryLoopJobs itself (now unread by any capability, but a generic hook a future routed capability with purge-as-a-pack semantics could still use — Tasks backs off one run at a time instead). Full suites still pass (2012 agents, 661 ai-chat, 888 think). Claude-Session: https://claude.ai/code/session_011DVPXpp9SyXM1ChxvkQCTB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Inverts alarm ownership: instead of capabilities keeping their own durable work and answering
getNextAlarm()polls, Lifecycle owns a timestamp-ordered queue of jobs and drives it as an alarm event loop. Capabilities just push work in.A job is a serialisable callback address — owning capability +
fn— with a due time (epoch ms) and a payload, stored in the Lifecycle-ownedcf_agents_jobstable. When the alarm fires, Lifecycle:tryNretries, platform-failure deferral (superseded isolate, transient, memory-limit) preserving the job for a fresh invocation,onJobErrorfor terminal application failures;{ rescheduleAt }= suspend,"yield"= leave due and wake immediately;onAlarm(), then re-arms the physical alarm purely from queue state (empty queue ⇒ delete alarm and hibernate;exclusivejobs suppress ordinary candidates).Every queue mutation re-arms automatically — there is no capability-visible
rearm()choreography. No lanes: thecapabilitycolumn already partitions the queue one lane per owner; a queue-name field can subdivide later.Removed (experimental surface)
Capability
getNextAlarm()/onAlarm(), hostgetNextAlarm(),LifecycleServices.alarms,AlarmContribution,Agent._getExtensionAlarm(), and Scheduler's__DO_NOT_USE_WILL_BREAK__handleAlarmMemoryLimitescape hatch — the one standing violation of the capability three-channels contract. HostonAlarm()survives (once per alarm invocation, after due jobs).Moved
The alarm memory-limit circuit breaker (#1825) moves from
Agent.alarm()into the Lifecycle event loop, where it can target the exact executing job (strike counter, backoff, seal — semantics preserved). Agent contributes domain policy (chat-recovery backoff/seal) through the newonAlarmMemoryLimit()host hook;Agent.alarm()keeps only the pending-destroy preamble.Reworked on top of the queue
fnis the callback name; intervals are single-flight jobs; recurrence is the drive result.cf_agents_schedulesis migrated into the queue on startup and dropped (seconds → ms; keep-alive orphans discarded).cf:keep-alive,cf:housekeeping(fiber/facet recovery with the existing backoff math), and the exclusivecf:destroy— re-derived from the durable destroy marker on every sync so a keepAlive-holding agent can't delay its own condemnation, including markers written by pre-queue releases.think:workflow-notificationshost job.Back-compat is preserved at the public
AgentAPI only (schedule(),getSchedules(),keepAlive(),destroy(), observabilityschedule:*events;schedule:duplicate_warningis replaced by a generic lifecyclejob:backlog_warning).Docs
docs/agents/lifecycle.md(job queue section),agent-class.md,AGENTS.md, the lifecycle example, and the design records:design/lifecycle-work-queue.md(spec) anddesign/alarm-coordination.mdrewritten to record the deliberate reversal of "Scheduler is one alarm contributor, not the general alarm service".Testing
agentsworkers suite: 1912/1912 (alarm arbitration, scheduler capability, hung/single-flight, OOM breaker seal/backoff, deferred destroy, keep-alive, eviction, sub-agents, migration)@cloudflare/thinkworkers suite: 887/887New migration tests seed legacy
cf_agents_schedulesschemas (pre-interval and full) and assert rows land in the queue with times converted, retry options kept, intervals single-flight, and the legacy table dropped, idempotently.