fix(cloud-agent-next) skip container wake on idle stop - #4902
Merged
Conversation
The idle sweep requests a physical wrapper stop only after confirming via DO state that no wrapper runtime or pending work remains. stopWrappers then inspected the container to re-derive that, and listProcesses is a container fetch, so a sleeping container was cold-booted to look for processes that activity expiry had already SIGTERMd. It then idled to the 900s sleepAfter. In 90 minutes of production logs this path was 233 inspections, all absent, none present. Confirm absence from container state instead when the container is stopped. A wrapper is a process and cannot outlive its container, so a stopped container cannot hide a leaked wrapper. Scoped to idle-timeout: every other stop reason still inspects, preserving the leaked wrapper recovery added in #3555.
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryThe idle-timeout stop path now short-circuits on DO-only container state ( Verification notes
Files Reviewed (4 files)
Reviewed by claude-opus-5 · Input: 64 · Output: 15.6K · Cached: 2.3M Review guidance: REVIEW.md from base branch |
jrf0110
approved these changes
Jul 30, 2026
pandemicsyn
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
stopWrappersinspects the sandbox by callinglistProcesses, which is acontainer fetch and therefore boots a sleeping container. On the idle timeout
path that boot is wasted work:
cleanupIdleKiloServeronly requests a stopafter
hasWrapperRuntimeOrPendingWork()has already confirmed from DurableObject state that no wrapper runtime and no pending work remain. The inspection
then wakes a stopped container to re-derive that same fact, and the woken
container sits idle until the 900s
sleepAfterexpires it.In 90 minutes of production logs,
reason=idle-timeoutproduced 233 stopinspections. Every one returned
absent. None returnedpresent.This confirms absence from container state instead, when the container is
stopped. A wrapper is a process, and a process cannot outlive its container
(activity expiry SIGTERMs the container, exit code 143), so a stopped container
cannot be hiding a leaked wrapper.
Changes
container-usage.ts: addsMeteredSandbox.isContainerRunning(), returningthis.ctx.container?.running === true. It reads Durable Object state only, socalling it over RPC does not boot the container.
container-usage-context.ts: adds the method toMeteredSandboxInstanceandadds
isSandboxContainerRunning(), using the same runtime guarded cast asconfigureSandboxBillingInput. It returnsboolean | undefined, yieldingundefinedwhen the method is missing or throws, so an unknown state is neverread as stopped.
cloudflare-agent-sandbox.ts: instopWrappers, when the reason isidle-timeoutand the container is confirmed not running, return{ status: 'absent' }without inspecting. Every other stop reason isunchanged.
cloudflare-agent-sandbox.ts: thewrapper_stop_inspectionlog is now emittedonce for all outcomes rather than once per branch, with a local
StopInspectiontype adding anabsent-no-containerstatus for the new case.Verification
has already gone to sleep plus a Durable Object alarm firing the idle sweep,
which is not reproducible locally. The problem itself was measured from
production logs (233 idle timeout inspections, all
absent).Visual Changes
N/A
Reviewer Notes
stopWrappersinspection was added byfix(cloud-agent-next): recover leaked sandbox wrappers safely #3555 ("recover leaked sandbox wrappers safely") to catch wrapper processes the
Durable Object lease does not track, including duplicates in shared sandboxes.
That recovery is preserved: it can only apply when a container is running, and
in that case this change falls through to the existing inspection untouched.
Only
idle-timeoutshort circuits.isSandboxContainerRunningsits behind a&&on the reason check, so the probedoes not run at all for other stop reasons. There is a test asserting this.
{ status: 'absent' }dropsstoppedInstanceIds. That field isoptional on
StopWrappersResultand has no readers anywhere in the codebase(produced in
stopWrappers, never consumed).ctx.container.running. If that flag were ever stale and reported false while acontainer was up, a legitimate leak check would be skipped. The same flag
already gates billing adoption in
configureBilling.(asserts
listProcessesis never called), idle timeout with a runningcontainer, a non idle timeout reason with a stopped container, and a sandbox
that does not expose
isContainerRunning.reason=idle-timeoutshould report
observation=absent-no-container, andcontainer_stoppedwithreason=activity_expiredshould fall relative toreason=exit.