fix(cloud-agent-next): recover exhausted wrapper cleanup leases - #5104
Merged
Conversation
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryRe-reviewed after rebase: the new wrapper-death reconciliation in Files Reviewed (14 files)
Previous Review Summary (commit 6da8fe0)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 6da8fe0)Status: No Issues Found | Recommendation: Merge Files Reviewed (14 files)
Reviewed by kimi-k3 · Input: 81.1K · Output: 13.8K · Cached: 2M Review guidance: REVIEW.md from base branch |
Recheck exhausted cleanups on a slow cadence (and force one probe when a user is waiting on flush) so a later-reaped sandbox can release the lease instead of fencing delivery for the rest of the session TTL.
eshurakov
force-pushed
the
eshurakov/hardy-ember
branch
from
August 6, 2026 18:27
6da8fe0 to
aa5d045
Compare
pandemicsyn
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
After physical wrapper stop attempts hit
WRAPPER_STOP_MAX_ATTEMPTS, the cleanup lease stayed exhausted for the rest of the session TTL. Delivery stayed fenced even when the container runtime later reaped the wedged sandbox, so follow-up messages could not start a fresh wrapper. Separately,prepareSessionrejected first-class tilde-prefixed model IDs such as~x-ai/grok-latest.What was done
noneonly after a confirmedabsentobservation.observeWrappersWithoutWakingso recovery never issues another stop and never wakes a stopped container just to inspect a process that cannot outlive it.WRAPPER_CLEANUP_EXHAUSTED_RECHECK_WINDOW_MS; explicit pending flushes force one out-of-cadence probe, then retry on a dedicatedWRAPPER_CLEANUP_EXHAUSTEDbudget before failing closed with an authoritative failure code.~inmodelIdSchemaso gateway latest-alias model IDs validate end-to-end.High-level architecture
sequenceDiagram participant DO as CloudAgentSession DO participant Sup as WrapperSupervisor participant Queue as SessionMessageQueue participant Sandbox as AgentSandbox Note over DO,Sandbox: Background recovery DO->>Sup: runMaintenance / reconcilePhysicalCleanup Sup->>Sup: exhausted lease due for recheck Sup->>Sandbox: observeWrappersWithoutWaking alt wrapper still present / inspection failed Sandbox-->>Sup: present or inspection-failed Sup->>Sup: schedule nextRecheckAt (within window) else wrapper gone Sandbox-->>Sup: absent Sup->>Sup: release_exhausted → lease none Sup->>Queue: requestPendingDrainIfNeeded end Note over DO,Sandbox: User waiting on flush Queue->>DO: recoverExhaustedDeliveryBlock DO->>Sup: recheckExhaustedCleanup (force) Sup->>Sandbox: observeWrappersWithoutWaking alt still exhausted Queue->>Queue: WRAPPER_CLEANUP_EXHAUSTED retry budget, then fail closed else lease released Queue->>Queue: deliver normally endArchitecture decision
Decision: Recover exhausted wrapper-cleanup leases with observation-only rechecks (cadenced in the background, forced on blocked flush), never by spending more stop attempts.
Context: Stop budget and rollback fence already say the supervisor will not stop again. The container runtime can still reap a wedged sandbox later; without a release path, that permanent fence bricks delivery for the session TTL. A user flush cannot wait on the full background cadence, but indefinite
queuedwithout a terminal signal is also unacceptable.Rationale: Confirmed absence is enough proof to free the lease because a wrapper process cannot outlive its container. Keeping recovery observation-only preserves the stop budget and avoids waking stopped containers. The flush path forces one probe, retries on a dedicated budget, then fails closed with
WRAPPER_CLEANUP_EXHAUSTEDso terminalization stays authoritative (INTERNALwould not be).Alternatives considered:
~model aliases at the web client. Rejected: tilde-prefixed latest aliases are first-class public model IDs in the AI gateway.Consequences: Exhausted sessions can resume after a later reap without weakening stop fencing. Background DO alarms re-arm only inside a one-hour recovery window; unrecoverable exhaustion stops automatic rechecks while explicit sends still probe once. Pending messages blocked on exhaustion get a short retry budget, then a clear terminal failure.
modelIdSchemaaccepts a slightly broader character set.Verification
Visual Changes
N/A
Reviewer Notes
stopWrappersafter exhaustion.WRAPPER_CLEANUP_EXHAUSTED, notINTERNAL, sorecordPendingFlushFailuretreats it as authoritative.mainby several unrelated commits; this PR is a single commit on top of its merge-base.