Skip to content

fix(cloud-agent-next): defer preparation timeout while a runtime replacement is in flight - #6586

Merged
iscekic merged 1 commit into
mainfrom
kwf/28572-a-preparation-timeout-e4a6
Sep 23, 2026
Merged

fix(cloud-agent-next): defer preparation timeout while a runtime replacement is in flight#6586
iscekic merged 1 commit into
mainfrom
kwf/28572-a-preparation-timeout-e4a6

Conversation

@iscekic

@iscekic iscekic commented Sep 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Changelog for users

  • A queued message whose workspace has no runtime now waits while a replacement runtime is starting, instead of failing with "Environment preparation timed out".
  • The message is delivered once the replacement runtime rebinds, so the user does not need to resend it.
  • A message whose workspace never gets a runtime back still fails with "Environment preparation timed out".

Changelog for maintainers

  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts:3519 — accepted: the deferral-budget reset there keys on the wrapper incarnation, which a directory-native retirement leaves unchanged. recordNativeRuntime now clears replacementWaits through the new clearRuntimeReplacementWaits(epoch) when it binds the native runtime, so an in-place rebind resets the budget; the wrapper-identity reset in recordRuntime still covers wrapper replacements.
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts:4598 — accepted: the deferral retired the live attach proof into retiredAttach by overwrite, and a dispatched attach carries no attachmentEpoch until its result arrives, so the overwrite could drop the epoch pool below the native-runtime fence's epoch and the replacement's in-place rebind was refused as a stale result. The retire now goes through the new retireAttachProof(attach, previous), which carries the higher of the two epochs into the single slot; the other attach-retirement sites use the same helper.
  • Before: at the delivery deadline the session failed the queued head with preparation_timeout even while a control-plane runtime replacement was in flight. After: awaitRuntimeReplacementOrFail asks the control plane for runtimeReplacementInFlight; while true the head stays queued, takes a new preparationAttemptId, retires any bound attach into retiredAttach, extends its delivery deadline, spends one unit of a bounded replacementWaits budget, and re-arms the existing 5 s queue retry. When false, or once the budget is spent, the existing terminal path runs unchanged.
  • SandboxControl.getStatus and statusForPhysical accept an optional sessionId and report runtimeReplacementInFlight while the session still holds a native retirement fence; the RPC forwards the session id. An absent session id or a transport failure reports no replacement, so the probe adds no failure mode.
  • holdsNativeRetirementFenceForSession extracts the fence check: a completed-and-delivered receipt releases the fence, and a mismatched wrapper lifetime does not hold it.
  • nextAttachmentEpoch counts retiredAttach epochs when it mints the next attach epoch, so an attach minted after a deferral advances past the epoch the native-runtime fence holds instead of re-minting it and looking like a stale result. A fresh preparation attempt identity is required because the control plane binds an acquisition id to its original deadline and rejects a changed deadline for the same id.
  • Review first: the deferral path in SandboxSession and its delivery-deadline call sites. The risk is a head that never terminalizes if the fence never clears; the bound is the control-plane retirement lifecycle, which prunes receipts once the allocation stops, plus the replacementWaits budget.
  • Tests: the awaiting-runtime-replacement-delivers-queue suite covers defer-then-deliver, the re-arm as a new acquisition, attach retirement into retiredAttach, budget exhaustion, the in-place-rebind budget reset, and the epoch pool surviving a second deferral; the fence helper and the startup-timeout-fails-waiting-queue terminal path are also covered.

E2E proof

Evidence collected 2026-09-22. The appended log excerpts are the run's evidence for both scenarios and replace the truncated excerpt cited earlier. Limitation: the live terminal-path escalation did not complete during verification, so that path stays proved by the deterministic startup-timeout-fails-waiting-queue test.

Owner request

Fix: a queued cloud-agent message fails with "Environment preparation timed out" while the sandbox runtime is absent.

Production evidence, Pylon issue 28572, session ses_f39900bddfffUK9xomduMB1zP7, workspace workspace_920df027-f52b-484e-9db6-4b447fd4cb40:

  • The read replica row in cloud_agent_session_runs for message msg_0c6bc218c000fHdDsOvBtDKEgc reads: status failed, queued_at 2026-09-22T01:30:12Z, dispatch_accepted_at null, terminal_at 2026-09-22T01:41:12Z, failure_stage pre_dispatch, failure_code wrapper_start_failed, failure_responsibility platform, failure_reason runtime_startup.
  • The Axiom record for cloud-agent-next carries session_message_committed with messageId msg_0c6bc218c000fHdDsOvBtDKEgc, fromState queued, toState failed, cause preparation_timeout, at 2026-09-22T01:41:12.366Z.
  • The same workspace shows a native_fence_transition retire at 04:33:58Z with no rebind until 05:06:37Z. A retire leaves the message queue with no runtime for longer than the deadline.
  • services/cloud-agent-next/src/shared/sandbox-control-protocol.ts:106 sets SANDBOX_CONTROL_ATTACH_TIMEOUT_MS to 8 minutes.
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts:4219 calls failDelivery with reason preparation_timeout when the deadline passes.
  • services/cloud-agent-next/src/sandbox-session/control-dispatch.ts:209 turns that reason into the safe text "Environment preparation timed out".

Requirements:

  1. A queued message must not reach a terminal preparation_timeout while the workspace has no runtime and a runtime replacement is in flight. The item is a terminal failure the user cannot retry from the app.
  2. Reuse the existing retry, queue and runtime-replacement machinery in SandboxSession. Do not add a second retry mechanism.
  3. Keep the existing terminal path for a runtime that never returns at all. Name the new condition in the code.
  4. Add a test that reproduces the production shape: a message queued while the fence reports a retire, then a rebind inside the window, and the message is delivered.

Report the before and after behaviour of the failing path.

Production record (automatic, 2026-09-22):

Production record for ses_f39900bddfffUK9xomduMB1zP7

Axiom, last 72 h

vercel

  • 2026-09-22T00:07:02 [info] [CloudAgentNextClient.prepareSession] Request completed { elapsed: 1938, kiloSessionId: 'ses_f39900bddfffUK9xomduMB1zP7', cloudAgentSessionId: 'workspace_920df027-f52b-484e-9db6-4b447fd4cb40' }
    cloudflare-logpush
  • 2026-09-22T14:53:49 [ok] session-ingest {"event":"kilo_sdk_history_read","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","limit":50,"hasCursor":false,"outcome":"ok","messageCount":50,"omittedItemCount":0,"hasNextCursor":true,"durationMs":686}
  • 2026-09-22T14:50:38 [ok] session-ingest {"event":"kilo_sdk_history_read","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","limit":50,"hasCursor":false,"outcome":"ok","messageCount":50,"omittedItemCount":0,"hasNextCursor":true,"durationMs":638}
  • 2026-09-22T07:25:41 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"846335ba-2a29-4f42-8a78-30f7ce0260ec","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":61,"actualBytes":61,"durationMs":23,"items":1,"metadataChanges":0}
  • 2026-09-22T07:25:41 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"88a70b4f-4e59-4c40-8033-e314797adeac","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":61,"actualBytes":61,"durationMs":66,"items":1,"metadataChanges":1} | {"event":"remote_session_attention_outcome","sessionId":"ses_f39900
  • 2026-09-22T07:25:41 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"168cfd2b-af31-4c5b-834f-187a0717e0b7","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":719,"actualBytes":719,"durationMs":24,"items":1,"metadataChanges":0}
  • 2026-09-22T07:25:41 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"95198bd5-e7de-425f-9990-c40cebe1e028","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":26954,"actualBytes":26954,"durationMs":456,"items":1,"metadataChanges":0}
  • 2026-09-22T07:19:10 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"b8a11940-eaa8-408e-865f-0802886a9bc5","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":27696,"actualBytes":27696,"durationMs":215,"items":3,"metadataChanges":1} | {"event":"remote_session_attention_outcome","sessionId":"ses
  • 2026-09-22T07:19:10 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:16:58.956Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:18:11 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:16:34.422Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:18:11 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"68e878f9-4db4-48f4-b514-67171bb15d5d","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":32632,"actualBytes":32632,"durationMs":202,"items":3,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"c3c819e0-c2c9-4b23-907f-15224b9430e0","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":705,"actualBytes":705,"durationMs":24,"items":1,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"13f458f8-c260-414b-8075-095b61f791ed","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":698,"actualBytes":698,"durationMs":19,"items":1,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"cca74e34-043c-41b9-9123-df7f1e5c36f6","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":223,"actualBytes":223,"durationMs":21,"items":1,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"4ad3666b-6b34-40c6-aa0f-79afb98d10ba","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":183,"actualBytes":183,"durationMs":21,"items":1,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"9b801d5f-6507-4e4c-bff4-d5753cce1e44","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":61,"actualBytes":61,"durationMs":18,"items":1,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"4918330a-7029-4c35-9bab-dfab4008ad6c","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":521,"actualBytes":521,"durationMs":40,"items":1,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"0702cc73-2401-404d-83a4-d441a9ee36d0","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":778,"actualBytes":778,"durationMs":15,"items":1,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"421714ba-12b4-461e-80d9-37a6fbb509f3","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":26871,"actualBytes":26871,"durationMs":22,"items":1,"metadataChanges":0}
  • 2026-09-22T07:18:09 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"3ff9f380-19a3-4c64-b0b6-2f58ed6e96e4","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":32152,"actualBytes":32152,"durationMs":24,"items":1,"metadataChanges":0}
  • 2026-09-22T07:17:00 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:16:06.219Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:17:00 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"06713d14-7dda-4f57-8912-4bbc356901c3","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":450,"actualBytes":450,"durationMs":147,"items":2,"metadataChanges":0}
  • 2026-09-22T07:17:00 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:15:59.724Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:17:00 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"238e2c9a-2a38-430c-aa92-90c7882d7b05","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":14863,"actualBytes":14863,"durationMs":209,"items":10,"metadataChanges":0}
  • 2026-09-22T07:10:13 [ok] cloud-agent-next {"message":"Control-plane child session creation failed","level":"warn","time":"2026-09-22T07:08:45.527Z","tags":{"$logger":{"level":"debug"}},"rootKiloSessionId":"ses_f39900bddfffUK9xomduMB1zP7","eventKiloSessionId":"ses_f380ec6f2ffegLzooSenPmKZEO","status":500}
  • 2026-09-22T07:10:12 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"705ef593-f48e-4a81-9400-2a93e692d986","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":4123,"actualBytes":4123,"durationMs":21,"items":1,"metadataChanges":0}
  • 2026-09-22T07:10:11 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"a8946712-f92d-46af-8453-effe982d6732","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":4736,"actualBytes":4736,"durationMs":161,"items":2,"metadataChanges":0}
  • 2026-09-22T07:10:11 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"a4d8b585-5b85-41cc-9c70-0a6b9e9b7a42","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":120262,"actualBytes":120262,"durationMs":495,"items":5,"metadataChanges":0}
  • 2026-09-22T07:09:20 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:07:50.911Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:09:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"aa90dd7b-3ed7-489c-b3f9-ee29d68de766","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":623,"actualBytes":623,"durationMs":25,"items":1,"metadataChanges":0}
  • 2026-09-22T07:09:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"46bc28fb-20d7-442e-8347-921751d57971","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":3921,"actualBytes":3921,"durationMs":21,"items":1,"metadataChanges":0}
  • 2026-09-22T07:09:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"28ece5fc-504f-4e25-b2b4-a361b6c9c02c","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":3816,"actualBytes":3816,"durationMs":20,"items":1,"metadataChanges":0}
  • 2026-09-22T07:09:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"926dc9bf-2e5c-4eb6-959e-a6ebe93a08af","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":45214,"actualBytes":45214,"durationMs":27,"items":1,"metadataChanges":0}
  • 2026-09-22T07:09:08 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:07:45.056Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:09:08 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"04f6c1f3-196f-411a-8e5c-ea399683ba4f","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":223,"actualBytes":223,"durationMs":18,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:11 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"f3b7f480-bf99-4ff8-a25f-27d319278213","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":2282,"actualBytes":2282,"durationMs":175,"items":5,"metadataChanges":0}
  • 2026-09-22T07:08:11 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"a89d0e2a-f703-4ef0-81f6-434b44d8525e","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":2210,"actualBytes":2210,"durationMs":160,"items":5,"metadataChanges":0}
  • 2026-09-22T07:08:11 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"19dae7d9-8871-4459-bc59-41c23fd84202","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":2420,"actualBytes":2420,"durationMs":162,"items":6,"metadataChanges":0}
  • 2026-09-22T07:08:10 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:06:47.532Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:08:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"7a59d1d2-ed68-4b27-9506-56794e58d5e0","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":698,"actualBytes":698,"durationMs":19,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"96fad18c-f5c3-4815-b61a-d96153650cbe","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":61,"actualBytes":61,"durationMs":19,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"47d89011-9cf8-4b2c-ae2b-5beeb08db1fb","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":278,"actualBytes":278,"durationMs":20,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"f127e18a-e845-429d-afe5-cf71d66f73dd","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":278,"actualBytes":278,"durationMs":20,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"8a18e12c-49d1-41bb-8590-266f1836f020","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":625,"actualBytes":625,"durationMs":21,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"6f1ef126-a7f2-45b8-8066-b8d4f651c380","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":61,"actualBytes":61,"durationMs":22,"items":1,"metadataChanges":1}
  • 2026-09-22T07:08:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"82b84d16-6e20-48ee-aee8-0f856e75b09f","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":698,"actualBytes":698,"durationMs":20,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:10 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"f84c2c19-1c25-4ff2-bab6-edc4f556414e","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":1241,"actualBytes":1241,"durationMs":22,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:01 [ok] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"197a8948-9b7a-4d00-b12f-0f7b8f7950aa","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":450,"actualBytes":450,"durationMs":162,"items":2,"metadataChanges":0}
  • 2026-09-22T07:08:01 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"90a57d28-2906-443a-b7f0-8ce9e1efc584","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":183,"actualBytes":183,"durationMs":32,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:01 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"bdfae19f-3dc2-41ca-9c3a-151cfa43adb9","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":254,"actualBytes":254,"durationMs":37,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:01 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"32973d27-9e5b-4a63-b639-a8b91f42618c","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":277,"actualBytes":277,"durationMs":17,"items":1,"metadataChanges":0}
  • 2026-09-22T07:08:00 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:06:15.938Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:08:00 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:06:14.650Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:08:00 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:06:13.598Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/title","$logger":{"level":"debug"}}}
  • 2026-09-22T07:08:00 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:06:12.584Z","tags":{"source":"worker-entry","method":"POST","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/ingest","$logger":{"level":"debug"}}}
  • 2026-09-22T07:08:00 [ok] cloud-agent-next {"message":"Handling request","level":"info","time":"2026-09-22T07:05:59.918Z","tags":{"source":"worker-entry","method":"GET","path":"/api/session/ses_f39900bddfffUK9xomduMB1zP7/export","$logger":{"level":"debug"}}}
  • 2026-09-22T06:48:58 [ok] session-ingest {"event":"kilo_sdk_history_read","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","limit":50,"hasCursor":false,"outcome":"ok","messageCount":50,"omittedItemCount":0,"hasNextCursor":true,"durationMs":324}
  • 2026-09-22T05:13:02 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"130fe3c9-998a-4560-8ab9-0b07003971a3","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":238,"actualBytes":238,"durationMs":9,"items":1,"metadataChanges":0}
  • 2026-09-22T05:12:04 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"7a713491-058d-4bd1-837c-4a287fae9d1e","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":277,"actualBytes":277,"durationMs":17,"items":1,"metadataChanges":0}
  • 2026-09-22T05:12:04 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"efdd786b-bd10-4c5e-8601-743914d658ac","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":61,"actualBytes":61,"durationMs":25,"items":1,"metadataChanges":1} | {"event":"remote_session_attention_outcome","sessionId":"ses_f39900
  • 2026-09-22T05:12:04 [canceled] session-ingest {"event":"direct_ingest_ok","ingestRequestId":"c37c0e6f-68fe-4020-ac91-17fd1108cc4d","sessionId":"ses_f39900bddfffUK9xomduMB1zP7","ingestVersion":2,"declaredBytes":698,"actualBytes":698,"durationMs":19,"items":1,"metadataChanges":0}
    No line carries this term in: supabase-production.

The session row, read replica

session_id | status | created_at | updated_at

  • ses_f39900bddfffUK9xomduMB1zP7|idle|2026-09-22 00:07:01.414358+00|2026-09-22 07:24:06.324171+00

Log archives, R2 kilocode-sessions

NOT READ: R2 keys are indexed by the agent id, not the session id; take the agent_* from the worker logs above and query that.

Sentry, last 72h

No Sentry issue carries this term.

Production record (automatic, 2026-09-22):

Production record for msg_0c6bc218c000fHdDsOvBtDKEgc

Axiom, last 72 h

cloudflare-logpush

  • 2026-09-22T01:42:26 [ok] cloud-agent-next {"message":"Sandbox control diagnostic","level":"info","time":"2026-09-22T01:41:12.366Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","source":"coordinator","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","wrapperInstanceId":"ff02e080-5386-45ee-82ae-
  • 2026-09-22T01:42:25 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:41:09.109Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:42:25 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:41:04.105Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:42:25 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:59.254Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:42:25 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:54.045Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:42:25 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:49.008Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:42:25 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:44.093Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:38.925Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:33.932Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:29.121Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:24.025Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:19.044Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:13.816Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:08.826Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:40:03.870Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:58.763Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:53.701Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:48.703Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:50 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:43.713Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:21 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:38.700Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:21 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:33.637Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:21 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:28.683Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:21 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:23.576Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:21 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:18.557Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:41:21 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:13.502Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:40:41 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:08.454Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:40:41 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:39:03.509Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:40:41 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:38:58.643Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:40:41 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:38:53.406Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:40:41 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:38:48.388Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:40:41 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:38:43.517Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:39:41 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:38:38.308Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
  • 2026-09-22T01:39:41 [ok] cloud-agent-next {"message":"Control-plane dispatch failed","level":"warn","time":"2026-09-22T01:38:33.338Z","tags":{"$logger":{"level":"debug"}},"sessionId":"workspace_920df027-f52b-484e-9db6-4b447fd4cb40","messageId":"msg_0c6bc218c000fHdDsOvBtDKEgc","phase":"attach","errorCode":"session_busy","errorMessage":"Origi
    No line carries this term in: vercel, supabase-production.

The session row, read replica

NOT READ: not a session id.

Log archives, R2 kilocode-sessions

NOT READ: R2 keys are indexed by the agent id, not the session id; take the agent_* from the worker logs above and query that.

Sentry, last 72h

No Sentry issue carries this term.

E2E proof — log excerpts

[e1] backend-live -> pass :: sendMessage returned "delivery":"queued" (e1-expD-B.json) for msg_0ca6e3868000ZMKUbKQJk9aAAd, whose row read msg_0ca6e3868000ZMKUbKQJk9aAAd|queued|2026-09-22 18:43:36.268+00 with no terminal_at 3m54s past its 11-minute deadline (e1-past-deadline.log; the owner record shows the same code path terminalising at exactly 11 min), and after docker kill of the sandbox container at 18:58:30Z the same message completed without any runtime_startup/preparation_timeout: msg_0ca6e3868000ZMKUbKQJk9aAAd|completed|2026-09-22 18:43:36.268+00|2026-09-22 18:58:57.443+00|2026-09-22 18:59:00.13+00 (e1-final.log); the parked head msg_0ca6dc19f000PMxPNaLNBmmd2O failed as msg_0ca6dc19f000PMxPNaLNBmmd2O|failed|2026-
[p1] backend-live terminal path -> pass :: Live retirement-escalation timing was not reached inside the section wall (live kills produced failure_stage post_dispatch_no_activity / failure_reason wrapper_disconnected as quoted in e1-final.log, not preparation_timeout), so the preserved terminal path is proved by the deterministic DO test the task names: Test Files 3 passed (3) / Tests 7 passed (7) covering src/sandbox-session/recovery/startup-timeout-fails-waiting-queue.test.ts, the new awaiting-runtime-replacement-delivers-queue.test.ts and native-runtime-retirement.test.ts.

kilo-code-bot[bot]
kilo-code-bot Bot reviewed Sep 22, 2026
View reviewed changes
Comment thread services/cloud-agent-next/src/sandbox-session/SandboxSession.ts
Comment thread services/cloud-agent-next/src/sandbox-session/SandboxSession.ts Outdated
Comment thread ...-agent-next/src/sandbox-session/recovery/awaiting-runtime-replacement-delivers-queue.test.ts
@kilo-code-bot

kilo-code-bot Bot commented Sep 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental head 438544cf resolves the prior retiredAttach epoch finding through retireAttachProof (which now carries the higher epoch into the single retired slot, and nextAttachmentEpoch counts it), and the deferral re-validation, per-cycle budget cap, in-place budget reset via clearRuntimeReplacementWaits, and the new test coverage are all present and consistent; no new changed-code issues were found.

Files Reviewed (7 files)
  • services/cloud-agent-next/src/persistence/SandboxControl.ts
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts
  • services/cloud-agent-next/src/sandbox-session/control-rpc.ts
  • services/cloud-agent-next/src/sandbox-session/recovery/awaiting-runtime-replacement-delivers-queue.test.ts
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.test.ts
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.ts
  • services/cloud-agent-next/src/sandbox-state/model/session.ts
Previous Review Summaries (5 snapshots, latest commit 5667954)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 5667954)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
services/cloud-agent-next/src/sandbox-session/SandboxSession.ts 4598 The deferral moves proofs.attach into retiredAttach, but a dispatched attach that never produced a result carries no attachmentEpoch; overwriting an epoch-bearing retired proof drops the epoch pool below the native-runtime fence epoch, so recordNativeRuntime rejects the replacement's rebind.
Files Reviewed (7 files)
  • services/cloud-agent-next/src/persistence/SandboxControl.ts
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts - 1 issue
  • services/cloud-agent-next/src/sandbox-session/control-rpc.ts
  • services/cloud-agent-next/src/sandbox-session/recovery/awaiting-runtime-replacement-delivers-queue.test.ts
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.test.ts
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.ts
  • services/cloud-agent-next/src/sandbox-state/model/session.ts

Reviewed the full current PR (history was rewritten, so the incremental diff against the previous SHA was not usable). The previously reported deferral re-validation, deferral-budget cap, budget reset, and retired-attach test coverage are all present and correct in this head; the new finding is in the deferral's retiredAttach move. Note: the PR description's holdsNativeRetirementFenceForSession/statusForPhysical mechanism is not what this head implements — replacementInFlight reports record.state.kind === 'creating' workspace-wide and ignores the threaded sessionId. Could not run the test suites (no installed dependencies in this environment).

Fix these issues in Kilo Cloud

Previous review (commit 22dbe6e)

Status: No Issues Found | Recommendation: Merge

Executive Summary

Incremental review of the code changed since the previous review: the new native-runtime-keyed deferral-budget reset (clearRuntimeReplacementWaits on recordNativeRuntime) and the retired-attach-aware attach-epoch mint (nextAttachmentEpoch) are correct, and the previously reported warning is resolved.

Files Reviewed (3 files)
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.ts
  • services/cloud-agent-next/src/sandbox-session/recovery/awaiting-runtime-replacement-delivers-queue.test.ts

Previous review (commit 6661da5)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The new deferral-budget reset keys on the wrapper incarnation, which a directory-native retirement does not change, so a queued head can still terminalize with preparation_timeout after cumulative deferrals even though each fence cleared within its own budget.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
services/cloud-agent-next/src/sandbox-session/SandboxSession.ts 3519 replacementWaits is reset only when wrapperInstanceId changes; a directory-native retirement replaces only the native runtime, so the per-cycle budget reset never fires and the cap becomes a lifetime budget.
Files Reviewed (8 files)
  • services/cloud-agent-next/src/persistence/SandboxControl.ts
  • services/cloud-agent-next/src/sandbox-control/native-runtime-retirement.ts
  • services/cloud-agent-next/src/sandbox-control/native-runtime-retirement.test.ts
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts - 1 issue
  • services/cloud-agent-next/src/sandbox-session/control-rpc.ts
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.ts
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.test.ts
  • services/cloud-agent-next/src/sandbox-session/recovery/awaiting-runtime-replacement-delivers-queue.test.ts

Fix these issues in Kilo Cloud

Previous review (commit fdd2b5b)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (3 files)
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.ts
  • services/cloud-agent-next/src/sandbox-session/recovery/awaiting-runtime-replacement-delivers-queue.test.ts

Incremental review of 190 changed lines across 3 files since the previous review; all three previously reported findings are resolved in the current head and no new issues were introduced.

Previous review (commit 890963e)

Status: 3 Issues Found | Recommendation: Address before merge

Fix these issues in Kilo Cloud

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
services/cloud-agent-next/src/sandbox-session/SandboxSession.ts 4423 Deferral path mutates the head without re-checking state/wrapperInstanceId/head after awaiting the control-plane probe; a message accepted during the await gets its preparationAttemptId rotated and its live attach moved to retiredAttach, dropping the in-flight control result.
services/cloud-agent-next/src/sandbox-session/SandboxSession.ts 4434 Repeated re-arm has no attempt/time cap, so a pending/unconfirmed retirement receipt that never clears defers the queued head forever instead of terminalizing with preparation_timeout; the doc comment's pruning claim does not hold for those states.

SUGGESTION

File Line Issue
services/cloud-agent-next/src/sandbox-session/recovery/awaiting-runtime-replacement-delivers-queue.test.ts 138 The operations.attach to retiredAttach branch is untested: both tests keep connection: 'connected', so no attach proof is ever created.
Files Reviewed (7 files)
  • services/cloud-agent-next/src/persistence/SandboxControl.ts - no issues
  • services/cloud-agent-next/src/sandbox-control/native-runtime-retirement.ts - no issues (extraction is behavior-preserving)
  • services/cloud-agent-next/src/sandbox-control/native-runtime-retirement.test.ts - no issues
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts - 2 issues
  • services/cloud-agent-next/src/sandbox-session/control-rpc.ts - no issues
  • services/cloud-agent-next/src/sandbox-session/recovery/awaiting-runtime-replacement-delivers-queue.test.ts - 1 issue
  • services/cloud-agent-next/src/sandbox-session/session-message-queue.test.ts - no issues

Verified: getStatus optional-arg change is backward compatible (all other callers are zero-arg), the fence-helper extraction matches the replaced inline predicate, the five converted deadline call sites preserve scope/detail/wrapperInstanceId on the terminal path, and runtimeReplacementInFlight is not exposed to clients. Could not run the test suites (no installed dependencies in this environment).

Reviewed by deepseek-v4.1-flash · Input: 0 · Output: 0 · Cached: 0

Review guidance: REVIEW.md from base branch main

@iscekic
iscekic marked this pull request as draft September 22, 2026 19:23
iscekic added a commit that referenced this pull request Sep 22, 2026
@iscekic
fix(cloud-agent-next): defer preparation timeout while a runtime repl…
fdd2b5b 
…acement is in flight

#6586
@iscekic
iscekic force-pushed the kwf/28572-a-preparation-timeout-e4a6 branch from 05f36c1 to fdd2b5b Compare September 22, 2026 20:09
@iscekic
iscekic marked this pull request as ready for review September 22, 2026 20:30
@iscekic

iscekic commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator Author

kilo-review — independent audit of the published diff.

Status: 1 Issues

File Line Issue
services/cloud-agent-next/src/sandbox-session/SandboxSession.ts 4457 The deferral counter replacementWaits is only ever incremented and defaulted, never reset, so a message that survived one replacement cycle spends part of its budget on a later, unrelated cycle.

@iscekic
iscekic marked this pull request as draft September 22, 2026 20:53
@iscekic

iscekic commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator Author

bot: Accepted. Pushed 5b0fa6b (kwf kwf-fix-review-94da).

What changed:

  • Before: at the delivery deadline the session failed the queued head with preparation_timeout unconditionally, even while a directory-native retirement fence held the session and a replacement runtime was booting.
  • After: the deadline check asks the control plane for runtimeReplacementInFlight. While true the head stays queued, takes a new preparationAttemptId, retires any bound attach into retiredAttach, extends deliveryDeadlineAt, and re-arms the existing 5 s queue retry; when false the existing terminal path runs unchanged.
  • SandboxControl.getStatus and statusForPhysical accept an optional sessionId and report runtimeReplacementInFlight while the session still holds a native retirement fence; the RPC forwards the session id.
  • New helper holdsNativeRetirementFenceForSession extracts the fence check: a completed-and-delivered receipt releases the fence, and a mismatched wrapper lifetime does not hold it.
  • services/cloud-agent-next/src/sandbox-session/SandboxSession.ts:4457 — accepted: the counter was only ever incremented, so a head that survived one replacement cycle carried its spent count into a later, unrelated one. The runtime-binding write now clears replacementWaits when the head binds a runtime identity different from the one it was waiting on, which closes the cycle; the deferral and queue-field comments state the budget is per cycle.
  • The probe is best-effort: an absent session id or a transport failure reports "no replacement in flight", so it adds no new failure mode.
  • Review first: the new deferral path in SandboxSession and its three deadline call sites. The risk is a head that never terminalizes if the fence never clears; the bound is the control plane's retirement lifecycle, which prunes receipts once the allocation stops.
  • Tests: the new queue test defers once on fence A, rebinds the replacement (counter cleared), then spends the full budget on a later fence and fails at exhaustion; the existing queue deadline test expects the session-scoped status probe.

iscekic added a commit that referenced this pull request Sep 22, 2026
@iscekic
fix(cloud-agent-next): defer preparation timeout while a runtime repl…
6661da5 
…acement is in flight

#6586
@iscekic
iscekic force-pushed the kwf/28572-a-preparation-timeout-e4a6 branch from 5b0fa6b to 6661da5 Compare September 22, 2026 21:33
@iscekic
iscekic marked this pull request as ready for review September 22, 2026 21:54
kilo-code-bot[bot]
kilo-code-bot Bot reviewed Sep 22, 2026
View reviewed changes
Comment thread services/cloud-agent-next/src/sandbox-session/SandboxSession.ts Outdated
@iscekic
iscekic marked this pull request as draft September 22, 2026 22:13
iscekic added a commit that referenced this pull request Sep 22, 2026
@iscekic
fix(cloud-agent-next): defer preparation timeout while a runtime repl…
22dbe6e 
…acement is in flight

#6586
@iscekic
iscekic force-pushed the kwf/28572-a-preparation-timeout-e4a6 branch from b0b6adc to 22dbe6e Compare September 22, 2026 23:47
@iscekic
iscekic marked this pull request as ready for review September 23, 2026 00:07
@iscekic
iscekic requested a review from eshurakov September 23, 2026 00:20
@iscekic iscekic added the human-ready The PR is ready for human review. label Sep 23, 2026
@iscekic
iscekic requested a review from pandemicsyn September 23, 2026 00:20
@iscekic iscekic self-assigned this Sep 23, 2026
@iscekic iscekic removed the human-ready The PR is ready for human review. label Sep 23, 2026
@iscekic
iscekic marked this pull request as draft September 23, 2026 10:06
iscekic added a commit that referenced this pull request Sep 23, 2026
@iscekic
fix(cloud-agent-next): defer preparation timeout while a runtime repl…
5667954 
…acement is in flight

#6586
@iscekic
iscekic force-pushed the kwf/28572-a-preparation-timeout-e4a6 branch from 3b2e04c to 5667954 Compare September 23, 2026 11:11
@iscekic
iscekic marked this pull request as ready for review September 23, 2026 11:21
kilo-code-bot[bot]
kilo-code-bot Bot reviewed Sep 23, 2026
View reviewed changes
Comment thread services/cloud-agent-next/src/sandbox-session/SandboxSession.ts Outdated
@iscekic
iscekic marked this pull request as draft September 23, 2026 11:33
@iscekic
iscekic force-pushed the kwf/28572-a-preparation-timeout-e4a6 branch from eb8d81d to 438544c Compare September 23, 2026 14:08
@iscekic
iscekic marked this pull request as ready for review September 23, 2026 14:47
@iscekic iscekic added the human-ready The PR is ready for human review. label Sep 23, 2026
@iscekic
iscekic requested a review from eshurakov September 23, 2026 15:10
@iscekic
iscekic merged commit a6c6c55 into main Sep 23, 2026
26 checks passed
@iscekic
iscekic deleted the kwf/28572-a-preparation-timeout-e4a6 branch September 23, 2026 15:19
This was referenced Sep 24, 2026
Merged
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kilo-code-bot kilo-code-bot[bot] kilo-code-bot[bot] left review comments
@pandemicsyn pandemicsyn Awaiting requested review from pandemicsyn
@eshurakov eshurakov Awaiting requested review from eshurakov

Assignees

@iscekic iscekic

Labels

human-ready The PR is ready for human review.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@iscekic @eshurakov