diff --git a/docs/design/background-agent-progress-watchdog.md b/docs/design/background-agent-progress-watchdog.md new file mode 100644 index 00000000000..6a71f05e0ad --- /dev/null +++ b/docs/design/background-agent-progress-watchdog.md @@ -0,0 +1,43 @@ +# Background Agent progress watchdog + +[中文](background-agent-progress-watchdog.zh-CN.md) + +## Problem + +An ordinary background Agent can remain registered as running while its model, +control flow, or one tool makes no progress. The existing workflow watchdog is +not suitable: it retries stalled work and suspends its deadline for every +running tool. Ordinary background Agents must settle once as failed instead. + +## Behavior + +Each fresh, restored, and resident-continuation background turn has two fixed +internal deadlines: + +- 15 minutes without model or control progress. +- 10 minutes without progress from each in-flight tool. + +Model streaming, round transitions, usage, and external input renew the model +deadline. Tool output and liveness heartbeats renew only that tool's deadline. +Retry delays surfaced by qwen-code extend the model deadline by at most six +hours; provider-internal retries remain covered by the ordinary deadline. A +tool's own deadline starts when the scheduler reports it executing, so a silent +tool is not charged to the model deadline. Parallel tools retain independent +deadlines. + +The relevant tool deadline is suspended while user approval is pending. The +model deadline is suspended only after a no-tool round enters a Monitor-owned +external-input wait, and resumes when input arrives. A timer delayed by host +suspend or a local event-loop gap is rearmed rather than charged to the Agent. + +On expiry the watchdog aborts the turn with an `AgentProgressTimeoutError`. +Cooperative model and tool paths map that reason to `TIMEOUT`; the background +registry and sidecar then settle once as `failed`. There is no retry. Existing +definition-level turn and wall-clock limits are unchanged. + +## Scope + +Workflow dispatch remains unchanged. An Agent that ignores the cooperative +abort retains its physical slot while the daemon drains and replaces that +Session's runtime generation, as described in +`background-agent-runtime-generations.md`. diff --git a/docs/design/background-agent-progress-watchdog.zh-CN.md b/docs/design/background-agent-progress-watchdog.zh-CN.md new file mode 100644 index 00000000000..5d517d86d38 --- /dev/null +++ b/docs/design/background-agent-progress-watchdog.zh-CN.md @@ -0,0 +1,24 @@ +# 后台 Agent 进度看门狗 + +[English](background-agent-progress-watchdog.md) + +## 问题 + +普通后台 Agent 可能在模型、控制流或某个工具长期没有进展时仍保持运行状态。现有 workflow 看门狗会重试停滞工作,并在任一工具运行期间暂停期限,不适用于这里。普通后台 Agent 应当只以失败状态结算一次。 + +## 行为 + +每个新建、恢复及 resident continuation 的后台 turn 都有两个固定的内部期限: + +- 模型或控制流 15 分钟无进展。 +- 每个执行中工具 10 分钟无进展。 + +模型流式输出、round 状态变化、用量更新和外部输入会续期模型期限。工具输出和存活心跳只续期该工具的期限。qwen-code 显式上报的重试延迟最多将模型期限延长 6 小时;provider 内部重试仍受普通期限约束。工具期限从调度器报告工具开始执行时起算,因此静默工具不会计入模型期限,并行工具各自计时。 + +等待用户审批时,相关工具期限暂停。模型期限仅在无工具 round 真正进入 Monitor 所属的外部输入等待后暂停,并在收到输入后恢复。因主机挂起或本地事件循环间隙而延迟的计时器会重新计时,不会把这段时间算作 Agent 停滞。 + +期限到达时,看门狗以 `AgentProgressTimeoutError` 中止 turn。可协作中止的模型和工具路径把原因映射为 `TIMEOUT`,后台 registry 与 sidecar 只结算一次 `failed`,不做重试。定义级别的 turn 数量和总耗时限制保持不变。 + +## 范围 + +Workflow 调度保持不变。若 Agent 忽略协作式中止,它会继续占用物理槽位,同时 daemon 排空并替换该 Session 的 runtime generation,详见 [后台 Agent runtime generation](background-agent-runtime-generations.zh-CN.md)。 diff --git a/docs/design/background-agent-runtime-generations.md b/docs/design/background-agent-runtime-generations.md new file mode 100644 index 00000000000..1eddff355c8 --- /dev/null +++ b/docs/design/background-agent-runtime-generations.md @@ -0,0 +1,17 @@ +# Background Agent runtime generations + +[中文](background-agent-runtime-generations.zh-CN.md) + +## Problem + +A background Agent that does not settle after cancellation can leave its ACP child usable enough to answer transport probes but unsafe for fresh work. Replacing that child must not move its existing Sessions, create unbounded children, or route new work back to the draining generation. + +## Design + +Each ACP bridge channel has one of three states: `active`, `draining`, or `dying`. Existing Session entries continue to route through their recorded channel while it drains. Explicit recycle requests mark only the affected generation as draining; fresh work then creates a new active generation. Existing timeout retirement keeps its previous reap-after-drain behavior without starting another generation. + +Fresh work admits at most two non-dying generations. If both are draining, admission fails with `503 runtime_recycling` until one exits. Restore and recycle recovery may start a replacement while dying processes await reap; dying generations remain tracked until channel exit so synchronous shutdown can still reach them. + +After a logical watchdog abort, the Agent gets a fixed five-second cooperative exit window. If it still has not settled, its registry entry and sidecar become failed once while the underlying run keeps its concurrency slot. The terminal notification is recorded and displayed without starting another model turn, then the trusted child-to-daemon route requests recycle for the Session's owner generation. A late Agent settlement releases the physical slot but cannot replace the failed terminal state. + +This changes no persisted Session format and adds no public timeout configuration. diff --git a/docs/design/background-agent-runtime-generations.zh-CN.md b/docs/design/background-agent-runtime-generations.zh-CN.md new file mode 100644 index 00000000000..10bcb3cfee0 --- /dev/null +++ b/docs/design/background-agent-runtime-generations.zh-CN.md @@ -0,0 +1,17 @@ +# 后台 Agent runtime generation + +[English](background-agent-runtime-generations.md) + +## 问题 + +后台 Agent 在取消后仍不结算时,其 ACP child 可能仍能响应传输探测,却已经不适合接收新工作。替换该 child 时不能移动它现有的 Session、无限创建 child,也不能把新工作重新路由到正在排空的 generation。 + +## 设计 + +每个 ACP bridge channel 处于 `active`、`draining` 或 `dying` 三种状态之一。已有 Session 在排空期间继续通过自己记录的 channel 路由。只有显式 recycle 请求会把受影响的 generation 标记为 `draining`,新工作随后创建新的 active generation。已有的超时退役路径保持原先的“排空后回收”行为,不会启动另一代 runtime。 + +新工作最多允许两个尚未 dying 的 generation。若两者都在 draining,admission 返回 `503 runtime_recycling`,直到其中一个退出。restore 和 recycle recovery 可以在 dying 进程等待回收时启动替代进程;dying generation 会持续被追踪到 channel 退出,保证同步 shutdown 仍可访问它们。 + +逻辑看门狗中止后,Agent 有固定 5 秒的协作退出时间。若仍未结算,registry entry 与 sidecar 只会被标记失败一次,而底层 run 继续占用并发槽。终态通知在不启动额外模型 turn 的情况下记录并展示,随后可信的 child-to-daemon route 请求回收该 Session 的 owner generation。迟到的 Agent 结算会释放物理槽,但不会覆盖已经发布的失败终态。 + +本设计不改变 Session 持久化格式,也不增加公开的超时配置。 diff --git a/packages/acp-bridge/src/bridge.test.ts b/packages/acp-bridge/src/bridge.test.ts index 034bce725cf..e36ace60f0d 100644 --- a/packages/acp-bridge/src/bridge.test.ts +++ b/packages/acp-bridge/src/bridge.test.ts @@ -30,6 +30,7 @@ import { BranchWhilePromptActiveError, InvalidClientIdError, BridgeChannelQuarantinedError, + BridgeRuntimeRecyclingError, InvalidPermissionOptionError, InvalidSessionMetadataError, InvalidSessionScopeError, @@ -29423,14 +29424,21 @@ describe('createAcpSessionBridge', () => { }, }); capturedConn = new AgentSideConnection(() => fakeAgent, agentStream); + // A killed child exits, and the bridge frees a generation slot + // only once `exited` fires — it admits no fresh work while two + // OS-live generations are still pending reap. + let resolveExited: (() => void) | undefined; + const exited = new Promise< + | { exitCode: number | null; signalCode: NodeJS.Signals | null } + | undefined + >((r) => { + resolveExited = () => r(undefined); + }); return { stream: clientStream, - exited: new Promise< - | { exitCode: number | null; signalCode: NodeJS.Signals | null } - | undefined - >(() => {}), - kill: async () => {}, - killSync: () => {}, + exited, + kill: async () => resolveExited!(), + killSync: () => resolveExited!(), }; }; const bridge = makeBridge({ channelFactory: factory }); @@ -32914,6 +32922,181 @@ describe('createAcpSessionBridge', () => { await bridge.shutdown(); }); }); + + // ============================================================ + // Runtime recycle (issue #8586) — a generation hand-off must not + // admit fresh session work onto the condemned generation. + // ============================================================ + describe('requestRuntimeRecycle — draining generation admission', () => { + it('rejects a newSession that resolves after its channel was condemned to drain', async () => { + // Two sessions multiplex on gen1. The second spawn is held inside + // `connection.newSession` while a runtime recycle is requested for + // the first: gen1 flips to `draining` and retirement is DEFERRED + // (the in-flight spawn counts as work), so `isDying` stays false. + // When `newSession` finally resolves, doSpawn's post-await re-check + // must reject the fresh session instead of installing it on the + // generation the daemon just judged unsafe for fresh work — which + // would also pin that generation open until the session closed. + const secondNewSessionStarted = deferred(); + const releaseSecondNewSession = deferred(); + let newSessionCalls = 0; + const gen1 = makeChannel({ + newSessionImpl: async () => { + newSessionCalls++; + if (newSessionCalls === 1) return { sessionId: 'sess-drain-a' }; + secondNewSessionStarted.resolve(); + await releaseSecondNewSession.promise; + return { sessionId: 'sess-drain-b' }; + }, + }); + const gen2 = makeChannel({}); + let channelSpawns = 0; + const bridge = makeBridge({ + channelFactory: async () => + channelSpawns++ === 0 ? gen1.channel : gen2.channel, + sessionScope: 'thread', + }); + + const first = await bridge.spawnOrAttach({ workspaceCwd: WS_A }); + expect(first.sessionId).toBe('sess-drain-a'); + + const spawningSecond = bridge.spawnOrAttach({ workspaceCwd: WS_A }); + await secondNewSessionStarted.promise; + + // Optional on the interface (older embedded bridges omit it); the + // non-null call fails loudly rather than silently skipping the recycle. + await bridge.requestRuntimeRecycle!(first.sessionId); + + // gen1 is condemned but must NOT be killed: `first` is still live on + // it, and retirement was deferred until that session drains. + expect(gen1.killed).toBe(false); + + releaseSecondNewSession.resolve(); + await expect(spawningSecond).rejects.toBeInstanceOf( + BridgeChannelClosedError, + ); + + // The late session was never installed on the draining generation. + expect(bridge.sessionCount).toBe(1); + expect(() => bridge.getSessionSummary('sess-drain-b')).toThrow( + SessionNotFoundError, + ); + // The surviving session keeps its own (draining) generation. + expect(bridge.getSessionSummary(first.sessionId).sessionId).toBe( + 'sess-drain-a', + ); + expect(gen1.killed).toBe(false); + + await bridge.shutdown(); + }); + + it('rejects a resumeSession that resolves after its channel was condemned to drain', async () => { + // Restore twin of the `newSession` case above. gen1 is condemned + // mid-`session/load`, but retirement is DEFERRED (the in-flight restore + // counts as work, so `isDying` stays false). The post-await re-check + // must reject the restored session instead of installing it on the + // generation the daemon just judged unsafe for fresh work — which would + // also pin that generation open until the session closed. + const restoreStarted = deferred(); + const releaseRestore = deferred(); + const gen1 = makeChannel({ + newSessionImpl: async () => ({ sessionId: 'sess-drain-a' }), + loadSessionImpl: async () => { + restoreStarted.resolve(); + await releaseRestore.promise; + return {}; + }, + }); + const gen2 = makeChannel({}); + let channelSpawns = 0; + const bridge = makeBridge({ + channelFactory: async () => + channelSpawns++ === 0 ? gen1.channel : gen2.channel, + sessionScope: 'thread', + }); + + const first = await bridge.spawnOrAttach({ workspaceCwd: WS_A }); + expect(first.sessionId).toBe('sess-drain-a'); + + const restoring = bridge.loadSession({ + sessionId: 'sess-drain-b', + workspaceCwd: WS_A, + }); + await restoreStarted.promise; + + await bridge.requestRuntimeRecycle!(first.sessionId); + // gen1 is condemned but must NOT be killed: `first` is still live on it + // and the in-flight restore defers retirement. + expect(gen1.killed).toBe(false); + expect(channelSpawns).toBe(2); + + releaseRestore.resolve(); + await expect(restoring).rejects.toThrow( + /Session sess-drain-b restored on a closed agent channel/, + ); + + // The restored session was never installed on the draining generation. + expect(bridge.sessionCount).toBe(1); + expect(() => bridge.getSessionSummary('sess-drain-b')).toThrow( + SessionNotFoundError, + ); + + // The surviving session keeps its own (draining) generation. + expect(bridge.getSessionSummary(first.sessionId).sessionId).toBe( + 'sess-drain-a', + ); + expect(gen1.killed).toBe(false); + + await bridge.shutdown(); + }); + + it('rolls a recycle target back to active when the generation cap refuses the replacement', async () => { + const gen1 = makeChannel({ + newSessionImpl: async () => ({ sessionId: 'sess-gen1' }), + }); + let gen2SessionCount = 0; + const gen2 = makeChannel({ + newSessionImpl: async () => ({ + sessionId: `sess-gen2-${++gen2SessionCount}`, + }), + }); + let channelSpawns = 0; + const bridge = makeBridge({ + channelFactory: async () => + channelSpawns++ === 0 ? gen1.channel : gen2.channel, + sessionScope: 'thread', + }); + + const first = await bridge.spawnOrAttach({ workspaceCwd: WS_A }); + expect(first.sessionId).toBe('sess-gen1'); + + // Recycle the first session: gen1 drains, recovery spawns gen2 (active). + await bridge.requestRuntimeRecycle!(first.sessionId); + expect(channelSpawns).toBe(2); + + // A fresh session attaches to the active gen2. + const second = await bridge.spawnOrAttach({ workspaceCwd: WS_A }); + expect(second.sessionId).toBe('sess-gen2-1'); + + // Recycling gen2's session leaves both generations draining, so the + // replacement hits the two-generation cap. The recycle must reject and + // roll gen2 back to active rather than stranding the workspace with no + // active generation. + await expect( + bridge.requestRuntimeRecycle!(second.sessionId), + ).rejects.toBeInstanceOf(BridgeRuntimeRecyclingError); + + // gen2 rolled back to active: a third session attaches without spawning + // a third generation. + const third = await bridge.spawnOrAttach({ workspaceCwd: WS_A }); + expect(third.sessionId).toBe('sess-gen2-2'); + expect(channelSpawns).toBe(2); + expect(gen1.killed).toBe(false); + expect(gen2.killed).toBe(false); + + await bridge.shutdown(); + }); + }); }); // ============================================================ diff --git a/packages/acp-bridge/src/bridge.ts b/packages/acp-bridge/src/bridge.ts index e92a48dde3c..0e2469d2258 100644 --- a/packages/acp-bridge/src/bridge.ts +++ b/packages/acp-bridge/src/bridge.ts @@ -124,6 +124,7 @@ import { InvalidRewindTargetError, PromptDeadlineExceededError, BridgeChannelQuarantinedError, + BridgeRuntimeRecyclingError, McpAuthenticationInProgressError, SessionResetPendingError, StandaloneSessionSpawnError, @@ -1098,9 +1099,10 @@ interface ChannelInfo { * `killAllSync` must still find the channel during the SIGTERM * grace window to fire SIGKILL on `process.exit(1)`. `aliveChannels` * holds the dying entry until `channel.exited` fires (OS-level - * reap); `isDying` is the "available-for-new-spawns" half of the - * two-bit (alive, dying) state. + * reap). Draining generations retain their Session owners but cannot accept + * fresh work; dying generations are unavailable while the OS reaps them. */ + state: 'active' | 'draining' | 'dying'; isDying: boolean; /** Existing sessions stay usable, but no fresh session work may enter. */ isQuarantined: boolean; @@ -2836,7 +2838,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { | { channel: ChannelInfo; reason: BridgeChannelUnavailableReason } | undefined => { for (const ci of aliveChannels) { - if (ci.isDying) continue; + if (ci.state !== 'active') continue; if (ci.isQuarantined) { return { channel: ci, reason: 'restore_cleanup_failed' }; } @@ -3091,8 +3093,8 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { // same-workspace attach under `single` scope reuses). Thread-scope // sessions add to `byId` but don't displace `defaultEntry`. let defaultEntry: SessionEntry | undefined; - // `channelInfo` is the SINGLE attach-available channel. Cleared - // ONLY by the `channel.exited` handler (see below) when the OS + // `channelInfo` is the newest generation. It is attach-available only while + // active, and is cleared ONLY by its `channel.exited` handler when the OS // reaps the underlying child process. Teardown initiators // (`killSession` last-session-leaving — via `startIdleTimer` -> // `killChannelWithLog` / `reapPendingEmptyChannel`, @@ -3828,7 +3830,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { ci: ChannelInfo, context: string, ): Promise { - if (ci.isDying) return; + if (ci.state === 'dying') return; if (hasNoSessionWork(ci)) { await killChannelWithLog(ci, context); return; @@ -3839,6 +3841,46 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { ); } + async function requestRuntimeRecycleForSession( + sessionId: string, + ): Promise { + const entry = byId.get(sessionId); + if (!entry) throw new SessionNotFoundError(sessionId); + const owner = channelInfoForEntry(entry); + if (!owner || owner.state === 'dying') { + throw new SessionNotFoundError(sessionId); + } + owner.state = 'draining'; + if (channelInfo === owner) cancelIdleTimer(); + // `retireWhenSessionsDrain` is a single sticky flag shared by every + // retire-after-drain condemnor on this channel. Capture whether a different + // path had already condemned it so the rollback below only clears the + // condemnation this recycle itself set, instead of erasing someone else's. + const wasReapPending = owner.retireWhenSessionsDrain; + await retireChannelAfterSessionsDrain( + owner, + `runtime recycle requested by session ${JSON.stringify(sessionId)}`, + ); + if (!owner.isDying) { + try { + await ensureChannel('recovery'); + } catch (error) { + // The two-generation cap refused the replacement spawn. Leaving the + // owner draining here would strand the workspace with no active + // generation (the draining owner never empties while its unresponsive + // session is still attached), so roll the owner back to active and let + // it keep serving as a degraded fallback until a generation drains. + if (error instanceof BridgeRuntimeRecyclingError) { + owner.state = 'active'; + if (!wasReapPending) { + owner.retireWhenSessionsDrain = false; + } + } + throw error; + } + } + } + async function retireChannelOnTimeout( ci: ChannelInfo, error: unknown, @@ -3870,7 +3912,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { ci: ChannelInfo, context?: string, ): Promise { - if (ci.isDying || liveChannelInfo() !== ci) return; + if (ci.isDying || admissibleChannelInfo() !== ci) return; const timeoutMs = resolvedChannelIdleTimeoutMs(); if (timeoutMs <= 0) { await killChannelWithLog(ci, context); @@ -4639,15 +4681,10 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { return clientId; }; - /** - * Get-or-create the daemon's single `qwen --acp` channel. N sessions - * multiplex onto it via `connection.newSession()`. Concurrent callers - * coalesce through `inFlightChannelSpawn` so we never spawn two - * children. Wires up the one-and-only `channel.exited` cleanup on - * first creation so the late-arriving event tears down ALL - * multiplexed sessions. - */ - async function ensureChannel(): Promise { + /** Get or create the active runtime generation. */ + async function ensureChannel( + admission: 'fresh' | 'recovery' = 'fresh', + ): Promise { if (shuttingDown) { throw new Error('AcpSessionBridge is shutting down'); } @@ -4656,8 +4693,23 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { // would either hang or land the caller with a sessionId that // immediately 404s on every follow-up. cancelIdleTimer(); - if (channelInfo && !channelInfo.isDying) return channelInfo; + const admissible = admissibleChannelInfo(); + if (admissible) return admissible; if (inFlightChannelSpawn) return await inFlightChannelSpawn; + const workOwningGenerations = [...aliveChannels].filter( + (info) => info.state !== 'dying', + ); + if (workOwningGenerations.length >= 2) { + writeStderrLine( + `qwen serve: runtime recycling blocked ${admission} work; generations=${workOwningGenerations + .map( + (info) => + `${info.id}:${info.state}:transportFailed=${info.transportFailed}`, + ) + .join(',')}`, + ); + throw new BridgeRuntimeRecyclingError(); + } const promise = (async () => { const privateParentCapability = randomBytes(32).toString('base64url'); @@ -4717,7 +4769,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { // instead of throwing. Surface that ambiguity loudly. (sessionId) => { if (sessionId) return byId.get(sessionId); - if (channelInfo && channelInfo.sessionIds.size > 1) { + if (sessionIds.size > 1) { throw new Error( 'BridgeClient: ACP call without sessionId on a ' + 'multi-session channel cannot be routed — workspace=' + @@ -4812,9 +4864,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { .catch(() => undefined); }, opts.onChannelDelivery, - () => - channelInfo?.sessionIds === sessionIds && - channelInfo.sessionSpawnsInFlight > 0, + () => (infoRef.current?.sessionSpawnsInFlight ?? 0) > 0, () => liveScreenContextCaptureHandler, () => liveTaskToolRequestHandler, () => liveSpeakToUserHandler, @@ -4831,6 +4881,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { // nothing else would settle what its last drain missed. settleMidTurnQueueAfterAutomaticTurn, opts.onCreateCurrentSessionScheduledTask, + requestRuntimeRecycleForSession, async (sessionId, turn, afterPromptId) => { const entry = byId.get(sessionId); if ( @@ -4940,7 +4991,13 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { newSessionCleanupFailed: false, transportFailed: false, transportFailureInitiatedTeardown: false, - isDying: false, + state: 'active', + get isDying() { + return this.state === 'dying'; + }, + set isDying(value) { + if (value) this.state = 'dying'; + }, isQuarantined: false, handshakeComplete: false, }; @@ -4961,22 +5018,15 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { markTransportFailed, ); aliveChannels.add(info); - // Belt-and-suspenders leak detection. The set is intentionally - // multi-entry to cover the `killSession`-then-`spawnOrAttach` - // overlap window (size 2 is legitimate: one dying + one fresh - // attach-target). Anything higher implies a `channel.exited` - // handler never fired for some prior channel — a real leak we'd - // otherwise notice only as gradually-growing RSS over hours. - // The warning surfaces it the moment it happens. Threshold is - // 2 because that's the design ceiling; bumping it requires - // updating both this guard and the comments around - // `aliveChannels` declaration. + // Recovery can temporarily exceed the fresh-work ceiling while dying + // children await OS reap. Surface that overlap so a persistent one is + // diagnosable; `channel.exited` remains the only removal authority. if (aliveChannels.size > 2) { writeStderrLine( `qwen serve: WARNING aliveChannels.size=${aliveChannels.size} ` + - `(expected 1, max 2 during killSession-then-spawnOrAttach ` + - `overlap) — possible channel leak; check that prior channels' ` + - `channel.exited fired and the handler ran cleanup.`, + `during runtime recovery; states=${[...aliveChannels] + .map((entry) => `${entry.id}:${entry.state}`) + .join(',')}`, ); } @@ -5331,10 +5381,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { }), onFailure: failChannelLiveness, isActive: () => - channelInfo === info && - aliveChannels.has(info) && - !info.isDying && - !shuttingDown, + aliveChannels.has(info) && !info.isDying && !shuttingDown, }); } telemetry.metrics?.channelLifecycle('spawn'); @@ -5551,7 +5598,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { // `ensureChannel`, never spawning a fresh one. Tear down the // empty channel so the next attempt gets a clean spawn. const channelPath = - channelInfo && !channelInfo.isDying + channelInfo?.state === 'active' ? 'reused' : inFlightChannelSpawn ? 'joined' @@ -5564,7 +5611,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { }, ensureChannel, ); - if (ci.isDying) { + if (ci.state !== 'active') { throw new BridgeChannelClosedError('before newSession'); } ci.sessionSpawnsInFlight++; @@ -5740,7 +5787,13 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { // lifecycle marker before installing a session from a response that was // admitted immediately ahead of the fatal frame. await Promise.resolve(); - if (ci.isDying) { + // Same three-state test as the pre-`newSession` twin above, NOT + // `ci.isDying`: a recycle landing inside the `newSession` round-trip + // leaves the channel `draining` with `isDying === false` (retirement is + // deferred while this very spawn is in flight, so nothing kills it), and + // installing the session would route fresh work back to the condemned + // generation — and pin it open until that session closed. + if (ci.state !== 'active') { throw new BridgeChannelClosedError('after newSession'); } @@ -6348,10 +6401,11 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { return workspaceKey; }; - const liveChannelInfo = (): ChannelInfo | undefined => { - if (!channelInfo || channelInfo.isDying) return undefined; - return channelInfo; - }; + const liveChannelInfo = (): ChannelInfo | undefined => + channelInfo && !channelInfo.isDying ? channelInfo : undefined; + + const admissibleChannelInfo = (): ChannelInfo | undefined => + channelInfo?.state === 'active' ? channelInfo : undefined; const channelInfoForEntry = ( entry: SessionEntry, @@ -8631,8 +8685,8 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { }; const promise = (async (): Promise => { pendingRestoreEvents.set(req.sessionId, restoreEvents); - const restoreChannel = await ensureChannel(); - if (restoreChannel.isDying) { + const restoreChannel = await ensureChannel('recovery'); + if (restoreChannel.state !== 'active') { throw new BridgeChannelClosedError(`before session/${action}`); } ci = restoreChannel; @@ -8868,7 +8922,16 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { restoreEvents.close(); throw new Error('AcpSessionBridge is shutting down'); } - if (ci.isDying || !aliveChannels.has(ci)) { + // Same three-state test as the post-`newSession` twin above, NOT + // `ci.isDying`: a recycle landing inside the restore round-trip leaves + // the channel `draining` with `isDying === false` — retirement is + // deferred while this very restore is in flight (`hasNoSessionWork` + // counts `pendingRestoreCount`), so nothing kills it. Installing the + // restored session would route fresh work back to the generation the + // daemon just judged unsafe for fresh work, and would pin it open until + // that session closed, because `retireWhenSessionsDrain` can no longer + // fire while `sessionIds.size > 0`. + if (ci.state !== 'active' || !aliveChannels.has(ci)) { restoreEvents.close(); throw new Error( `Session ${req.sessionId} restored on a closed agent channel`, @@ -9794,7 +9857,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { }, isChannelLive() { - return !!liveChannelInfo(); + return liveChannelInfo() !== undefined; }, getWorkspaceRuntimeLifecycleSnapshot() { @@ -9811,7 +9874,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { } const starting = inFlightChannelSpawn !== undefined; const stopping = Array.from(aliveChannels).some( - (candidate) => candidate.isDying, + (candidate) => candidate.state !== 'active', ); const reservedWork = runtimeOperationReservations > 0 || @@ -9840,6 +9903,10 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { }; }, + async requestRuntimeRecycle(sessionId) { + await requestRuntimeRecycleForSession(sessionId); + }, + get pendingPermissionCount() { return permissionMediator.pendingCount; }, @@ -11626,7 +11693,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { }; } - const ci = await ensureChannel(); + const ci = await ensureChannel('recovery'); let restored; try { const hideInheritedHistory = req.replayInheritedHistory === false; @@ -14830,7 +14897,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { }, async generateWorkspaceAgent(description, _originatorClientId) { - const info = liveChannelInfo(); + const info = admissibleChannelInfo(); if (!info) { throw new SessionNotFoundError('agents:generate'); } @@ -14952,7 +15019,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { // success. Soft-refuse (`budget_warning_only`) returns the skip // shape without emitting — the caller (HTTP route) decides how to // surface the skip to the SDK consumer. - const info = liveChannelInfo(); + const info = admissibleChannelInfo(); if (!info) { throw Object.assign( new Error(`No live ACP channel for runtime MCP add: ${name}`), @@ -15008,7 +15075,7 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { // Round-trip the runtime-remove ext-method through // the live ACP child and broadcast `mcp_server_removed` on success. // Idempotent skip (`not_present`) returns without emitting. - const info = liveChannelInfo(); + const info = admissibleChannelInfo(); if (!info) { throw Object.assign( new Error(`No live ACP channel for runtime MCP remove: ${name}`), diff --git a/packages/acp-bridge/src/bridgeClient.ts b/packages/acp-bridge/src/bridgeClient.ts index 9dec778a2f4..11425593eb0 100644 --- a/packages/acp-bridge/src/bridgeClient.ts +++ b/packages/acp-bridge/src/bridgeClient.ts @@ -964,6 +964,8 @@ export class BridgeClient implements Client { */ private readonly onAutomaticTurnEnded?: (sessionId: string) => void, private readonly onCreateCurrentSessionScheduledTask?: CurrentSessionScheduledTaskCreateHandler, + /** Owner-scoped runtime recycle; wired only by the managed daemon bridge. */ + private readonly onRuntimeRecycle?: (sessionId: string) => Promise, private readonly onBackgroundTurnStart?: ( sessionId: string, turn: BackgroundNotificationTurn, @@ -1434,7 +1436,9 @@ export class BridgeClient implements Client { * `qwen/control/client_mcp/message` (reverse tool channel), * `qwen/control/create-sub-session` (the `create_sub_session` tool → daemon * spawns a sub-session and, for `'first-turn'`, returns its first-turn - * result), and `craft/drainMidTurnQueue`: the ACP child calls the last one + * result), `qwen/control/session/runtime/recycle` (trusted owner-generation + * recycle after an Agent ignores abort), and `craft/drainMidTurnQueue`: the + * ACP child calls the last one * between tool batches to pull any messages the browser queued mid-turn. We splice the per-session * queue, return them to the child as the response, and — when non-empty — * publish a `mid_turn_message_injected` SSE frame so the browser can move @@ -1448,6 +1452,23 @@ export class BridgeClient implements Client { method: string, params: Record, ): Promise> { + if (method === SERVE_CONTROL_EXT_METHODS.sessionRuntimeRecycle) { + if (!this.onRuntimeRecycle) throw RequestError.methodNotFound(method); + const sessionId = params['sessionId']; + if ( + typeof sessionId !== 'string' || + !this.ownsSession(sessionId) || + !this.resolveEntry(sessionId) || + params['reason'] !== 'unresponsive_agent' + ) { + throw RequestError.invalidParams( + undefined, + 'Invalid unresponsive Agent runtime recycle request.', + ); + } + await this.onRuntimeRecycle(sessionId); + return { accepted: true }; + } if (method === '_qwencode/start_turn') { const sessionId = params['sessionId']; const turn = parseBackgroundNotificationTurn(params); diff --git a/packages/acp-bridge/src/bridgeErrors.ts b/packages/acp-bridge/src/bridgeErrors.ts index 6171efd6a9f..2ed7a9d356f 100644 --- a/packages/acp-bridge/src/bridgeErrors.ts +++ b/packages/acp-bridge/src/bridgeErrors.ts @@ -696,6 +696,17 @@ export class BridgeChannelQuarantinedError extends Error { } } +export class BridgeRuntimeRecyclingError extends Error { + readonly code = 'runtime_recycling'; + + constructor() { + super( + 'The ACP runtime is recycling; retry after an older generation exits', + ); + this.name = 'BridgeRuntimeRecyclingError'; + } +} + export class InvalidRewindTargetError extends Error { readonly sessionId: string; constructor(sessionId: string, message?: string) { diff --git a/packages/acp-bridge/src/bridgeTypes.ts b/packages/acp-bridge/src/bridgeTypes.ts index be4f8337f60..e181f0069f2 100644 --- a/packages/acp-bridge/src/bridgeTypes.ts +++ b/packages/acp-bridge/src/bridgeTypes.ts @@ -2588,13 +2588,10 @@ export interface AcpSessionBridge extends WorkspaceEventBridge { readonly sessionCount: number; /** - * Whether an ACP channel is currently live (spawned and not dying). - * Distinct from `sessionCount > 0`: a channel can be live with zero - * attached sessions during the cold-spawn window, and conversely a - * killed channel may briefly retain sessions before reaping. Consumers - * that need true channel liveness (e.g. the workspace service's - * `acpChannelLive` envelope field) must use this rather than the - * session count. + * Whether an ACP channel is active and can accept fresh workspace work. + * Distinct from `sessionCount > 0`: a channel can be active with zero + * attached sessions during the cold-spawn window, while a draining + * generation still owns existing sessions but cannot accept new work. */ isChannelLive(): boolean; @@ -2605,6 +2602,12 @@ export interface AcpSessionBridge extends WorkspaceEventBridge { */ getWorkspaceRuntimeLifecycleSnapshot?(): BridgeWorkspaceRuntimeLifecycleSnapshot; + /** + * Stop admitting fresh work to the generation that owns `sessionId` and + * prepare a replacement without moving existing Sessions between children. + */ + requestRuntimeRecycle?(sessionId: string): Promise; + /** Number of sessions with an active prompt. */ readonly activePromptCount: number; diff --git a/packages/acp-bridge/src/status.ts b/packages/acp-bridge/src/status.ts index 3c94586c26e..b80905f36f5 100644 --- a/packages/acp-bridge/src/status.ts +++ b/packages/acp-bridge/src/status.ts @@ -191,6 +191,8 @@ export const SERVE_CONTROL_EXT_METHODS = { sessionLiveConversation: 'qwen/control/session/live-conversation', sessionLiveTranscript: 'qwen/control/session/live-transcript', sessionBackgroundNotification: 'qwen/control/session/background_notification', + /** Private child→daemon request for an abort-ignoring background Agent. */ + sessionRuntimeRecycle: 'qwen/control/session/runtime/recycle', sessionArtifactsPersist: 'qwen/control/session/artifacts/persist', workspaceMcpRestart: 'qwen/control/workspace/mcp/restart', workspaceMcpManage: 'qwen/control/workspace/mcp/manage', diff --git a/packages/cli/src/acp-integration/session/Session.ts b/packages/cli/src/acp-integration/session/Session.ts index 3d0f56f3016..a5c31a3a343 100644 --- a/packages/cli/src/acp-integration/session/Session.ts +++ b/packages/cli/src/acp-integration/session/Session.ts @@ -1509,6 +1509,7 @@ export interface BackgroundNotificationQueueItem { interface QueuedBackgroundNotification extends BackgroundNotificationQueueItem { continuesTodoStopGuardWorkChain: boolean; persisted?: true; + recordOnly?: true; turn?: BackgroundNotificationTurn; admissionRetries?: number; } @@ -10271,15 +10272,13 @@ export class Session implements SessionContext { (entry ? buildBackgroundEntryLabel(entry, { includePrefix: false }) : undefined); - this.#enqueueBackgroundNotification({ + const item: BackgroundNotificationQueueItem = { displayText, modelText, taskId: meta.agentId, sourceTurnId: meta.sourceTurnId, status: meta.status, kind: 'agent', - continuesTodoStopGuardWorkChain: - this.#agentContinuesTodoStopGuardWorkChain(meta.agentId), toolUseId: meta.toolUseId, todoWorkChainId: meta.todoWorkChainId, label: label ? truncateNotificationLabel(label) : undefined, @@ -10290,6 +10289,15 @@ export class Session implements SessionContext { ), } : undefined, + }; + if (meta.recordOnly) { + void this.#recordUnresponsiveAgentNotification(item); + return; + } + this.#enqueueBackgroundNotification({ + ...item, + continuesTodoStopGuardWorkChain: + this.#agentContinuesTodoStopGuardWorkChain(meta.agentId), }); }, ); @@ -10513,6 +10521,48 @@ export class Session implements SessionContext { void this.#drainNotificationQueue(); } + async #recordUnresponsiveAgentNotification( + item: BackgroundNotificationQueueItem, + ): Promise { + this.activeNotificationAcceptances.add(item.taskId); + this.#activeWorkChanged(); + try { + const accepted = await this.#persistDaemonBackgroundNotification( + item, + false, + ); + if (!this.disposed && !this.closing) { + this.#enqueueBackgroundNotification({ + ...item, + continuesTodoStopGuardWorkChain: + this.#agentContinuesTodoStopGuardWorkChain(item.taskId), + ...(accepted ? { persisted: true } : {}), + recordOnly: true, + }); + } + } catch (error) { + debugLogger.warn( + `Unresponsive Agent notification failed [session ${this.sessionId}, task ${item.taskId}]: ${this.#formatError(error)}`, + ); + } finally { + try { + await this.client.extMethod( + SERVE_CONTROL_EXT_METHODS.sessionRuntimeRecycle, + { + sessionId: this.sessionId, + reason: 'unresponsive_agent', + }, + ); + } catch (error) { + debugLogger.warn( + `Unresponsive Agent runtime recycle failed [session ${this.sessionId}, task ${item.taskId}]: ${this.#formatError(error)}`, + ); + } + this.activeNotificationAcceptances.delete(item.taskId); + this.#activeWorkChanged(); + } + } + async enqueueBackgroundNotification( item: BackgroundNotificationQueueItem, ): Promise<{ accepted: boolean }> { @@ -10545,6 +10595,7 @@ export class Session implements SessionContext { async #persistDaemonBackgroundNotification( item: BackgroundNotificationQueueItem, + enqueue = true, ): Promise { if (this.disposed || this.closing) return false; const recording = this.config.getChatRecordingService(); @@ -10570,7 +10621,7 @@ export class Session implements SessionContext { } this.persistedBackgroundNotificationTaskIds.add(item.taskId); - if (!this.disposed && !this.closing) { + if (enqueue && !this.disposed && !this.closing) { this.#enqueueBackgroundNotification({ ...item, continuesTodoStopGuardWorkChain: @@ -10767,6 +10818,20 @@ export class Session implements SessionContext { this.currentNotificationWorkChainId = item.todoWorkChainId; this.#activeWorkChanged(); try { + if (item.recordOnly) { + try { + await this.#emitBackgroundNotificationDisplay(item).catch( + (error) => { + debugLogger.warn( + `Unresponsive Agent notification display failed [session ${this.sessionId}, task ${item.taskId}]: ${this.#formatError(error)}`, + ); + }, + ); + } finally { + await this.#emitBackgroundNotificationEndTurn('end_turn'); + } + continue; + } // A notification fires from async resources created inside the // turn that spawned the task, so a Goal permit can reach here by // lineage after that turn is long over. This is not a Goal turn: @@ -11406,6 +11471,10 @@ export class Session implements SessionContext { async #emitBackgroundNotificationEndTurn( reason: PromptResponse['stopReason'], + // Omitted by the record-only path, which displays a terminal notification + // without running a model turn. There is no background turn to close, so + // the bridge settles it as a bare `background_notification_turn_complete`; + // a turnId would name no live turn and be dropped. turnId?: string, ): Promise { try { diff --git a/packages/cli/src/nonInteractiveCli.ts b/packages/cli/src/nonInteractiveCli.ts index bfb9d559cf7..efcc94b54a2 100644 --- a/packages/cli/src/nonInteractiveCli.ts +++ b/packages/cli/src/nonInteractiveCli.ts @@ -879,6 +879,7 @@ export async function runNonInteractive( sendMessageType: SendMessageType; todoWorkChainId?: string; monitorId?: string; + recordOnly?: true; sdkNotification?: { task_id: string; tool_use_id?: string; @@ -1513,7 +1514,7 @@ export async function runNonInteractive( // tool-call chain can push completions onto the queue. const registry = config.getBackgroundTaskRegistry(); registry.setNotificationCallback((displayText, modelText, meta) => { - localQueue.push({ + const item: LocalQueueItem = { displayText, modelText, sendMessageType: SendMessageType.Notification, @@ -1530,7 +1531,12 @@ export async function runNonInteractive( } : undefined, }, - }); + }; + if (meta.recordOnly) { + localQueue.push({ ...item, recordOnly: true }); + return; + } + localQueue.push(item); }); registry.setRegisterCallback((entry) => { @@ -2801,11 +2807,14 @@ export async function runNonInteractive( emitNotificationToSdk(queueItem); } + const modelBatch = batch.filter((item) => !item.recordOnly); + if (modelBatch.length === 0) return; + const item = { - displayText: batch.map((i) => i.displayText).join('; '), - modelText: batch.map((i) => i.modelText).join('\n\n'), + displayText: modelBatch.map((i) => i.displayText).join('; '), + modelText: modelBatch.map((i) => i.modelText).join('\n\n'), sendMessageType: targetType, - todoWorkChainId: batch[0]?.todoWorkChainId, + todoWorkChainId: modelBatch[0]?.todoWorkChainId, }; turnCount++; diff --git a/packages/cli/src/serve/acp-http/dispatch.ts b/packages/cli/src/serve/acp-http/dispatch.ts index 275ef89df37..af67e44fb37 100644 --- a/packages/cli/src/serve/acp-http/dispatch.ts +++ b/packages/cli/src/serve/acp-http/dispatch.ts @@ -41,6 +41,7 @@ import { } from '../acp-session-bridge.js'; import type { BridgeChannelQuarantinedError, + BridgeRuntimeRecyclingError, BridgeTimeoutError, RestoreInProgressError, SessionRestoreTimeoutError, @@ -932,6 +933,19 @@ export function toRpcError(err: unknown): { }, }; } + case 'BridgeRuntimeRecyclingError': { + const recyclingError = err as BridgeRuntimeRecyclingError; + return { + code: RPC.INTERNAL_ERROR, + message: recyclingError.message, + data: { + code: recyclingError.code, + errorKind: recyclingError.code, + httpStatus: 503, + retryable: true, + }, + }; + } case 'SessionArchivedError': return { code: RPC.INTERNAL_ERROR, diff --git a/packages/cli/src/serve/acp-session-bridge.ts b/packages/cli/src/serve/acp-session-bridge.ts index af8d4e3c7d4..25740130158 100644 --- a/packages/cli/src/serve/acp-session-bridge.ts +++ b/packages/cli/src/serve/acp-session-bridge.ts @@ -143,6 +143,7 @@ export { SessionResetPendingError, WorkspaceDrainingError, BridgeChannelQuarantinedError, + BridgeRuntimeRecyclingError, InvalidRewindTargetError, TotalSessionLimitExceededError, NOT_CURRENTLY_GENERATING_CANCEL_MESSAGE, diff --git a/packages/cli/src/serve/server/error-response.ts b/packages/cli/src/serve/server/error-response.ts index d182a273ceb..92bfb109b8c 100644 --- a/packages/cli/src/serve/server/error-response.ts +++ b/packages/cli/src/serve/server/error-response.ts @@ -25,6 +25,7 @@ import { AcpChildCapacityExceededError, BranchWhilePromptActiveError, BridgeChannelQuarantinedError, + BridgeRuntimeRecyclingError, BridgeTimeoutError, CancelSentinelCollisionError, CdWhilePromptActiveError, @@ -380,6 +381,16 @@ export function sendBridgeError( }); return; } + if (err instanceof BridgeRuntimeRecyclingError) { + recordExpectedBridgeError(err, ctx, daemonLog); + res.status(503).json({ + error: err.message, + code: err.code, + errorKind: err.code, + retryable: true, + }); + return; + } if (err instanceof DaemonDrainingError) { res.status(503).json({ error: err.message, diff --git a/packages/cli/src/ui/hooks/use-llm-stream.ts b/packages/cli/src/ui/hooks/use-llm-stream.ts index 995fc4f313a..0720fbf7d3b 100644 --- a/packages/cli/src/ui/hooks/use-llm-stream.ts +++ b/packages/cli/src/ui/hooks/use-llm-stream.ts @@ -6311,6 +6311,7 @@ export const useLlmStream = ( useEffect(() => { const registry = config.getBackgroundTaskRegistry(); registry.setNotificationCallback((displayText, modelText, meta) => { + if (meta?.recordOnly) return; admitNotification({ displayText, modelText, diff --git a/packages/cli/src/ui/utils/backgroundWorkUtils.ts b/packages/cli/src/ui/utils/backgroundWorkUtils.ts index d1376a98cb3..2b5fbf3cfa6 100644 --- a/packages/cli/src/ui/utils/backgroundWorkUtils.ts +++ b/packages/cli/src/ui/utils/backgroundWorkUtils.ts @@ -75,12 +75,12 @@ export interface BlockingBackgroundWork { /** * Enumerates the entries that make `hasBlockingBackgroundWork()` true, * mirroring its per-registry predicate exactly (background agents: - * `isBackgrounded` + `running`; monitors: `running`; shells: `running`; - * workflow runs: `running` or `pausing`, plus the reserved-but- - * unregistered runs `hasRunningEntries()` counts via `starting`). Returns `undefined` - * when nothing is enumerated — e.g. an entry settled between the gate check - * and this call — so callers fall back to their base message instead of - * rendering an empty list. + * `isBackgrounded` + `running` or a retained physical slot; monitors: + * `running`; shells: `running`; workflow runs: `running` or `pausing`, plus + * the reserved-but-unregistered runs `hasRunningEntries()` counts via + * `starting`). Returns `undefined` when nothing is enumerated — e.g. an entry + * settled between the gate check and this call — so callers fall back to their + * base message instead of rendering an empty list. */ export function describeBlockingBackgroundWork( config: Config, @@ -96,11 +96,16 @@ export function describeBlockingBackgroundWork( }> = []; for (const entry of config.getBackgroundTaskRegistry().getAll()) { - if (!entry.isBackgrounded || entry.status !== 'running') continue; + if ( + !entry.isBackgrounded || + (entry.status !== 'running' && !entry.retainsPhysicalSlot) + ) + continue; + const label = buildBackgroundEntryLabel(entry); entries.push({ startTime: entry.startTime, id: entry.agentId, - label: buildBackgroundEntryLabel(entry), + label: entry.retainsPhysicalSlot ? `${label} — still stopping` : label, status: entry.status, isWorkflowRun: false, }); diff --git a/packages/core/src/agents/background-agent-resume.ts b/packages/core/src/agents/background-agent-resume.ts index be24a7a3f63..d0d931c2c06 100644 --- a/packages/core/src/agents/background-agent-resume.ts +++ b/packages/core/src/agents/background-agent-resume.ts @@ -18,6 +18,10 @@ import { import { AgentTerminateMode } from './runtime/agent-types.js'; import { AgentHeadless, ContextState } from './runtime/agent-headless.js'; import type { SubagentExecutor } from './runtime/subagent-executor.js'; +import { + attachAgentProgressWatchdog, + getAgentProgressTimeout, +} from './runtime/agent-progress-watchdog.js'; import { buildAgentTranscriptAttach, getAgentJsonlPath, @@ -1276,7 +1280,11 @@ export class BackgroundAgentResumeService { }); } - const terminateMode = subagent.getTerminateMode(); + const terminateMode = getAgentProgressTimeout( + turnAbortController.signal, + ) + ? AgentTerminateMode.TIMEOUT + : subagent.getTerminateMode(); const modelVisibleText = toModelVisibleSubagentResult( subagent.getFinalText(), terminateMode, @@ -1289,6 +1297,7 @@ export class BackgroundAgentResumeService { stopHookWarning, ); const stats = getCompletionStats(subagent, liveToolCallCount); + if (registry.get(meta.agentId)?.retainsPhysicalSlot) break; if (terminateMode === AgentTerminateMode.GOAL) { const pending = registry.drainMessages(meta.agentId); if (pending.length > 0) { @@ -1316,7 +1325,10 @@ export class BackgroundAgentResumeService { : {}), }); registry.complete(meta.agentId, finalText, stats); - } else if (terminateMode === AgentTerminateMode.CANCELLED) { + } else if ( + terminateMode === AgentTerminateMode.CANCELLED || + registry.get(meta.agentId)?.status === 'cancelled' + ) { registry.finalizeCancelled(meta.agentId, finalText, stats); persistBackgroundCancellation( metaPath, @@ -1345,12 +1357,21 @@ export class BackgroundAgentResumeService { break; } } catch (error) { + const progressTimeout = getAgentProgressTimeout( + turnAbortController.signal, + ); const errorMessage = - error instanceof Error ? error.message : String(error); + progressTimeout?.message ?? + (error instanceof Error ? error.message : String(error)); debugLogger.error( `[BackgroundAgentResume] Background agent failed: ${errorMessage}`, ); - if (turnAbortController.signal.aborted) { + if (registry.get(meta.agentId)?.retainsPhysicalSlot) return; + if ( + turnAbortController.signal.aborted && + (!progressTimeout || + registry.get(meta.agentId)?.status === 'cancelled') + ) { const stats = getCompletionStats(subagent, liveToolCallCount); registry.finalizeCancelled(meta.agentId, errorMessage, stats); persistBackgroundCancellation( @@ -1390,6 +1411,12 @@ export class BackgroundAgentResumeService { turnAbortController: AbortController, fireStartHook: boolean, ) => { + const disposeWatchdog = attachAgentProgressWatchdog( + bgEmitter, + turnAbortController, + () => monitorRegistry.hasRunningForOwner(meta.agentId), + (error) => registry.failUnresponsive(meta.agentId, error.message), + ); // Restore the persisted launch depth so a resumed nested agent keeps // its original nesting level (and spawn eligibility) instead of // recomputing to depth 0 from this top-level resume frame. @@ -1401,9 +1428,14 @@ export class BackgroundAgentResumeService { ); const invocationRunBody = () => runWithInvocationContext(undefined, framedRunBody); - return target.isFork - ? runInForkContext(invocationRunBody) - : invocationRunBody(); + return ( + target.isFork + ? runInForkContext(invocationRunBody) + : invocationRunBody() + ).finally(() => { + disposeWatchdog(); + registry.releaseRetainedPhysicalSlot(meta.agentId); + }); }; const reportUnexpectedBackgroundError = (error: unknown) => { diff --git a/packages/core/src/agents/background-tasks.test.ts b/packages/core/src/agents/background-tasks.test.ts index 65b1015d9c4..63d20776cfe 100644 --- a/packages/core/src/agents/background-tasks.test.ts +++ b/packages/core/src/agents/background-tasks.test.ts @@ -15,6 +15,7 @@ import { type AgentTaskRegistration, type BackgroundApproval, type BackgroundTaskEntry, + type NotificationMeta, type ResidentBackgroundAgent, } from './background-tasks.js'; import { @@ -342,6 +343,85 @@ describe('BackgroundTaskRegistry', () => { expect(displayText).toContain('failed'); }); + it('retains the physical slot when the watchdog escalates a cancelled agent', () => { + const callback = vi.fn(); + registry.setNotificationCallback(callback); + + registry.register({ + agentId: 'test-1', + description: 'test agent', + status: 'running', + startTime: Date.now(), + abortController: new AbortController(), + isBackgrounded: true, + outputFile: '/tmp/test.jsonl', + }); + + // task_stop wins the abort-grace race: the watchdog aborted first, but + // the user cancels before the five-second escalation fires. The + // abort-ignoring execution is still holding its physical slot, so the + // escalation must retain it and request runtime recycle even though the + // visible status became `cancelled`. + registry.cancel('test-1'); + expect(registry.get('test-1')!.status).toBe('cancelled'); + + registry.failUnresponsive( + 'test-1', + 'Background agent made no model/control progress for 900000ms.', + ); + + const entry = registry.get('test-1')!; + expect(entry.status).toBe('failed'); + expect(entry.retainsPhysicalSlot).toBe(true); + expect(callback).toHaveBeenCalledOnce(); + const [, , meta] = callback.mock.calls[0] as [ + string, + string, + NotificationMeta, + ]; + expect(meta.recordOnly).toBe(true); + }); + + it('retains the physical slot when the cancel grace timer finalizes before the escalation', () => { + const callback = vi.fn(); + registry.setNotificationCallback(callback); + + registry.register({ + agentId: 'test-1', + description: 'test agent', + status: 'running', + startTime: Date.now(), + abortController: new AbortController(), + isBackgrounded: true, + outputFile: '/tmp/test.jsonl', + }); + + // The cancel grace timer wins the race this time instead of the user's + // `task_stop`: the escalation timer is drift-guarded and re-arms when the + // event loop runs more than a second past its due time, while + // `CANCEL_GRACE_MS` is a bare setTimeout — so the cancellation is already + // finalized (terminal notification delivered, `notified` set) by the time + // the escalation callback lands. `onUnresponsive` can only fire while the + // execution is provably still alive, so the slot must still be retained. + registry.cancel('test-1'); + registry.finalizeCancellationIfPending('test-1'); + expect(registry.get('test-1')!.status).toBe('cancelled'); + expect(registry.get('test-1')!.notified).toBe(true); + expect(registry.get('test-1')!.retainsPhysicalSlot).toBeUndefined(); + + registry.failUnresponsive( + 'test-1', + 'Background agent made no model/control progress for 900000ms.', + ); + + const entry = registry.get('test-1')!; + expect(entry.retainsPhysicalSlot).toBe(true); + // The occupied slot stays visible to the reset / session-switch guards. + expect(registry.hasRunningTasks()).toBe(true); + // ...and the already-delivered terminal notification is not re-fired. + expect(callback).toHaveBeenCalledOnce(); + }); + describe('resident background agents', () => { function makeResident( overrides: Partial = {}, diff --git a/packages/core/src/agents/background-tasks.ts b/packages/core/src/agents/background-tasks.ts index 4ae2f742e78..8ede9114c6d 100644 --- a/packages/core/src/agents/background-tasks.ts +++ b/packages/core/src/agents/background-tasks.ts @@ -373,6 +373,8 @@ export interface AgentTask extends TaskBase { * `running` so `/resume` can recover the work later. */ persistedCancellationStatus?: Extract; + /** The underlying run ignored abort and still occupies a physical slot. */ + retainsPhysicalSlot?: true; } /** @@ -404,6 +406,7 @@ export interface NotificationMeta { toolUseId?: string; todoWorkChainId?: string; label?: string; + recordOnly?: true; } export type BackgroundNotificationCallback = ( @@ -924,6 +927,52 @@ export class BackgroundTaskRegistry { this.drainWaitQueue(); } + // Deliberately NOT gated on `entry.notified`. The cancel-grace timer + // (`CANCEL_GRACE_MS`) can finalize the cancellation — and emit the terminal + // "was cancelled" notification, setting `notified` — *before* this + // escalation lands: the escalation timer is drift-guarded and re-arms + // instead of firing when the event loop runs more than a second past its + // due time, while the cancel grace timer is a bare `setTimeout`, so a single + // stall is enough for the cancel side to win. Reaching this method at all + // proves the execution is still alive (the watchdog is detached once it + // settles), so the physical slot must still be retained and the entry + // settled — otherwise `getRunningBackgroundCount` and `hasRunningTasks()` + // free a concurrency slot that is still occupied, and `/clear`, `/resume`, + // `/branch` and session switches all proceed over live work. Only the + // notification is suppressed, and `emitNotification` is itself idempotent + // (`if (entry.notified) return`), so the already-delivered terminal + // notification is never re-fired. + failUnresponsive(agentId: string, error: string): void { + const entry = this.agents.get(agentId); + if (!entry) return; + if (entry.status !== 'running' && entry.status !== 'cancelled') return; + + entry.status = 'failed'; + entry.endTime = Date.now(); + entry.error = error; + entry.retainsPhysicalSlot = true; + if (entry.metaPath) { + patchAgentMeta(entry.metaPath, { + status: 'failed', + lastUpdatedAt: new Date().toISOString(), + lastError: error, + }); + } + this.releaseFinishingWaiters(agentId, true); + this.rejectPendingApprovals(entry); + this.emitNotification(entry, true); + this.emitStatusChange(entry); + this.disposeResidentAgent(agentId); + } + + releaseRetainedPhysicalSlot(agentId: string): void { + const entry = this.agents.get(agentId); + if (!entry?.retainsPhysicalSlot) return; + delete entry.retainsPhysicalSlot; + this.emitStatusChange(entry); + this.drainWaitQueue(); + } + // Cancellation aborts the signal and marks the entry as cancelled, but // does *not* emit the terminal notification immediately. The natural // completion path (bgBody) fires complete()/fail()/finalizeCancelled() @@ -1296,16 +1345,17 @@ export class BackgroundTaskRegistry { return Array.from(this.agents.values()); } - // Counts backgrounded agents that still occupy a slot: running, or - // cancelled-but-not-yet-finalized. When `model` is given, only agents on - // that model are counted (per-model cap); otherwise all of them (global). + // Counts backgrounded agents that still occupy a slot: running, + // cancelled-but-not-yet-finalized, or watchdog-terminal but not physically + // settled. When `model` is given, only agents on that model are counted. private getRunningBackgroundCount(model?: string): number { let count = 0; for (const entry of this.agents.values()) { const occupiesSlot = entry.isBackgrounded && (entry.status === 'running' || - (entry.status === 'cancelled' && !entry.notified)); + (entry.status === 'cancelled' && !entry.notified) || + entry.retainsPhysicalSlot === true); if (!occupiesSlot) { continue; } @@ -1461,13 +1511,15 @@ export class BackgroundTaskRegistry { * registry right after passing the gate, which suppresses that very * notification, so blocking on it made the command silently no-op * when the user cleared immediately after cancelling (issue #5949). + * A watchdog-terminal run still counts while its underlying execution holds + * a physical slot, so session reset cannot erase the only remaining owner. * Headless holdback loops must keep using `hasUnfinalizedTasks()` so * every task_started still pairs with a task_notification. */ hasRunningTasks(): boolean { for (const entry of this.agents.values()) { if (!entry.isBackgrounded) continue; - if (entry.status === 'running') return true; + if (entry.status === 'running' || entry.retainsPhysicalSlot) return true; } return false; } @@ -1704,7 +1756,7 @@ export class BackgroundTaskRegistry { return buildBackgroundEntryLabel(entry); } - private emitNotification(entry: AgentTask): void { + private emitNotification(entry: AgentTask, recordOnly = false): void { // Mark notified *before* invoking the callback so that a re-entrant // terminal call inside the callback chain (cancel → complete race) // sees the flag and short-circuits, rather than firing twice. @@ -1784,6 +1836,7 @@ export class BackgroundTaskRegistry { stats: entry.stats, toolUseId: entry.toolUseId, todoWorkChainId: entry.todoWorkChainId, + ...(recordOnly ? { recordOnly: true } : {}), label: buildBackgroundEntryLabel(entry, { includePrefix: false }), }; @@ -1845,6 +1898,7 @@ export class BackgroundTaskRegistry { private pruneTerminalEntries(): void { const evictable = Array.from(this.agents.values()) .filter((entry) => entry.notified === true) + .filter((entry) => !entry.retainsPhysicalSlot) .sort( (a, b) => (a.endTime ?? a.startTime) - (b.endTime ?? b.startTime) || diff --git a/packages/core/src/agents/runtime/agent-core.ts b/packages/core/src/agents/runtime/agent-core.ts index 44a4b1f40cd..0cacb98592f 100644 --- a/packages/core/src/agents/runtime/agent-core.ts +++ b/packages/core/src/agents/runtime/agent-core.ts @@ -96,12 +96,14 @@ import type { AgentExternalInput, } from './agent-types.js'; import { AgentTerminateMode } from './agent-types.js'; +import { getAgentProgressTimeout } from './agent-progress-watchdog.js'; import type { AgentRoundEvent, AgentRoundTextEvent, AgentToolCallEvent, AgentToolResultEvent, AgentToolOutputUpdateEvent, + AgentToolProgressEvent, AgentUsageEvent, AgentHooks, AgentExternalMessageEvent, @@ -1042,7 +1044,9 @@ export class AgentCore { // Check abort before starting a new round — prevents unnecessary API // calls after processFunctionCalls was unblocked by an abort signal. if (abortController.signal.aborted) { - terminateMode = AgentTerminateMode.CANCELLED; + terminateMode = getAgentProgressTimeout(abortController.signal) + ? AgentTerminateMode.TIMEOUT + : AgentTerminateMode.CANCELLED; break; } @@ -1086,6 +1090,17 @@ export class AgentCore { DEFAULT_QWEN_MODEL, messageParams, promptId, + undefined, + { + onRetry: (retryDelayMs) => + this.eventEmitter?.emit(AgentEventType.MODEL_RETRY, { + subagentId: this.subagentId, + round: turnCounter, + promptId, + retryDelayMs, + timestamp: Date.now(), + } as AgentRoundEvent), + }, ); this.eventEmitter?.emit(AgentEventType.ROUND_START, { subagentId: this.subagentId, @@ -1114,7 +1129,11 @@ export class AgentCore { if (roundAbortController.signal.aborted) { return { text: finalText, - terminateMode: AgentTerminateMode.CANCELLED, + terminateMode: getAgentProgressTimeout( + roundAbortController.signal, + ) + ? AgentTerminateMode.TIMEOUT + : AgentTerminateMode.CANCELLED, turnsUsed: turnCounter, }; } @@ -1123,6 +1142,13 @@ export class AgentCore { // retry does not inherit stale data (e.g. wasOutputTruncated) from a // previous attempt that may have hit MAX_TOKENS. if (streamEvent.type === 'retry') { + this.eventEmitter?.emit(AgentEventType.MODEL_RETRY, { + subagentId: this.subagentId, + round: turnCounter, + promptId, + retryDelayMs: streamEvent.retryInfo?.delayMs, + timestamp: Date.now(), + } as AgentRoundEvent); if ( checkSubagentLoop({ type: LlmEventType.Retry, @@ -1372,6 +1398,7 @@ export class AgentCore { subagentId: this.subagentId, round: turnCounter, promptId, + waitingForExternalInput: true, timestamp: Date.now(), } as AgentRoundEvent); @@ -1539,7 +1566,12 @@ export class AgentCore { } if (abortController.signal.aborted) { - return { inputs: [], terminateMode: AgentTerminateMode.CANCELLED }; + return { + inputs: [], + terminateMode: getAgentProgressTimeout(abortController.signal) + ? AgentTerminateMode.TIMEOUT + : AgentTerminateMode.CANCELLED, + }; } if (!this.hasTurnBudgetForAnotherRound(options, turnCounter)) { @@ -1575,7 +1607,12 @@ export class AgentCore { waitAbortController.signal, ); if (abortController.signal.aborted) { - return { inputs: [], terminateMode: AgentTerminateMode.CANCELLED }; + return { + inputs: [], + terminateMode: getAgentProgressTimeout(abortController.signal) + ? AgentTerminateMode.TIMEOUT + : AgentTerminateMode.CANCELLED, + }; } if (timedOut) { return { inputs: [], terminateMode: AgentTerminateMode.TIMEOUT }; @@ -1588,7 +1625,12 @@ export class AgentCore { } } catch (error) { if (abortController.signal.aborted) { - return { inputs: [], terminateMode: AgentTerminateMode.CANCELLED }; + return { + inputs: [], + terminateMode: getAgentProgressTimeout(abortController.signal) + ? AgentTerminateMode.TIMEOUT + : AgentTerminateMode.CANCELLED, + }; } if (timedOut) { return { inputs: [], terminateMode: AgentTerminateMode.TIMEOUT }; @@ -2041,6 +2083,7 @@ export class AgentCore { ); } }; + const executingToolCallIds = new Set(); const scheduler = new CoreToolScheduler({ config: this.runtimeContext, shouldObserveProducer: (callId) => !emittedCallIds.has(callId), @@ -2049,8 +2092,24 @@ export class AgentCore { // for why the registry cannot answer this and what the predicate owes. hasSkillTool: () => this.canInvokeSkill(declaredToolNames), outputUpdateHandler: (callId, outputChunk) => { - // Shell liveness heartbeats have no subagent consumer; broadcasting - // one would overwrite the live output view kept in liveOutputs. + const isTaskExecutionChunk = + typeof outputChunk === 'object' && + outputChunk !== null && + 'type' in outputChunk && + outputChunk.type === 'task_execution'; + const waitingForExternalInput = + isTaskExecutionChunk && outputChunk.waitingForExternalInput === true; + const awaitingApproval = + isTaskExecutionChunk && outputChunk.awaitingApproval === true; + this.eventEmitter?.emit(AgentEventType.TOOL_PROGRESS, { + subagentId: this.subagentId, + round: currentRound, + callId, + ...(waitingForExternalInput ? { waitingForExternalInput: true } : {}), + ...(awaitingApproval ? { awaitingApproval: true } : {}), + timestamp: Date.now(), + } as AgentToolProgressEvent); + // Keep Shell liveness heartbeats out of the live output view. if (isShellProgressData(outputChunk)) { return; } @@ -2126,6 +2185,41 @@ export class AgentCore { resolveBatch?.(); }, onToolCallsUpdate: (calls: ToolCall[]) => { + for (const call of calls) { + if ( + call.status === 'success' || + call.status === 'error' || + call.status === 'cancelled' + ) { + this.eventEmitter?.emit(AgentEventType.TOOL_PROGRESS, { + subagentId: this.subagentId, + round: currentRound, + callId: call.request.callId, + settled: true, + timestamp: Date.now(), + } as AgentToolProgressEvent); + } + } + const started = calls.filter( + (call) => + call.status === 'executing' && + !executingToolCallIds.has(call.request.callId), + ); + executingToolCallIds.clear(); + for (const call of calls) { + if (call.status === 'executing') { + executingToolCallIds.add(call.request.callId); + } + } + for (const call of started) { + this.eventEmitter?.emit(AgentEventType.TOOL_PROGRESS, { + subagentId: this.subagentId, + round: currentRound, + callId: call.request.callId, + timestamp: Date.now(), + } as AgentToolProgressEvent); + } + const awaitingByCallId = new Map( calls .filter( diff --git a/packages/core/src/agents/runtime/agent-events.ts b/packages/core/src/agents/runtime/agent-events.ts index 0736216d48a..5b9456c4ea8 100644 --- a/packages/core/src/agents/runtime/agent-events.ts +++ b/packages/core/src/agents/runtime/agent-events.ts @@ -38,10 +38,12 @@ export type AgentEvent = | 'round_end' | 'round_text' | 'stream_text' + | 'model_retry' | 'tool_call' | 'tool_result' | 'tool_responses_finalized' | 'tool_output_update' + | 'tool_progress' | 'tool_waiting_approval' | 'usage_metadata' | 'external_message' @@ -56,10 +58,12 @@ export enum AgentEventType { /** Complete round text, emitted once after streaming before tool calls. */ ROUND_TEXT = 'round_text', STREAM_TEXT = 'stream_text', + MODEL_RETRY = 'model_retry', TOOL_CALL = 'tool_call', TOOL_RESULT = 'tool_result', TOOL_RESPONSES_FINALIZED = 'tool_responses_finalized', TOOL_OUTPUT_UPDATE = 'tool_output_update', + TOOL_PROGRESS = 'tool_progress', TOOL_WAITING_APPROVAL = 'tool_waiting_approval', USAGE_METADATA = 'usage_metadata', /** External user message injected mid-run (e.g. via send_message). */ @@ -83,6 +87,9 @@ export interface AgentRoundEvent { subagentId: string; round: number; promptId: string; + waitingForExternalInput?: true; + /** Expected model backoff before the next attempt starts. */ + retryDelayMs?: number; timestamp: number; } @@ -171,6 +178,21 @@ export interface AgentToolOutputUpdateEvent { timestamp: number; } +export interface AgentToolProgressEvent { + subagentId: string; + round: number; + callId: string; + /** Clears the call's watchdog deadline before batch finalization finishes. */ + settled?: true; + /** Suspends the call's watchdog deadline while nested work is parked on + Monitor-owned external input. */ + waitingForExternalInput?: true; + /** Suspends the call's watchdog deadline while nested work is parked on a + user approval (approval waits must not cause false watchdog failures). */ + awaitingApproval?: true; + timestamp: number; +} + export interface AgentApprovalRequestEvent { subagentId: string; round: number; @@ -247,10 +269,12 @@ export interface AgentEventMap { [AgentEventType.ROUND_END]: AgentRoundEvent; [AgentEventType.ROUND_TEXT]: AgentRoundTextEvent; [AgentEventType.STREAM_TEXT]: AgentStreamTextEvent; + [AgentEventType.MODEL_RETRY]: AgentRoundEvent; [AgentEventType.TOOL_CALL]: AgentToolCallEvent; [AgentEventType.TOOL_RESULT]: AgentToolResultEvent; [AgentEventType.TOOL_RESPONSES_FINALIZED]: AgentToolResponsesFinalizedEvent; [AgentEventType.TOOL_OUTPUT_UPDATE]: AgentToolOutputUpdateEvent; + [AgentEventType.TOOL_PROGRESS]: AgentToolProgressEvent; [AgentEventType.TOOL_WAITING_APPROVAL]: AgentApprovalRequestEvent; [AgentEventType.USAGE_METADATA]: AgentUsageEvent; [AgentEventType.EXTERNAL_MESSAGE]: AgentExternalMessageEvent; diff --git a/packages/core/src/agents/runtime/agent-headless.ts b/packages/core/src/agents/runtime/agent-headless.ts index 2c567d705cb..63512bf6cfa 100644 --- a/packages/core/src/agents/runtime/agent-headless.ts +++ b/packages/core/src/agents/runtime/agent-headless.ts @@ -38,6 +38,7 @@ import type { AgentExternalInput, } from './agent-types.js'; import { AgentTerminateMode } from './agent-types.js'; +import { getAgentProgressTimeout } from './agent-progress-watchdog.js'; import { logSubagentExecution } from '../../telemetry/loggers.js'; import { SubagentExecutionEvent } from '../../telemetry/types.js'; import { AgentCore, EXTERNAL_MESSAGE_PREFIX } from './agent-core.js'; @@ -390,10 +391,20 @@ export class AgentHeadless implements SubagentExecutor { }, ); - this.finalText = result.text; this.terminateMode = result.terminateMode ?? AgentTerminateMode.GOAL; + this.finalText = + result.text || + (this.terminateMode === AgentTerminateMode.TIMEOUT + ? (getAgentProgressTimeout(abortController.signal)?.message ?? '') + : ''); this.loopType = result.loopType ?? null; } catch (error) { + const progressTimeout = getAgentProgressTimeout(abortController.signal); + if (progressTimeout) { + this.finalText = progressTimeout.message; + this.terminateMode = AgentTerminateMode.TIMEOUT; + return; + } debugLogger.error('Error during subagent execution:', error); this.terminateMode = AgentTerminateMode.ERROR; this.core.eventEmitter?.emit(AgentEventType.ERROR, { diff --git a/packages/core/src/agents/runtime/agent-progress-watchdog.test.ts b/packages/core/src/agents/runtime/agent-progress-watchdog.test.ts new file mode 100644 index 00000000000..995c681b800 --- /dev/null +++ b/packages/core/src/agents/runtime/agent-progress-watchdog.test.ts @@ -0,0 +1,192 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { + AgentEventEmitter, + AgentEventType, + type AgentApprovalRequestEvent, + type AgentRoundEvent, + type AgentToolCallEvent, + type AgentToolProgressEvent, + type AgentToolResultEvent, +} from './agent-events.js'; +import { + attachAgentProgressWatchdog, + getAgentProgressTimeout, +} from './agent-progress-watchdog.js'; + +const TOOL_TIMEOUT_MS = 10 * 60_000; +const MODEL_TIMEOUT_MS = 15 * 60_000; + +describe('attachAgentProgressWatchdog', () => { + let emitter: AgentEventEmitter; + let controller: AbortController; + let onUnresponsive: ReturnType; + let detach: () => void; + + const attach = (isWaitingForExternalInput: () => boolean = () => false) => { + detach = attachAgentProgressWatchdog( + emitter, + controller, + isWaitingForExternalInput, + onUnresponsive, + ); + }; + const toolCall = (callId: string, name = 'task') => + emitter.emit(AgentEventType.TOOL_CALL, { + callId, + name, + } as AgentToolCallEvent); + const toolProgress = ( + callId: string, + flags: Partial = {}, + ) => + emitter.emit(AgentEventType.TOOL_PROGRESS, { + callId, + timestamp: Date.now(), + ...flags, + } as AgentToolProgressEvent); + const toolResult = (callId: string) => + emitter.emit(AgentEventType.TOOL_RESULT, { + callId, + } as AgentToolResultEvent); + const abortPhase = () => + controller.signal.aborted + ? getAgentProgressTimeout(controller.signal)?.phase + : undefined; + + beforeEach(() => { + vi.useFakeTimers(); + emitter = new AgentEventEmitter(); + controller = new AbortController(); + onUnresponsive = vi.fn(); + }); + + afterEach(() => { + detach(); + vi.useRealTimers(); + }); + + it('aborts a genuinely stalled tool after the tool deadline', () => { + attach(); + toolCall('t1'); + toolProgress('t1'); + vi.advanceTimersByTime(TOOL_TIMEOUT_MS - 1); + expect(abortPhase()).toBeUndefined(); + vi.advanceTimersByTime(1); + expect(abortPhase()).toBe('tool'); + }); + + it('suspends the model deadline while a direct approval is pending', () => { + attach(); + toolCall('t1'); + toolProgress('t1'); + emitter.emit(AgentEventType.TOOL_WAITING_APPROVAL, { + callId: 't1', + } as AgentApprovalRequestEvent); + vi.advanceTimersByTime(2 * MODEL_TIMEOUT_MS); + expect(abortPhase()).toBeUndefined(); + + // Approval answered → the tool resumes executing → tool deadline re-arms. + toolProgress('t1'); + vi.advanceTimersByTime(TOOL_TIMEOUT_MS); + expect(abortPhase()).toBe('tool'); + }); + + it('keeps a nested external-input wait free of any deadline until progress resumes', () => { + attach(); + toolCall('t1'); + toolProgress('t1'); + toolProgress('t1', { waitingForExternalInput: true }); + vi.advanceTimersByTime(2 * MODEL_TIMEOUT_MS); + expect(abortPhase()).toBeUndefined(); + + toolProgress('t1'); + vi.advanceTimersByTime(TOOL_TIMEOUT_MS); + expect(abortPhase()).toBe('tool'); + }); + + it('suspends the model deadline during a nested approval wait', () => { + attach(); + toolCall('t1'); + toolProgress('t1'); + toolProgress('t1', { awaitingApproval: true }); + vi.advanceTimersByTime(2 * MODEL_TIMEOUT_MS); + expect(abortPhase()).toBeUndefined(); + + toolProgress('t1'); + vi.advanceTimersByTime(TOOL_TIMEOUT_MS); + expect(abortPhase()).toBe('tool'); + }); + + it('keeps the model deadline suspended while a nested input wait outlives sibling tools', () => { + attach(); + toolCall('parked'); + toolProgress('parked'); + toolProgress('parked', { waitingForExternalInput: true }); + toolCall('sibling'); + toolProgress('sibling'); + toolResult('sibling'); + vi.advanceTimersByTime(2 * MODEL_TIMEOUT_MS); + expect(abortPhase()).toBeUndefined(); + + toolResult('parked'); + vi.advanceTimersByTime(MODEL_TIMEOUT_MS); + expect(abortPhase()).toBe('model/control'); + }); + + it('suspends the model deadline during a top-level external-input wait', () => { + attach(() => true); + emitter.emit(AgentEventType.ROUND_START, {} as AgentRoundEvent); + emitter.emit(AgentEventType.ROUND_END, { + waitingForExternalInput: true, + } as AgentRoundEvent); + vi.advanceTimersByTime(2 * MODEL_TIMEOUT_MS); + expect(abortPhase()).toBeUndefined(); + + emitter.emit(AgentEventType.EXTERNAL_MESSAGE, {} as never); + vi.advanceTimersByTime(MODEL_TIMEOUT_MS); + expect(abortPhase()).toBe('model/control'); + }); + + it('keeps a provider-backoff deadline extension across a clock-drift re-arm', () => { + // performance.now() drives the drift guard. Couple it to the fake clock + // and add a constant offset to stand in for a timer that lands late + // (laptop resume / loaded host). + let clock = 0; + let drift = 0; + const nowSpy = vi + .spyOn(performance, 'now') + .mockImplementation(() => clock + drift); + const advance = (ms: number) => { + clock += ms; + vi.advanceTimersByTime(ms); + }; + try { + attach(); + const retryDelayMs = 60 * 60_000; + const extendedMs = MODEL_TIMEOUT_MS + retryDelayMs; + emitter.emit(AgentEventType.MODEL_RETRY, { + retryDelayMs, + } as AgentRoundEvent); + + // The extended deadline fires 2s late, so the drift branch re-arms it. + drift = 2_000; + advance(extendedMs); + + // The re-arm must keep the extension: the base deadline alone has now + // elapsed, and the extension has not. + advance(MODEL_TIMEOUT_MS + 1); + expect(abortPhase()).toBeUndefined(); + + advance(retryDelayMs); + expect(abortPhase()).toBe('model/control'); + } finally { + nowSpy.mockRestore(); + } + }); +}); diff --git a/packages/core/src/agents/runtime/agent-progress-watchdog.ts b/packages/core/src/agents/runtime/agent-progress-watchdog.ts new file mode 100644 index 00000000000..db381677387 --- /dev/null +++ b/packages/core/src/agents/runtime/agent-progress-watchdog.ts @@ -0,0 +1,256 @@ +/** + * @license + * Copyright 2026 Qwen + * SPDX-License-Identifier: Apache-2.0 + */ + +import { AgentEventType } from './agent-events.js'; +import type { + AgentEventEmitter, + AgentApprovalRequestEvent, + AgentRoundEvent, + AgentToolCallEvent, + AgentToolProgressEvent, + AgentToolResultEvent, +} from './agent-events.js'; + +const MODEL_CONTROL_PROGRESS_TIMEOUT_MS = 15 * 60_000; +const TOOL_PROGRESS_TIMEOUT_MS = 10 * 60_000; +const UNRESPONSIVE_ABORT_GRACE_MS = 5_000; +const MAX_RETRY_DEADLINE_EXTENSION_MS = 6 * 60 * 60_000; + +export class AgentProgressTimeoutError extends Error { + constructor( + readonly phase: 'model/control' | 'tool', + readonly timeoutMs: number, + readonly toolName?: string, + ) { + super( + phase === 'tool' + ? `Background agent tool "${toolName ?? 'unknown'}" made no progress for ${timeoutMs}ms.` + : `Background agent made no model/control progress for ${timeoutMs}ms.`, + ); + this.name = 'AgentProgressTimeoutError'; + } +} + +export function getAgentProgressTimeout( + signal: AbortSignal, +): AgentProgressTimeoutError | undefined { + return signal.reason instanceof AgentProgressTimeoutError + ? signal.reason + : undefined; +} + +interface ToolDeadline { + name: string; + state: 'queued' | 'approval' | 'executing'; + /** Nested run parked on Monitor-owned external input: suppresses the + model deadline like a top-level external-input wait does. */ + parkedOnInput?: true; + timer?: ReturnType; +} + +export function attachAgentProgressWatchdog( + emitter: AgentEventEmitter, + controller: AbortController, + isWaitingForExternalInput: () => boolean, + onUnresponsive: (error: AgentProgressTimeoutError) => void, +): () => void { + let disposed = false; + let waitingForExternalInput = false; + let roundHadToolCalls = false; + let modelTimer: ReturnType | undefined; + let escalationTimer: ReturnType | undefined; + const tools = new Map(); + + const abort = (error: AgentProgressTimeoutError) => { + if (disposed || controller.signal.aborted) return; + controller.abort(error); + const armEscalation = () => { + escalationTimer = schedule( + UNRESPONSIVE_ABORT_GRACE_MS, + () => onUnresponsive(error), + armEscalation, + ); + }; + armEscalation(); + }; + const schedule = ( + timeoutMs: number, + callback: () => void, + rearm: () => void, + ): ReturnType => { + const expectedAt = performance.now() + timeoutMs; + const timer = setTimeout(() => { + if (performance.now() - expectedAt > 1_000) { + rearm(); + return; + } + callback(); + }, timeoutMs); + timer.unref?.(); + return timer; + }; + const clearModel = () => { + if (modelTimer) clearTimeout(modelTimer); + modelTimer = undefined; + }; + const armModel = (retryDelayMs = 0) => { + clearModel(); + if ( + disposed || + waitingForExternalInput || + [...tools.values()].some( + (tool) => + tool.state === 'executing' || + tool.state === 'approval' || + tool.parkedOnInput === true, + ) + ) + return; + modelTimer = schedule( + MODEL_CONTROL_PROGRESS_TIMEOUT_MS + + Math.min(retryDelayMs, MAX_RETRY_DEADLINE_EXTENSION_MS), + () => + abort( + new AgentProgressTimeoutError( + 'model/control', + MODEL_CONTROL_PROGRESS_TIMEOUT_MS, + ), + ), + // Re-arm through a closure so the granted extension survives the + // clock-drift re-arm: passing the bare reference would drop the + // argument and collapse the deadline back to the base timeout. + () => armModel(retryDelayMs), + ); + }; + const armTool = (callId: string) => { + const tool = tools.get(callId); + if (!tool || tool.state !== 'executing' || disposed) return; + if (tool.timer) clearTimeout(tool.timer); + tool.timer = schedule( + TOOL_PROGRESS_TIMEOUT_MS, + () => + abort( + new AgentProgressTimeoutError( + 'tool', + TOOL_PROGRESS_TIMEOUT_MS, + tool.name, + ), + ), + () => armTool(callId), + ); + }; + const onActivity = () => { + armModel(); + }; + const onRoundStart = () => { + waitingForExternalInput = false; + roundHadToolCalls = false; + armModel(); + }; + const onRoundEnd = (event: AgentRoundEvent) => { + waitingForExternalInput = + event.waitingForExternalInput === true && + !roundHadToolCalls && + isWaitingForExternalInput(); + armModel(); + }; + const onModelRetry = (event: AgentRoundEvent) => { + armModel(event.retryDelayMs); + }; + const onExternalInput = () => { + waitingForExternalInput = false; + armModel(); + }; + const onToolCall = (event: AgentToolCallEvent) => { + roundHadToolCalls = true; + tools.set(event.callId, { name: event.name, state: 'queued' }); + armModel(); + }; + const onToolHeartbeat = (event: AgentToolProgressEvent) => { + const tool = tools.get(event.callId); + if (!tool) return; + if (event.settled) { + if (tool.timer) clearTimeout(tool.timer); + tools.delete(event.callId); + armModel(); + return; + } + if (event.awaitingApproval) { + // Nested run parked on a user approval: no deadline, matching direct + // approvals (approval waits must not cause false watchdog failures). + tool.state = 'approval'; + delete tool.parkedOnInput; + clearTimeout(tool.timer); + tool.timer = undefined; + armModel(); + return; + } + if (event.waitingForExternalInput) { + // Nested run parked on Monitor-owned external input: no deadline, same + // as a top-level external-input wait. + tool.state = 'approval'; + tool.parkedOnInput = true; + clearTimeout(tool.timer); + tool.timer = undefined; + clearModel(); + return; + } + tool.state = 'executing'; + delete tool.parkedOnInput; + armTool(event.callId); + clearModel(); + }; + const onApproval = (event: AgentApprovalRequestEvent) => { + const tool = tools.get(event.callId); + if (!tool) return; + tool.state = 'approval'; + delete tool.parkedOnInput; + clearTimeout(tool.timer); + tool.timer = undefined; + armModel(); + }; + const onToolResult = (event: AgentToolResultEvent) => { + const tool = tools.get(event.callId); + if (tool?.timer) clearTimeout(tool.timer); + tools.delete(event.callId); + armModel(); + }; + + emitter.on(AgentEventType.START, onActivity); + emitter.on(AgentEventType.ROUND_START, onRoundStart); + emitter.on(AgentEventType.ROUND_END, onRoundEnd); + emitter.on(AgentEventType.STREAM_TEXT, onActivity); + emitter.on(AgentEventType.USAGE_METADATA, onActivity); + emitter.on(AgentEventType.MODEL_RETRY, onModelRetry); + emitter.on(AgentEventType.EXTERNAL_MESSAGE, onExternalInput); + emitter.on(AgentEventType.TOOL_CALL, onToolCall); + emitter.on(AgentEventType.TOOL_PROGRESS, onToolHeartbeat); + emitter.on(AgentEventType.TOOL_WAITING_APPROVAL, onApproval); + emitter.on(AgentEventType.TOOL_RESULT, onToolResult); + armModel(); + + return () => { + if (disposed) return; + disposed = true; + clearModel(); + if (escalationTimer) clearTimeout(escalationTimer); + for (const tool of tools.values()) { + if (tool.timer) clearTimeout(tool.timer); + } + tools.clear(); + emitter.off(AgentEventType.START, onActivity); + emitter.off(AgentEventType.ROUND_START, onRoundStart); + emitter.off(AgentEventType.ROUND_END, onRoundEnd); + emitter.off(AgentEventType.STREAM_TEXT, onActivity); + emitter.off(AgentEventType.USAGE_METADATA, onActivity); + emitter.off(AgentEventType.MODEL_RETRY, onModelRetry); + emitter.off(AgentEventType.EXTERNAL_MESSAGE, onExternalInput); + emitter.off(AgentEventType.TOOL_CALL, onToolCall); + emitter.off(AgentEventType.TOOL_PROGRESS, onToolHeartbeat); + emitter.off(AgentEventType.TOOL_WAITING_APPROVAL, onApproval); + emitter.off(AgentEventType.TOOL_RESULT, onToolResult); + }; +} diff --git a/packages/core/src/core/llm-chat.ts b/packages/core/src/core/llm-chat.ts index 7bb76b4a0af..fd5f38bc05d 100644 --- a/packages/core/src/core/llm-chat.ts +++ b/packages/core/src/core/llm-chat.ts @@ -587,6 +587,8 @@ export type StreamEvent = export interface LlmChatSendOptions { /** Skip only the configured model fallback chain for this request. */ disableModelFallbacks?: boolean; + /** Reports retry backoff so background-agent liveness can extend its deadline. */ + onRetry?: (delayMs: number) => void; } /** @deprecated Use `LlmChatSendOptions`; retained until a future major release. */ @@ -3617,6 +3619,7 @@ export class LlmChat { ? transportContinuationPrefix : undefined, acceptQuietToolResultCompletion, + options?.onRetry, ); streamEstablished = true; @@ -4411,6 +4414,7 @@ export class LlmChat { turnGoalContext, undefined, acceptQuietToolResultCompletion, + options?.onRetry, ); for await (const chunk of stream) { yield { type: StreamEventType.CHUNK, value: chunk }; @@ -4843,6 +4847,7 @@ export class LlmChat { fallbackRetryErrorCodes, requestRouteKey, turnGoalContext, + options?.onRetry, )) { const emittedUserVisibleOutput = event.type !== StreamEventType.CHUNK || @@ -5038,6 +5043,7 @@ export class LlmChat { goalContext?: GoalTurnPermit, transportContinuationPrefix?: Part[], acceptQuietToolResultCompletion = false, + onRetry?: (delayMs: number) => void, ): Promise> { const generator = overrides?.contentGenerator ?? this.config.getContentGenerator(); @@ -5107,6 +5113,7 @@ export class LlmChat { } : {}), onRetry: (info) => { + onRetry?.(info.delayMs); logApiRetry( this.config, new ApiRetryEvent({ @@ -5144,6 +5151,7 @@ export class LlmChat { retryErrorCodes?: readonly number[], routeKey?: string, goalContext?: GoalTurnPermit, + onRetry?: (delayMs: number) => void, ): AsyncGenerator { const stream = await this.makeApiCallAndProcessStream( model, @@ -5153,6 +5161,9 @@ export class LlmChat { { contentGenerator, retryAuthType, retryErrorCodes }, routeKey, goalContext, + undefined, + false, + onRetry, ); for await (const chunk of stream) { diff --git a/packages/core/src/tools/agent/agent.ts b/packages/core/src/tools/agent/agent.ts index 6daa1177dc0..8e38e0944eb 100644 --- a/packages/core/src/tools/agent/agent.ts +++ b/packages/core/src/tools/agent/agent.ts @@ -36,6 +36,10 @@ import { } from '../../agents/runtime/agent-headless.js'; import type { SubagentExecutor } from '../../agents/runtime/subagent-executor.js'; import type { AgentExternalInput } from '../../agents/runtime/agent-types.js'; +import { + attachAgentProgressWatchdog, + getAgentProgressTimeout, +} from '../../agents/runtime/agent-progress-watchdog.js'; import type { Content } from '@google/genai'; import { FORK_AGENT, @@ -92,6 +96,7 @@ import type { AgentFinishEvent, AgentErrorEvent, AgentApprovalRequestEvent, + AgentRoundEvent, AgentUsageEvent, } from '../../agents/runtime/agent-events.js'; import { @@ -1459,11 +1464,44 @@ class AgentToolInvocation extends BaseToolInvocation { ): void { let pendingConfirmationCallId: string | undefined; const preserveProtocolPayloads = !this.config.isInteractive(); + const waitingForApproval = () => + this.currentToolCalls!.some( + (call) => call.status === 'awaiting_approval', + ) && !this.currentToolCalls!.some((call) => call.status === 'executing'); eventEmitter.on(AgentEventType.START, () => { this.updateDisplay({ status: 'running' }, updateOutput); }); + let lastForwardedProgressAt = 0; + const forwardProgress = () => { + const now = Date.now(); + if (now - lastForwardedProgressAt < 1_000) return; + lastForwardedProgressAt = now; + this.updateDisplay({}, updateOutput); + }; + eventEmitter.on(AgentEventType.STREAM_TEXT, forwardProgress); + eventEmitter.on(AgentEventType.MODEL_RETRY, forwardProgress); + eventEmitter.on(AgentEventType.TOOL_PROGRESS, forwardProgress); + + eventEmitter.on(AgentEventType.ROUND_END, (...args: unknown[]) => { + const event = args[0] as AgentRoundEvent; + if (event.waitingForExternalInput) { + this.updateDisplay({ waitingForExternalInput: true }, updateOutput); + } + }); + eventEmitter.on(AgentEventType.ROUND_START, () => { + if ( + this.currentDisplay?.waitingForExternalInput || + this.currentDisplay?.awaitingApproval + ) { + this.updateDisplay( + { waitingForExternalInput: undefined, awaitingApproval: undefined }, + updateOutput, + ); + } + }); + eventEmitter.on(AgentEventType.TOOL_CALL, (...args: unknown[]) => { const event = args[0] as AgentToolCallEvent; const skill = @@ -1533,6 +1571,7 @@ class AgentToolInvocation extends BaseToolInvocation { this.updateDisplay( { toolCalls: [...this.currentToolCalls!], + awaitingApproval: waitingForApproval() ? true : undefined, ...clearPending, }, updateOutput, @@ -1638,12 +1677,18 @@ class AgentToolInvocation extends BaseToolInvocation { { toolCalls: [...this.currentToolCalls!], pendingConfirmation: undefined, + waitingForExternalInput: undefined, + awaitingApproval: undefined, }, updateOutput, ); } else { this.updateDisplay( - { pendingConfirmation: undefined }, + { + pendingConfirmation: undefined, + waitingForExternalInput: undefined, + awaitingApproval: undefined, + }, updateOutput, ); } @@ -1656,6 +1701,7 @@ class AgentToolInvocation extends BaseToolInvocation { { toolCalls: [...this.currentToolCalls!], pendingConfirmation: details, + awaitingApproval: waitingForApproval() ? true : undefined, }, updateOutput, ); @@ -3576,14 +3622,20 @@ class AgentToolInvocation extends BaseToolInvocation { // the parent model (and the UI) don't treat incomplete runs as // completed. // - const terminateMode = bgSubagent.getTerminateMode(); + const progressTimeout = getAgentProgressTimeout( + turnAbortController.signal, + ); + const terminateMode = progressTimeout + ? AgentTerminateMode.TIMEOUT + : bgSubagent.getTerminateMode(); const subagentRawText = bgSubagent.getFinalText(); const hadWorktreeIsolation = worktreeIsolation !== null; const recordTerminalOutcome = () => recordSpanOutcome( deriveSubagentOutcomeMetadata({ terminateMode, - signalAborted: turnAbortController.signal.aborted, + signalAborted: + turnAbortController.signal.aborted && !progressTimeout, resultSummaryPresent: Boolean( subagentRawText && subagentRawText.length > 0, ), @@ -3652,6 +3704,8 @@ class AgentToolInvocation extends BaseToolInvocation { recordTerminalOutcome(); } + if (registry.get(hookOpts.agentId)?.retainsPhysicalSlot) break; + if (terminateMode === AgentTerminateMode.GOAL) { keepResident = residentRegistered && !needsAutoPermissionLease(); @@ -3680,7 +3734,8 @@ class AgentToolInvocation extends BaseToolInvocation { ); } else if ( terminateMode === AgentTerminateMode.CANCELLED || - terminateMode === AgentTerminateMode.SHUTDOWN + terminateMode === AgentTerminateMode.SHUTDOWN || + registry.get(hookOpts.agentId)?.status === 'cancelled' ) { // SHUTDOWN is grouped with CANCELLED in the span taxonomy // (deriveSubagentOutcomeMetadata); align the registry side @@ -3729,15 +3784,25 @@ class AgentToolInvocation extends BaseToolInvocation { // so release keepResident here to let the finally block dispose the // runtime instead of leaking a zombie resident. keepResident = false; + const progressTimeout = getAgentProgressTimeout( + turnAbortController.signal, + ); // Publish first — same reason as the success path. recordSpanOutcome( - deriveSubagentExceptionMetadata( - error, - turnAbortController.signal.aborted, - ), + progressTimeout + ? deriveSubagentOutcomeMetadata({ + terminateMode: AgentTerminateMode.TIMEOUT, + signalAborted: false, + resultSummaryPresent: false, + }) + : deriveSubagentExceptionMetadata( + error, + turnAbortController.signal.aborted, + ), ); const baseErrorMsg = - error instanceof Error ? error.message : String(error); + progressTimeout?.message ?? + (error instanceof Error ? error.message : String(error)); debugLogger.error( `[Agent] Background agent failed: ${baseErrorMsg}`, ); @@ -3757,10 +3822,16 @@ class AgentToolInvocation extends BaseToolInvocation { } const errorMsg = baseErrorMsg + wtSuffix; + if (registry.get(hookOpts.agentId)?.retainsPhysicalSlot) return; + // If the error came from a cancellation, preserve the cancelled // status so the model's notification matches what task_stop // requested rather than reporting it as a generic failure. - if (turnAbortController.signal.aborted) { + if ( + turnAbortController.signal.aborted && + (!progressTimeout || + registry.get(hookOpts.agentId)?.status === 'cancelled') + ) { const completionStats = getCompletionStats(); registry.finalizeCancelled( hookOpts.agentId, @@ -3807,6 +3878,16 @@ class AgentToolInvocation extends BaseToolInvocation { turnAbortController: AbortController, fireStartHook: boolean, ) => { + const disposeWatchdog = attachAgentProgressWatchdog( + bgEmitter, + turnAbortController, + () => + this.config + .getMonitorRegistry() + .hasRunningForOwner(hookOpts.agentId), + (error) => + registry.failUnresponsive(hookOpts.agentId, error.message), + ); const framedBgBody = () => this.runWithSubagentSpan( this.buildSubagentSpanSpec( @@ -3828,7 +3909,12 @@ class AgentToolInvocation extends BaseToolInvocation { launchDepth, ), ); - return isFork ? runInForkContext(framedBgBody) : framedBgBody(); + return ( + isFork ? runInForkContext(framedBgBody) : framedBgBody() + ).finally(() => { + disposeWatchdog(); + registry.releaseRetainedPhysicalSlot(hookOpts.agentId); + }); }; const reportUnexpectedBackgroundError = (err: unknown) => { diff --git a/packages/core/src/tools/tools.ts b/packages/core/src/tools/tools.ts index c6463c1f13a..e0266b48a33 100644 --- a/packages/core/src/tools/tools.ts +++ b/packages/core/src/tools/tools.ts @@ -685,6 +685,10 @@ export interface AgentResultDisplay { // If the subagent is awaiting approval for a tool call, // this contains the confirmation details for inline UI rendering. pendingConfirmation?: ToolCallConfirmationDetails; + /** Whether the subagent is parked on Monitor-owned external input. */ + waitingForExternalInput?: true; + /** Whether the subagent is parked on a pending tool approval. */ + awaitingApproval?: true; toolCalls?: Array<{ callId: string;