fix(core): time out stalled background agents

docs/design/background-agent-progress-watchdog.md

@@ -0,0 +1,43 @@
+# Background Agent progress watchdog
+
+[中文](background-agent-progress-watchdog.zh-CN.md)
+
+## Problem
+
+An ordinary background Agent can remain registered as running while its model,
+control flow, or one tool makes no progress. The existing workflow watchdog is
+not suitable: it retries stalled work and suspends its deadline for every
+running tool. Ordinary background Agents must settle once as failed instead.
+
+## Behavior
+
+Each fresh, restored, and resident-continuation background turn has two fixed
+internal deadlines:
+
+- 15 minutes without model or control progress.
+- 10 minutes without progress from each in-flight tool.
+
+Model streaming, round transitions, usage, and external input renew the model
+deadline. Tool output and liveness heartbeats renew only that tool's deadline.
+Retry delays surfaced by qwen-code extend the model deadline by at most six
+hours; provider-internal retries remain covered by the ordinary deadline. A
+tool's own deadline starts when the scheduler reports it executing, so a silent
+tool is not charged to the model deadline. Parallel tools retain independent
+deadlines.
+
+The relevant tool deadline is suspended while user approval is pending. The
+model deadline is suspended only after a no-tool round enters a Monitor-owned
+external-input wait, and resumes when input arrives. A timer delayed by host
+suspend or a local event-loop gap is rearmed rather than charged to the Agent.
+
+On expiry the watchdog aborts the turn with an `AgentProgressTimeoutError`.
+Cooperative model and tool paths map that reason to `TIMEOUT`; the background
+registry and sidecar then settle once as `failed`. There is no retry. Existing
+definition-level turn and wall-clock limits are unchanged.
+
+## Scope
+
+Workflow dispatch remains unchanged. An Agent that ignores the cooperative
+abort retains its physical slot while the daemon drains and replaces that
+Session's runtime generation, as described in
+`background-agent-runtime-generations.md`.

docs/design/background-agent-progress-watchdog.zh-CN.md

@@ -0,0 +1,24 @@
+# 后台 Agent 进度看门狗
+
+[English](background-agent-progress-watchdog.md)
+
+## 问题
+
+普通后台 Agent 可能在模型、控制流或某个工具长期没有进展时仍保持运行状态。现有 workflow 看门狗会重试停滞工作，并在任一工具运行期间暂停期限，不适用于这里。普通后台 Agent 应当只以失败状态结算一次。
+
+## 行为
+
+每个新建、恢复及 resident continuation 的后台 turn 都有两个固定的内部期限：
+
+- 模型或控制流 15 分钟无进展。
+- 每个执行中工具 10 分钟无进展。
+
+模型流式输出、round 状态变化、用量更新和外部输入会续期模型期限。工具输出和存活心跳只续期该工具的期限。qwen-code 显式上报的重试延迟最多将模型期限延长 6 小时；provider 内部重试仍受普通期限约束。工具期限从调度器报告工具开始执行时起算，因此静默工具不会计入模型期限，并行工具各自计时。
+
+等待用户审批时，相关工具期限暂停。模型期限仅在无工具 round 真正进入 Monitor 所属的外部输入等待后暂停，并在收到输入后恢复。因主机挂起或本地事件循环间隙而延迟的计时器会重新计时，不会把这段时间算作 Agent 停滞。
+
+期限到达时，看门狗以 `AgentProgressTimeoutError` 中止 turn。可协作中止的模型和工具路径把原因映射为 `TIMEOUT`，后台 registry 与 sidecar 只结算一次 `failed`，不做重试。定义级别的 turn 数量和总耗时限制保持不变。
+
+## 范围
+
+Workflow 调度保持不变。若 Agent 忽略协作式中止，它会继续占用物理槽位，同时 daemon 排空并替换该 Session 的 runtime generation，详见 [后台 Agent runtime generation](background-agent-runtime-generations.zh-CN.md)。

docs/design/background-agent-runtime-generations.md

@@ -0,0 +1,17 @@
+# Background Agent runtime generations
+
+[中文](background-agent-runtime-generations.zh-CN.md)
+
+## Problem
+
+A background Agent that does not settle after cancellation can leave its ACP child usable enough to answer transport probes but unsafe for fresh work. Replacing that child must not move its existing Sessions, create unbounded children, or route new work back to the draining generation.
+
+## Design
+
+Each ACP bridge channel has one of three states: `active`, `draining`, or `dying`. Existing Session entries continue to route through their recorded channel while it drains. Explicit recycle requests mark only the affected generation as draining; fresh work then creates a new active generation. Existing timeout retirement keeps its previous reap-after-drain behavior without starting another generation.
+
+Fresh work admits at most two non-dying generations. If both are draining, admission fails with `503 runtime_recycling` until one exits. Restore and recycle recovery may start a replacement while dying processes await reap; dying generations remain tracked until channel exit so synchronous shutdown can still reach them.
+
+After a logical watchdog abort, the Agent gets a fixed five-second cooperative exit window. If it still has not settled, its registry entry and sidecar become failed once while the underlying run keeps its concurrency slot. The terminal notification is recorded and displayed without starting another model turn, then the trusted child-to-daemon route requests recycle for the Session's owner generation. A late Agent settlement releases the physical slot but cannot replace the failed terminal state.
+
+This changes no persisted Session format and adds no public timeout configuration.

docs/design/background-agent-runtime-generations.zh-CN.md

@@ -0,0 +1,17 @@
+# 后台 Agent runtime generation
+
+[English](background-agent-runtime-generations.md)
+
+## 问题
+
+后台 Agent 在取消后仍不结算时，其 ACP child 可能仍能响应传输探测，却已经不适合接收新工作。替换该 child 时不能移动它现有的 Session、无限创建 child，也不能把新工作重新路由到正在排空的 generation。
+
+## 设计
+
+每个 ACP bridge channel 处于 `active`、`draining` 或 `dying` 三种状态之一。已有 Session 在排空期间继续通过自己记录的 channel 路由。只有显式 recycle 请求会把受影响的 generation 标记为 `draining`，新工作随后创建新的 active generation。已有的超时退役路径保持原先的“排空后回收”行为，不会启动另一代 runtime。
+
+新工作最多允许两个尚未 dying 的 generation。若两者都在 draining，admission 返回 `503 runtime_recycling`，直到其中一个退出。restore 和 recycle recovery 可以在 dying 进程等待回收时启动替代进程；dying generation 会持续被追踪到 channel 退出，保证同步 shutdown 仍可访问它们。
+
+逻辑看门狗中止后，Agent 有固定 5 秒的协作退出时间。若仍未结算，registry entry 与 sidecar 只会被标记失败一次，而底层 run 继续占用并发槽。终态通知在不启动额外模型 turn 的情况下记录并展示，随后可信的 child-to-daemon route 请求回收该 Session 的 owner generation。迟到的 Agent 结算会释放物理槽，但不会覆盖已经发布的失败终态。
+
+本设计不改变 Session 持久化格式，也不增加公开的超时配置。

packages/acp-bridge/src/bridge.test.ts

@@ -30,6 +30,7 @@ import {
   BranchWhilePromptActiveError,
   InvalidClientIdError,
   BridgeChannelQuarantinedError,
+  BridgeRuntimeRecyclingError,
   InvalidPermissionOptionError,
   InvalidSessionMetadataError,
   InvalidSessionScopeError,
@@ -29423,14 +29424,21 @@ describe('createAcpSessionBridge', () => {
           },
         });
         capturedConn = new AgentSideConnection(() => fakeAgent, agentStream);
+        // A killed child exits, and the bridge frees a generation slot
+        // only once `exited` fires — it admits no fresh work while two
+        // OS-live generations are still pending reap.
+        let resolveExited: (() => void) | undefined;
+        const exited = new Promise<
+          | { exitCode: number | null; signalCode: NodeJS.Signals | null }
+          | undefined
+        >((r) => {
+          resolveExited = () => r(undefined);
+        });
         return {
           stream: clientStream,
-          exited: new Promise<
-            | { exitCode: number | null; signalCode: NodeJS.Signals | null }
-            | undefined
-          >(() => {}),
-          kill: async () => {},
-          killSync: () => {},
+          exited,
+          kill: async () => resolveExited!(),
+          killSync: () => resolveExited!(),
         };
       };
       const bridge = makeBridge({ channelFactory: factory });
@@ -32914,6 +32922,181 @@ describe('createAcpSessionBridge', () => {
       await bridge.shutdown();
     });
   });
+
+  // ============================================================
+  // Runtime recycle (issue #8586) — a generation hand-off must not
+  // admit fresh session work onto the condemned generation.
+  // ============================================================
+  describe('requestRuntimeRecycle — draining generation admission', () => {
+    it('rejects a newSession that resolves after its channel was condemned to drain', async () => {
+      // Two sessions multiplex on gen1. The second spawn is held inside
+      // `connection.newSession` while a runtime recycle is requested for
+      // the first: gen1 flips to `draining` and retirement is DEFERRED
+      // (the in-flight spawn counts as work), so `isDying` stays false.
+      // When `newSession` finally resolves, doSpawn's post-await re-check
+      // must reject the fresh session instead of installing it on the
+      // generation the daemon just judged unsafe for fresh work — which
+      // would also pin that generation open until the session closed.
+      const secondNewSessionStarted = deferred<void>();
+      const releaseSecondNewSession = deferred<void>();
+      let newSessionCalls = 0;
+      const gen1 = makeChannel({
+        newSessionImpl: async () => {
+          newSessionCalls++;
+          if (newSessionCalls === 1) return { sessionId: 'sess-drain-a' };
+          secondNewSessionStarted.resolve();
+          await releaseSecondNewSession.promise;
+          return { sessionId: 'sess-drain-b' };
+        },
+      });
+      const gen2 = makeChannel({});
+      let channelSpawns = 0;
+      const bridge = makeBridge({
+        channelFactory: async () =>
+          channelSpawns++ === 0 ? gen1.channel : gen2.channel,
+        sessionScope: 'thread',
+      });
+
+      const first = await bridge.spawnOrAttach({ workspaceCwd: WS_A });
+      expect(first.sessionId).toBe('sess-drain-a');
+
+      const spawningSecond = bridge.spawnOrAttach({ workspaceCwd: WS_A });
+      await secondNewSessionStarted.promise;
+
+      // Optional on the interface (older embedded bridges omit it); the
+      // non-null call fails loudly rather than silently skipping the recycle.
+      await bridge.requestRuntimeRecycle!(first.sessionId);
+
+      // gen1 is condemned but must NOT be killed: `first` is still live on
+      // it, and retirement was deferred until that session drains.
+      expect(gen1.killed).toBe(false);
+
+      releaseSecondNewSession.resolve();
+      await expect(spawningSecond).rejects.toBeInstanceOf(
+        BridgeChannelClosedError,
+      );
+
+      // The late session was never installed on the draining generation.
+      expect(bridge.sessionCount).toBe(1);
+      expect(() => bridge.getSessionSummary('sess-drain-b')).toThrow(
+        SessionNotFoundError,
+      );
+      // The surviving session keeps its own (draining) generation.
+      expect(bridge.getSessionSummary(first.sessionId).sessionId).toBe(
+        'sess-drain-a',
+      );
+      expect(gen1.killed).toBe(false);
+
+      await bridge.shutdown();
+    });
+
+    it('rejects a resumeSession that resolves after its channel was condemned to drain', async () => {
+      // Restore twin of the `newSession` case above. gen1 is condemned
+      // mid-`session/load`, but retirement is DEFERRED (the in-flight restore
+      // counts as work, so `isDying` stays false). The post-await re-check
+      // must reject the restored session instead of installing it on the
+      // generation the daemon just judged unsafe for fresh work — which would
+      // also pin that generation open until the session closed.
+      const restoreStarted = deferred<void>();
+      const releaseRestore = deferred<void>();
+      const gen1 = makeChannel({
+        newSessionImpl: async () => ({ sessionId: 'sess-drain-a' }),
+        loadSessionImpl: async () => {
+          restoreStarted.resolve();
+          await releaseRestore.promise;
+          return {};
+        },
+      });
+      const gen2 = makeChannel({});
+      let channelSpawns = 0;
+      const bridge = makeBridge({
+        channelFactory: async () =>
+          channelSpawns++ === 0 ? gen1.channel : gen2.channel,
+        sessionScope: 'thread',
+      });
+
+      const first = await bridge.spawnOrAttach({ workspaceCwd: WS_A });
+      expect(first.sessionId).toBe('sess-drain-a');
+
+      const restoring = bridge.loadSession({
+        sessionId: 'sess-drain-b',
+        workspaceCwd: WS_A,
+      });
+      await restoreStarted.promise;
+
+      await bridge.requestRuntimeRecycle!(first.sessionId);
+      // gen1 is condemned but must NOT be killed: `first` is still live on it
+      // and the in-flight restore defers retirement.
+      expect(gen1.killed).toBe(false);
+      expect(channelSpawns).toBe(2);
+
+      releaseRestore.resolve();
+      await expect(restoring).rejects.toThrow(
+        /Session sess-drain-b restored on a closed agent channel/,
+      );
+
+      // The restored session was never installed on the draining generation.
+      expect(bridge.sessionCount).toBe(1);
+      expect(() => bridge.getSessionSummary('sess-drain-b')).toThrow(
+        SessionNotFoundError,
+      );
+
+      // The surviving session keeps its own (draining) generation.
+      expect(bridge.getSessionSummary(first.sessionId).sessionId).toBe(
+        'sess-drain-a',
+      );
+      expect(gen1.killed).toBe(false);
+
+      await bridge.shutdown();
+    });
+
+    it('rolls a recycle target back to active when the generation cap refuses the replacement', async () => {
+      const gen1 = makeChannel({
+        newSessionImpl: async () => ({ sessionId: 'sess-gen1' }),
+      });
+      let gen2SessionCount = 0;
+      const gen2 = makeChannel({
+        newSessionImpl: async () => ({
+          sessionId: `sess-gen2-${++gen2SessionCount}`,
+        }),
+      });
+      let channelSpawns = 0;
+      const bridge = makeBridge({
+        channelFactory: async () =>
+          channelSpawns++ === 0 ? gen1.channel : gen2.channel,
+        sessionScope: 'thread',
+      });
+
+      const first = await bridge.spawnOrAttach({ workspaceCwd: WS_A });
+      expect(first.sessionId).toBe('sess-gen1');
+
+      // Recycle the first session: gen1 drains, recovery spawns gen2 (active).
+      await bridge.requestRuntimeRecycle!(first.sessionId);
+      expect(channelSpawns).toBe(2);
+
+      // A fresh session attaches to the active gen2.
+      const second = await bridge.spawnOrAttach({ workspaceCwd: WS_A });
+      expect(second.sessionId).toBe('sess-gen2-1');
+
+      // Recycling gen2's session leaves both generations draining, so the
+      // replacement hits the two-generation cap. The recycle must reject and
+      // roll gen2 back to active rather than stranding the workspace with no
+      // active generation.
+      await expect(
+        bridge.requestRuntimeRecycle!(second.sessionId),
+      ).rejects.toBeInstanceOf(BridgeRuntimeRecyclingError);
+
+      // gen2 rolled back to active: a third session attaches without spawning
+      // a third generation.
+      const third = await bridge.spawnOrAttach({ workspaceCwd: WS_A });
+      expect(third.sessionId).toBe('sess-gen2-2');
+      expect(channelSpawns).toBe(2);
+      expect(gen1.killed).toBe(false);
+      expect(gen2.killed).toBe(false);
+
+      await bridge.shutdown();
+    });
+  });
 });
 
 // ============================================================