Skip to content
Closed
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions docs/design/2026-09-08-pooled-mcp-recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Recovering lost MCP connections in active ACP sessions

[中文设计](./2026-09-08-pooled-mcp-recovery.zh-CN.md) · Issue: https://github.com/QwenLM/qwen-code/issues/11272

## Reproduction and ownership

Baseline `bb79319ce5e0e2bbe64307c50dc3621621afc25c` was exercised through a real loopback `qwen serve` daemon and ACP WebSocket sessions, with a deterministic model endpoint and a counted stdio MCP fixture.

Ordinary cancellation delivered `notifications/cancelled`, left the child alive, and allowed both sessions to continue. A fixture that deliberately exited on cancellation exposed a stale CONNECTED status; a fixture that emitted an invalid JSON-RPC frame triggered `silent_drop`, removed session tool registrations, and left later turns unable to call tools. These are controlled fault injections, not proof of the original reporter's teardown mechanism.

There are three lifetimes:

- A tool invocation owns arguments, an abort signal, and one permission decision. It may already have applied a side effect when its response disappears.
- The pool owns the transport and stdio process, shared by connection fingerprint. Its cleanup owns process and descendant teardown.
- Each live session owns a pool reference and filtered tool/prompt/resource registrations. Removing an entry does not close the session, but previously nothing restored these registrations before its next model send.

The SDK's clean-close callback was not reflected in McpClient status. Separately, the invocation reconnect path could use the shared snapshot's bootstrap Config to spawn a private client while the real sessions retained stale handles.

## Design

1. Reflect unexpected SDK close in the disconnected status path for pool-managed clients. Standalone clients retain their existing health-monitor behavior. An intentional disconnect remains guarded by `isDisconnecting`. Reject a discovery snapshot if the connection closed while any listing was in flight, even when another listing returned data.
2. Record a recovery candidate when a held connection emits `failed`, or when an acquired handle has already failed before the session attaches its listener. Retain its transport fingerprint, not a tool invocation. Explicit disconnect/stop removes that candidate; initial acquisition failure does not create one.
3. Before an ACP model send, recover eligible candidates through the pool and refresh declarations. Preserve the normal folder trust, disabled-server, approval, configuration fingerprint, transport authentication and per-tool permission checks. Recheck eligibility after acquisition so a late result cannot override revocation or shutdown.
4. Serialize a session's recovery, discovery and runtime additions. A queued configuration refresh reconciles after either success or failure; stop, removal and explicit disconnect invalidate older work. Every discovery request retains its own invalidations; an older request skips all reads and writes for a superseded server, and a newer explicit refresh reconciles after it. A successful add clears failure records; failed additions and no-op removals preserve them. Reapply current trust and tool filters before accepting a recovered or runtime-added handle, since these fields are excluded from transport identity. Use the pool's existing spawn-in-flight coordination across sessions. Acquisition waits for the previous entry's cleanup; unpooled HTTP/SSE cleanup belongs to the session and transport fingerprint. Shutdown includes evicted entries still being cleaned within its existing deadline. A shared five-second cooldown covers acquisition and session-acceptance failures. Each model send can demand recovery, including tool-loop continuations and scheduled model turns; there are no background reconnect timers or invocation retries. Explicit management discovery is not throttled by this cooldown.
5. Pool-projected tools delegate recovery to their session. They do not invoke the standalone reconnect/replay path, regardless of annotations. Child Agent registries retain their source registry and refresh inherited MCP objects before each model send, after recovering that source. Local overrides keep their own discovery ownership. Replacement copies must pass the child configuration and permission checks; shutdown removes the source link so a late refresh cannot repopulate the registry.
6. Synchronize tools and ACP prompt commands before every pooled ACP model send, including when recovery returns no notice or throws. Registry state, rather than the presence of a diagnostic, determines what the model can use. Expose recovery success/failure as an ACP diagnostic. Suppress consecutive identical states per server within a logical turn, while retaining transitions such as success → failure → success. Each send still refreshes registrations after recovery. Inherited child and resource recovery leave the source manager’s notices pending for ACP. An agent that owns independent MCP discovery retains notices in its own manager; forwarding those diagnostics to ACP is outside this change, and its failures remain in the debug log and manager status. If a turn is cancelled while waiting for a shared connection, stop waiting without cancelling that connection or executing a tool; retain its notice until the next send or superseding configuration/management action. Recovery errors remain best effort and cannot prevent the ordinary model request from proceeding.

Runtime add and recovery use the same effective transport configuration, including the session cwd for implicit stdio servers. The raw runtime overlay remains unstamped so it can follow a workspace relocation. The synthetic `mcp.serverCommand` name cannot be overwritten by a runtime addition. Retiring a transport settles the SDK's old pending requests and removes its callbacks before reconnecting; a late child-close event cannot clear the SDK's replacement transport. A handle releases only its own session attachment, so an older handle cannot remove its replacement. Resource reads retain failed candidates and demand recovery for the named server through the pool; a client that is temporarily disconnected during a restart keeps its live pool seat. Failed entries can recover; force-closed entries are not made recovery candidates; a missing handle never creates a private standalone client in pool mode.

Parent and child registries can share a logical session ID. Pool subscriptions therefore include the ToolRegistry identity, allowing each registry to receive failures and release its own registrations. Releasing the logical session removes all of its registry subscriptions. Children with their own MCP configuration use separate prompt and resource registries, so their cleanup preserves the parent's registrations. An inheriting child routes resource reads through the source registry’s manager and refreshes its copied tools; after waiting for recovery, it rechecks the source links and each child’s current eligibility before reading. A manager that performed discovery owns its configured servers, including failed and explicitly disconnected ones. Such a child reads its own HTTP/SSE session even if its recipe matches the parent. Independent overrides compare effective stdio cwd as well as transport and session policy; ordinary borrowers retain source ownership when they share the same raw recipe. The legacy command-derived `mcp` recipe takes precedence over a raw entry of that name; borrowers with the same command retain the source cwd.

Pool-owned tool failures keep an explicit unknown-outcome/no-replay warning when a request may have reached the server, even though they skip invocation-level reconnect. A closed SDK transport is rejected before sending with a not-sent message; a JSON-RPC business error from a healthy connection retains its original meaning, even if its message mentions a disconnected downstream service. Read-only or idempotent annotations cannot prove that a request was never sent. Preventing the SDK from replaying is insufficient if a bare connection error encourages the model to issue the same operation again. This warning does not prove that arbitrary future model decisions will never duplicate a business operation; the client recovery itself performs no tool calls.

A session-acceptance failure can impose the remaining cooldown on another session even if the pool still has a healthy entry. This conservative bound avoids repeated registration failures and concurrent successful returns erasing a failure record. Partial recovery preserves unrelated workspace budget refusals and clears the recovered server's refusal; budget failures receive budget-specific advice and do not impose a transport cooldown. Refusal snapshots retain outstanding names; refusal events contain only newly refused attempts, so successful recovery cannot emit an empty refusal batch. Published pool reference counts remain counts of logical sessions, regardless of registry seats. When pooling is disabled with `QWEN_SERVE_NO_MCP_POOL=1`, legacy per-session recovery remains outside this change.

Runtime removal and failed additions ask the pool to release an unused budget reservation. The pool checks live entries and pending acquisitions of the same server name first, preserving the reservation while another session still owns that server or an idle process is draining. Removal does not promise instant capacity before process retirement.

## Existing management interfaces and scope

PR #7309 remains open at `b5f111c02300e39c2c1563af7abbf229e8455341` at investigation time. Its workspace runtime management, authentication, approval and restart routes overlap the management surface, not this demand recovery implementation. Current main already owns a workspace MCP pool and the restart route. This change adds no route, chat command, management platform, authentication bypass, or generic retry framework. PR #11145 concerns initial workspace discovery after preheat; recovery here is limited to previously acquired connections.

## Validation contract

Use counted tools, PID logs, actual resource results and tool declarations in the next model request, not model prose or `end_turn`, to judge success. Cover ordinary cancellation, cancellation plus controlled exit/transport error, independent child exit, simultaneous sessions, concurrent acquisition and failure cooldown, explicit disable/disconnect, late shutdown/revocation, discovery timeout, permission denial, runtime add/remove, child agents and shutdown cleanup. Keep the cancelled invocation count at one. Distinguish local macOS daemon evidence from untested production DingTalk infrastructure, Windows/Linux process semantics and real OAuth providers.

## Review follow-up

Discovery, demand recovery and runtime addition retain subscriptions to an active entry while its client is restarting. A transient CONNECTING or DISCONNECTED client status does not terminate pool ownership; closed and failed entries are rejected. Same-recipe runtime additions update the existing projection without releasing its subscription. A failed replacement rolls back its own tentative in-memory configuration when the overlay still has that request's object identity. Restoring the previous configuration does not authorize a connection: disconnect/remove/stop invalidations separately prevent recreating its recovery candidate, and a later queued add cannot restore that authority. Removal invalidates an older queued add even before it writes its overlay; removing a settings-only name with no pending add remains a no-op. Queued discovery rechecks folder trust when the pass actually starts, before acquiring or spawning any server.

Per-agent MCP configuration snapshots retain both decorated metadata and the original inherited recipes. SDK-only discovery leaves other servers borrowed from the parent, so a failed SDK override must not remove a healthy inherited extension tool during refresh. Agent overrides shadow the parent in both views.

Extension metadata decoration creates a fresh configuration object on each read. Borrowing compares original recipes through the existing configuration merge and admission filters; effective transport and session policy checks still apply to independent overrides. This adds no configuration cache. Invocation error classification reads the owning McpClient's status, because an SDK transport can remain assigned after a network error. The public SDK transport getter separately identifies a closed connection. The status accessor is preserved through tool projections and standalone retry construction. Legacy discovery without a McpClient owner retains its standalone name-status fallback. A pooled callable without that client uses error evidence, not process-global server-name status, so closing a same-named sibling cannot hide a healthy connection's business error or execution timeout.

Session projection deliberately clones a discovered tool to disable standalone reconnect; it is not a zero-allocation path. Tests use an unprojected discovery snapshot and verify that the shared snapshot keeps standalone behavior; production session projection still applies include/exclude filters. Cooldown records expire after five seconds even if their configuration is removed. These expiry timers perform bookkeeping, never connection attempts.

Recovery before model send can delay a turn: stdio discovery defaults to 30 seconds (remote discovery to five seconds). Waiting for previous-entry cleanup has a five-second caller deadline. On expiry, acquisition reports failure and keeps the cleanup barrier: a timeout does not prove the old process exited, so no replacement starts until cleanup finishes. The five-second retry cooldown is separate from this wait and is not a turn deadline. Cancellation stops the caller waiting. Discovery timeout configuration remains unchanged. With SDK 2.0.0, the real stdio fixture that ignored EOF and SIGTERM exited after about four seconds through the SDK SIGKILL path; a never-settling teardown was exercised only by fault injection, not reproduced with the built-in transports.
Loading
Loading