fix(v1): interception server owns and multiplexes model clients

[mikasenghaas]

Summary

• Move model-client ownership from the rollout to the interception server: InterceptionServer keeps one live Client per distinct endpoint config (keyed by the config's JSON — configs carry an env-var name, never the key itself), assigns it to each RolloutSession at register, and closes it with the server via its exit stack.
• RolloutSession.client is no longer a constructor argument; the server sets it at registration. Rollout no longer builds (resolve_client) or closes a client.
• Rollouts multiplexed onto a server (multiplex, default 32) now share one keepalive connection pool: since each rollout issues turns sequentially, a server's client never holds more than ~multiplex requests in flight — comfortably inside the existing transport limits — so upstream connections are reused warm instead of opened and torn down per rollout.
• Mixed-config sessions on one server (e.g. multi-agent envs with per-role endpoints) fall out of the config-keyed cache. TrainClient is safe to share: all per-turn state is local, and the renderer pool has been process-wide since feat: build one client per rollout, share renderers process-wide #2218.

Context

#2218 made each rollout build and close its own httpx client. At ~768 concurrent rollouts that means constant TCP churn against the inference router — the load pattern implicated in the vllm-router wedges (router stops processing mid-stream, engine drains and idles, sessions hashed to it die at the solver timeout). This restores shared, bounded connection reuse while keeping the per-rollout client configs introduced there: the sharing rate is naturally the interception server's multiplex rate, with no new knob.

Note on cancellation: previously Rollout.abort() closed its own client, killing any in-flight upstream request with it. The shared client stays open; in-flight handlers are cancelled via session.release() (which already existed) and their connections return to the pool.

Verification

• uv run ruff check / ruff format --check clean; full non-e2e v1 suite: 84 passed.
• Live e2e (prime endpoint): test_single_turn, test_tool, test_interaction, test_multi_agent_env legs pass, including the tunneled harness-in-subprocess-with-tool-in-docker leg. The modal legs fail identically on unmodified main (local modal environment issue, unrelated).

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes upstream connection lifecycle and abort/cancellation behavior on the inference path at high concurrency; mitigated by per-config sharing and existing session.release() for stragglers.

Overview
Moves live model Client ownership from each Rollout to InterceptionServer: the server keeps one resolved client per distinct BaseClientConfig (keyed by config JSON), assigns it on RolloutSession register, and closes clients with the server exit stack.

Rollouts no longer call resolve_client or client.close() in abort/close; multiplexed sessions on the same server reuse one upstream keepalive pool instead of per-rollout TCP churn. RolloutSession.client is optional until registration.

Docs/comments are updated across client config, GEPA runner, TrainClient, and session to describe server-owned sharing. Cancellation no longer tears down a per-rollout client on abort; in-flight work is cancelled via existing session.release() while the shared client stays open.

^{Reviewed by Cursor Bugbot for commit 30de027. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Move model client ownership from Rollout to InterceptionServer with shared caching

• InterceptionServer now owns model clients, caching them by serialized endpoint config in a clients dict and closing them via its async exit stack on teardown.
• A new _client helper on InterceptionServer returns a cached Client for a given BaseClientConfig, creating it via resolve_client on first use.
• InterceptionServer.register assigns the server-owned client to the RolloutSession, replacing per-rollout client construction.
• Rollout no longer creates or closes a Client; RolloutSession.client is now optional (None until assigned at registration).
• Behavioral Change: multiplexed rollouts sharing the same endpoint config now reuse a single connection pool and ElasticRendererPool slots rather than each maintaining their own.

^{Macroscope summarized 30de027.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR refactors client ownership from individual rollouts to the interception server, changing connection lifecycle semantics (creation timing, sharing across rollouts, teardown timing). While well-structured for connection pooling efficiency, such resource management changes warrant human review to verify the new lifecycle is correct.

^{You can customize Macroscope's approvability policy. Learn more.}