Match EvalClient pool to V1 concurrency

[xeophon]

Overview

Align the shared V1 EvalClient HTTP connection pool with the default 128 concurrent rollouts while retaining HTTPX's existing 20-connection idle keepalive bound.

Why

V1 can admit 128 rollouts concurrently, but the implicit HTTPX pool admits only 100 active connections. At full default concurrency, the remaining 28 requests wait for a pool slot and form a second latency wave. This change removes that internal queue without making the pool unbounded.

Implementation

Construct the existing httpx.AsyncClient with max_connections=128 and max_keepalive_connections=20. Request timeouts, keepalive expiry, headers, bodies, response parsing, streaming, error handling, and client shutdown remain unchanged.

Performance and resources

A local PEP 723 benchmark ran five repetitions of 128 concurrent requests with 500 ms server latency, using fresh processes for each pool configuration.

Metric	100 active / 20 idle	128 active / 20 idle
Median wall time	1.085869 s	0.572788 s
Absolute time saved	—	0.513081 s
Relative time saved	—	47.251% (1.896×)
Peak RSS	68,534,272 B	70,270,976 B
RSS above baseline	22,052,864 B	23,773,184 B
Median event-loop stall	24.288 ms	22.163 ms
Peak active requests	100	128
Peak combined benchmark tasks	330	386
Idle connections after each burst	20	20
Successful responses	128/128	128/128
Response bytes	1,536 B	1,536 B

The measured cost is 1,736,704 additional peak RSS bytes and more concurrently active transport tasks; idle resource retention does not grow.

---

Note

Low Risk
Single transport tuning knob on the eval relay client; no auth, request shaping, or response handling changes.

Overview
EvalClient now constructs its shared httpx.AsyncClient with explicit httpx.Limits: max_connections=128 (aligned with V1’s default --max-concurrent) and max_keepalive_connections=20 (HTTPX’s default idle cap).

Previously the client relied on HTTPX’s implicit 100 active connection limit, so at full default concurrency some upstream eval requests could block waiting for a pool slot instead of running in parallel. Timeouts, headers, relay/streaming behavior, and shutdown are unchanged.

^{Reviewed by Cursor Bugbot for commit ce0260b. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Set explicit connection pool limits on EvalClient to match V1 concurrency

Sets httpx.Limits(max_connections=128, max_keepalive_connections=20) on the httpx.AsyncClient in EvalClient.__init__, replacing HTTPX's defaults. The 128-connection cap aligns with V1's default concurrency level; the 20 keepalive cap retains HTTPX's built-in bound.

^{Macroscope summarized ce0260b.}

[macroscopeapp]

Approvability

Verdict: Approved

This is a minor configuration change that adjusts HTTP connection pool limits to align with V1 concurrency settings. The change is self-contained, well-commented, and low risk.

^{You can customize Macroscope's approvability policy. Learn more.}