fix(deps): raise aiohttp floor to 3.14.2 to clear pooled-connection timeouts - #35337
Conversation
…imeouts aiohttp 3.14.0 and 3.14.1 re-arm the sock_read timer on a keep-alive connection after it has already been returned to the idle pool. The stray timer stamps a SocketTimeoutError on the pooled connection without closing it, so the pool keeps handing it out and the next request to pick it up fails instantly on an error left behind by an earlier, unrelated request. Because a single pool is shared across providers, the failures appear simultaneously across Vertex AI, Bedrock, Anthropic and OpenAI-compatible deployments as sub-millisecond "Connection timed out" errors. uv.lock resolved aiohttp 3.14.1 and the published images install via `uv sync --frozen`, so every image built from that lock shipped the regression. The wheel's own metadata declared `aiohttp>=3.10,<4.0`, which also left pip consumers free to resolve into the same broken window, so both the runtime floor and the uv constraint move to >=3.14.2. Upstream fixed this in aio-libs/aiohttp#12954, released in aiohttp 3.14.2; the lock now resolves 3.14.3. Raising the floor rather than capping below 3.14 keeps the advisories that the existing 3.14.1 floor cleared, so no osv-scanner ignores are needed. litellm requires Python >=3.10 and aiohttp 3.14.2 requires >=3.10, so no supported interpreter loses support. Both new tests fail on the previous pins and pass on these.
Greptile SummaryRaises the minimum supported aiohttp version to avoid pooled-connection timeout regressions.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| pyproject.toml | Raises both aiohttp dependency declarations to a fixed release floor. |
| tests/local_testing/test_basic_python_version.py | Adds packaging and lockfile assertions excluding the affected aiohttp releases. |
| uv.lock | Relocks aiohttp at 3.14.3 and updates its distribution artifacts. |
Reviews (2): Last reviewed commit: "fix(deps): raise aiohttp floor to 3.14.2..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "click>=8.0.0,<9.0", | ||
| "jinja2>=3.1.6,<4.0", | ||
| "aiohttp>=3.10,<4.0", | ||
| "aiohttp>=3.14.2,<4.0", |
There was a problem hiding this comment.
When the scheduled price and context-window update runs with --with 'aiohttp==3.13.3', the new aiohttp>=3.14.2 project floor makes the requirements unsatisfiable, causing dependency resolution to fail before the update script runs.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
@greptile uv --with installs an overlay layer, so the pin shadows the project floor instead of conflicting. Ran the exact command; aiohttp resolves 3.13.3, workflow unaffected. |
|
You're correct. |
|
@greptile review again please |
3 participants
TLDR
Problem this solves:
uv.lockpinned aiohttp 3.14.1, which poisons pooled connectionsaiohttp>=3.10, so pip could land there tooHow it solves it:
aiohttp>=3.14.2Relevant issues
Fixes #33820
Linear ticket
Resolves LIT-4726
Pre-Submission checklist
@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Both runs below are at commit
ffd6ac52c5, against the real Anthropic API. The only variable is the aiohttp version, because the aiohttp version is the entire change: "before" is the 3.14.1 thatuv.lockpinned, "after" is the 3.14.3 this PR locks.The probe opens one pooled connection, then issues real
POST https://api.anthropic.com/v1/messagescalls with a 4s idle gap against a 2ssock_read, so each call lands on a connection that has been sitting in the keepalive pool. aiohttp's own tracing hooks report whether each call got a fresh connection (NEW) or a pooled one (REUSE).Before, on aiohttp 3.14.1:
Every fresh connection succeeds and almost every pooled one fails, which is the reported symptom: the failure belongs to whichever request happens to pick up a connection an earlier, unrelated request left poisoned.
After, on aiohttp 3.14.3:
One connection now serves all 14 calls, so keepalive reuse is intact rather than merely avoided.
The same before/after holds without a provider account, using the upstream reproducer shape against a local aiohttp server, where the timing signature is unambiguous:
Those failures land in 0.1-0.3 ms against a 1s timeout, matching the
time taken=0.001 secondsin the issue report; they are not real waits.Also confirmed that a proxy on the new lock serves live traffic normally, via
POST /v1/chat/completionsthroughlitellm/proxy/proxy_cli.pytoanthropic/claude-haiku-4-5, 8/8 HTTP 200 with distinct real completions at 1.1-2.4s.Type
🐛 Bug Fix
Changes
aiohttp 3.14.0 and 3.14.1 re-arm the
sock_readtimer on a keep-alive connection after it has already been returned to the idle pool. The stray timer stamps aSocketTimeoutErroron the pooled connection without closing it, and aiohttp's pool acquisition never rechecks that flag, so it keeps handing the connection out. The next request to pick it up fails immediately on an error left behind by an earlier request. Because one pool is shared across providers, the failures surface simultaneously across Vertex AI, Bedrock, Anthropic and OpenAI-compatible deployments, which is what made this look provider-independent in the field.Upstream fixed it in aio-libs/aiohttp#12954, released in aiohttp 3.14.2. That release did not exist when this was first triaged, which is why the earlier attempts had to work around it.
The exposure was through two separate declarations.
uv.lockresolved 3.14.1, and the published images install withuv sync --frozen, so every image built from that lock shipped the regression. Separately the wheel's own metadata declaredaiohttp>=3.10,<4.0, which left anyone installing via pip free to resolve into the same broken window regardless of the lock; a user reporting this on a version whose lock was clean is explained by exactly that. Both move to>=3.14.2.Raising the floor rather than capping below 3.14 matters for CI hygiene: the existing 3.14.1 floor is what cleared the aiohttp advisories that
osv-scannerchecks, so moving up keeps that coverage and needs no new ignore entries, whereas capping below 3.14 would reintroduce nine of them.litellm requires Python
>=3.10and aiohttp 3.14.2 requires>=3.10, so no supported interpreter loses support. Verified withuv pip install --resolution=lowest-direct .on 3.10 through 3.13, each resolving aiohttp to exactly 3.14.2. Two pre-existing failures surfaced during that check and are unrelated to this change, reproducing identically on the base commit: the lowest-direct resolve cannot buildtiktoken==0.8.0on Python 3.14, and a base-SDK install cannotimport litellmbecauselitellm/integrations/otel/model/config.pyimportspydantic_settings, which ships only in theproxyextra. Both are worth their own PRs.The relock is contained; comparing resolved name/version pairs across the whole lock,
aiohttp 3.14.1 -> 3.14.3is the only package that moved, with 436 packages before and after.Two tests are added to
tests/local_testing/test_basic_python_version.py, which already holds the packaging-metadata assertions and runs in CI. One asserts the declared runtime requirement admits neither 3.14.0 nor 3.14.1, covering the wheel that pip consumers resolve against. The other asserts the versionuv.lockresolves is outside that range, covering the images. Both fail on the previous pins with the offending version named in the assertion message, and pass on these.This supersedes #33822, which capped below 3.14 and was blocked on accepting the nine reintroduced advisories, and #33828, which added a
HardenedTCPConnectorto re-check the flag at acquisition. Both were correct responses to there being no fixed aiohttp release at the time. Now that 3.14.2 is out, the dependency bump makes the workaround code unnecessary, so I would suggest closing both rather than carrying a permanent connector subclass for a fixed upstream bug.Final Attestation