Skip to content

fix(deps): pin aiohttp below 3.14 to avoid pooled-connection timeout regression - #33822

Open
devin-ai-integration[bot] wants to merge 3 commits into
litellm_internal_stagingfrom
litellm_aiohttp_pin_below_314
Open

fix(deps): pin aiohttp below 3.14 to avoid pooled-connection timeout regression#33822
devin-ai-integration[bot] wants to merge 3 commits into
litellm_internal_stagingfrom
litellm_aiohttp_pin_below_314

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Relevant issues

Fixes #33820

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

The regression lives entirely in aiohttp, so the proof runs the self-contained repro from aio-libs/aiohttp#12953 against the two aiohttp versions this PR moves between. No mocks; a real aiohttp server and a real keep-alive pooled connection. Captured at commit d21795864c9f6beb5b8e12419d7743627bb8626f

Before (what v1.91.0+ resolves today, aiohttp==3.14.1):

$ uv run --no-project --with "aiohttp==3.14.1" python repro.py
aiohttp version = 3.14.1
after req1: read_timeout_handle armed = True
after idle: protocol exception = SocketTimeoutError
Traceback (most recent call last):
  ...
  File ".../aiohttp/streams.py", line 713, in read
    raise self._exception
aiohttp.client_exceptions.SocketTimeoutError: Timeout on reading data from socket

After (what this PR resolves, aiohttp==3.13.5):

$ uv run --no-project --with "aiohttp==3.13.5" python repro.py
aiohttp version = 3.13.5
after req1: read_timeout_handle armed = False
after idle: protocol exception = None
RESULT: OK

The repro (from the upstream issue) opens a local aiohttp server, makes one request whose buffered body is read, checks the pooled connection, idles past sock_read but under keepalive_timeout, then makes a second request that reuses the pooled connection:

timeout = aiohttp.ClientTimeout(total=30, sock_read=1)
conn = aiohttp.TCPConnector(keepalive_timeout=75)
async with aiohttp.ClientSession(timeout=timeout, connector=conn) as s:
    async with s.get("http://127.0.0.1:8080/") as r:
        await r.read()
    proto = next(iter(conn._conns.values()))[0][0]
    print("armed =", proto._read_timeout_handle is not None)
    await asyncio.sleep(2)
    print("exc =", type(proto.exception()).__name__ if proto.exception() else None)
    async with s.get("http://127.0.0.1:8080/") as r:  # reuses poisoned conn
        await r.read()
    print("RESULT: OK")

The lockfile confirms the resolution moves with the constraint:

$ uv lock
Updated aiohttp v3.14.1 -> v3.13.5

Type

🐛 Bug Fix

Changes

Since v1.91.0 the uv constraint-dependencies pin raises the aiohttp floor to >=3.14.1,<4.0, and 3.14.x carries a keep-alive pool poisoning regression (aio-libs/aiohttp#12953). Reading the already-buffered body of a completed response calls resume_reading(), which in 3.14 lost its _reading_paused guard and re-arms the sock_read timer on a connection that has already been returned to the keep-alive pool. When keepalive_timeout > sock_read, that timer later stamps a SocketTimeoutError on the idle pooled ResponseHandler without closing the transport, so the next request to grab it fails instantly with litellm.Timeout: Connection timed out ... time taken=0.001 seconds. Because it is the shared pool, this shows up as sporadic sub-millisecond timeouts across every provider at once (Vertex, Bedrock, Anthropic, OpenAI-compatible)

The upstream fix (aio-libs/aiohttp#12954) merged on 2026-06-18 but has not shipped in any release; the latest aiohttp is still 3.14.1 (uploaded 2026-06-07), so no aiohttp release currently has both the two medium CVEs fixed and the regression fixed. This PR caps the constraint back to >=3.13.5,<3.14, matching what v1.90.0 already shipped, and relocks (aiohttp 3.14.1 -> 3.13.5)

Holding below 3.14 reintroduces the two medium advisories that the 3.14.1 bump cleared, so their osv-scanner ignore entries come back. Both barely apply to litellm: GHSA-hg6j-4rv6-33pg only leaks cookies set via the per-request cookies parameter across a cross-origin redirect, and GHSA-jg22-mg44-37j8 only bites CookieJar.load() on attacker-controlled files; a sporadic cross-provider outage is the worse tradeoff. The ignore entries and the constraint should both be dropped once a 3.14.x release ships the #12954 fix

tests/local_testing/test_basic_python_version.py::test_aiohttp_constraint_excludes_314 parses the constraint and fails if the pin ever admits 3.14.0 or 3.14.1 again, so this cannot silently flip back the way it did between v1.89.3 and v1.91.0

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/97741dd6c63a4f7f9bef188126178284

@devin-ai-integration
devin-ai-integration Bot requested a review from a team July 18, 2026 14:10
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR pins aiohttp below 3.14 in pyproject.toml's [tool.uv] constraint-dependencies to work around a keep-alive connection-pool poisoning bug introduced in aiohttp 3.14.x, and re-adds two CVE ignore entries in osv-scanner.toml that had been cleared by the original 3.14.1 bump. The tradeoff is clearly documented.

  • pyproject.toml: Constraint downgraded from >=3.14.1,<4.0 to >=3.13.5,<3.14; lockfile re-solved accordingly (aiohttp 3.14.1 → 3.13.5).
  • osv-scanner.toml: Two new [[IgnoredVulns]] entries added with a 2026-10-15 expiry and detailed rationale.
  • tests/local_testing/test_basic_python_version.py: New regression guard test parses the live pyproject.toml and fails if the constraint ever re-admits 3.14.0 or 3.14.1; note that constraint-dependencies is uv-specific and does not protect pip users.

Confidence Score: 4/5

Safe to merge; the change is a targeted dependency pin with well-documented rationale, a matching lockfile update, and a regression guard test.

The core change (constraint pin + lockfile) is straightforward and low-risk. The new test correctly enforces the constraint but will throw an unhandled FileNotFoundError rather than skipping if pyproject.toml is absent from the expected relative path, which could cause confusing failures in non-checkout environments.

tests/local_testing/test_basic_python_version.py — the path-resolution logic for pyproject.toml deserves a second look.

Important Files Changed

Filename Overview
pyproject.toml Constraint-dependencies pin for aiohttp downgraded from >=3.14.1,<4.0 to >=3.13.5,<3.14 to avoid the keep-alive pool poisoning regression in 3.14.x; well-justified and matches the intent of the PR.
osv-scanner.toml Adds two CVE ignore entries (GHSA-hg6j-4rv6-33pg, GHSA-jg22-mg44-37j8) with expiry 2026-10-15 and detailed justifications; tradeoff is clearly documented.
tests/local_testing/test_basic_python_version.py Adds test_aiohttp_constraint_excludes_314 to enforce the version pin; pyproject.toml path resolution will raise FileNotFoundError (rather than skipping) in non-editable-install environments.
uv.lock Lockfile re-solved with aiohttp 3.14.1 -> 3.13.5; exclude-newer timestamp also bumped, which is expected when relocking.

Reviews (1): Last reviewed commit: "fix(deps): pin aiohttp below 3.14 to avo..." | Re-trigger Greptile

Comment on lines +166 to +168
pyproject_path = pathlib.Path(litellm.__file__).parent.parent / "pyproject.toml"
with open(pyproject_path, "rb") as f:
constraints = tomli.load(f)["tool"]["uv"]["constraint-dependencies"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 If pyproject.toml isn't found at the computed path (e.g. when running against a non-editable wheel install), open() raises FileNotFoundError and the test fails with an unhandled exception instead of skipping gracefully. Wrapping the file open in a similar guard makes the test self-contained regardless of install mode.

Suggested change
pyproject_path = pathlib.Path(litellm.__file__).parent.parent / "pyproject.toml"
with open(pyproject_path, "rb") as f:
constraints = tomli.load(f)["tool"]["uv"]["constraint-dependencies"]
pyproject_path = pathlib.Path(litellm.__file__).parent.parent / "pyproject.toml"
if not pyproject_path.exists():
pytest.skip(f"pyproject.toml not found at {pyproject_path} - skipping dependency check")
with open(pyproject_path, "rb") as f:
constraints = tomli.load(f)["tool"]["uv"]["constraint-dependencies"]

@codecov

codecov Bot commented Jul 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_aiohttp_pin_below_314 (61b01e7) with litellm_internal_staging (e238e89)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (3f9b71c) during the generation of this report, so e238e89 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: aiohttp 3.14.x connection-pool poisoning causes cross-provider "Connection timed out" failures (litellm pins aiohttp>=3.14.1, v1.91.0+)

1 participant