Random ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport...

[JulienPalard]

Describe the bug

Since I recently learned that #1925 is closed, I happily poked around and boom got a new RecourceWarning.

Sadly it does not happen every time, it really depends on the number of domains I have in my list, if I try to reduce the list, in the hope to find a culpit domain, the warning vanishes.

To Reproduce

Here's a reproducer (I'm sorry for the APHP, I have nothing against them, I hope some 10s of requests is not noticable in their load average, I am not trying to hurt in any way... just happen I reproduced the bug using a dataset of their URLs).

reproducer

import aiohttp
import asyncio

urls = [
    "aaa.aphp.fr",
    "addictovigilance.aphp.fr",
    "administration-goodmeeting.aphp.fr",
    "affairesjuridiques.aphp.fr",
    "ageps.aphp.fr",
    "ambroisepare.aphp.fr",
    "ami-dd.aphp.fr",
    "anomaliescervelet.aphp.fr",
    "antithrombotiques.aphp.fr",
    "aphp.aphp.fr",
    "aphp.fr",
    "archives.aphp.fr",
    "blogs.aphp.fr",
    "bo-myaphp.aphp.fr",
    "campus.aphp.fr",
    "cancer-estparisien.aphp.fr",
    "cancer-ghparis10.aphp.fr",
    "cancer-grossesse.aphp.fr",
    "cancer.aphp.fr",
    "cedit.aphp.fr",
    "cfdc.aphp.fr",
    "charlesfoix.aphp.fr",
    "chirurgie-digestive-sat.aphp.fr",
    "chirurgie-refractive.aphp.fr",
    "chu-mondor.aphp.fr",
    "chu93.aphp.fr",
    "cleanweb.aphp.fr",
    "cme.aphp.fr",
    "cnr-escherichiacoli-robertdebre.aphp.fr",
    "compare.aphp.fr",
    "corevih-idfcentre.aphp.fr",
    "covid-documentation.aphp.fr",
    "darsaintantoine.aphp.fr",
    "dhu.aphp.fr",
    "diabete-robertdebre.aphp.fr",
    "direct.aphp.fr",
    "disphelif.aphp.fr",
    "eds.aphp.fr",
    "electionscommissionsoins.aphp.fr",
    "endocrino-sat.aphp.fr",
    "epilepsie-robertdebre.aphp.fr",
    "espace-emploi.aphp.fr",
    "ethique-clinique.aphp.fr",
    "evenementiels.aphp.fr",
    "fcm.aphp.fr",
    "fertilite-bichat.aphp.fr",
    "formulairesrendezvous.aphp.fr",
    "frequentationdesurgences.aphp.fr",
    "ghparis10.aphp.fr",
]


async def hit(url, session):
    try:
        async with session.head(f"https://{url}/") as r:
            print(url, r.status)
    except Exception as err:
        print(url, err)


async def main():
    async with aiohttp.ClientSession(
        connector=aiohttp.TCPConnector(limit=10, limit_per_host=1),
    ) as session:
        await asyncio.gather(*[hit(url, session=session) for url in urls])


asyncio.run(main())

It does not reproduce the bug every time (sorry again for the servers in front).

Expected behavior

No ResourceWarning, as they could lead in fd leaks on long running client apps.

Logs/tracebacks

/home/mdk/.local/lib/python3.9/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x7fa436ef44b0>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
Object allocated at (most recent call last):
  File "/home/mdk/clones/noms-de-domaine-organismes-publics/sources/.venv/lib/python3.9/site-packages/aiohttp/client.py", lineno 1077
    self._resp = await self._coro
  File "/home/mdk/clones/noms-de-domaine-organismes-publics/sources/.venv/lib/python3.9/site-packages/aiohttp/client.py", lineno 479
    conn = await self._connector.connect(
  File "/home/mdk/clones/noms-de-domaine-organismes-publics/sources/.venv/lib/python3.9/site-packages/aiohttp/connector.py", lineno 508
    proto = await self._create_connection(req, traces, timeout)
  File "/home/mdk/clones/noms-de-domaine-organismes-publics/sources/.venv/lib/python3.9/site-packages/aiohttp/connector.py", lineno 869
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/mdk/clones/noms-de-domaine-organismes-publics/sources/.venv/lib/python3.9/site-packages/aiohttp/connector.py", lineno 1093
    transp, proto = await self._wrap_create_connection(
  File "/home/mdk/clones/noms-de-domaine-organismes-publics/sources/.venv/lib/python3.9/site-packages/aiohttp/connector.py", lineno 948
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/home/mdk/.local/lib/python3.9/asyncio/base_events.py", lineno 1081
    transport, protocol = await self._create_connection_transport(
  File "/home/mdk/.local/lib/python3.9/asyncio/base_events.py", lineno 1103
    transport = self._make_ssl_transport(
  File "/home/mdk/.local/lib/python3.9/asyncio/selector_events.py", lineno 74
    ssl_protocol = sslproto.SSLProtocol(
  File "/home/mdk/.local/lib/python3.9/asyncio/sslproto.py", lineno 445
    self._app_transport = _SSLProtocolTransport(self._loop, self)

Python Version

3.9.7

aiohttp Version

4.0.0a1 from pip install --pre, also tried on `master` branch, at 4ebd52ed1c29c7db52039f213cd2ab268c85bc74.

multidict Version

Name: multidict
Version: 4.7.6
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/mdk/clones/noms-de-domaine-organismes-publics/sources/.venv/lib/python3.9/site-packages
Requires: 
Required-by: yarl, aiohttp

yarl Version

Name: yarl
Version: 1.7.0
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/mdk/clones/noms-de-domaine-organismes-publics/sources/.venv/lib/python3.9/site-packages
Requires: idna, multidict
Required-by: aiohttp

OS

Debian Bookworm

Related component

Client

Additional context

No proxy, very good connection to the internet, but laptop on Wi-Fi so like 60Mbps up 60Mbps down.

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[webknjaz]

Hey @JulienPalard, 4.0.0a1 is quite an old alpha. You said you tried master, does it mean that it's reproducible there as well?

[JulienPalard]

Hey @JulienPalard, 4.0.0a1 is quite an old alpha.

Oh, I missed that, I blindly trusted pip install --pre to give me something fresh :D

You said you tried master, does it mean that it's reproducible there as well?

Yes, I can try again for good measure: I'm on 9723abf, I run pip install ., then python repro.py and yes:

/home/mdk/.local/lib/python3.9/asyncio/sslproto.py:320: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x7f80acf9fd70>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)

[JulienPalard]

Looks like it's not an aiohttp issue, opened a cpython issue: https://bugs.python.org/issue46318

[andrewfogg]

@JulienPalard I have had a similar issue that has been driving me nuts for a couple of months. Have you found a workaround?

[JulienPalard]

I worked on this a bit. I have not found a workaround, but I think I slowly surrounded the root cause which is around asyncio/sslproto.py and/or _ssl.c (cpython side), probably really near SSH_shutdown which handling looks a bit on the simple side (like "try it twice and that's it.") (handling it the right way looks not trivial, easily leading to security issues, if I understand correctly).

[Alexander-Serov]

For those interested in how to solve it, this workaround worked for me:

await asyncio.sleep(0)

See here