Timeout exception when reading a response body

[myarik]

Long story short

Hello
I have a strange issue with aiohttp.client, sometimes when I try to read a response body, it crashes by timeout.

My function:

async def test(self, value: str) -> dict:
       timeout = aiohttp.ClientTimeout(total=30)
       async with aiohttp.ClientSession(timeout=self.timeout) as session:
            resp = await session.get(
                urljoin(url, "api/"), params={"value": value}
            )
       if resp.status == 200:
            response = await resp.json()
            return response
       else:
            err_message = await resp.text()
            raise MyException(resp.status, err_message)

Traceback:

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
<ipython-input-95-032af7fa0ee9> in <module>
----> 1 resp = asyncio.run(st.query("select * from sources * where source matches \"groennieuws.nl\" and author matches \"Wesley Van Der Linde\";"))

~/.pyenv/versions/3.7.3/lib/python3.7/asyncio/runners.py in run(main, debug)
     41         events.set_event_loop(loop)
     42         loop.set_debug(debug)
---> 43         return loop.run_until_complete(main)
     44     finally:
     45         try:

~/.pyenv/versions/3.7.3/lib/python3.7/asyncio/base_events.py in run_until_complete(self, future)
    582             raise RuntimeError('Event loop stopped before Future completed.')
    583 
--> 584         return future.result()
    585 
    586     def stop(self):

~/scripts/consumer.py in query(self, yql_query)
     77         if resp.status == 200:
     78             response = await resp.json()
---> 79             return response
     80         else:
     81             err_message = await resp.text()

~/.pyenv/versions/3.7.3/envs/test/lib/python3.7/site-packages/aiohttp/client_reqrep.py in json(self, encoding, loads, content_type)
   1015         """Read and decodes JSON response."""
   1016         if self._body is None:
-> 1017             await self.read()
   1018 
   1019         if content_type:

~/.pyenv/versions/3.7.3/envs/test/lib/python3.7/site-packages/aiohttp/client_reqrep.py in read(self)
    967         if self._body is None:
    968             try:
--> 969                 self._body = await self.content.read()
    970                 for trace in self._traces:
    971                     await trace.send_response_chunk_received(self._body)

~/.pyenv/versions/3.7.3/envs/test/lib/python3.7/site-packages/aiohttp/streams.py in read(self, n)
    357             blocks = []
    358             while True:
--> 359                 block = await self.readany()
    360                 if not block:
    361                     break

~/.pyenv/versions/3.7.3/envs/test/lib/python3.7/site-packages/aiohttp/streams.py in readany(self)
    379         # without feeding any data
    380         while not self._buffer and not self._eof:
--> 381             await self._wait('readany')
    382 
    383         return self._read_nowait(-1)

~/.pyenv/versions/3.7.3/envs/test/lib/python3.7/site-packages/aiohttp/streams.py in _wait(self, func_name)
    295             if self._timer:
    296                 with self._timer:
--> 297                     await waiter
    298             else:
    299                 await waiter

~/.pyenv/versions/3.7.3/envs/test/lib/python3.7/site-packages/aiohttp/helpers.py in __exit__(self, exc_type, exc_val, exc_tb)
    583 
    584         if exc_type is asyncio.CancelledError and self._cancelled:
--> 585             raise asyncio.TimeoutError from None
    586         return None
    587 

TimeoutError:

Steps to reproduce

resp = asyncio.run(test("test"))

Your environment

aiohttp==3.5.4
python3.7.3

[webknjaz]

You're using a 30-second timeout. What did you expect to happen instead?

[myarik]

I don't expect to get the timeout error when I read a response body.
Also, I don't have the same issue with the requests library

Example:

[requests.get('http://localhost:8080/test/', params=params) for i in range(150)] 

It works fine

Aiohttp.client example:

[test(value) for i in range(150)] 
TimeoutError ...

It craches

[webknjaz]

I think this is by design. The timeout is applied to the whole client session, not just to exchanging headers.
requests is a synchronous lib and I bet they read the whole payload in the request function. This approach is dangerous because it can crash your process if you hit a huge file (or a number of those) which would consume all your memory.
With aiohttp, you can process such things by chunks making it safer and possible to discard processed chunks freeing up the memory.

[webknjaz]

Plz fix the indentation in your snippet: it's unclear whether your if-blocks are in the async CM or not.

[myarik]

@webknjaz Thanks for explaining.
I put the if-blocks in the async and it fixed the issue.
I really appreciate your help ;)

[ikebo]

same issue, and also can be fixed by put the if-blocks in the async, but i cannot explain it, would you please tell me why? @myarik @webknjaz

[uhmseohun]

I am struggling with same issue now.
I get timeout exception at the read statement in aiohttp.
But all the statements are in the async with block in my case.
Is there anybody who are struggling with similar issue with me?

[sio-svv]

Has anyone figured out how to resolve this error?

[Dreamsorcerer]

The async with controls the lifecycle of the session. Once you exit the async with the session is closed. Naturally, you won't be able to read a response when the session is closed (why there was a Timeout, rather than another exception, I'm not sure...).

If you have an issue while the ClientSession is still open, it is not this issue. Search for another issue or create a new one.

[SDF39679]

The async with controls the lifecycle of the session. Once you exit the async with the session is closed. Naturally, you won't be able to read a response when the session is closed (why there was a Timeout, rather than another exception, I'm not sure...).

If you have an issue while the ClientSession is still open, it is not this issue. Search for another issue or create a new one.

I have this problem after 2000-3000 requests via aiohttp. I tried to return json from the fetch function instead of the request object itself, but the error did not disappear. I solved it with the help of backoff.
@backoff.on_exception( backoff.expo, asyncio.exceptions.TimeoutError, on_backoff=lambda details: logger.write_to_log( f"ERROR(KEEP WORKING) | {details["target"]} | {type(details["exception"])}" ), max_tries=30 )