Remote end closed connection without response

[Nubuki-all]

Describe the bug
aria2p suddenly hung up and disconnects after adding torrent with many files
aria2c daemon process is also stopped after this
The bug doesn't occur if torrent were added directly with aria2c.

To Reproduce
Steps to reproduce the behavior:

1. using aria2p as a library add torrent uri or magnetic link with aria2p.add
2. If magnetic link error happens after getting metadata
3. If torrent link error occurs after downloading torrent file

Expected behavior
Download should proceed normally

Screenshots
If applicable, add screenshots to help explain your problem.

System (please complete the following information):

• aria2p version: 0.11.3
• Python version: 3.11
• OS: Linux

Additional context
Error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/lib64/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/lib64/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bot/bot/workers/downloaders/download.py", line 373, in progress_for_aria2
    download = await sync_to_async(self.aria2.get_download, gid)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/bot/bot/utils/bot_utils.py", line 635, in sync_to_async
    return await future if wait else future
           ^^^^^^^^^^^^
  File "/usr/lib64/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aria2p/api.py", line 287, in get_download
    return Download(self, self.client.tell_status(gid))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aria2p/client.py", line 880, in tell_status
    return self.call(self.TELL_STATUS, [gid, keys])  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aria2p/client.py", line 265, in call
    return self.res_or_raise(self.post(payload))
                             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aria2p/client.py", line 361, in post
    return requests.post(self.server, data=payload, timeout=self.timeout).json()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[pawamoy]

Hello, thanks for the report.

Does it occur reliably, or is this issue sporadic? I'm unsure how and why the client would crash the server itself. I'll try to replicate anyway.

[hannesweisbach]

Hello,

I think I experience the same issue. I'm running aria2c 1.36.0, aria2p 0.12.0 and python 3.11.6. Trying to add a torrent file results in a similar backtrace:

aria2p -H http://<ip> -s <mysecret> add-torrents  ~/Downloads/NMT.torrent 
Traceback (most recent call last):
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 1378, in getresponse
    response.begin()
  File "/usr/lib/python3.11/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hannesweisbach/.local/venvs/aria2p/bin/aria2p", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/cli/main.py", line 115, in main
    return commands[subcommand](api, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/cli/commands/add_torrent.py", line 48, in add_torrents
    new_download = api.add_torrent(torrent_file, options=options, position=position)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/api.py", line 151, in add_torrent
    gid = self.client.add_torrent(encoded_contents, uris, client_options, position)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/client.py", line 528, in add_torrent
    return self.call(self.ADD_TORRENT, [torrent, uris, options, position])  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/client.py", line 258, in call
    return self.res_or_raise(self.post(payload))
                             ^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/aria2p/client.py", line 351, in post
    return requests.post(self.server, data=payload, timeout=self.timeout).json()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hannesweisbach/.local/venvs/aria2p/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I was never able to add this torrent file at all. So as far as I can tell, it's always reproducible. I cannot rule out that the torrent file is maliciously crafted or that the error is with aria2c. But I thought I'd let you know, since it seemed relevant.

[pawamoy]

Thanks! Have you tried passing this file directly to aria2c? Does it accept it without error?

[Nubuki-all]

Hello, thanks for the report.

Does it occur reliably, or is this issue sporadic? I'm unsure how and why the client would crash the server itself. I'll try to replicate anyway.

Sorry I didn't realize this has been replied to
It occurs reliably especially on certain torrents

[pawamoy]

Thanks, and no worries. I don't think I'll have time to investigate why a particular torrent makes it fail and how to fix that, but we can definitely improve the error handling here 🙂

[by-justin]

If a torrent is too large (larger than 2M) , aria2c drops rpc connection.

Document says cranking this value up will solve the issue. I tried to set "rpc-max-request-size": "1073741824" in global options, it doesn't work, but other options do work.

So in the mean time, the simplest work around would be

aria2c --enable-rpc --daemon=true --rpc-max-request-size=1024M

[hannesweisbach]

Thanks! Have you tried passing this file directly to aria2c? Does it accept it without error?

Sorry for the late reply. I did test it with aria2c directly and I did not observe aria2c throwing any errors nor did I see any errors in the log file.

[pawamoy]

So I don't think there's anything meaningful we can do after catching such an error, so I've decided to just log an error hinting at --rpc-max-request-size and just re-raise the exception.