use epoll instead of select #1053

liuqiang1999 · 2017-04-08T05:18:14Z

hi, I get errors "ValueError: filedescriptor out of range in select()" when uploading files in my application. Looks like it's because boto3 uses select() instead of epoll for socket. Can this be changed? Thanks

stealthycoin · 2017-04-10T16:20:01Z

What command are you running and can you provide the output of adding --debug to it?

liuqiang1999 · 2017-04-11T03:47:28Z

Hi I don't know how to add --debug, but here is the exception info:

I looked at the log again, maybe I should ask the question in "botocore" project? Thanks

Traceback (most recent call last):
File "{mycode}.py", line 100, in func
boto3.client('s3').upload_file(Filename=file, Bucket=bucket, Key=file)
File "/usr/local/lib/python3.5/site-packages/boto3/s3/inject.py", line 106, in upload_file
extra_args=ExtraArgs, callback=Callback)
File "/usr/local/lib/python3.5/site-packages/boto3/s3/transfer.py", line 275, in upload_file
future.result()
File "/usr/local/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
return self._coordinator.result()
File "/usr/local/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
raise self._exception
File "/usr/local/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in call
return self._execute_main(kwargs)
File "/usr/local/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/usr/local/lib/python3.5/site-packages/s3transfer/upload.py", line 679, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 253, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 531, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 170, in _send_request
success_response, exception):
File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 249, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/usr/local/lib/python3.5/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.5/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 183, in call
if self._checker(attempts, response, caught_exception):
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 251, in call
caught_exception)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 269, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 317, in call
caught_exception)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 223, in call
attempt_number, caught_exception)
File "/usr/local/lib/python3.5/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/usr/local/lib/python3.5/site-packages/botocore/endpoint.py", line 204, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/local/lib/python3.5/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.5/site-packages/botocore/vendored/requests/adapters.py", line 370, in send
timeout=timeout
File "/usr/local/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python3.5/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 349, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib64/python3.5/http/client.py", line 1083, in request
self._send_request(method, url, body, headers)
File "/usr/local/lib/python3.5/site-packages/botocore/awsrequest.py", line 129, in _send_request
self, method, url, body, headers, *args, **kwargs)
File "/usr/lib64/python3.5/http/client.py", line 1128, in _send_request
self.endheaders(body)
File "/usr/lib64/python3.5/http/client.py", line 1079, in endheaders
self._send_output(message_body)
File "/usr/local/lib/python3.5/site-packages/botocore/awsrequest.py", line 162, in _send_output
read, write, exc = select.select([self.sock], [], [self.sock], 1)
ValueError: filedescriptor out of range in select()

stablerg · 2017-06-17T18:10:49Z

Has there been any updates on this? I did not see an issue in botocore for it. I am running into the issue in my program after it runs for several hours and I haven't found a workaround for it.

My client code is running the following inside of its own thread:

session = boto3.session.Session(aws_access_key_id=AWS_ACCESS_KEY_ID,
                                aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
s3 = session.client('s3')
s3_path = local_path.replace(LOCAL_LOGS_DIR, 'data')
s3.upload_file(local_path, S3_BUCKET, s3_path)

I'm running inside the Python 3.6.1 Docker container with the following library versions:
boto3==1.4.4
botocore==1.5.68
requests==2.18.1
s3transfer==0.1.10
urllib3==1.21.1

Here's the stacktrace for my version:

File "./data_collector/extractor.py", line 483, in upload_file_to_s3
    s3_transfer.upload_file(local_path, settings.S3_BUCKET, s3_path)
  File "/usr/local/lib/python3.6/site-packages/boto3/s3/transfer.py", line 275, in upload_file
    future.result()
  File "/usr/local/lib/python3.6/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.6/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/usr/local/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/s3transfer/upload.py", line 679, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 544, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 141, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 170, in _send_request
    success_response, exception):
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 249, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 204, in _get_response
    proxies=self.proxies, timeout=self.timeout)
  File "/usr/local/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/site-packages/botocore/awsrequest.py", line 130, in _send_request
    self, method, url, body, headers, *args, **kwargs)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/site-packages/botocore/awsrequest.py", line 163, in _send_output
    read, write, exc = select.select([self.sock], [], [self.sock], 1)
ValueError: filedescriptor out of range in select()

sethmlarson · 2017-06-17T22:21:15Z

May I suggest vendoring/using selectors2 or an equivalent library?

(Disclosure: I'm the author of selectors2)

liuqiang1999 · 2017-06-18T00:25:20Z

No I don't have a solution. I agree the fix should be in botocore, to change the library of select.
I just simply reduced my file descriptor usage, and avoid it.

stablerg · 2017-06-19T19:58:21Z

While digging into it more, I discovered my program was leaking threads, which led to the high file descriptor count and hitting the error. After fixing the thread leak, the problem went away.

joguSD · 2017-07-21T22:01:06Z

Looks like this is an issue in the urllib3 library. Was fixed in this pr: urllib3/urllib3#1001

zxwx · 2017-08-03T23:32:04Z

overselect.py:

import boto3

files = [open("/tmp/{}".format(i), 'w') for i in range(1024)]
boto3.client('s3').upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')

$ python overselect.py
Traceback (most recent call last):
  File "select.py", line 3, in <module>
    boto3.client('s3').upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')
  File "/usr/local/lib/python2.7/dist-packages/boto3/s3/inject.py", line 106, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 275, in upload_file
    future.result()
  File "/usr/local/lib/python2.7/dist-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/usr/local/lib/python2.7/dist-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
ValueError: filedescriptor out of range in select()

Looks like select is imported both in the vendored version of urllib3 and in botocore/awsrequest.py

DamZiobro · 2017-09-14T23:40:08Z

Linux allows to open 1024 files per process at the same time (http://docs.oracle.com/cd/E19450-01/820-6168/file-descriptor-requirements.html). Therefore the above overselect.py does not work. Is this 'boto' task to omitting OS limitations?

zxwx · 2017-10-18T22:06:13Z

@JordonPhillips @kyleknap Any updates?

@XMementoIT You are mistaken; linux does not have a fixed limitation on the number of open files per process. In fact, the link you shared explains how to adjust some of the relevant system limits.

lqs · 2018-01-16T08:33:03Z

Python's select.select() throws an exception if any fd value is larger than FD_SETSIZE.

To avoid the select() call in awsrequest.py, add the workaround to skip it:

from botocore import handlers
handlers.BUILTIN_HANDLERS = filter(lambda el: el[1] is not handlers.add_expect_header,
                                   handlers.BUILTIN_HANDLERS)

tuulos · 2018-05-24T20:22:46Z

any updates? We hit this issue too

swetashre · 2020-03-26T23:34:27Z

Following up on this issue. Considering it's a old issue, is anyone still getting this error with latest version of sdk ? If anyone is still getting the error please reopen a new issue.

no-response · 2020-04-02T23:36:57Z

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

stealthycoin added closing-soon This issue will automatically close in 4 days unless further comments are made. question labels Apr 10, 2017

joguSD mentioned this issue Jul 21, 2017

A urllib3 connection is not put back in the pool after HEAD boto/botocore#1248

Closed

kyleknap added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed closing-soon This issue will automatically close in 4 days unless further comments are made. question labels Aug 14, 2017

kyleknap assigned JordonPhillips Aug 14, 2017

swetashre added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Mar 23, 2020

swetashre added closing-soon This issue will automatically close in 4 days unless further comments are made. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Mar 26, 2020

no-response bot closed this as completed Apr 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use epoll instead of select #1053

use epoll instead of select #1053

liuqiang1999 commented Apr 8, 2017

stealthycoin commented Apr 10, 2017

liuqiang1999 commented Apr 11, 2017 •

edited

Loading

stablerg commented Jun 17, 2017

sethmlarson commented Jun 17, 2017

liuqiang1999 commented Jun 18, 2017

stablerg commented Jun 19, 2017

joguSD commented Jul 21, 2017

zxwx commented Aug 3, 2017 •

edited

Loading

DamZiobro commented Sep 14, 2017

zxwx commented Oct 18, 2017

lqs commented Jan 16, 2018

tuulos commented May 24, 2018

swetashre commented Mar 26, 2020

no-response bot commented Apr 2, 2020

use epoll instead of select #1053

use epoll instead of select #1053

Comments

liuqiang1999 commented Apr 8, 2017

stealthycoin commented Apr 10, 2017

liuqiang1999 commented Apr 11, 2017 • edited Loading

stablerg commented Jun 17, 2017

sethmlarson commented Jun 17, 2017

liuqiang1999 commented Jun 18, 2017

stablerg commented Jun 19, 2017

joguSD commented Jul 21, 2017

zxwx commented Aug 3, 2017 • edited Loading

DamZiobro commented Sep 14, 2017

zxwx commented Oct 18, 2017

lqs commented Jan 16, 2018

tuulos commented May 24, 2018

swetashre commented Mar 26, 2020

no-response bot commented Apr 2, 2020

liuqiang1999 commented Apr 11, 2017 •

edited

Loading

zxwx commented Aug 3, 2017 •

edited

Loading