SL validation failed for https://sts.amazonaws.com/ EOF occurred in violation of protocol (_ssl.c:618)

[andrealai75]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
We upgraded from 1.18.173 to 1.18.183 and now when we execute 'aws sts get-caller-identity' we get the following error: "SSL validation failed for https://sts.amazonaws.com/ EOF occurred in violation of protocol (_ssl.c:618)"

SDK version number
aws-cli/1.18.183 Python/2.7.5 Linux/3.10.0-1160.6.1.el7.x86_64 botocore/1.19.23
Python 2.7.5
pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)

Platform/OS/Hardware/Device
Red Hat Enterprise Linux Server release 7.7 (Maipo)

To Reproduce (observed behavior)
Using 1.18.183 just run 'aws sts get-caller-identity'. We also get the same error running for example 'aws s3 ls'.

If we downgrade using 'pip install --upgrade awscli==1.18.173' works fine on the same machine and user.

Expected behavior
'aws sts get-caller-identity' should work.

Logs/output

2020-11-23 14:53:52,987 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 217, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 358, in call
return command_table[parsed_args.operation](remaining, parsed_globals)
File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 530, in call
call_parameters, parsed_globals)
File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 650, in invoke
client, operation_name, parameters, parsed_globals)
File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 662, in _make_client_call
**parameters)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 663, in _make_api_call
operation_model, request_dict, request_context)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 682, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/lib/python2.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/lib/python2.7/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/usr/lib/python2.7/site-packages/botocore/endpoint.py", line 256, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/usr/lib/python2.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/lib/python2.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/usr/lib/python2.7/site-packages/botocore/retryhandler.py", line 183, in call
if self._checker(attempts, response, caught_exception):
File "/usr/lib/python2.7/site-packages/botocore/retryhandler.py", line 251, in call
caught_exception)
File "/usr/lib/python2.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/lib/python2.7/site-packages/botocore/retryhandler.py", line 317, in call
caught_exception)
File "/usr/lib/python2.7/site-packages/botocore/retryhandler.py", line 223, in call
attempt_number, caught_exception)
File "/usr/lib/python2.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
SSLError: SSL validation failed for https://sts.amazonaws.com/ EOF occurred in violation of protocol (_ssl.c:618)
2020-11-23 14:53:52,988 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

SSL validation failed for https://sts.amazonaws.com/ EOF occurred in violation of protocol (_ssl.c:618)

Additional context

[kdaily]

Hi @andrealai75,

Are you using a proxy?

I note that the urllib3 version was bumped in the AWS CLI dependency botocore between the versions you mentioned, related to this issue: urllib3/urllib3#1850.

[andrealai75]

Yes we are behind proxy with endpoint restrictions. The .amazonaws.com domain is allowed.

[kdaily]

Can you provide how you are configuring the proxy? It's likely that the urllib3 change is the culprit. As noted in the urllib3 issue linked above, if you are using a plaintext HTTP connection to the proxy you need to specify the proxy as such using http.

[andrealai75]

We only set the usual proxy environment variables:
export http_proxy=http://x.x.x.x:3128
export https_proxy=http://x.x.x.x:3128
export no_proxy=127.0.0.1,localhost,169.254.169.254
export no_proxy_aws=169.254.169.254

[andrealai75]

You are right we are contacting the proxy over HTTP. I read urllib3/urllib3#1850 and it looks I should be getting a Warning and not a failure.

[julb]

Hello,
I confirm I have the exact same issue with a version a bit earlier than @andrealai75 :

$ aws --version
aws-cli/1.18.178 Python/3.6.8 Linux/3.10.0-1127.19.1.el7.x86_64 botocore/1.19.18

We are behind a corporate proxy, configured with http_proxy and https_proxy env variables.

The exact same job is however working with a previous version:

$ aws --version
aws-cli/1.18.160 Python/3.6.8 Linux/3.10.0-1127.19.1.el7.x86_64 botocore/1.19.0

[sonnyhui98]

I reverted to version 1.18.165 and it‘s working fine for me.

[kdaily]

@andrealai75 -

If you go back in the debug traceback, you should see an error from urllib3 like the following:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 756, in urlopen
File "urllib3/util/retry.py", line 506, in increment
File "urllib3/packages/six.py", line 734, in reraise
File "urllib3/connectionpool.py", line 696, in urlopen
File "urllib3/connectionpool.py", line 964, in prepare_proxy
File "urllib3/connection.py", line 359, in connect
File "urllib3/connection.py", line 502, in connect_tls_proxy
File "urllib3/util/ssl.py", line 421, in ssl_wrap_socket
File "urllib3/util/ssl.py", line 464, in _ssl_wrap_socket_impl
File "ssl.py", line 412, in wrap_socket
File "ssl.py", line 853, in _create
File "ssl.py", line 1117, in do_handshake
urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:1056)

We're capturing the output of that error that you see, but it's coming from urllib3.

@sonnyhui98, @julb - the change was made in botocore==1.19.17, which was used for awscli==1.18.177.

[nateprewitt]

As an extra note, if you install urllib3 < 1.26 you can continue installing newer versions of the AWS CLI for the time being. That will support the old proxy configuration. However, we'd highly recommend reviewing the proxy setup you're using since there are security concerns that prompted urllib3 to change this behavior starting in 1.26.

[kdaily]

Thanks @nateprewitt! I think this is resolved. I've discussed with our doc writers and we will review the user guide for configuring a proxy to be sure it reflects any changes with this.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[huangtouge]

Remember to set ca bundle for the awscli
https://docs.aws.amazon.com/sdkref/latest/guide/setting-global-ca_bundle.html