Modified the code and added user notification about Python certificat…

[Hillsie]

Modified the code and added user notification about Python certificates issue to reduce the number of future git issues being raised about the Python ... urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate ... exception. This issue has resulted in continuous stream of issues with common problems in: #133 #136 #140 #176 #6275 #6548 #6723 #9036 ...

A notification during installation would help the user resolve the issue without raising a new issue request.

[kripken]

Thanks @Hillsie! Great idea to show a helpful message here.

[kripken]

Thanks @Hillsie! I think this can be simplifed a bit more, then it's perfect.

[Hillsie]

@kripken Updated and ready for review

[kripken]

This is wrong, I think: it will be interpreted as if "a" or ("b" in c). I think we need if "a" in c or "b" in c.

[Hillsie]

Oh my, of course. Haven't resolved the Linux version, so might also change the link once I have it worked out on Linux.

[Hillsie]

@kripken I think I have gotten to the bottom of what’s happening. It has taken some time and thinking through the problem.

Here’s my view, sorry a bit long, but necessary to understand the conclusion

Background

The recommended installation on OS X/Linux/Unix is to use ./emsdk install latest which runs a shell command and checks the python version and calls emsdk.py from the shell script.

https://webassembly.org/getting-started/developers-guide/

Issue - OS X/Linux/Unix

During installation there is an install issue

SSL: certificates_verify_failed on Mac and
urlopen error unknown url type: https on Linux/Unix

It’s the storage locations rejecting the Python TLS certificate and Python error notifying that the storage locations of WebAssembly downloads do not accept 1024-bit TLS certificates.

Current recommended solutions - Upgrade Certs however, nil impact

Current solutions recommended upgrading the Python certificates to 2048-bit or greater

• On Mac a way to install the newer TLS is to download the latest Python version (currently 3.7.4) and run open /Applications/python\3.x/Install\ Certificates.command in the terminal for x version

• Another suggestion is to install certificates via pip install certifi, which upgrades the certificates for Mac/Linux/Unix

However, running ./emsdk install latest results in the same error

There are prolly edge cases where this works.

The shell script on OS X/ Linux/Unix ./emsdk

There appears to be confusings issues with certificates and Python version being used. The complication is the shell script during the installation process calls a specific Python version.

• The shell script executes the shell command which python3. What if it’s higher than 3, or you have skipped version 3, as in my case I have 3, 3.6 and 3.7 installed. In this case the shell script only uses python3 or python2 and certs are installed under a higher version and located in a different location.

Some Work Arounds

After installing the new certs with pip install certifi

• Change the version of python used in the shell script to the most currently installed version is one … for me. For instance, change this line to use python3.7 instead of python3
  python=$(which python3.7 2> /dev/null)
  And the download works. Probably not a robust solution.

• ./emsdk.py install latest is a second way. Relies on the Python shebang line. I think this is a more robust option.

• python3.x ./emsdk.py install latest means the user can do it with any version of python they choose

Question

Based on above, why is a shell script required?

#!/usr/bin/env python is the python shebang line used in the emdsk.py that executes the latest installed version of python based on the symbolic links /usr/bin/

./emsdk.py install latest works and negates the need for a shell script.

Over to you

IMHO, the installation process might be an obstacle to adoption.

Could the Python shebang line be trusted to work on each flavour of the installation and more emphasis placed on the Python scripting? Is this possible?

I’ll raise a pull request for the change I’ve made, but the impact to the TLS rejection error and issues raised around it is prolly going to be limited as there seem to be a number of issue variables.

[Hillsie]

@kripken Updated and ready for review.

[kripken]

(I think there's still one review comment left to be addressed, see above.)

Thanks for the detailed writeup!

This is definitely not easy to fix. Not sure what's best here, but reading this comment I wonder if maybe we should have an --ignore-cert or something like that, to let users ignore this error (at their own risk, obviously)?

[Hillsie]

@kripken I've updated the code and removed the comments. #354 (comment)

Ready for review.

[Hillsie]

This is definitely not easy to fix. Not sure what's best here, but reading this comment I wonder if maybe we should have an --ignore-cert or something like that, to let users ignore this error (at their own risk, obviously)?

Not entirely sure if an ignore-cert is the way to go? This is not exclusively a WebAssembly problem. I guess we'd need to answer the question, why have TLS? The core issue is that the 1024-bit certificates are no longer trusted. The installation of 2048-bit upgrade Python has implemented, doesn't impact all locally installed versions, so the symbolic Python linked versions seem to pick the old TLS certificate.

Personally, I'd prefer to maintain confidence in the WebAssembly download and not bypass the certificate. This is probably important for the entire community. A good writeup would help resolve the confusion.

[kripken]

Thanks @Hillsie, yeah, I think those are good points.

This should improve things for users a lot. If there's a good place to write more docs we can do that separately later.

[ziocleto]

Hi, all of a sudden I had the same problem, I was doing some work in a Linux environment and I was hit by the lethal SSL certificate madness. I've followed every possible step of installing certifi from pip, pip3, making sure all the pythons have the correct 2048 root cert but nothing...

Still broked.

Until...

I've changed the script to include this magic line:

from:
urlopen(url)
to:
urlopen(url, context=ssl.create_default_context(cafile=certifi.where()))

Now I can run

./emsdk install latest

again! Now is that magic create_default_context a proper fix? I have no idea what that is! 😄

[sbc100]

Can you tell me how you reproduced this on linux? Which version linux and which version of python were you able to reproduce this on?

[ziocleto]

Can you tell me how you reproduced this on linux? Which version linux and which version of python were you able to reproduce this on?

Sure, I have been working on Linux Mint 19.3, kernel 5.3, this is the complete machine info:

linux SSL madness machine

I'm not familiar with python but I've been doing all sorts of work in every possible computer language so I ended up with, strangely, and I don't know why, a python3.7 python3.6, python3.6m, python2.7, python2, python all installed at the same time. I've ran the pip install certifi for every python, pip and pip3. I've checked manually and the certificates are all there for every version.

It was after a lot of poking around with all sort of other devs that I came up with the deaded SSL MADNESS, so unfortunately I don't have a dockerized example to reproduce it.

The thing I've noticed from that log in the link it's that it says "unknown python3.6 version".

Just to recap I had the error following step by step the instruction guidelines provided here:

Emscripten install instructions