Use SecureTransport on macOS

[dstufft]

The OpenSSL available on macOS is really old, so old in fact that in the future it will no longer be able to talk to PyPI. Thanks to the great work of @Lukasa we can utilize the SecureTransport library instead of OpenSSL which is new enough to support this.

Ideally this should not change any user visible behavior, but that cannot be guaranteed. In addition this really only matters for people using a Python that is linking against the system OpenSSL library, which is /usr/bin/python and Python.org prior to 3.6. Homebrew and Python.org 3.6+ installers use their own OpenSSL which is fine.

So that means we have a few options here:

• We can just unambiguously use SecureTransport on macOS to have guaranteed consistent behavior on all Python installs regardless of what OpenSSL they link to. The downside to this is that the securetransport bindings is written in ctypes (because we can't use a C compiler) and isn't as battle tested as the OpenSSL code so it may run into more errors. Obviously there is also a risk of behavior changing between OpenSSL and SecureTransport.
• We can detect if the OpenSSL is the ancient old one shipped by macOS and only enable the securetransport library in that case. The downside here is that if there are behavior diffrences, you can get different behaviors depending on if you're using a Python that links against the system OpenSSL or not. The upside is that if you're not using one of those Pythons linked against the system OpenSSL, then this PR is effectively a no-op for you.

Thoughts @pypa/pip-committers?

Note: This is a WIP because I pulled in the branch to test it, this wouldn't actually be merged until both urllib3 and requests are released.

[Lukasa]

So as a note, you may not want to ship this slightly-bastardised Requests in a production version of pip. I'm hoping to get a version of Requests containing the relevant code out this week or next, though it naturally will not be enabled by default.

Other than that, I think you've outlined the two best choices for how to handle this on macOS. I can't help the pip team make the choice between them really. I'm just excited to see this progress!

[dstufft]

Yea, this is just a temporary thing to allow folks to test it easier on their own machines.

[dstufft]

I think I'm leaning towards only enabling this if we need to, ideally at some point macOS fixes it's system Python so that it can speak TLSv1.2 (this reason we need this) natively with the ssl library, and Python.org is switching over to bundling OpenSSL. Those two things will eventually mean we won't need to actually do this anymore, and we can just again rely on the ssl module to handle our needs.

[ncoghlan]

I'd also vote for the "only if needed" option, as that means folks will only end up running the new code in cases where the system OpenSSL would have broken entirely, and will otherwise stick to the better tested OpenSSL based implementation.

Preferring SecureTransport by default on Mac OS X would then happen whenever the standard library itself is updated to do that, rather than pip having to blaze that trail on its own.

[pfmoore]

FWIW "only if needed" seems safer to me, too. Differences in behaviour shouldn't be an issue - after all, we already have a difference in those cases, it's just that it's currently "broken" vs "working" 😄

[dstufft]

It was reported that the securetransport code seems to be a lot slower than the previous code, one person measured (in an non-scientific benchmark) that:

$ pip install -I Django --no-cache-dir

Took 12 seconds on our latest release and this branch took 56 seconds, which I'm going to guess is due to the ctypes interface. Which further suggests using this only as added.

[dstufft]

This should be ready to go now. The slow down mentioned above is gone now (It was a bug in the bindings, which @Lukasa has fixed). Even though that has been fixed, I still went ahead and only used this in the case where we actually need to use it in order to reduce the chances this breaks something for someone.