bootstrap: enable https for bootstrap curl using wolfssl

[mcmtroffaes]

This was spurred by NixOS/hydra#257 ; also see #11221.

I used wolfssl for this patch, because it is a lot smaller than openssl so perhaps more suitable for a bootstrapping environment (only 300kb as opposed to about 3Mb for openssl), but if openssl (or libressl) is more desirable then it is fairly easy to change the patch accordingly. I've added myself as a maintainer of the wolfssl package (of course I'm happy to pass it on to someone else too).

Tested with:

NIX_PATH=nixpkgs=/path/to/my/local/nixpkgs/ nix-build /path/to/my/local/nixpkgs/pkgs/stdenv/linux/make-bootstrap-tools.nix -A test

(Note that this takes a very long time to complete, a few hours on my machine.)

I've also successfully downloaded files over https with the new curl.

A note about the changes to unpack-bootstrap-tools.sh: for the test to complete, this script has to work on both the old bootstrap package as well as on the new bootstrap package. The old one does not have the wolfssl shared library, so there's an extra check to ensure that patchelf isn't run on a file that isn't present even if it is in the list.

cc @vcunat @peti

[mention-bot]

By analyzing the blame information on this pull request, we identified @edolstra, @ambrop72 and @vcunat to be potential reviewers

[vcunat]

/cc @wkennington because of WIP with a different approach.

[vcunat]

Is the commit adding wolfssl of independent interest as well? That one could be picked to master and even 15.09 immediately.

I do like the general approach of using a lighter version during bootstrapping.

[mcmtroffaes]

@wkennington: Apologies - I wasn't aware of another approach being in the works - as I ran into this curl https issue, it seemed like a nice little exercise for me to get to know the internals of nix a bit better.

@vcunat: Yes, the wolfssl commit could be cherrypicked into master. Do you want me to set up a separate pull request for it?

[mcmtroffaes]

I now found the work of @wkennington on this. Bits of #8081 could be reused to improve this pull request. For example, the ca certificates probably needs to be copied into the bootstrap bundle - not sure why my test worked without doing so (maybe because I was testing from fedora?).

Thoughts, @wkennington?

[vcunat]

I pushed the new package.

[mcmtroffaes]

Thanks @vcunat

[mcmtroffaes]

I've rebased this onto current staging (which already picked up the wolfssl patch previously included in this pull request).

[zimbatm]

I know it's been a long time. Would you mind making another rebase ?

[mcmtroffaes]

Yes, it was no longer merging without conflicts, that's now resolved. A fresh rebase pushed against the latest upstream. I have not tested after rebase as I don't have the proper resources at this moment, I will be able to test it on Monday.

I was hoping for some feedback from @wkennington on what else is needed from #8081, in particular whether or not a similar patch needs to be applied to pkgs/stdenv/linux/make-bootstrap-tools-cross.nix. If so, then as the two files duplicate a lot, maybe we can also think of ways to reduce that duplication.

Finally, #8081 has another nice feature: copying the required libraries by inspecting the binaries directly. I think that can be handled and discussed in a separate pull request as it does not pertain to https support for curl specifically.

[globin]

We shouldn't enable SSLv3 support!

[vcunat]

• cross: I think we can leave cross-compiling bootstrap without https support for now. The duplication might be good to reduce, but let this PR not depend on that. WKennington has stopped contributing here and rolls his own fork; I don't expect him to react much anymore :-(
• SSLv3: we only do it for fetchurl which already knows the hashes of the results, and it's only for a few packages used during bootstrapping, so I think such details don't matter at all.

[edolstra]

I would rather use Nix's builtin fetchurl function. That would allow us to get rid of curl in the bootstrap altogether. I'll see if I can get that to work in the next few days.

[vcunat]

OK, closing this then.

[mcmtroffaes]

Looking forward to seeing the improved fetchurl!

For history's sake, just noting here that upstream curl has been patched in the mean time to compile against a vanilla version of wolfssl, that is without SSLv3 support and in fact also without get_peer_certificate:

curl/curl@1ff3a07#diff-d7cd75d19525a9177c7b19826b3cb84a

curl/curl@151da51#diff-d7cd75d19525a9177c7b19826b3cb84a

That will make it easier to use curl with wolfssl if ever desired in the future.