Use libcurl rather than curl command line

[BillyONeal]

Restores #1856
Replaces #1860

Internal demonstration build: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=13191343&view=logs&j=4193bbc9-f3be-5e45-5485-eeda0c4f14df

Adopts libcurl for all http transactions rather than shelling out to the curl command line on some platforms, or winhttp sometimes.

• Windows:
  • libcurl is built with (a previous version) of vcpkg; typically expected to be the copy bundled with Visual Studio. This allows turning on features like brotli decoding since vcpkg takes care of resolving all those dependencies for us.
• macOS:
  • Use the copy of libcurl that comes with the operating system.
• Linux:
  • Download headers for libcurl, but make all function references using dlopen/dlsym. Our telemetry showed approximately 1% of Linux customers unable to find a libcurl4.so; however we observed that some very old Debian derivatives have the curl command line tool linking with libcurl4-gnutls.so instead, so we'll try that one too.
  • The version of libcurl headers used on amd64 is 7.29 because that's the version RHEL7 ships with. That's much older than the oldest Linux we support, so using headers that old should make things work out.
  • On arm64 we needed a more recent version of curl headers as 7.29 won't compile on arm64 without patches. The option VCPKG_LIBCURL_DLSYM_UPDATED_HEADERS tells us to use updated headers.
  • From source Linux customers can pass -DVCPKG_LIBCURL_DLSYM=OFF to build with a system-provided copy of curl (like macOS) instead.

Because this opens the can of worms to build vcpkg on vcpkg we may want to consider moving the other two dependencies (cmrc and fmt) to use that as well but I haven't done that here.

Other changes from #1856

• Several curl APIs used there are not present in the 7.29 headers, so I had to do several polyfills:
  • CURL_SSLVERSION_TLSv1_0 and friends are polyfilled with their values taken from a later version of curl
  • curl_multi_poll is replaced with curl_multi_wait at the dlopen/dlsym stage if the curl is old enough to not have _wait. That's a suboptimal perf situation but customers who care can fix it themselves by updating their libcurl4.so
  • CURLOPT_HEADEROPT / CURLHEADER_SEPARATE don't exist if the copy of curl is too old; those values are just cast from ints. Customers who have curl that old may get extra headers sent to their proxies; they can solve this by updating their copy of curl.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/vcpkg/base/downloads.cpp:464

• The switch statement in check_combine_download_prognosis is missing a case for DownloadPrognosis::TransientNetworkError. This enum value was added in line 440 but there's no corresponding case in the switch statement that handles it. This means if individual_call is TransientNetworkError, the function will hit the default case and call Checks::unreachable, which will terminate the program unexpectedly. You should add a case for TransientNetworkError that determines how to combine this state with the target prognosis.

    static bool check_combine_download_prognosis(DownloadPrognosis& target, DownloadPrognosis individual_call)
    {
        switch (individual_call)
        {
            case DownloadPrognosis::Success: return true;
            case DownloadPrognosis::OtherError:
                if (target == DownloadPrognosis::Success)
                {
                    target = DownloadPrognosis::OtherError;
                }

                return false;
            case DownloadPrognosis::NetworkErrorProxyMightHelp:
                if (target == DownloadPrognosis::Success || target == DownloadPrognosis::OtherError)
                {
                    target = DownloadPrognosis::NetworkErrorProxyMightHelp;
                }

                return false;
            default: Checks::unreachable(VCPKG_LINE_INFO);
        }
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[vicroms]

LGTM 🚢

[vicroms]

Retry on FTP 4xx error codes

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.