[curl] http3 (with ngtcp2) non experimental feature#46459
[curl] http3 (with ngtcp2) non experimental feature#46459BillyONeal merged 2 commits intomicrosoft:masterfrom
Conversation
f3b02ef to
57c4afd
Compare
65ccdec to
0d3a8e2
Compare
|
This is different compilation of openssl from the original PR. |
c4538e0 to
cd10ec6
Compare
|
I sent a msg to curl Debian maintainers about http3 feature. |
7138ecb to
1b16a27
Compare
| vcpkg_check_features(OUT_FEATURE_OPTIONS FEATURE_OPTIONS | ||
| FEATURES | ||
| http2 USE_NGHTTP2 | ||
| http3 USE_NGTCP2 |
There was a problem hiding this comment.
Shouln't http3 also be wired to control the use of nghttp3?
There was a problem hiding this comment.
It adding nghttp3 inside curl cmake automatic.
I don't want it will be confusion with openssl + nghttp3 the experimental way for http3 with openssl.
There was a problem hiding this comment.
"automatic" without context doesn't answer the question.
I looked it up now: USE_NGTCP2 is (almost*) sufficient to control the use of the nghttp3 dependency (USE_QUICHE left aside):
https://github.com/curl/curl/blob/cfbfb65047e85e6b08af65fe9cdbcf68e9ad496a/CMakeLists.txt#L1087-L1131
*) There is the multi-ssl problem again: curl could support this with wolfssl, but the port won't allow that combination - feature "http3" can depend only on one particlar SSL option.
There was a problem hiding this comment.
Given that USE_NGHTTP3 does not appear to be an input I'm happy with this as @talregev has written
| curl[core,http3,gnutls]=options | ||
| curl[core,http3,mbedtls]=options | ||
| curl[core,http3,schannel]=options | ||
| curl[core,http3,wolfssl]=options |
There was a problem hiding this comment.
Is this meant to enforce single-backend configuration in feature testing?
AFAICS nghttp3+ngtcp2 would also work with gnutls and wolfssl (and the "fork family").
There was a problem hiding this comment.
You cannot have multi ssl when you do http3 in curl. You have to choose one backend.
You can see my previous PR that I give alternative options for http3:
http3-gnutls, http3-wolfssl. but they exclusive to each other.
#40219
You can also see Daniel comment:
curl/curl#14308 (comment)
https://github.com/curl/curl/blob/master/docs/INSTALL.md?plain=1#L155
There was a problem hiding this comment.
You cannot have multi ssl when you do http3 in curl. You have to choose one backend.
So the original barrier for adding http3 to this port is not resolved.
There was a problem hiding this comment.
You cannot have multi ssl when you do http3 in curl. You have to choose one backend.
So the original barrier for adding http3 to this port is not resolved.
The barrieries was if the http3 + openssl will be non experimental, and also that other distributions is going to adapt it.
So now I propose http3 + ngtp2 + openssl the non experimental way, also debian is going to adapt it, and ubuntu will follow.
look on these comments:
#37146 (comment)
#37146 (comment)
There was a problem hiding this comment.
I believe the expected outcome from #37146 was that we find it likely that http3 is more useful to more people than multissl, and that when we added http3 the gnutls, mbedtls, etc. backends would be removed.
The things that matter are:
- The out of the box experience vcpkg customers get from
vcpkg install curlneeds to produce a libcurl that can talk to TLS servers. OpenSSL is the defacto backend on *nix, schannel is the defacto backend on Windows, and Secure Transport is the defacto backend on macOS. Choosing a backend that is not the defacto one is likely to result in a libcurl that can't talk to TLS servers due to missing root certificates. Of course, Apple has broken this by removing Secure Transport for macOS, but they also provide a system copy of libcurl users can use.... too bad we don't have a great way of exposing that difference in vcpkg.... - If we are going to, in effect, remove the lesser used TLS backends from the port in favor of adding http3, that needs to be marked non-experimental by the curl project, and needs to actually work on at least one of the triplets we test for in the curated registry.
There was a problem hiding this comment.
After some discussion yesterday we've decided to allow this limited exception to https://learn.microsoft.com/vcpkg/contributing/maintainer-guide#do-not-use-features-to-implement-alternatives because:
- The features that would create alternative issues are already here. If we were adding curl as a new port today, we would forbid them.
- Removing them may break customers who are already using those other features due to a conflict with the http3 feature they are likely to not care about.
- We believe http3 support is important and likely to be more important than the non-default TLS backends.
The big differences between now and the last time we considered this is:
- http3 implemented in terms of one of our canonical default TLS backends is no longer considered experimental by the curl project: http3 on ngtcp2 on openssl appears to be the path being taken here.
- It was our understanding last time that we would have had to forbid schannel and Secure Transport in order to allow http3 which would break the out of the box experience on Windows and macOS, respectively, due to missing root certificates. However, as @MichalPetryka explained below, getting at the platform's root certificates with the OpenSSL TLS implementation is possible which we didn't know. And curl themselves broke Secure Transport because macOS broke Secure Transport.
| curl[core,http3,gnutls]=options | ||
| curl[core,http3,mbedtls]=options | ||
| curl[core,http3,schannel]=options | ||
| curl[core,http3,wolfssl]=options |
There was a problem hiding this comment.
I believe the expected outcome from #37146 was that we find it likely that http3 is more useful to more people than multissl, and that when we added http3 the gnutls, mbedtls, etc. backends would be removed.
The things that matter are:
- The out of the box experience vcpkg customers get from
vcpkg install curlneeds to produce a libcurl that can talk to TLS servers. OpenSSL is the defacto backend on *nix, schannel is the defacto backend on Windows, and Secure Transport is the defacto backend on macOS. Choosing a backend that is not the defacto one is likely to result in a libcurl that can't talk to TLS servers due to missing root certificates. Of course, Apple has broken this by removing Secure Transport for macOS, but they also provide a system copy of libcurl users can use.... too bad we don't have a great way of exposing that difference in vcpkg.... - If we are going to, in effect, remove the lesser used TLS backends from the port in favor of adding http3, that needs to be marked non-experimental by the curl project, and needs to actually work on at least one of the triplets we test for in the curated registry.
| }, | ||
| "http3": { | ||
| "description": "Enable http3 with openssl backend", | ||
| "supports": "(uwp | !windows) & !mingw", |
There was a problem hiding this comment.
Wait, how does it work for UWP?!!?
There was a problem hiding this comment.
it always has.
and Secure Transport is the defacto backend on macOS.
Curl remove the support in Secure Transport. So now in vcpkg curl the openssl is defacto backend on all platforms except for windows or mingw.
|
Just confirming that no schannel on Windows means no TLS: C:\Dev\test>ninja -C build
ninja: Entering directory `build'
[0/1] Re-running CMake...-- Running vcpkg install
Detecting compiler hash for triplet x64-windows...
Compiler found: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.44.35207/bin/Hostx64/x64/cl.exe
Detecting compiler hash for triplet x64-windows-static...
Compiler found: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.44.35207/bin/Hostx64/x64/cl.exe
The following packages are already installed:
* vcpkg-cmake:x64-windows@2024-04-23
* vcpkg-cmake-config:x64-windows@2024-05-23
* zlib:x64-windows-static@1.3.1
The following packages will be rebuilt:
curl[core,openssl]:x64-windows-static@8.15.0#1
The following packages will be built and installed:
* openssl:x64-windows-static@3.5.1
* vcpkg-cmake-get-vars:x64-windows@2025-05-29
Additional packages (*) will be modified to complete this operation.
Restored 2 package(s) from C:\Users\bion\AppData\Local\vcpkg\archives in 759 ms. Use --debug to see more details.
Removing 1/4 curl:x64-windows-static
Elapsed time to handle curl:x64-windows-static: 20.2 ms
Installing 2/4 vcpkg-cmake-get-vars:x64-windows@2025-05-29...
Elapsed time to handle vcpkg-cmake-get-vars:x64-windows: 9.49 ms
vcpkg-cmake-get-vars:x64-windows package ABI: 929a3dceafe103b4e83cf87ffa8e8d902cb0c7214d27a11b7430ee690a04d1cf
Installing 3/4 openssl:x64-windows-static@3.5.1...
Elapsed time to handle openssl:x64-windows-static: 86.4 ms
openssl:x64-windows-static package ABI: 20db3173dc6f2473d486d5101997b9c11e9cfb1ccd5f4534bbd403ee9bd0e1f6
Installing 4/4 curl[core,openssl]:x64-windows-static@8.15.0#1...
Building curl[core,openssl]:x64-windows-static@8.15.0#1...
-- Using cached curl-curl-curl-8_15_0.tar.gz
-- Cleaning sources at C:/Dev/vcpkg/buildtrees/curl/src/url-8_15_0-8a95464ca7.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source C:/Dev/vcpkg-downloads/curl-curl-curl-8_15_0.tar.gz
-- Applying patch dependencies.patch
-- Applying patch pkgconfig-curl-config.patch
-- Using source at C:/Dev/vcpkg/buildtrees/curl/src/url-8_15_0-8a95464ca7.clean
-- Using cached msys2-mingw-w64-x86_64-pkgconf-1~2.4.3-1-any.pkg.tar.zst
-- Using cached msys2-msys2-runtime-3.6.2-2-x86_64.pkg.tar.zst
-- Using msys root at C:/Dev/vcpkg-downloads/tools/msys2/9272adbcaf19caef
-- Found external ninja('1.12.1').
-- Configuring x64-windows-static
-- Building x64-windows-static-dbg
-- Building x64-windows-static-rel
-- Fixing pkgconfig file: C:/Dev/vcpkg/packages/curl_x64-windows-static/lib/pkgconfig/libcurl.pc
-- Fixing pkgconfig file: C:/Dev/vcpkg/packages/curl_x64-windows-static/debug/lib/pkgconfig/libcurl.pc
-- Installing: C:/Dev/vcpkg/packages/curl_x64-windows-static/share/curl/vcpkg-cmake-wrapper.cmake
-- Installing: C:/Dev/vcpkg/packages/curl_x64-windows-static/share/curl/usage
-- Performing post-build validation
Starting submission of curl[core,openssl]:x64-windows-static@8.15.0#1 to 1 binary cache(s) in the background
Elapsed time to handle curl:x64-windows-static: 23 s
curl:x64-windows-static package ABI: cec329afe0220f390ead87af32cb423c3d880dba13c2c818edbfa97ec8b3ff47
Installed contents are licensed to you by owners. Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Packages installed in this vcpkg installation declare the following licenses:
Apache-2.0
BSD-3-Clause
ISC
MIT
curl
curl is compatible with built-in CMake targets:
find_package(CURL REQUIRED)
target_link_libraries(main PRIVATE CURL::libcurl)
Waiting for 1 remaining binary cache submissions...
Completed submission of curl[core,openssl]:x64-windows-static@8.15.0#1 to 1 binary cache(s) in 858 ms (1/1)
All requested installations completed successfully in: 24 s
-- Running vcpkg install - done
-- Found OpenSSL: optimized;C:/Dev/test/build/vcpkg_installed/x64-windows-static/lib/libcrypto.lib;debug;C:/Dev/test/build/vcpkg_installed/x64-windows-static/debug/lib/libcrypto.lib (found suitable version "3.5.1", minimum required is "3")
-- Configuring done (30.4s)
-- Generating done (0.0s)
-- Build files have been written to: C:/Dev/test/build
[2/3] Linking CXX executable curl_hello_world.exe
C:\Dev\test>.\build\curl_hello_world.exe
Hello World! Downloading https://nghttp2.org/ using curl...
Making request to https://nghttp2.org/...
curl_easy_perform() failed: SSL peer certificate or SSH remote key was not OK
C:\Dev\test> |
|
I guess we could just always pick schannel on Windows and always pick openssl on everything else which would work and meet the requirements. Going to ask other maintainers about it. |
You need to pass |
cf656a9 to
6386063
Compare
@BillyONeal I understand the problem thanks to your hint. |
BillyONeal
left a comment
There was a problem hiding this comment.
Thanks for the feature! It's been a long time coming....
| ctemplate:x64-linux=fail | ||
| ctemplate:x64-osx=fail | ||
| cuda:x64-osx=fail | ||
| curl[core,http3,gnutls]=options |
There was a problem hiding this comment.
Note for other reviewers: This has the effect of testing http3 and no longer testing the other TLS backends in our 'combined' feature test.
There was a problem hiding this comment.
We do test the other tls (multissl) in the vcpkg-ci-curl.
| "name": "ngtcp2", | ||
| "default-features": false, |
There was a problem hiding this comment.
Is it even possible to set default-features to false for other dependencies than "self" (i.e. curl trying to set ngtcp2 to non-default)? I read somewhere in the issue tracker that this can only be explicitly set by the root project and otherwise has no effect. My Gradle build script wrapper around vcpkg at least complains about this now. Or is my check too strict?
There was a problem hiding this comment.
Having false here is the necessary condition for having control in the top-level manifest / install command.
Otherwise this feature would explicitly depend on ngtcp2's default features, and then the user couldn't ask for less.
|
@BillyONeal Thank you for your review and your changes. Thank you again for your time and idea for the review. |
Yes, that's the combination built in curl-for-win for a while now. It's using As for building with native CA enabled by default for the curl tool, I plan |
|
@BillyONeal @vicroms @vszakats Thank you for the merge, and also thank you all in this PR that support for this. |
I asking these because when you compile curl with nghttp2 or nghttp3 or ngtcp2 in static windows compilation you need to add these symbols to the compilations:
Thank you! that will be wonderful. I hope in your future PR the maintainers can comments too, and explain the need in here in vcpkg. |
I configure manually, and set these dependency
Seeing your reasons will be helpful, thanks! |
Usage: - cmake: `-DCURL_CA_NATIVE_BY_DEFAULT=ON` - autotools: `--enable-ca-native-by-default` - CPPFLAGS: `-DCURL_CA_NATIVE_BY_DEFAULT` If enabled a built-time, the native CA be disabled at runtime with the `--no-ca-native` command-line option. I'm opening again, because: - there is repeat and active interest in this option. - it helps integrating curl with Windows for those who need this. - it frees applications/user from the task of securely distributing, and keeping up-to-date, a CA bundle. - it frees potentially many curl tool users from configuring a CA bundle manually to access HTTPS URLs. This is traditionally difficult on Windows because there is no concept of a protected, non-world-writable directory to securely store a CA bundle. Ref: curl#16181 (previous attempt) Ref: curl#9348 Ref: microsoft/vcpkg#46459 (comment)
|
Native CA build option PR attempt → curl/curl#18279 |
You write a wrong link to your PR: |
Ah, my bad, thanks for the correction. Should be fixed now. |
Usage: - cmake: `-DCURL_CA_NATIVE_BY_DEFAULT=ON` - autotools: `--enable-ca-native-by-default` - CPPFLAGS: `-DCURL_CA_NATIVE_BY_DEFAULT` If enabled a built-time, the native CA be disabled at runtime with the `--no-ca-native` command-line option. I'm opening again, because: - there is repeat and active interest in this option. - it helps integrating curl with Windows for those who need this. - it frees applications/user from the task of securely distributing, and keeping up-to-date, a CA bundle. - it frees potentially many curl tool users from configuring a CA bundle manually to access HTTPS URLs. This is traditionally difficult on Windows because there is no concept of a protected, non-world-writable directory to securely store a CA bundle. Ref: curl#16181 (previous attempt) Ref: curl#9348 Ref: microsoft/vcpkg#46459 (comment)
Usage: - cmake: `-DCURL_CA_NATIVE_BY_DEFAULT=ON` - autotools: `--enable-ca-native-by-default` - CPPFLAGS: `-DCURL_CA_NATIVE_BY_DEFAULT` If enabled a built-time, the native CA be disabled at runtime with the `--no-ca-native` command-line option. I'm opening again, because: - there is repeat and active interest in this option. - it helps integrating curl with Windows for those who need this. - it frees applications/user from the task of securely distributing, and keeping up-to-date, a CA bundle. - it frees potentially many curl tool users from configuring a CA bundle manually to access HTTPS URLs. This is traditionally difficult on Windows because there is no concept of a protected, non-world-writable directory to securely store a CA bundle. Ref: curl#16181 (previous attempt) Ref: curl#9348 Ref: microsoft/vcpkg#46459 (comment)
Usage: - cmake: `-DCURL_CA_NATIVE_BY_DEFAULT=ON` - autotools: `--enable-ca-native-by-default` - CPPFLAGS: `-DCURL_CA_NATIVE_BY_DEFAULT` If enabled a built-time, the native CA be disabled at runtime with the `--no-ca-native` command-line option. I'm opening again, because: - there is repeat and active interest in this option. - it helps integrating curl with Windows for those who need this. - it frees applications/user from the task of securely distributing, and keeping up-to-date, a CA bundle. - it frees potentially many curl tool users from configuring a CA bundle manually to access HTTPS URLs. This is traditionally difficult on Windows because there is no concept of a protected, non-world-writable directory to securely store a CA bundle. Ref: curl#16181 (previous attempt) Ref: curl#9348 Ref: microsoft/vcpkg#46459 (comment)
Usage: - cmake: `-DCURL_CA_NATIVE_BY_DEFAULT=ON` - autotools: `--enable-ca-native-by-default` - CPPFLAGS: `-DCURL_CA_NATIVE_BY_DEFAULT` If enabled a built-time, the native CA be disabled at runtime with the `--no-ca-native` command-line option. I'm opening again, because: - there is repeat and active interest in this option. - it helps integrating curl with Windows for those who need this. - it frees applications/user from the task of securely distributing, and keeping up-to-date, a CA bundle. - it frees potentially many curl tool users from configuring a CA bundle manually to access HTTPS URLs. This is traditionally difficult on Windows because there is no concept of a protected, non-world-writable directory to securely store a CA bundle. Ref: curl#16181 (previous attempt) Ref: curl#9348 Ref: microsoft/vcpkg#46459 (comment)
Usage: - cmake: `-DCURL_CA_NATIVE_BY_DEFAULT=ON` - autotools: `--enable-ca-native-by-default` - CPPFLAGS: `-DCURL_CA_NATIVE_BY_DEFAULT` If enabled a built-time, the native CA be disabled at runtime with the `--no-ca-native` command-line option. I'm opening again, because: - there is repeat and active interest in this option. - it helps integrating curl with Windows for those who need this. - it frees applications/user from the task of securely distributing, and keeping up-to-date, a CA bundle. - it frees potentially many curl tool users from configuring a CA bundle manually to access HTTPS URLs. This is traditionally difficult on Windows because there is no concept of a protected, non-world-writable directory to securely store a CA bundle. Ref: curl#16181 (previous attempt) Ref: curl#9348 Ref: microsoft/vcpkg#46459 (comment)
Usage: - cmake: `-DCURL_CA_NATIVE_BY_DEFAULT=ON` - autotools: `--enable-ca-native-by-default` - CPPFLAGS: `-DCURL_CA_NATIVE_BY_DEFAULT` If enabled a built-time, the native CA be disabled at runtime with the `--no-ca-native` command-line option. I'm opening again, because: - there is repeat and active interest in this option. - it helps integrating curl with Windows for those who need this. - it frees applications/user from the task of securely distributing, and keeping up-to-date, a CA bundle. - it frees potentially many curl tool users from configuring a CA bundle manually to access HTTPS URLs. This is traditionally difficult on Windows because there is no concept of a protected, non-world-writable directory to securely store a CA bundle. Ref: curl#16181 (previous attempt) Ref: curl#9348 Ref: microsoft/vcpkg#46459 (comment)
With the same semantics as Apple SecTrust, in both libcurl and the curl tool, when using non-Schannel TLS backends. In practice it means that it makes TLS work without manually or implicitly configuring a CA bundle `.crt` file, such as `curl-ca-bundle.crt`. To enable: - autotools: `--enable-ca-native` - cmake: `-DCURL_CA_NATIVE=ON` - CPPFLAGS: `-DCURL_CA_NATIVE` When enabled: - enables `CURLSSLOPT_NATIVE_CA` (libcurl) / `--ca-native` and `--proxy-ca-native` (curl tool) options by default. - unsafe search for an on-disk CA bundle gets disabled by default. Equivalent to `--disable-ca-search` with autotools, `-DCURL_DISABLE_CA_SEARCH=ON` with CMake. - build-time detection of CA bundle and CA path gets disabled. As with Apple SecTrust. This was already the default for Windows. - native CA can be disabled at run-time with the `--no-ca-native` and/or `--no-proxy-ca-native` command-line options. Rationale: This build option: - has a repeat and active interest from packagers and users. - helps integrating curl with Windows for those who need this. - it also applies to macOS: #17525 Shipped in curl 8.17.0. - makes it trivial to use custom certs configured on the OS. - frees applications/packagers/users from the task of securely distributing, and keeping up-to-date, a CA bundle. - frees potentially many curl tool from configuring a CA bundle manually to access HTTPS (and other TLS) URLs. This is traditionally difficult on Windows because there is no concept of a universal, protected, non-world-writable, location on the file system to securely store a CA bundle. - allows using modern features regardless of Windows version. Some of these features are not supported with Schannel (e.g. HTTP/3, ECH) on any Windows version. - is necessary for HTTP/3 builds, where bootstrapping a CA bundle is not possible with Schannel, because MultiSSL is not an option, and HTTP/3 is not supported with Schannel. Ref: #16181 (previous attempt) Ref: #9348 Ref: #9350 Ref: #13111 Ref: microsoft/vcpkg#46459 (comment) Ref: 22652a5 #14582 Ref: eefd03c #18703 Closes #18279

./vcpkg x-add-version --alland committing the result.This is the continue of this PR:
#37146