Add triplet option VCPKG_DISABLE_COMPILER_TRACKING

[klalumiere]

Describe the pull request

• What does your PR fix?

  Implement tool changes required for [vcpkg] Add triplet option VCPKG_NO_COMPILER_TRACKING vcpkg#17526
  Fixes [vcpkg] Possibility to control the compiler hash for binary caching vcpkg#16631

TL;DR: Implements @ras0219-msft suggestion in the pull request microsoft/vcpkg#16632

Alternatively, we could expose (effectively) --feature-flags=-compilertracking as a triplet setting, which would omit the detect compiler step entirely and rely on the hash of the triplet file + toolchain to distinguish package ABI. In this case, the user could add a comment to their triplet file containing any information they want to distinguish (such as compiler version). I think this is a more promising approach because it has the least interface churn.

[julianxhokaxhiu]

What's the status of this PR? Currently looking for exactly this feature as on Azure DevOps I'm cluttering the nuget feed with thousands of very same artifacts builds, despite the manifest never changed ( I'm using fixed version and port-version syntax ).

Thank you in advance.

[klalumiere]

@julianxhokaxhiu

What's the status of this PR?

Oups, sorry I tend to leave this PR on the back burner. 😳 One of the reason I do this is that for now, we set the environment variable VCPKG_FEATURE_FLAGS to "-compilertracking,manifests,registries,versions" in order to turn compiler tracking off. The reason I made this PR is actually because I suspected that the feature flag -compilertracking might disappear one day since this is a feature flag.

I've implemented the suggestions and the PR is ready for a second round of review. Hopefully it's also ready to be merged 🙂.

[ras0219-msft]

This looks really good to me!

However, testing it is challenging at the moment since we have the behavior split across both repos. I'd like to change this so that the source for scripts/vcpkg_get_tags.cmake is actually embedded in the binary. I'll do that in this PR so that I can independently confirm the behavior, then merge this.

[ras0219-msft]

While testing, I found that the compiler was still being analyzed even if given this flag. I've fixed the implementation to no longer perform any compiler tracking when this has been set.

Additionally, it works with per-port customization:

if(PORT STREQUAL "openssl")
    set(VCPKG_DISABLE_COMPILER_TRACKING 1)
endif()

So if you have C libraries that don't care about having the latest-and-greatest, you can disable them on a per-port basis.

Finally, I added testing :)

[Be-ing]

Woohoo! Thank you! I'll give vcpkg's binary caching feature another try. Hopefully Mixxx can stop relying on vcpkg archive and downloading 1 GB ZIP files now.

[Be-ing]

Oh wait, this hasn't been merged yet... does it require more review?

[julianxhokaxhiu]

When can we expect a new release that includes this patch? Thank you for merging btw!

[ras0219-msft]

Expect an average of 1 week -- we've been publishing ~once per two weeks.

[Be-ing]

Is this available in the vcpkg master branch yet?

[julianxhokaxhiu]

Is this available in the vcpkg master branch yet?

if you mean https://github.com/microsoft/vcpkg then no, but that repo will fetch the binary ( using bootstrap-vcpkg.sh/bat ) from this one. Unfortunately though there was yet no new release since this PR was merged.

Until then, you can declare this environment variable as a workaround, before running vcpkg on top of the latest current available release:

VCPKG_FEATURE_FLAGS=-compilertracking

[Be-ing]

VCPKG_FEATURE_FLAGS=-compilertracking

Unfortunately this is not sufficient. Packages built on GitHub Actions are still not matching the cache hashes in local builds. I suppose there is some other input to the hash that needs to be optionally disabled as well...

ping @emabrey

[autoantwort]

It is now released and you can use it since microsoft/vcpkg#20105

[emabrey]

VCPKG_FEATURE_FLAGS=-compilertracking

Unfortunately this is not sufficient. Packages built on GitHub Actions are still not matching the cache hashes in local builds. I suppose there is some other input to the hash that needs to be optionally disabled as well...

ping @emabrey

I mean we don't know why it doesn't match. Is there some reason that even with compiler tracking off the hash wouldn't match between dependencies built with Visual Studio 2019 Enterprise vs Visual Studio 2019 Community?

[klalumiere]

@emabrey

I mean we don't know why it doesn't match. Is there some reason that even with compiler tracking off the hash wouldn't match between dependencies built with Visual Studio 2019 Enterprise vs Visual Studio 2019 Community?

In the share installed directory, you can find vcpkg_abi_info.txt files, for instance installed/x64-osx/share/zlib/vcpkg_abi_info.txt files. This file is what is hashed. Hence, any change in this file (line ending included if I remember well (LF vs CRLF)) will change the final hash. It means that you can use these files to figure out why did the final hash changed. To continue with my example, my file contains, for an old version of vcpkg,

0001-Prevent-invalid-inclusions-when-HAVE_-is-set-to-0.patch 45ea73be70815c5520877f347f0b9af7804d9e65
CONTROL 196dc885ee80e1c31c2ddf996f3e1615005895de
LICENSE e23c6e5f40647ee2f817439d3858531fb0753c9d
add_debug_postfix_on_mingw.patch 5116458a27323c83389ea15ddf12156e8f63e0ec
cmake 3.19.2
cmake_dont_build_more_than_needed.patch 380b131349bf645f05637612e4a9a04ed8ba6349
features core
portfile.cmake f6bb80c53f45830cb9f72c5d6b27cad5bf154eba
post_build_checks 2
triplet 59a71dd77f4b3cd6ddb6d962f523c06a3379e50c-94be8c046f9e0595c199a36690d288f70945643b-634fd98ff6200ef41082d9e6793ed075532e79e3
usage b446f7466dead962527355c13bbb617213098938
vcpkg_configure_cmake 9df1c6098c9f7f00ed45e5daa2e5c047d9e047a8
vcpkg_copy_pdbs 77501408e9111fbc567621d90d587244ee4ccb3b
vcpkg_download_distfile 9e59f2d7ac7193568b301812a84c7129d25c952f
vcpkg_extract_source_archive d800c9f9fc4df8eeeabc5acc8c8a337bafc9ab6d
vcpkg_extract_source_archive_ex cc8f8ace5195315de748ddf9898f6469843b8c27
vcpkg_fixup_pkgconfig 136ac43b23096add4709daf756a7a52b75b9ff52
vcpkg_install_cmake 0ca42e314ef99ccbc75ace6664a7934f396ab391
vcpkg_replace_string af7320f3c2a1a02077549eddfc48a584bb3fbc24

As you see, the scripts file used are hashed, for instance vcpkg_replace_string.cmake. It means that if any of these scripts are updated when you do a git pull on the vcpkg repository, the final hash will change. The same goes for CMake version: it's usually control by vcpkg, not by your machine. Hence, a git pull on vcpkg might update CMake which might trigger a rebuild on everything.

On vcpkg roadmap, there is Ship vcpkg binary in Visual Studio. I thought that it means that most (if not all) of vcpkg's scripts were going to be moved to helper ports like vcpkg-cmake-config. The advantage is that these helper ports can be versioned, so you're confident that machine with slightly different version of vcpkg will build the same binaries. However, a while ago, I browsed vcpkg's issues and I'm not sure anymore that moving every scripts to helper port is a goal.