[testing ci] add tests for unicode tarballs, fix utf-8 on macOS CI

[strega-nil]

see the errors in #19963

[BillyONeal]

I think this belongs as an e2e test rather than here?

[strega-nil-ms]

@BillyONeal it's testing vcpkg_extract_source_archive, so having it as a port test is fine, no?

[BillyONeal]

@BillyONeal it's testing vcpkg_extract_source_archive, so having it as a port test is fine, no?

Kinda related to #19034 where I made the same comment. As far as I know this is testing scripts/cmake meaning it logically belongs in the tool repo.

[strega-nil-ms]

@BillyONeal At that point, we should probably pull out the test_ports directory, but until then the scripts still live in vcpkg, right?

[dg0yt]

At that point, we should probably pull out the test_ports directory, but until then the scripts still live in vcpkg, right?

Note the table I posted in #19034 (comment). IMO key question is what triggers the test. If scripts are moved out of vcpkg, #20060 can be moved out, too. But some tests in test_ports are really meant to be triggered by changes to ports (i.e. to vcpkg repo), in order to ensure usability of the ports or selected non-default feature combinations.

[BillyONeal]

It seems like maybe vcpkg itself should be setting this rather than hiding it in CI? It would be in keeping with other environment variable handling we do....

[strega-nil-ms]

I don't know how to set this variable in general; I don't even know if it's possible. Perhaps we could do some locale parsing; however, for CI, it makes sense that we set it to en_US.UTF-8.

[BillyONeal]

I think the correct fix here is to have vcpkg set the locale by testing a few common ones like:

• C.UTF-8
• POSIX.UTF-8
• en_US.UTF-8

and choosing the first matching one.

Notably, this change plus #19963 will break existing macos customers, most of whom aren't passing LC_ALL like this.

[JackBoosY]

So, should we wait for the next vcpkg release?

[BillyONeal]

So, should we wait for the next vcpkg release?

It looks like the current tool should have the fix:

[strega-nil-ms]

Closing in favor of the next tool