vcpkg ci: Parse ci.baseline.txt to determine regressions by autoantwort · Pull Request #345 · microsoft/vcpkg-tool

autoantwort · 2022-02-07T19:12:57Z

Probably you don't want to use iostreams. Should I add write_unlocalized_text_to_stderr to write to stderr?

This makes afaik the whole powershell parsing stuff in the ci obsolete

BillyONeal · 2022-02-09T02:36:34Z

Stop doing things on my todo list before I do them 😅

Thanks for the submission! Will take a look soon

…-passing

strega-nil-ms · 2022-03-01T19:40:27Z

https://gist.githubusercontent.com/strega-nil-ms/03628b7f6febec5ff5355be78167593d/raw/8d7aab62a8872244ea71e5fae11ecd402f36f2e1/0001-Nicole-CRs.patch

My CRs are given in the patch above ^

BillyONeal · 2022-03-11T02:12:15Z

src/vcpkg/commands.ci.cpp

+                        case Build::BuildResult::FILE_CONFLICTS:
+                            if (expected_failures.find(port_result.spec) == expected_failures.end())
+                            {
+                                std::cerr


I think this should also use print2, and we certainly should avoiding new use of iostreams if at all possible.

BillyONeal · 2022-03-11T02:12:59Z

Sorry it has taken me so long to get to this; I didn't leave comments back at the time because I thought I was going to respond quickly with the bits I wanted fixed but that didn't work out.

It looks like Nicole already commented about effectively everything I was going to say, except I would like to see a unit test put around parse_ci_baseline. At that point I would be happy to merge this and make the corresponding infrastructure changes.

If don't want to do that because you've been kept hanging too long I understand :)

BillyONeal · 2022-03-16T03:35:25Z

If don't want to do that because you've been kept hanging too long I understand :)

Are you interested in that? Otherwise I can push that into this branch.

autoantwort · 2022-03-16T19:08:20Z

Otherwise I can push that into this branch.

That would be cool. I am currently writing exams and don't have that much time :)

BillyONeal · 2022-03-16T20:14:33Z

OK I have to fix the bug where the "one liner" doesn't update correctly and this is next on my list.

* Delete namespace Parse:: * Add forwarding headers for a lot of the parse machinery. * Extract some supporting structures for parsing ci.baseline.txt into a header so that tests can be added.

# Conflicts: # include/vcpkg/base/parse.h # src/vcpkg/versions.cpp

Extract exclusions / ci baseline infrastructure.

# Conflicts: # src/vcpkg/commands.ci.cpp

BillyONeal · 2022-03-18T22:32:35Z

src/vcpkg/commands.ci.cpp

+            Checks::check_exit(VCPKG_LINE_INFO, equal_loc != std::string::npos, "Line '%s' must contain a '='", line);
+            parsed_line.triplet_name =
+                Strings::trim(StringView{line.data() + colon_loc + 1, line.data() + equal_loc}).to_string();


FYI I'm changing this behavior because it fails, for example, with

catch-classic:arm64-windows = skip catch-classic:arm-uwp = skip catch-classic:x64-linux = skip catch-classic:x64-osx = skip catch-classic:x64-uwp = skip catch-classic:x64-windows = skip catch-classic:x64-windows-static = skip catch-classic:x64-windows-static-md=skip catch-classic:x86-windows = skip

…rs_or_warnings, merge match_zero_or_more and match_until as merge_while, delocalize locale-invariant FormattedParseMessageLocation, start implementing actual parsing with ParserBase.

BillyONeal

Whoops, I forgot to get rid of cerr

include/vcpkg/base/parse.h

include/vcpkg/base/sortedvector.h

strega-nil · 2022-03-21T18:06:42Z

locales/messages.json

  "ChecksUnreachableCode": "unreachable code was reached",
  "ChecksUpdateVcpkg": "updating vcpkg by rerunning bootstrap-vcpkg may resolve this failure.",
+  "CiBaselineRegression": "REGRESSION: {spec} failed with {build_result}. If expected, add {spec}=fail to {path}.",
+  "_CiBaselineRegression.comment": "example of {spec} is 'zlib:x64-windows'.\nexample of {build_result} is 'One of the BuildResultXxx messages (such as BuildResultSucceeded/SUCCEEDED)'.\nexample of {path} is '/foo/bar'.\n",


this should be {Locked}.

I wasn't sure if it should be. The "spec=fail" part needs to be but there's no reason the rest of the message couldn't be localized.

strega-nil · 2022-03-21T18:08:02Z

src/vcpkg/binarycaching.cpp

            for (;;)
            {
-                auto n = match_until([](char32_t ch) { return ch == ',' || ch == '`' || ch == ';'; });
+                auto n = match_while([](char32_t ch) { return ch != ',' && ch != '`' && ch != ';'; });


why this change?

To merge match_until and match_zero_or_more into one function.

In microsoft#345 (comment) Nicole indicated that the workaround here to support features with _ in their names is no longer relevant because the feature in question was already renamed before versioning. Since this is a change in the usable surface of the tool I extracted the change into its own more targeted PR.

In #345 (comment) Nicole indicated that the workaround here to support features with _ in their names is no longer relevant because the feature in question was already renamed before versioning. Since this is a change in the usable surface of the tool I extracted the change into its own more targeted PR.

# Conflicts: # locales/messages.en.json # locales/messages.json # src/vcpkg/export.prefab.cpp # src/vcpkg/packagespec.cpp

Extracted from microsoft#345 to make that PR easier to review.

BillyONeal · 2022-03-22T00:46:36Z

I sucked out #447 "Delete Parse::" to make this easier to review if you want to land that one first.

* Delete Parse::. Extracted from #345 to make that PR easier to review. * ... and IParseError, as requested by Nicole

# Conflicts: # include/vcpkg-test/util.h # include/vcpkg/base/fwd/parse.h # include/vcpkg/base/json.h # include/vcpkg/base/parse.h # src/vcpkg/base/json.cpp # src/vcpkg/base/parse.cpp # src/vcpkg/packagespec.cpp

strega-nil

Approved, but I still prefer to have match_until as well.

include/vcpkg/base/sortedvector.h

strega-nil · 2022-03-23T00:18:20Z

locales/messages.json

  "ChecksUnreachableCode": "unreachable code was reached",
  "ChecksUpdateVcpkg": "updating vcpkg by rerunning bootstrap-vcpkg may resolve this failure.",
+  "CiBaselineRegression": "REGRESSION: {spec} failed with {build_result}. If expected, add {spec}=fail to {path}.",
+  "_CiBaselineRegression.comment": "example of {spec} is 'zlib:x64-windows'.\nexample of {build_result} is 'One of the BuildResultXxx messages (such as BuildResultSucceeded/SUCCEEDED)'.\nexample of {path} is '/foo/bar'.\n",


BillyONeal · 2022-03-23T00:31:22Z

Can we inherit from Compare then?

I really don't want to do that because it gives SortedVector an operator(). I can see an argument for a compressed_pair optimization in the future in order to implement this but I think that is a separate improvement and we shouldn't block over that.

Alternately, we can change the SortedVector design such that the caller always needs to pass the comparison functor every time they do something that needs it.... let me try that.

… storing it.

strega-nil

The one I want changed is try_match_keyword.

include/vcpkg/base/parse.h

include/vcpkg/base/sortedvector.h

src/vcpkg/base/parse.cpp

src/vcpkg/commands.ci.cpp

…her than storing it." This reverts commit b99de1d.

… provided via --ci-baseline error.

src/vcpkg-test/ci-baseline.cpp

BillyONeal · 2022-03-24T23:33:34Z

Thanks @autoantwort :D

@Neumann-A

* Update vcpkg-tool to 2022-03-24 * Hook up microsoft/vcpkg-tool#345 * Hook up microsoft/vcpkg-tool#442 * Update vcpkg-tool to 2022-03-25 * Analysis of failures. * [Most recent nightly build failed](https://dev.azure.com/vcpkg/public/_build/results?buildId=69427) * [Validation of this tool update failed](https://dev.azure.com/vcpkg/public/_build/results?buildId=69417) ## Common to both: PASSING, REMOVE FROM FAIL LIST: chartdir:x64-windows (.\scripts\ci.baseline.txt) PASSING, REMOVE FROM FAIL LIST: chartdir:x64-windows-static-md (.\scripts\ci.baseline.txt) PASSING, REMOVE FROM FAIL LIST: chartdir:x86-windows (.\scripts\ci.baseline.txt) Probably fixed by #23701 PASSING, REMOVE FROM FAIL LIST: gmp:x64-uwp (.\scripts\ci.baseline.txt) PASSING, REMOVE FROM FAIL LIST: gmp:x64-windows-static-md (.\scripts\ci.baseline.txt) Probably fixed by #23466 ? REGRESSION: colmap:x64-windows-static-md failed with BUILD_FAILED. If expected, add colmap:x64-windows-static-md=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. I don't know exactly what changed. I observe that * this thing depends on a *lot* of stuff * on March 14 we didn't even attempt to build this * the x64-windows ones are already in the baseline so I skipped it. REGRESSION: qtdeclarative:x64-windows. If expected, add qtdeclarative:x64-windows=fail to .\scripts\ci.baseline.txt. REGRESSION: qtdeclarative:x64-windows failed with BUILD_FAILED. If expected, add qtdeclarative:x64-windows=fail to C:\a\1\s\scripts\azure-pipelines/../ci.baseline.txt. REGRESSION: qtdeclarative:x64-windows failed with BUILD_FAILED. If expected, add qtdeclarative:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. REGRESSION: qtdeclarative:x64-windows failed with BUILD_FAILED. If expected, add qtdeclarative:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. REGRESSION: qtdeclarative:x64-windows failed with BUILD_FAILED. If expected, add qtdeclarative:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. REGRESSION: qtdeclarative:x64-windows failed with BUILD_FAILED. If expected, add qtdeclarative:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. REGRESSION: qtdeclarative:x64-windows failed with BUILD_FAILED. If expected, add qtdeclarative:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. REGRESSION: qtdeclarative:x64-windows failed with BUILD_FAILED. If expected, add qtdeclarative:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. This is a reporting change: The new world order also includes host build failures which is why it's duplicated. See also #23714 See also #23490 I'm nervous about baslining this because it seems most of the qt world is built on top of this port I filed #23824 about this and @Neumann-A indicated this should be fixed by #23755 REGRESSION: nettle:x64-uwp. If expected, add nettle:x64-uwp=fail to .\scripts\ci.baseline.txt. REGRESSION: nettle:x64-windows-static-md. If expected, add nettle:x64-windows-static-md=fail to .\scripts\ci.baseline.txt. REGRESSION: nettle:x64-uwp failed with BUILD_FAILED. If expected, add nettle:x64-uwp=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. REGRESSION: nettle:x64-windows-static-md failed with POST_BUILD_CHECKS_FAILED. If expected, add nettle:x64-windows-static-md=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. Didn't analyze, probably fixed by #23519 ? REGRESSION: libgpg-error:x64-uwp. If expected, add libgpg-error:x64-uwp=fail to .\scripts\ci.baseline.txt. REGRESSION: libgpg-error:x64-uwp failed with BUILD_FAILED. If expected, add libgpg-error:x64-uwp=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. This was broken by VS2022 update: ``` C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VisualStudio\v17.0\AppxPackage\Microsoft.AppXPackage.Targets(892,25): error MSB4086: A numeric comparison was attempted on "$(TargetPlatformMinVersion)" that evaluates to "" instead of a number, in condition "'$(TargetPlatformMinVersion)' >= '10.0.17200.0'". [C:\Dev\vcpkg\buildtrees\libgpg-error\x64-uwp-rel\error-1.42-2324ddbc71.clean\SMP\libgpg-error_winrt.vcxproj] ``` REGRESSION: libmikmod:x64-osx. If expected, add libmikmod:x64-osx=fail to .\scripts\ci.baseline.txt. REGRESSION: libmikmod:x64-osx failed with BUILD_FAILED. If expected, add libmikmod:x64-osx=fail to /Users/vagrant/Data/work/2/s/scripts/azure-pipelines/../ci.baseline.txt. Broken between [2022-03-16](https://dev.azure.com/vcpkg/public/_build/results?buildId=68947) and [2022-03-18](https://dev.azure.com/vcpkg/public/_build/results?buildId=69051). Unfortunately I don't see obvious reasons why. Nothing else depends on this and nobody has noticed in 2 weeks, so I'm baslining it for now. (Will investigate shortly...) ## Only broken in tool update: REGRESSION: mesa:x64-windows failed with BUILD_FAILED. If expected, add mesa:x64-windows=fail to C:\a\2\s\scripts\azure-pipelines/../ci.baseline.txt. ``` -- Downloading https://gitlab.freedesktop.org/mesa/mesa/-/archive/mesa-21.2.5/mesa-mesa-21.2.5.tar.gz -> mesa-mesa-mesa-21.2.5-1.tar.gz... -- Extracting source /Users/vagrant/Data/downloads/mesa-mesa-mesa-21.2.5-1.tar.gz -- Applying patch swravx512-post-static-link.patch -- Applying patch swr-msvc-2.patch -- Applying patch swr-llvm13.patch -- Applying patch radv-msvc-llvm13-2.patch -- Applying patch d3d10sw.patch -- Using source at /Users/vagrant/Data/buildtrees/mesa/src/esa-21.2.5-2df234d2b1.clean Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'mako' CMake Error at ports/mesa/portfile.cmake:85 (message): Python package 'mako' needs to be installed for port 'mesa'. Complete list of required python packages: setuptools;mako Call Stack (most recent call first): ports/mesa/portfile.cmake:91 (vcpkg_get_python_package) scripts/ports.cmake:145 (include) ``` Looks like this is being tracked by #23089 ; perhaps that we don't have as aggressive a recycling strategy for macos boxes as we do for the others has let different machines give different results? ## Only broken without tool update: REGRESSION: chromium-base:x64-osx. If expected, add chromium-base:x64-osx=fail to .\scripts\ci.baseline.txt. This one has been constantly flaky; I baselined it. REGRESSION: libxml2:x64-osx. If expected, add libxml2:x64-osx=fail to .\scripts\ci.baseline.txt. This port uses vcpkg_from_git and the upstream server was down during the build. * Restore chartdir to the baseline, I thought #23732 had been merged.

autoantwort added 2 commits February 21, 2022 14:01

vcpkg ci: Parse ci.baseline.txt to determine regressions

c17e406

Fix bug and implement PASSING, REMOVE FROM FAIL LIST and --passing-is…

a555645

…-passing

autoantwort force-pushed the parse-ci.baseline.txt branch from 4936f38 to a555645 Compare February 21, 2022 13:01

Apply Nicole's CR

fc2cbb5

autoantwort force-pushed the parse-ci.baseline.txt branch from a365c6d to fc2cbb5 Compare March 6, 2022 13:00

autoantwort added 3 commits March 6, 2022 14:06

Merge branch 'main' into parse-ci.baseline.txt

4671c2f

Fix build

7549a57

Fix Formatting

a3797dc

BillyONeal reviewed Mar 11, 2022

View reviewed changes

dg0yt mentioned this pull request Mar 15, 2022

[ci] Publish test results microsoft/vcpkg#23477

Closed

BillyONeal mentioned this pull request Mar 16, 2022

Implement published test results built in and reduce console chattiness. #437

Merged

BillyONeal added 4 commits March 17, 2022 23:56

WIP

8fe0d2c

* Delete namespace Parse:: * Add forwarding headers for a lot of the parse machinery. * Extract some supporting structures for parsing ci.baseline.txt into a header so that tests can be added.

Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt

d9da60b

# Conflicts: # include/vcpkg/base/parse.h # src/vcpkg/versions.cpp

Extend SortedVector to allow merging different SortedVectors together.

d5f0498

Extract exclusions / ci baseline infrastructure.

Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt

6aa5eaa

# Conflicts: # src/vcpkg/commands.ci.cpp

BillyONeal reviewed Mar 18, 2022

View reviewed changes

BillyONeal added 3 commits March 18, 2022 16:30

Move some parsing stuff into the cpp, add ParseMessages::exit_if_erro…

81c0ccb

…rs_or_warnings, merge match_zero_or_more and match_until as merge_while, delocalize locale-invariant FormattedParseMessageLocation, start implementing actual parsing with ParserBase.

Finish implementing parser in terms of ParserBase.

f573df3

Tests and bugfixes!

13010c2

BillyONeal approved these changes Mar 19, 2022

View reviewed changes

BillyONeal requested changes Mar 19, 2022

View reviewed changes

BillyONeal added 2 commits March 21, 2022 09:32

Fix MacOS and Linux build failures.

2bc6f62

Remove std::cerr.

f3dea2a

BillyONeal approved these changes Mar 21, 2022

View reviewed changes

Ensure all named triplets are in the exclusions map.

b089a02

strega-nil reviewed Mar 21, 2022

View reviewed changes

BillyONeal mentioned this pull request Mar 21, 2022

Block features containing _ in their names. #445

Merged

Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt

8eca723

# Conflicts: # locales/messages.en.json # locales/messages.json # src/vcpkg/export.prefab.cpp # src/vcpkg/packagespec.cpp

BillyONeal added a commit to BillyONeal/vcpkg-tool that referenced this pull request Mar 22, 2022

Delete Parse::.

898d1c6

Extracted from microsoft#345 to make that PR easier to review.

BillyONeal mentioned this pull request Mar 22, 2022

Delete Parse:: and IParseError. #447

Merged

BillyONeal added a commit that referenced this pull request Mar 22, 2022

Delete Parse:: and IParseError. (#447)

3425b09

* Delete Parse::. Extracted from #345 to make that PR easier to review. * ... and IParseError, as requested by Nicole

BillyONeal added 2 commits March 22, 2022 11:21

Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt

7f7b54e

# Conflicts: # include/vcpkg-test/util.h # include/vcpkg/base/fwd/parse.h # include/vcpkg/base/json.h # include/vcpkg/base/parse.h # src/vcpkg/base/json.cpp # src/vcpkg/base/parse.cpp # src/vcpkg/packagespec.cpp

Fix bad merge.

bfbbe6e

strega-nil approved these changes Mar 23, 2022

View reviewed changes

BillyONeal added 3 commits March 22, 2022 17:53

Require the caller to pass the predicate in sorted vector rather than…

b99de1d

… storing it.

Revert the match_until change.

e97eac1

Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt

686d7cd

strega-nil suggested changes Mar 23, 2022

View reviewed changes

include/vcpkg/base/parse.h Outdated Show resolved Hide resolved

include/vcpkg/base/sortedvector.h Outdated Show resolved Hide resolved

src/vcpkg/base/parse.cpp Outdated Show resolved Hide resolved

src/vcpkg/commands.ci.cpp Outdated Show resolved Hide resolved

BillyONeal added 4 commits March 23, 2022 12:56

Revert "Require the caller to pass the predicate in sorted vector rat…

ffa2c8b

…her than storing it." This reverts commit b99de1d.

Localize --allow-unexpected-passing can only be used if a baseline is…

a3e92af

… provided via --ci-baseline error.

Revert ref qualification of extract_Xxx functions.

463db01

Simplify try_match_keyword.

19ae76d

ras0219-msft reviewed Mar 24, 2022

View reviewed changes

src/vcpkg-test/ci-baseline.cpp Show resolved Hide resolved

BillyONeal added 2 commits March 24, 2022 13:55

Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt

c2a64a0

Add more tests requested by Robert.

319ab06

BillyONeal merged commit 2a984ce into microsoft:main Mar 24, 2022

BillyONeal added a commit to BillyONeal/vcpkg that referenced this pull request Mar 25, 2022

Hook up microsoft/vcpkg-tool#345

add0407

autoantwort deleted the parse-ci.baseline.txt branch August 9, 2022 15:31

Conversation

autoantwort commented Feb 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BillyONeal commented Feb 9, 2022

Uh oh!

strega-nil-ms commented Mar 1, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BillyONeal commented Mar 11, 2022

Uh oh!

BillyONeal commented Mar 16, 2022

Uh oh!

autoantwort commented Mar 16, 2022

Uh oh!

BillyONeal commented Mar 16, 2022

Uh oh!

BillyONeal Mar 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BillyONeal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BillyONeal commented Mar 22, 2022

Uh oh!

strega-nil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BillyONeal commented Mar 23, 2022

Uh oh!

strega-nil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BillyONeal commented Mar 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

autoantwort commented Feb 7, 2022 •

edited

Loading

BillyONeal Mar 18, 2022 •

edited

Loading