Skip to content

vcpkg ci: Parse ci.baseline.txt to determine regressions #345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BillyONeal merged 30 commits into from
Mar 24, 2022
Merged

vcpkg ci: Parse ci.baseline.txt to determine regressions #345

Changes from 6 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
c17e406
vcpkg ci: Parse ci.baseline.txt to determine regressions
autoantwort Feb 7, 2022
a555645
Fix bug and implement PASSING, REMOVE FROM FAIL LIST and --passing-is…
autoantwort Feb 11, 2022
fc2cbb5
Apply Nicole's CR
autoantwort Mar 6, 2022
4671c2f
Merge branch 'main' into parse-ci.baseline.txt
autoantwort Mar 6, 2022
7549a57
Fix build
autoantwort Mar 6, 2022
a3797dc
Fix Formatting
autoantwort Mar 6, 2022
8fe0d2c
WIP
BillyONeal Mar 18, 2022
d9da60b
Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt
BillyONeal Mar 18, 2022
d5f0498
Extend SortedVector to allow merging different SortedVectors together.
BillyONeal Mar 18, 2022
6aa5eaa
Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt
BillyONeal Mar 18, 2022
81c0ccb
Move some parsing stuff into the cpp, add ParseMessages::exit_if_erro…
BillyONeal Mar 18, 2022
f573df3
Finish implementing parser in terms of ParserBase.
BillyONeal Mar 19, 2022
13010c2
Tests and bugfixes!
BillyONeal Mar 19, 2022
2bc6f62
Fix MacOS and Linux build failures.
BillyONeal Mar 21, 2022
f3dea2a
Remove std::cerr.
BillyONeal Mar 21, 2022
1f41c11
Generate messages map
BillyONeal Mar 21, 2022
c207828
Try to fix macos and linux again.
BillyONeal Mar 21, 2022
b089a02
Ensure all named triplets are in the exclusions map.
BillyONeal Mar 21, 2022
8eca723
Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt
BillyONeal Mar 22, 2022
7f7b54e
Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt
BillyONeal Mar 22, 2022
bfbbe6e
Fix bad merge.
BillyONeal Mar 22, 2022
b99de1d
Require the caller to pass the predicate in sorted vector rather than…
BillyONeal Mar 23, 2022
e97eac1
Revert the match_until change.
BillyONeal Mar 23, 2022
686d7cd
Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt
BillyONeal Mar 23, 2022
ffa2c8b
Revert "Require the caller to pass the predicate in sorted vector rat…
BillyONeal Mar 23, 2022
a3e92af
Localize --allow-unexpected-passing can only be used if a baseline is…
BillyONeal Mar 23, 2022
463db01
Revert ref qualification of extract_Xxx functions.
BillyONeal Mar 23, 2022
19ae76d
Simplify try_match_keyword.
BillyONeal Mar 23, 2022
c2a64a0
Merge remote-tracking branch 'origin/main' into parse-ci.baseline.txt
BillyONeal Mar 24, 2022
319ab06
Add more tests requested by Robert.
BillyONeal Mar 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 139 additions & 5 deletions src/vcpkg/commands.ci.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#include <vcpkg/base/system.h>
#include <vcpkg/base/system.print.h>
#include <vcpkg/base/util.h>
#include <vcpkg/base/view.h>

#include <vcpkg/binarycaching.h>
#include <vcpkg/build.h>
Expand All @@ -25,6 +26,8 @@
#include <vcpkg/vcpkglib.h>
#include <vcpkg/vcpkgpaths.h>

#include <iostream>

using namespace vcpkg;

namespace
Expand Down Expand Up @@ -74,6 +77,53 @@ namespace
private:
Path base_path;
};

enum class CiBaselineState
{
Skip,
Fail,
};

struct CiBaselineLine
{
std::string port_name;
std::string triplet_name;
CiBaselineState state;
};

std::vector<CiBaselineLine> parse_ci_baseline(View<std::string> lines)
{
std::vector<CiBaselineLine> result;
for (auto& line : lines)
{
if (line.empty() || line[0] == '#') continue;
CiBaselineLine parsed_line;

auto colon_loc = line.find(':');
Checks::check_exit(VCPKG_LINE_INFO, colon_loc != std::string::npos, "Line '%s' must contain a ':'", line);
parsed_line.port_name = Strings::trim(StringView{line.data(), line.data() + colon_loc}).to_string();

auto equal_loc = line.find('=', colon_loc + 1);
Checks::check_exit(VCPKG_LINE_INFO, equal_loc != std::string::npos, "Line '%s' must contain a '='", line);
parsed_line.triplet_name =
Strings::trim(StringView{line.data() + colon_loc + 1, line.data() + equal_loc}).to_string();

@BillyONeal BillyONeal Mar 18, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI I'm changing this behavior because it fails, for example, with

catch-classic:arm64-windows      = skip
catch-classic:arm-uwp            = skip
catch-classic:x64-linux          = skip
catch-classic:x64-osx            = skip
catch-classic:x64-uwp            = skip
catch-classic:x64-windows        = skip
catch-classic:x64-windows-static = skip
catch-classic:x64-windows-static-md=skip
catch-classic:x86-windows        = skip


auto baseline_value = Strings::trim(StringView{line.data() + equal_loc + 1, line.data() + line.size()});
if (baseline_value == "fail")
{
parsed_line.state = CiBaselineState::Fail;
}
else if (baseline_value == "skip")
{
parsed_line.state = CiBaselineState::Skip;
}
else
{
Checks::exit_with_message(VCPKG_LINE_INFO, "Unknown value '%s'", baseline_value);
}
}
return result;
}
}

namespace vcpkg::Commands::CI
Expand All @@ -93,25 +143,30 @@ namespace vcpkg::Commands::CI
static constexpr StringLiteral OPTION_HOST_EXCLUDE = "host-exclude";
static constexpr StringLiteral OPTION_FAILURE_LOGS = "failure-logs";
static constexpr StringLiteral OPTION_XUNIT = "x-xunit";
static constexpr StringLiteral OPTION_CI_BASELINE = "ci-baseline";
static constexpr StringLiteral OPTION_ALLOW_UNEXPECTED_PASSING = "allow-unexpected-passing";
static constexpr StringLiteral OPTION_RANDOMIZE = "x-randomize";
static constexpr StringLiteral OPTION_OUTPUT_HASHES = "output-hashes";
static constexpr StringLiteral OPTION_PARENT_HASHES = "parent-hashes";
static constexpr StringLiteral OPTION_SKIPPED_CASCADE_COUNT = "x-skipped-cascade-count";

static constexpr std::array<CommandSetting, 7> CI_SETTINGS = {
static constexpr std::array<CommandSetting, 8> CI_SETTINGS = {
{{OPTION_EXCLUDE, "Comma separated list of ports to skip"},
{OPTION_HOST_EXCLUDE, "Comma separated list of ports to skip for the host triplet"},
{OPTION_XUNIT, "File to output results in XUnit format (internal)"},
{OPTION_CI_BASELINE, "Path to the ci.baseline.txt file. Used to skip ports and detect regressions."},
{OPTION_FAILURE_LOGS, "Directory to which failure logs will be copied"},
{OPTION_OUTPUT_HASHES, "File to output all determined package hashes"},
{OPTION_PARENT_HASHES,
"File to read package hashes for a parent CI state, to reduce the set of changed packages"},
{OPTION_SKIPPED_CASCADE_COUNT,
"Asserts that the number of --exclude and supports skips exactly equal this number"}}};

static constexpr std::array<CommandSwitch, 2> CI_SWITCHES = {{
static constexpr std::array<CommandSwitch, 3> CI_SWITCHES = {{
{OPTION_DRY_RUN, "Print out plan without execution"},
{OPTION_RANDOMIZE, "Randomize the install order"},
{OPTION_ALLOW_UNEXPECTED_PASSING,
"Indicates that 'Passing, remove from fail list' results should not be emitted."},
}};

const CommandStructure COMMAND_STRUCTURE = {
Expand Down Expand Up @@ -484,10 +539,57 @@ namespace vcpkg::Commands::CI
const auto& settings = options.settings;

BinaryCache binary_cache{args, paths};
Triplet target_triplet = Triplet::from_canonical_name(std::string(args.command_arguments[0]));
Triplet target_triplet = Triplet::from_canonical_name(args.command_arguments[0]);

auto exclusions = parse_exclusions(settings, OPTION_EXCLUDE);
auto host_exclusions = parse_exclusions(settings, OPTION_HOST_EXCLUDE);
auto baseline_iter = settings.find(OPTION_CI_BASELINE);
const bool allow_unexpected_passing = Util::Sets::contains(options.switches, OPTION_ALLOW_UNEXPECTED_PASSING);
std::set<PackageSpec> expected_failures;
if (baseline_iter != settings.end())
{
auto baseline =
parse_ci_baseline(paths.get_filesystem().read_lines(baseline_iter->second, VCPKG_LINE_INFO));
auto target_triplet_string = args.command_arguments[0];
auto host_triplet_string = host_triplet.canonical_name();
for (auto& line : baseline)
{
if (line.triplet_name == target_triplet_string)
{
if (line.state == CiBaselineState::Skip)
{
exclusions.insert(line.port_name);
}
else if (line.state == CiBaselineState::Fail)
{
expected_failures.emplace(line.port_name, target_triplet);
}
}
else if (line.triplet_name == host_triplet_string)
{
if (line.state == CiBaselineState::Skip)
{
host_exclusions.insert(line.port_name);
}
else if (line.state == CiBaselineState::Fail)
{
expected_failures.emplace(line.port_name, host_triplet);
}
}
}
}
else
{
Checks::check_exit(VCPKG_LINE_INFO,
Comment thread
BillyONeal marked this conversation as resolved.
Outdated
!allow_unexpected_passing,
Strings::concat("--",
OPTION_ALLOW_UNEXPECTED_PASSING,
" can only be used if a ci baseline is provided via --",
OPTION_CI_BASELINE));
}
ExclusionPredicate is_excluded{
parse_exclusions(settings, OPTION_EXCLUDE),
parse_exclusions(settings, OPTION_HOST_EXCLUDE),
exclusions,
host_exclusions,
target_triplet,
host_triplet,
};
Expand Down Expand Up @@ -678,6 +780,38 @@ namespace vcpkg::Commands::CI
print2("\nTriplet: ", result.triplet, "\n");
print2("Total elapsed time: ", GlobalState::timer.to_string(), "\n");
result.summary.print();

if (baseline_iter != settings.end())
{
print2("\nREGRESSIONS:\n");
for (auto&& port_result : result.summary.results)
{
switch (port_result.build_result.code)
{
case Build::BuildResult::BUILD_FAILED:
case Build::BuildResult::POST_BUILD_CHECKS_FAILED:
case Build::BuildResult::FILE_CONFLICTS:
if (expected_failures.find(port_result.spec) == expected_failures.end())
{
std::cerr

@BillyONeal BillyONeal Mar 11, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should also use print2, and we certainly should avoiding new use of iostreams if at all possible.

<< " REGRESSION: " << port_result.spec.to_string() << " failed with "
<< Build::to_string_locale_invariant(port_result.build_result.code).to_string()
<< ". If expected, add " << port_result.spec.to_string() << "=fail to "
<< baseline_iter->second << std::endl;
}
break;
case Build::BuildResult::SUCCEEDED:
if (!allow_unexpected_passing &&
expected_failures.find(port_result.spec) != expected_failures.end())
{
std::cerr << " PASSING, REMOVE FROM FAIL LIST: " << port_result.spec.to_string()
<< " (" << baseline_iter->second << ")" << std::endl;
}
break;
default: break;
}
}
}
}

auto it_xunit = settings.find(OPTION_XUNIT);
Expand Down