Skip to content

[ci] Remove unaffected ports from pull request builds #21078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dan-shaw merged 4 commits into from
Nov 1, 2021
Merged

[ci] Remove unaffected ports from pull request builds #21078

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
15815ae
Rewrite to single invovation of 'vcpkg ci'
dg0yt Jul 30, 2021
d7d08df
Split caching args from common args
dg0yt Oct 30, 2021
336f088
Use parent hashes for pull request CI builds
dg0yt Oct 30, 2021
f7121a7
Completely switch to parent revision state and back
dg0yt Oct 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 35 additions & 10 deletions scripts/azure-pipelines/test-modified-ports.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,14 @@ The type and parameters of the binary source. Shared across runs of this script.
this parameter is not set, binary caching will not be used. Example: "files,W:\"

.PARAMETER BuildReason
The reason Azure Pipelines is running this script (controls in which mode Binary Caching is used).
If BinarySourceStub is not set, this parameter has no effect. If BinarySourceStub is set and this is
not, binary caching will default to read-write mode.
The reason Azure Pipelines is running this script. For invocations caused by `PullRequest`,
modified ports are identified by changed hashes with regard to git HEAD~1 (subject to NoParentHashes),
and ports marked as failing in the CI baseline (or which depend on such ports) are skipped.
If BinarySourceStub is set and this parameter is set to a non-empty value other than `PullRequest`,
binary caching will be in write-only mode.

.PARAMETER NoParentHashes
Indicates to not use parent hashes even for pull requests.

.PARAMETER PassingIsPassing
Indicates that 'Passing, remove from fail list' results should not be emitted as failures. (For example, this is used
Expand All @@ -49,6 +54,7 @@ Param(
[String]$BuildReason = $null,
[String[]]$AdditionalSkips = @(),
[String[]]$OnlyTest = $null,
[switch]$NoParentHashes = $false,
[switch]$PassingIsPassing = $false
)

Expand Down Expand Up @@ -77,12 +83,13 @@ $commonArgs = @(
"--x-packages-root=$packagesRoot",
"--overlay-ports=scripts/test_ports"
)
$cachingArgs = @()

$skipFailures = $false
if ([string]::IsNullOrWhiteSpace($BinarySourceStub)) {
$commonArgs += @('--no-binarycaching')
$cachingArgs = @('--no-binarycaching')
} else {
$commonArgs += @('--binarycaching')
$cachingArgs = @('--binarycaching')
$binaryCachingMode = 'readwrite'
if ([string]::IsNullOrWhiteSpace($BuildReason)) {
Write-Host 'Build reason not specified, defaulting to using binary caching in read write mode.'
Expand All @@ -96,7 +103,7 @@ if ([string]::IsNullOrWhiteSpace($BinarySourceStub)) {
$binaryCachingMode = 'write'
}

$commonArgs += @("--binarysource=clear;$BinarySourceStub,$binaryCachingMode")
$cachingArgs += @("--binarysource=clear;$BinarySourceStub,$binaryCachingMode")
}

if ($Triplet -eq 'x64-linux') {
Expand Down Expand Up @@ -141,7 +148,7 @@ if ($null -ne $OnlyTest)
{
$OnlyTest | % {
$portName = $_
& "./vcpkg$executableExtension" install --triplet $Triplet @commonArgs $portName
& "./vcpkg$executableExtension" install --triplet $Triplet @commonArgs @cachingArgs $portName
if (-not $?)
{
[System.Console]::Error.WriteLine( `
Expand All @@ -155,17 +162,35 @@ if ($null -ne $OnlyTest)
}
else
{
$hostArgs = @()
if ($Triplet -in @('x64-windows', 'x64-osx', 'x64-linux'))
{
# WORKAROUND: These triplets are native-targetting which triggers an issue in how vcpkg handles the skip list.
# The workaround is to pass the skip list as host-excludes as well.
& "./vcpkg$executableExtension" ci $Triplet --x-xunit=$xmlFile --exclude=$skipList --host-exclude=$skipList --failure-logs=$failureLogs @commonArgs
$hostArgs = @("--host-exclude=$skipList")
}
else

$parentHashes = @()
if (($BuildReason -eq 'PullRequest') -and -not $NoParentHashes)
{
& "./vcpkg$executableExtension" ci $Triplet --x-xunit=$xmlFile --exclude=$skipList --failure-logs=$failureLogs @commonArgs
# Prefetch tools for better output
& "./vcpkg$executableExtension" fetch cmake
& "./vcpkg$executableExtension" fetch ninja
& "./vcpkg$executableExtension" fetch git
Copy link
Member

@BillyONeal BillyONeal Nov 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically this git and the one used on line 184 are not the same git. Our UX for fetch is... not great :)

Copy link
Member

@BillyONeal BillyONeal Nov 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(But this doesn't matter)

Copy link
Contributor Author

@dg0yt dg0yt Nov 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is also what I thought.
What we might do on another revision is moving the fetches out of the script, next to the bootstrap step.


Write-Host "Determining parent hashes using HEAD~1"
$parentHashesFile = Join-Path $WorkingRoot 'parent-hashes.json'
$parentHashes = @("--parent-hashes=$parentHashesFile")
& git revert -n -m 1 HEAD
& "./vcpkg$executableExtension" ci $Triplet --dry-run --exclude=$skipList @hostArgs @commonArgs --no-binarycaching "--output-hashes=$parentHashesFile" `
| ForEach-Object { if ($_ -match ' dependency information| determine pass') { Write-Host $_ } }
& git reset --hard HEAD

Write-Host "Running CI using parent hashes"
}

& "./vcpkg$executableExtension" ci $Triplet --x-xunit=$xmlFile --exclude=$skipList --failure-logs=$failureLogs @hostArgs @commonArgs @cachingArgs @parentHashes

$failureLogsEmpty = (-Not (Test-Path $failureLogs) -Or ((Get-ChildItem $failureLogs).count -eq 0))
Write-Host "##vso[task.setvariable variable=FAILURE_LOGS_EMPTY]$failureLogsEmpty"

Expand Down