Skip to content

[ci] Publish test results #23477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dg0yt wants to merge 5 commits into from
Closed

[ci] Publish test results #23477

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b699c66
Publish CI test results
dg0yt Mar 10, 2022
bf66a5b
Process xunit file for AZP
dg0yt Mar 11, 2022
0c27a88
Simulate build error for testing
dg0yt Mar 30, 2022
710ab24
Merge remote-tracking branch 'origin/master' into test-results
dg0yt Jun 11, 2022
ed4585c
Restore xunit output and update result processing
dg0yt Jun 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ports/librttopo/portfile.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
message(FATAL_ERROR "SIMULATED BUILD ERROR. DO NOT MERGE")

# NOTE: update the version and checksum for new LIBRTTOPO release
set(LIBRTTOPO_VERSION_STR "1.1.0")
set(LIBRTTOPO_PACKAGE_SUM "d9c2f4db1261cc942152d348abb7f03e6053a63b6966e081c5381d40bbebd3c7ca1963224487355f384d7562a90287fb24d7af9e7eda4a1e230ee6441cef5de9")
Expand Down
9 changes: 9 additions & 0 deletions scripts/azure-pipelines/linux/azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,12 @@ jobs:
inputs:
PathtoPublish: scripts/list_files
ArtifactName: 'file lists for x64-linux'
- task: PublishTestResults@2
displayName: 'Publish Test Results'
condition: ne(variables['XML_RESULTS_FILE'], '')
inputs:
testRunTitle: x64-linux
testResultsFormat: xUnit
testResultsFiles: $(XML_RESULTS_FILE)
platform: x64-linux
configuration: static
9 changes: 9 additions & 0 deletions scripts/azure-pipelines/osx/azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,12 @@ jobs:
inputs:
PathtoPublish: scripts/list_files
ArtifactName: 'file lists for x64-osx${{ variables.Postfix }}'
- task: PublishTestResults@2
displayName: 'Publish Test Results'
condition: ne(variables['XML_RESULTS_FILE'], '')
inputs:
testRunTitle: x64-osx
testResultsFormat: xUnit
testResultsFiles: $(XML_RESULTS_FILE)
platform: x64-osx
configuration: static
38 changes: 38 additions & 0 deletions scripts/azure-pipelines/process-test-results.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: MIT
#

<#
.SYNOPSIS
Process the raw xUnit result file from vcpkg tool for uploading to AZP.
Copy link
Member

@BillyONeal BillyONeal Mar 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of adding yet another .ps1 that we wash output through can we just fix https://github.com/microsoft/vcpkg-tool/blob/29206722417f9b5ec3d1c1123c96776cd355e633/src/vcpkg/commands.ci.cpp#L218 to emit output that does the right thing?

Copy link
Member

@BillyONeal BillyONeal Mar 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the changes are:

  • Don't emit things excluded by --parent-hashes
  • Don't emit EXCLUDED things
  • Add the "owner/method" bit to each entry

Other than the EXCLUDED one I think it's clear we should always just do that, and for the excluded one it would be easy to add an additional --xunit-without-excluded thing or something like that?

Copy link
Contributor Author

@dg0yt dg0yt Mar 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can move it to vcpkg-tool once there is enough support for how to represent the data. This PR is the playground for direct testing of changes. I cannot do that with vpckg-tool PRs.

And it needs to wait at least for microsoft/vcpkg-tool#345 to be merged.

Copy link
Member

@BillyONeal BillyONeal Mar 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can move it to vcpkg-tool once there is enough support for how to represent the data. This PR is the playground for direct testing of changes. I cannot do that with vpckg-tool PRs.

I see, in that case this looks great, thank you!

I agree we need an easier way to test full e2e integration including a build. Unfortunately I don't know a way to set things up such that people can try changes from the internet directly; perhaps we can put in a switch in the .yml which chooses to build instead of bootstrap.

And it needs to wait at least for microsoft/vcpkg-tool#345 to be merged.

Does it? The only interesting thing your script consumes is parent hashes and the tool already consumes those.

Copy link
Member

@BillyONeal BillyONeal Mar 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Hmmm apparently I don't understand what the intended effect of parent hashes is because I'm seeing ones that are in the parent survive processing here.

Copy link
Member

@BillyONeal BillyONeal Mar 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also this is duplicating assembly/collection for every test, is that the intent?

Copy link
Member

@BillyONeal BillyONeal Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And it needs to wait at least for microsoft/vcpkg-tool#345 to be merged.

Does it? The only interesting thing your script consumes is parent hashes and the tool already consumes those.

Ah, I see, because the current analyze-test-results.ps1 is implemented in terms of that XUnit output.

Copy link
Contributor Author

@dg0yt dg0yt Mar 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is duplicating assembly/collection for every test, is that the intent?

As written in source and PR comments: The xunit assemby name is the AZP "Test file" name. I found it useful to have the port name in this field.


.DESCRIPTION
process-test-result takes the path a xUnit file generated by `vcpkg ci` and
modifies it in-place for reporting port builds as tests in Azure pipelines.

.PARAMETER xunit
The path to the xUnit file.
#>
[CmdletBinding()]
Param(
[Parameter(Mandatory = $true)]
[string]$xunit
)

if (-not(Test-Path $xunit)) {
write-error "No such file: $xunit"
}

# Fix invalid XML
(Get-Content -Path $xunit -Raw) `
-replace ' method="(Skip|Fail|Pass)">', ' result="$1">' `
-replace '</failure></message>', '</message></failure>' |
Set-Content -Path $xunit

[xml]$xmlContents = Get-Content $xunit
$node = $xmlContents.SelectSingleNode('/assemblies/assembly[@run-date="1970-01-01"]')
while ($null -ne $node) {
$node.ParentNode.RemoveChild($node) | Out-Null
$node = $xmlContents.SelectSingleNode('/assemblies/assembly[@run-date="1970-01-01"]')
}
$xmlContents.save($xunit)
6 changes: 5 additions & 1 deletion scripts/azure-pipelines/test-modified-ports.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ else {
}

$failureLogs = Join-Path $ArtifactStagingDirectory 'failure-logs'
$xunitFile = Join-Path $ArtifactStagingDirectory "$Triplet-results.xml"

if ($IsWindows)
{
Expand Down Expand Up @@ -162,7 +163,7 @@ if (($BuildReason -eq 'PullRequest') -and -not $NoParentHashes)
# but changes must trigger at least some testing.
Copy-Item "scripts/buildsystems/vcpkg.cmake" -Destination "scripts/test_ports/cmake"
Copy-Item "scripts/buildsystems/vcpkg.cmake" -Destination "scripts/test_ports/cmake-user"
& "./vcpkg$executableExtension" ci "--triplet=$Triplet" --failure-logs=$failureLogs "--ci-baseline=$PSScriptRoot/../ci.baseline.txt" @commonArgs @cachingArgs @parentHashes @skipFailuresArg
& "./vcpkg$executableExtension" ci "--triplet=$Triplet" --failure-logs=$failureLogs --x-xunit=$xunitFile "--ci-baseline=$PSScriptRoot/../ci.baseline.txt" @commonArgs @cachingArgs @parentHashes @skipFailuresArg

$failureLogsEmpty = (-Not (Test-Path $failureLogs) -Or ((Get-ChildItem $failureLogs).count -eq 0))
Write-Host "##vso[task.setvariable variable=FAILURE_LOGS_EMPTY]$failureLogsEmpty"
Expand All @@ -171,3 +172,6 @@ if ($LASTEXITCODE -ne 0)
{
throw "vcpkg ci failed"
}

& "$PSScriptRoot/process-test-results.ps1" -xunit $xunitFile
Write-Host "##vso[task.setvariable variable=XML_RESULTS_FILE]$xunitFile"
9 changes: 9 additions & 0 deletions scripts/azure-pipelines/windows/azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,3 +96,12 @@ jobs:
inputs:
PathtoPublish: scripts/list_files
ArtifactName: 'file lists for ${{ parameters.triplet }}'
- task: PublishTestResults@2
displayName: 'Publish Test Results'
condition: ne(variables['XML_RESULTS_FILE'], '')
inputs:
testRunTitle: ${{ parameters.triplet }}
testResultsFormat: xUnit
testResultsFiles: $(XML_RESULTS_FILE)
platform: ${{ parameters.triplet }}