Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Helix test results are not getting uploaded for a lot of jobs #74699

Closed
radical opened this issue Aug 27, 2022 · 2 comments · Fixed by #74788
Closed

Helix test results are not getting uploaded for a lot of jobs #74699

radical opened this issue Aug 27, 2022 · 2 comments · Fixed by #74788
Labels
area-Infrastructure blocking Marks issues that we want to fast track in order to unblock other important work

Comments

@radical
Copy link
Member

radical commented Aug 27, 2022 •
edited
Loading

For runtime pipeline, with rolling builds.

Working case in wasm/runtime-tests:

   JIT.CodeGenBringUpTests.XUnitWrapper  Total: 29, Errors: 0, Failed: 0, Skipped: 0, Time: 30.775s
+ export _commandExitCode=0
+ /usr/bin/python3 -u /datadisks/disk1/work/BB7B0A43/w/A9DB096A/u/xharness-event-processor.py
/usr/bin/python3: can't open file '/datadisks/disk1/work/BB7B0A43/w/A9DB096A/u/xharness-event-processor.py': [Errno 2] No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/BB7B0A43/p/reporter/run.py https://dev.azure.com/dnceng/ public 50501656 eyJ0eXAiOiJKV1 ....
2022-08-26T21:27:53.422Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2022-08-26T21:27:53.422Z	INFO   	run.py	__init__(42)	read_results	Searching '/datadisks/disk1/work/BB7B0A43/w/A9DB096A/e' for test results files
2022-08-26T21:27:53.422Z	INFO   	run.py	__init__(48)	read_results	Found results file /datadisks/disk1/work/BB7B0A43/w/A9DB096A/e/testResults.xml with format xunit
2022-08-26T21:27:53.427Z	INFO   	run.py	__init__(42)	read_results	Searching '/datadisks/disk1/work/BB7B0A43/w/A9DB096A/uploads' for test results files
2022-08-26T21:27:53.428Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 29 test reports to '/datadisks/disk1/work/BB7B0A43/w/A9DB096A/e/__test_report.json'
2022-08-26T21:27:53.428Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 7735 bytes

Broken cases

  1. Wasm/AOT/windows:
XHarness exit code: 0

C:\helix\work\workitem\e\wasm_build\AppBundle>set EXIT_CODE=0 
----- end Fri 08/26/2022 21:54:44.42 ----- exit code 0 ----------------------------------------------------------
XHarness artifacts: C:\helix\work\workitem\uploads\xharness-output
python: can't open file 'C:\helix\work\workitem\u\xharness-event-processor.py': [Errno 2] No such file or directory
ERROR: The process "corerun.exe" not found.

[END EXECUTION]
Exit Code:0
  1. Wasm/linux:
XHarness exit code: 0
/datadisks/disk1/work/B24B0983/w/A9D4099F/e /datadisks/disk1/work/B24B0983/w/A9D4099F/e
----- end Fri Aug 26 21:08:51 UTC 2022 ----- exit code 0 ----------------------------------------------------------
XHarness artifacts: /datadisks/disk1/work/B24B0983/w/A9D4099F/uploads/xharness-output
+ export _commandExitCode=0
+ /usr/bin/python3 -u /datadisks/disk1/work/B24B0983/w/A9D4099F/u/xharness-event-processor.py
/usr/bin/python3: can't open file '/datadisks/disk1/work/B24B0983/w/A9D4099F/u/xharness-event-processor.py': [Errno 2] No such file or directory
+ exit 0
['normal-Microsoft.CSharp.Tests' END OF WORK ITEM LOG: Command exited with 0]

  • The above 2 cases are not running reporter/run.py. The test results can't be seen via kusto.

  • It seems to be working with this, but failing on the next rolling build. And the changes between those - 7b0227d...8755447 .

  • This is for wasm, rolling builds, and each column represents a different CI job:

Screen Shot 2022-08-27 at 5 23 12 AM

Note: the xharness-event-processor.py error can be ignored.

cc @lewing @steveisok @akoeplinger @ViktorHofer @premun
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 27, 2022
@ghost
Copy link

ghost commented Aug 27, 2022

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

For runtime pipeline, with rolling builds.

Working case in wasm/runtime-tests:

   JIT.CodeGenBringUpTests.XUnitWrapper  Total: 29, Errors: 0, Failed: 0, Skipped: 0, Time: 30.775s
+ export _commandExitCode=0
+ /usr/bin/python3 -u /datadisks/disk1/work/BB7B0A43/w/A9DB096A/u/xharness-event-processor.py
/usr/bin/python3: can't open file '/datadisks/disk1/work/BB7B0A43/w/A9DB096A/u/xharness-event-processor.py': [Errno 2] No such file or directory
+ /usr/bin/python3 /datadisks/disk1/work/BB7B0A43/p/reporter/run.py https://dev.azure.com/dnceng/ public 50501656 eyJ0eXAiOiJKV1 ....
2022-08-26T21:27:53.422Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2022-08-26T21:27:53.422Z	INFO   	run.py	__init__(42)	read_results	Searching '/datadisks/disk1/work/BB7B0A43/w/A9DB096A/e' for test results files
2022-08-26T21:27:53.422Z	INFO   	run.py	__init__(48)	read_results	Found results file /datadisks/disk1/work/BB7B0A43/w/A9DB096A/e/testResults.xml with format xunit
2022-08-26T21:27:53.427Z	INFO   	run.py	__init__(42)	read_results	Searching '/datadisks/disk1/work/BB7B0A43/w/A9DB096A/uploads' for test results files
2022-08-26T21:27:53.428Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 29 test reports to '/datadisks/disk1/work/BB7B0A43/w/A9DB096A/e/__test_report.json'
2022-08-26T21:27:53.428Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 7735 bytes
  1. Wasm/AOT/windows:
XHarness exit code: 0

C:\helix\work\workitem\e\wasm_build\AppBundle>set EXIT_CODE=0 
----- end Fri 08/26/2022 21:54:44.42 ----- exit code 0 ----------------------------------------------------------
XHarness artifacts: C:\helix\work\workitem\uploads\xharness-output
python: can't open file 'C:\helix\work\workitem\u\xharness-event-processor.py': [Errno 2] No such file or directory
ERROR: The process "corerun.exe" not found.

[END EXECUTION]
Exit Code:0
  1. Wasm/linux:
XHarness exit code: 0
/datadisks/disk1/work/B24B0983/w/A9D4099F/e /datadisks/disk1/work/B24B0983/w/A9D4099F/e
----- end Fri Aug 26 21:08:51 UTC 2022 ----- exit code 0 ----------------------------------------------------------
XHarness artifacts: /datadisks/disk1/work/B24B0983/w/A9D4099F/uploads/xharness-output
+ export _commandExitCode=0
+ /usr/bin/python3 -u /datadisks/disk1/work/B24B0983/w/A9D4099F/u/xharness-event-processor.py
/usr/bin/python3: can't open file '/datadisks/disk1/work/B24B0983/w/A9D4099F/u/xharness-event-processor.py': [Errno 2] No such file or directory
+ exit 0
['normal-Microsoft.CSharp.Tests' END OF WORK ITEM LOG: Command exited with 0]

All the test results are missing, and can't be seen with kusto.

This is for wasm, rolling builds, and each column represents a different CI job:

Screen Shot 2022-08-27 at 5 23 12 AM

Note: the xharness-event-processor.py error can be ignored.

cc @lewing @steveisok @akoeplinger @ViktorHofer

Author: radical
Assignees: -
Labels:

ci-health, area-Infrastructure
Milestone: -

@radical radical added the blocking Marks issues that we want to fast track in order to unblock other important work label Aug 29, 2022
@ChadNedzlek
Copy link
Member

This is caused by 7a4b0af#diff-1d68a522b60c1ab69fb537ce06f5d7e9a176fa4f75e16425fc164d4b738d7678 from PR #73060 (which, if you look at the last run of it, didn't run hardly any tests)

It made breaking changes in "sendtohelixhelper.proj". Currently runtime has almost ZERO test coverage, so that change should be reverted immediately.

radical added a commit to radical/runtime that referenced this issue Aug 29, 2022
@radical
CI: Fix helix test results reporting
749dcc6 
PR dotnet#73060 broke uploading of helix test results. This was caused by the
change:

```xml
      <HelixPostCommands>@(HelixPostCommand)</HelixPostCommands>
```

This is overwriting the existing value of `$(HelixPostCommands)`, which
gets set to have the upload script invocation in https://github.com/dotnet/arcade/blob/34dff939b4a91e4693f78a856e0e055c1a3f3fba/src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/AzurePipelines.MonoQueue.targets#L8-L15 at evaluation time.

Fix by *appending* to the property.

Thanks to ChadNedzlek for finding the cause!

Fixes dotnet#74699 .
@radical radical mentioned this issue Aug 29, 2022
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 29, 2022
radical added a commit that referenced this issue Aug 30, 2022
@radical @akoeplinger
CI: Fix helix test results reporting (#74788)
2ed931d 
* CI: Fix helix test results reporting

PR #73060 broke uploading of helix test results. This was caused by the
change:

```xml
      <HelixPostCommands>@(HelixPostCommand)</HelixPostCommands>
```

This is overwriting the existing value of `$(HelixPostCommands)`, which
gets set to have the upload script invocation in https://github.com/dotnet/arcade/blob/34dff939b4a91e4693f78a856e0e055c1a3f3fba/src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/AzurePipelines.MonoQueue.targets#L8-L15 at evaluation time.

Fix by *appending* to the property.

Thanks to ChadNedzlek for finding the cause!

Fixes #74699 .

* Disable failing tests, see #74781

Co-authored-by: Alexander Köplinger <[email protected]>
@ghost ghost removed in-pr There is an active PR which will close this issue when it is merged untriaged New issue has not been triaged by the area owner labels Aug 30, 2022
@jerryk414 jerryk414 mentioned this issue Sep 21, 2022
Closed
@ghost ghost locked as resolved and limited conversation to collaborators Sep 29, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-Infrastructure blocking Marks issues that we want to fast track in order to unblock other important work
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

CI: Fix helix test results reporting radical/runtime
2 participants
@radical @ChadNedzlek