Refine corerun instructions. Improve argument parsing in src/tests/build. #81733

jeffhandley · 2023-02-07T01:38:04Z

While working through the corerun instructions, I got tripped up in a few ways.

Some of the instructions were confusing to me
There were places where the path to corerun was unclear or misleading
The argument parsing of src/tests/build.cmd was not behaving the same as src/tests/build.sh
- There seemed to be a few bugs lurking in there too

This PR updates the instructions and also improves the argument parsing in src/tests/build based on these findings. For the argument parsing, I also introduced a mechanism for testing the results of the argument parsing for better maintainability going forward.

ghost · 2023-02-07T01:38:14Z

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

While working through the corerun instructions, I got tripped up in a few ways.

Some of the instructions were confusing to me
There were places where the path to corerun was unclear or misleading
The argument parsing of src/tests/build.cmd was not behaving the same as src/tests/build.sh
- There seemed to be a few bugs lurking in there too

This PR updates the instructions and also improves the argument parsing in src/tests/build based on these findings. For the argument parsing, I also introduced a mechanism for testing the results of the argument parsing for better maintainability going forward.

Author:	jeffhandley
Assignees:	jeffhandley
Labels:	`area-Infrastructure-coreclr`
Milestone:	-

jeffhandley · 2023-02-07T01:38:34Z

docs/workflow/testing/coreclr/running-aspnet-benchmarks-with-crossgen2.md

@@ -192,7 +192,7 @@ CoreRun.exe \runtime\artifacts\bin\coreclr\windows.x64.Release\crossgen2\crossge
 On Linux:

 ```bash
-./corerun /runtime/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll
+corerun /runtime/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll


corerun is executed from $PATH, not from the current folder

jeffhandley · 2023-02-07T01:39:19Z

docs/workflow/testing/using-corerun-and-coreroot.md

@@ -53,9 +53,9 @@ dotnet new console
 dotnet build
 ```

-Now, instead of running our app the usual way, we will use our newly built `corerun` to execute it using our build of the runtime. For this, we will follow the steps denoted below:
+Now, instead of running our app the usual way, we will use `corerun` to execute it using our build of the runtime. The `corerun` executable is created as part of building the `clr` subset, and it will exist in the `<repo root>/artifacts/bin/coreclr/<OS>.<Arch>.<Configuration>` folder. For this, we will follow the steps denoted below:


"Our newly built" was confusing to me. I removed that and aimed to add clarity that it should have been built already (and where it exists so that can be confirmed).

jeffhandley · 2023-02-07T01:40:43Z

docs/workflow/testing/using-corerun-and-coreroot.md


 ```powershell
 # Note the '+=' since we're appending to the already existing PATH variable.
 # Also, replace the ';' with ':' if on Linux or macOS.
 $Env:PATH += ';<repo_root>\artifacts\tests\coreclr\windows.x64.Debug\Tests\Core_Root'
-.\corerun HelloWorld.dll
+corerun HelloWorld.dll


corerun is found via path, not from the current folder.

jeffhandley · 2023-02-07T01:45:37Z

src/tests/build.cmd

@@ -30,6 +30,7 @@ set "__MsbuildDebugLogsDir=%__LogsDir%\MsbuildDebugLogs"
 set __Exclude=%__RepoRootDir%\src\tests\issues.targets

 REM __UnprocessedBuildArgs are args that we pass to msbuild (e.g. /p:TargetArchitecture=x64)
+set __commandName=%~nx0


Down in Usage, we need to print the command name, but we've already shifted the args by then. This will grab the file name and extension before we do any shifting.

jeffhandley · 2023-02-07T01:46:53Z

src/tests/build.cmd

-if /i "%1" == "-help"  goto Usage
-if /i "%1" == "--help" goto Usage
+@REM All arguments following this tag will be passed directly to msbuild (as unprocessed arguments)
+if /i "%1" == "--"                    (set processedArgs=!processedArgs! %1&shift&goto ArgsDone)


Previously, handling of -- seemed broken. It was just continuing the loop, which could then recognize and process arguments that were expected to be passed through. Chances are, such args were going unrecognized and by happenstance, they were still getting passed through. But this changes the behavior to match the Usage notes.

jeffhandley · 2023-02-07T01:51:30Z

src/tests/build.cmd

+if /i "%arg%" == "skipgeneratelayout"    (set __SkipGenerateLayout=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+
+if /i "%arg%" == "copynativeonly"        (set __CopyNativeTestBinaries=1&set __SkipNative=1&set __CopyNativeProjectsAfterCombinedTestBuild=false&set __SkipGenerateLayout=1&set __SkipTestWrappers=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%arg%" == "generatelayoutonly"    (set __GenerateLayoutOnly=1&set __SkipManaged=1&set __SkipNative=1&set __CopyNativeProjectsAfterCombinedTestBuild=false&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)


Previously, the generatelayoutonly argument didn't set __GenerateLayoutOnly=1, which made it do more work than intended when specified.

jeffhandley · 2023-02-07T01:52:22Z

src/tests/build.cmd

+if /i "%arg%" == "tree"                  (set __BuildTestTree=!__BuildTestTree!%2%%3B&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop)
+if /i "%arg%" == "log"                   (set __BuildLogRootName=%2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop)
+if /i "%arg%" == "Exclude"               (set __Exclude=%2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop)
+if /i "%arg%" == "priority"              (set __Priority=%2&set processedArgs=!processedArgs! %1=%2&shift&shift&goto Arg_Loop)


Despite a comment about priority above that it needs special handling; it was still incorrect. It was shifting at the wrong place, and it was resulting in the processed/unprocessed args not being correct.

jeffhandley · 2023-02-07T01:53:08Z

src/tests/build.cmd

+if /i "%arg%" == "priority"              (set __Priority=%2&set processedArgs=!processedArgs! %1=%2&shift&shift&goto Arg_Loop)
+
+@REM The following arguments also consume two subsequent arguments
+if /i "%arg%" == "cmakeargs"             (set __CMakeArgs="%2=%3" %__CMakeArgs%&set "processedArgs=!processedArgs! %1 %2 %3"&shift&shift&shift&goto Arg_Loop)


The cmakeargs argument was also not correctly shifting the arguments, so extra/incorrect arguments would end up flowing through to msbuild.

jeffhandley · 2023-02-07T01:54:15Z

src/tests/build.cmd

@@ -130,7 +150,40 @@ if [!processedArgs!]==[] (
    )
 )


This logic was unexpectedly above the :ArgsDone label and was being hit whenever there was an unrecognized argument (vs. when we were done processing arguments).

This was working by coincidence that argument parsing aborts upon the first unrecognized argument. There are existing pipelines relying on that behavior.

Nonetheless, the logic is moved below the label now, even though we still exit processing upon the first unrecognized argument.

src/tests/build.cmd

…ine usage

trylek

Very nice, thanks Jeff for all the fixes, improvements and documentation updates!

Refine corerun instructions. Improve arg parsing in src/tests/build.

ee3c24e

dotnet-issue-labeler bot added the area-Infrastructure-coreclr label Feb 7, 2023

ghost assigned jeffhandley Feb 7, 2023

jeffhandley commented Feb 7, 2023

View reviewed changes

build-analysis bot mentioned this pull request Feb 7, 2023

Tracking issue for CI build timeouts #76454

Closed

jeffhandley added 2 commits February 7, 2023 15:00

Revert the libraries configuration default. Refine usage and docs.

9b0efe5

Exit argument processing at first unrecognized arg per existing pipel…

e4f796c

…ine usage

jeffhandley requested a review from jkotas February 8, 2023 08:20

jkotas requested review from a team and trylek February 8, 2023 15:03

trylek approved these changes Feb 8, 2023

View reviewed changes

jeffhandley merged commit 1166bba into dotnet:main Feb 8, 2023

jeffhandley mentioned this pull request Feb 8, 2023

Fix src/tests/build.sh argument processing #81836

Merged

BruceForstall mentioned this pull request Feb 10, 2023

Tweak coreclr test build script #81975

Merged

jeffhandley deleted the jeffhandley/test-instructions branch February 11, 2023 03:39

ghost locked as resolved and limited conversation to collaborators Mar 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refine corerun instructions. Improve argument parsing in src/tests/build. #81733

Refine corerun instructions. Improve argument parsing in src/tests/build. #81733

jeffhandley commented Feb 7, 2023

ghost commented Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 7, 2023

jeffhandley Feb 8, 2023

trylek left a comment

Refine corerun instructions. Improve argument parsing in src/tests/build. #81733

Refine corerun instructions. Improve argument parsing in src/tests/build. #81733

Conversation

jeffhandley commented Feb 7, 2023

ghost commented Feb 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trylek left a comment

Choose a reason for hiding this comment