src/tests tree test xunit-based source generated runner

[jkoritzinsky]

This draft PR includes a Roslyn source generator that detects public methods defined in the current (being compiled) assembly or any referenced assembly that are marked with XUnit attributes and generates a basic Main method that runs all of the discovered tests in order and returns 100 if all tests finish with no exceptions or 101 otherwise. It also includes support for a merged runner for future usage that reports test results with in the XUnit results format and runtime test filtering. The generator supports test filtering at compile time as well to simplify debugging.

It has support for [Fact] and [ConditionalFact] tests, [Theory] and [ConditionalTheory] parameterized tests, and all XUnitExtensions attributes except SkipOnCIAttribute.

The enum files are copied from Microsoft.DotNet.XUnitExtensions in dotnet/arcade.

This PR also includes examples of using the new [Fact]-based approach with three test suites:

• Interop/PInvoke/Vector2_3_4
  • Demonstrates basic usage
• Interop/PInvoke/Decimal
  • Demonstrates platform-specific test support
• JIT/IL_Conformance/Old/directed/AutoInit + Common/ILTestRunner
  • Demonstrates a test marking design (putting the [Fact] attribute on the .entrypoint method and making that method parameterless) that enables both standalone running of an IL test and using the source generator to generate a wrapper assembly in C# that runs the same tests.

cc: @trylek

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This draft PR includes a Roslyn source generator that detects public methods defined in the current (being compiled) assembly or any referenced assembly that are marked with XUnit attributes and generates a basic Main method that runs all of the discovered tests in order and returns 100 if all tests finish with no exceptions or 101 otherwise.

It has support for [Fact] and [ConditionalFact] tests, [SkipOnPlatform] support, as well as partial [ActiveIssue] support.

The enum files are copied from Microsoft.DotNet.XUnitExtensions in dotnet/arcade.

This PR also includes examples of using the new [Fact]-based approach with three test suites:

• Interop/PInvoke/Vector2_3_4
  • Demonstrates basic usage
• Interop/PInvoke/Decimal
  • Demonstrates platform-specific test support
• JIT/IL_Conformance/Old/directed/AutoInit + Common/ILTestRunner
  • Demonstrates a test marking design (putting the [Fact] attribute on the .entrypoint method and making that method parameterless) that enables both standalone running of an IL test and using the source generator to generate a wrapper assembly in C# that runs the same tests.

cc: @trylek

Author:	jkoritzinsky
Assignees:	-
Labels:	area-Infrastructure
Milestone:	-

[am11]

Can we reflect on Microsoft.DotNet.XUnitExtensions in generator to avoid duplicating these types?

[jkoritzinsky]

We can, but using the Roslyn symbol APIs to reflect like that is not fast. I think eventually making this generator live in dotnet/arcade might be a better solution.

[jkoritzinsky]

In future versions of the generator (v1 milestone of the new test infra), we need to support spitting out an XUnit-style results file as well as basic test filtering, so living in arcade would make more sense then.

[trylek]

We should be careful not to regress managed build time too significantly. While ultimately we'll optimize it by compiling tests as larger apps, it remains to be seen whether compiling tests in merged form makes sense for all pipelines - some of the GC stress pipelines come to mind where some tests are timing out already today even without merging - so that the "standalone" mode is likely to stay as a valid way of running tests at least for niche scenarios. If we pay the reflection startup cost just once per Roslyn generation of all the wrappers, it's probably just fine but paying the cost for every single test may be significant.

[jkoritzinsky]

I was thinking in generator case, startup performance is not a major issue (e.g. we can discover types using reflection (or perhaps Roslyn symbol APIs) to collect required infos in a static constructor just once during the lifetime of compilation).

I don't like caching it like that due to the fact that it's technically not correct and can break some assumptions that Roslyn has about the lifetime of a generator (as Roslyn can throw them away or reuse them in many different ways including in compiler server scenarios)

[trylek]

putting the [Fact] attribute on the .entrypoint method and making that method parameterless

@jkoritzinsky, what is your thinking about those cases where we execute a CLRTestProjectToRun multiple times with commandline variations? I think we can either put these as multiple [Fact] entrypoints in the app - but that would mean that the functionality of the "old-style" .entrypoint would diverge from the Roslyn-generated runner-based test execution - or continue honoring them in the individual projects in a backward-compatible manner and just somehow modify the generator to generate multiple calls to the entrypoint, but that would probably require some knowledge of the generator about the project files involved and I'm not sure whether that's doable and practical.

[jkoritzinsky]

what is your thinking about those cases where we execute a CLRTestProjectToRun multiple times with commandline variations? I think we can either put these as multiple [Fact] entrypoints in the app - but that would mean that the functionality of the "old-style" .entrypoint would diverge from the Roslyn-generated runner-based test execution - or continue honoring them in the individual projects in a backward-compatible manner and just somehow modify the generator to generate multiple calls to the entrypoint, but that would probably require some knowledge of the generator about the project files involved and I'm not sure whether that's doable and practical.

There are a few options here:

1. If all of the different variations of the test can run in-process, then just add multiple [Fact] methods for each variation and let the generated Main run all the tests at once. With the v1 version of the generator, we'll have test filtering that could be used at this level if people so desired.
2. If the test is an IL test, refactor each case into a separate method that has a [Fact] attribute, and then keep the existing Main method as-is for now.
3. Use the xunit [Theory] attributes for parameterized testing as after I add support to the generator (planned for v0)
4. If all variations need to run out-of-proc from each other, refactor the "Main" method into a separate library and build separate entry-point assemblies which have the [Fact] attributes.

[jkoritzinsky]

From a cursory look, many of the tests that set CLRTestProjectToRun (mostly "negative test" style tests or GCSimulator) will likely need to continue running out-of-proc, so we can move these to be out-of-proc tests using RemoteExecutor or something similar that spits out the target command line and PID for easier debugging.

If we really want to source-generate the entry-points for these tests, we could write a generator specifically for RunOnly-style tests that generates the out-of-proc test invocation and return code assert [Fact] methods and keep the target executables as-is (or using the standalone version of the generator).

[trylek]

Thanks Jeremy, that sounds reasonable; naturally it would be ideal to simplify the test project structure wherever possible and the RunOnly tests with CLRTestProjectToRun (in fact there are more tests of this type that are incorrectly marked as BuildAndRun as discussed in the issue thread) certainly do add complexity so if we're on track for something simpler, I'm all for it. Further down the road we'll also need to resolve other corner cases like tests returning weird exit codes (some or all of these cannot be merged as the negative exit codes signify intended runtime crashes in negative tests).

[trylek]

One other interesting side note regarding the ilproj projects: Many projects come in groups with the same entrypoint name (e.g. the 1500 or so TypeGeneratorTests under Loader all have the entrypoint Framework.main). So far I've been assuming we'll need to clean them up by making the class names unique, however considering we currently don't plan to actually merge IL source code, we wouldn't need that if we were able to express an assembly-qualified method call in the generated C# wrapper. I'm not sure whether that's doable in the C# syntax, we could probably express this if we generated the wrapper directly as IL source code but I doubt we want to go that way.

[jkoritzinsky]

Using extern aliases that should be possible, albeit not clean.

[jkoritzinsky]

@trylek I've added support for reference aliases (to handle the TypeGeneratorTests problem) and theory support (to handle a large part of the RunOnly problem) with examples for each.

[trylek]

Beautiful, thanks Jeremy! Just for my education, where do the reference alias infos actually live? Is that part of the reference assembly table in the ECMA metadata within the MSIL file?

[jkoritzinsky]

The alias name is passed to Roslyn through the project file. After compilation, I think it's just represented the same way an assembly-qualified reference written in IL is represented.

The extern alias statement is only for the C# language to know which alias names should be available in the current file. Not sure why they aren't just globally available (maybe was something to due with netmodules back in the day?)

[trylek]

I've become able to expand my ilproj analyzer to run larger tests and I hit an interesting new twist: as you probably remember, last year Steve MacLean refactored platform-specific tests to use variant projects, e.g. here:

https://github.com/dotnet/runtime/blob/main/src/tests/JIT/Regression/JitBlue/DevDiv_278523/DevDiv_278523_Target_32Bit.ilproj
https://github.com/dotnet/runtime/blob/main/src/tests/JIT/Regression/JitBlue/DevDiv_278523/DevDiv_278523_Target_64Bit.ilproj

I think this particular combo is relatively easy to implement but there are other similar tests that only apply to certain OSes, architectures and / or other build modes. As these tests only build conditionally, their output assembly also gets only produced conditionally and to my knowledge there's no equivalent functionality to make the "extern alias" clauses conditional - when the conditional assembly is not produced (because the current build configuration doesn't support the test), the extern alias becomes kind of orphaned and fails the build. This is definitely a problem for constructing a target-agnostic wrapper. I'm almost done for today but I make it my priority to figure out the solution for this case tomorrow.

[jkoritzinsky]

For the arch-specific tests, you can add a ConditionalFact attribute that points to a property that checks if the pointer size is 32 or 64 bit. For OSes, the SkipOnPlatform attribute can be used to skip on specific operating systems.

Basically we'll always compile the tests, but we'll either conditionally compile them into the runner, or conditionally execute them based on the detected platform.

[trylek]

Looks great to me, thanks Jeremy! I haven't noticed any major blocking issues, just a few nitpicks.

[trylek]

Would it be a big problem to postpone the platform / runtime check to the actual runtime execution? We can do it in a future change but ideally we should be able to use the same managed build for Mono and CoreCLR (I don't believe we're doing that now, we have two different legs for that purpose but that's also cost that could be reduced).

[jkoritzinsky]

The only mechanism known today to detect runtime flavor at runtime easily is though reflection, which is very heavyweight for the src/tests test tree. I'm trying to keep the platform checks as cheap as possible so as to not make the "bring up" tests in the JIT significantly less useful, which is why I try to hard-code them at generation time when possible.

[trylek]

I see. I think it's OK to leave as is for now, I can look into it as part of the new-style Helix publishing; in particular I believe we're setting up about two dozens of environment variables that get set on the Helix execution machine and these include target architecture and OS so I guess we could use these for the runtime checks. We can also add an arbitrary number of additional environment variables as we need to express any of the execution characteristics.

[trylek]

Would it be a big problem to postpone the platform / runtime check to the actual runtime execution? We can do it in a future change but ideally we should be able to use the same managed build for Mono and CoreCLR (I don't believe we're doing that now, we have two different legs for that purpose but that's also cost that could be reduced).

[jkoritzinsky]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jkoritzinsky]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jkoritzinsky]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).