Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src/tests tree test xunit-based source generated runner #60846

Merged
merged 39 commits into from
Nov 17, 2021
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
1bbfc9c
Add a basic xunit runner generator that generates a top-level-stateme…
jkoritzinsky Oct 22, 2021
6523a11
Fix some bugs in the generator. Convert the Interop/PInvoke/Vector2_3…
jkoritzinsky Oct 22, 2021
6143a48
Implement support for [Fact] methods that are instance methods on IDi…
jkoritzinsky Oct 22, 2021
9b9ae95
Import enum sources from XUnitExtensions and hook up support for some…
jkoritzinsky Oct 25, 2021
54e93eb
Add one more newline
jkoritzinsky Oct 25, 2021
14ca067
Add preliminary support for building an IL test runner using the sour…
jkoritzinsky Oct 25, 2021
149ffe2
Rename to make the logic clearer.
jkoritzinsky Oct 25, 2021
23b7975
Update src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.cs
jkoritzinsky Oct 25, 2021
623cc48
Add support for extern-alias with references (to handle conflicting n…
jkoritzinsky Oct 26, 2021
a20864e
Apply suggestions from code review
jkoritzinsky Nov 2, 2021
4a7ebe9
Merge branch 'main' of github.com:dotnet/runtime into xunit-runner-ge…
jkoritzinsky Nov 2, 2021
f5009c8
Update src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.cs
jkoritzinsky Nov 2, 2021
59960aa
Add the XUnit wrapper generator to the list of ambient project depend…
trylek Nov 2, 2021
191ac42
Fix nullability bug in XUnitWrapperGenerator
trylek Nov 2, 2021
a2bdb78
Merge branch 'main' of github.com:dotnet/runtime into xunit-runner-ge…
jkoritzinsky Nov 3, 2021
e597fec
Merge branch 'main' of github.com:dotnet/runtime into xunit-runner-ge…
jkoritzinsky Nov 8, 2021
e783f9e
Write out the exception that caused the test to fail to the console.
jkoritzinsky Nov 8, 2021
c978193
Convert the Castable test to use the generator
jkoritzinsky Nov 8, 2021
270144b
Support running the xunit wrapper generator in "standalone" mode for …
jkoritzinsky Nov 8, 2021
4c73d6f
Add first draft of result reporting infrastructure.
jkoritzinsky Nov 8, 2021
e81deda
Fix IL .assembly directive to be the assembly name that can be resolv…
jkoritzinsky Nov 9, 2021
4f9dcbb
Add test reporting for "merged test runner" assemblies.
jkoritzinsky Nov 10, 2021
d5961f6
Add Microsoft.DotNet.XUnitExtensions to the targetting pack.
jkoritzinsky Nov 10, 2021
4d412d0
Add support for the rest of ActiveIssueAttribute
jkoritzinsky Nov 10, 2021
377b8a0
Add support for more attributes and clean up code.
jkoritzinsky Nov 10, 2021
835f56d
Add support for SkipOnCoreClrAttribute.
jkoritzinsky Nov 10, 2021
c84c5d1
Merge branch 'main' of github.com:dotnet/runtime into xunit-runner-ge…
jkoritzinsky Nov 10, 2021
b7fb243
Support running tests with the [Fact] attribute that return 100 for s…
jkoritzinsky Nov 10, 2021
d49c1f4
Fix failures and update comment
jkoritzinsky Nov 10, 2021
1b19bed
Update src/tests/Common/XUnitWrapperGenerator/ITestInfo.cs
jkoritzinsky Nov 11, 2021
040877c
Fix handling of SkipOnCoreClrAttribute and SkipOnMonoAttribute with R…
jkoritzinsky Nov 11, 2021
d736995
Add support for running tests marked with <RequiresProcessIsolation>t…
jkoritzinsky Nov 12, 2021
a77d13c
Merge branch 'main' into xunit-runner-generator
jkoritzinsky Nov 12, 2021
04650fd
Add support for substring-based test filtering and add infrastructure…
jkoritzinsky Nov 15, 2021
8ef73bd
Integrate work from #61224 with some modifications based on offline d…
jkoritzinsky Nov 15, 2021
5881fa1
Fix Castable.csproj and fix SkipOnMono/SkipOnCoreCLR attribute support.
jkoritzinsky Nov 15, 2021
eccf29a
Work around tests that hit compiler limits. Move tests that used Xuni…
jkoritzinsky Nov 16, 2021
ccd08a2
Rewrite how superpmicollect sets up its testing to make it more frien…
jkoritzinsky Nov 16, 2021
db68ce4
Add back the _BuildSpmiTestProjectScripts target since I'm not sure w…
jkoritzinsky Nov 17, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions src/tests/Common/ILTestRunner/ILTestRunner.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net6.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="$(RepoRoot)/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
</ItemGroup>

<Import Project="$(RepoRoot)/src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.props" />

<ItemGroup>
<ProjectReference Include="$(RepoRoot)\src\tests\JIT\IL_Conformance\Old\directed\AutoInit.ilproj" />
</ItemGroup>
</Project>
118 changes: 118 additions & 0 deletions src/tests/Common/XUnitWrapperGenerator/ITestInfo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.Collections.Generic;
using Microsoft.CodeAnalysis;

namespace XUnitWrapperGenerator;

interface ITestInfo
{
string ExecutionStatement { get; }
}

sealed class StaticFactMethod : ITestInfo
{
public StaticFactMethod(IMethodSymbol method)
{
ExecutionStatement = $"{method.ContainingType.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat)}.{method.Name}();";
}

public string ExecutionStatement { get; }

public override bool Equals(object obj)
{
return obj is StaticFactMethod other && ExecutionStatement == other.ExecutionStatement;
}
}

sealed class InstanceFactMethod : ITestInfo
{
public InstanceFactMethod(IMethodSymbol method)
{
ExecutionStatement = $"using ({method.ContainingType.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat)} obj = new()) obj.{method.Name}();";
}

public string ExecutionStatement { get; }

public override bool Equals(object obj)
{
return obj is InstanceFactMethod other && ExecutionStatement == other.ExecutionStatement;
}
}

sealed class ConditionalTest : ITestInfo
{
public ConditionalTest(ITestInfo innerTest, string condition)
{
ExecutionStatement = $"if ({condition}) {{ {innerTest.ExecutionStatement} }}";
}

public string ExecutionStatement { get; }

public override bool Equals(object obj)
{
return obj is ConditionalTest other && ExecutionStatement == other.ExecutionStatement;
}
}

sealed class PlatformSpecificTest : ITestInfo
{
public PlatformSpecificTest(ITestInfo innerTest, Xunit.TestPlatforms platform)
{
List<string> platformCheckConditions = new();
if (platform.HasFlag(Xunit.TestPlatforms.Windows))
{
platformCheckConditions.Add("global::System.OperatingSystem.IsWindows()");
}
if (platform.HasFlag(Xunit.TestPlatforms.Linux))
{
platformCheckConditions.Add("global::System.OperatingSystem.IsLinux()");
}
if (platform.HasFlag(Xunit.TestPlatforms.OSX))
{
platformCheckConditions.Add("global::System.OperatingSystem.IsMacOS()");
}
if (platform.HasFlag(Xunit.TestPlatforms.illumos))
{
platformCheckConditions.Add(@"global::System.OperatingSystem.IsOSPlatform(""illumos"")");
}
if (platform.HasFlag(Xunit.TestPlatforms.Solaris))
{
platformCheckConditions.Add(@"global::System.OperatingSystem.IsOSPlatform(""Solaris"")");
}
if (platform.HasFlag(Xunit.TestPlatforms.Android))
{
platformCheckConditions.Add("global::System.OperatingSystem.IsAndroid()");
}
if (platform.HasFlag(Xunit.TestPlatforms.iOS))
{
platformCheckConditions.Add("global::System.OperatingSystem.IsIOS()");
jkoritzinsky marked this conversation as resolved.
Show resolved Hide resolved
}
if (platform.HasFlag(Xunit.TestPlatforms.tvOS))
{
platformCheckConditions.Add("global::System.OperatingSystem.IsAndroid()");
jkoritzinsky marked this conversation as resolved.
Show resolved Hide resolved
}
if (platform.HasFlag(Xunit.TestPlatforms.MacCatalyst))
{
platformCheckConditions.Add(@"global::System.OperatingSystem.IsOSPlatform(""maccatalyst"")");
jkoritzinsky marked this conversation as resolved.
Show resolved Hide resolved
}
if (platform.HasFlag(Xunit.TestPlatforms.Browser))
{
platformCheckConditions.Add(@"global::System.OperatingSystem.IsOSPlatform(""browser"")");
jkoritzinsky marked this conversation as resolved.
Show resolved Hide resolved
}
if (platform.HasFlag(Xunit.TestPlatforms.FreeBSD))
{
platformCheckConditions.Add(@"global::System.OperatingSystem.IsFreeBSD()");
}
if (platform.HasFlag(Xunit.TestPlatforms.NetBSD))
{
platformCheckConditions.Add(@"global::System.OperatingSystem.IsOSPlatform(""NetBSD"")");
safern marked this conversation as resolved.
Show resolved Hide resolved
}
ExecutionStatement = $"if ({string.Join(" || ", platformCheckConditions)}) {{ {innerTest.ExecutionStatement} }}";
}

public string ExecutionStatement { get; }
}

16 changes: 16 additions & 0 deletions src/tests/Common/XUnitWrapperGenerator/RuntimeConfiguration.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;

namespace Xunit
{
[Flags]
public enum RuntimeConfiguration
{
Any = ~0,
Checked = 1,
Debug = 1 << 1,
Release = 1 << 2
}
}
38 changes: 38 additions & 0 deletions src/tests/Common/XUnitWrapperGenerator/RuntimeTestModes.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;

namespace Xunit
{
[Flags]
public enum RuntimeTestModes
{
// Disable always when using coreclr runtime.
Any = ~0,

// We're running regular tests with no runtime stress modes
RegularRun = 1,

// JitStress, JitStressRegs, JitMinOpts and TailcallStress enable
// various modes in the JIT that cause us to exercise more code paths,
// and generate different kinds of code
JitStress = 1 << 1, // COMPlus_JitStress is set.
JitStressRegs = 1 << 2, // COMPlus_JitStressRegs is set.
JitMinOpts = 1 << 3, // COMPlus_JITMinOpts is set.
TailcallStress = 1 << 4, // COMPlus_TailcallStress is set.

// ZapDisable says to not use NGEN or ReadyToRun images.
// This means we JIT everything.
ZapDisable = 1 << 5, // COMPlus_ZapDisable is set.

// GCStress3 forces a GC at various locations, typically transitions
// to/from the VM from managed code.
GCStress3 = 1 << 6, // COMPlus_GCStress includes mode 0x3.

// GCStressC forces a GC at every JIT-generated code instruction,
// including in NGEN/ReadyToRun code.
GCStressC = 1 << 7, // COMPlus_GCStress includes mode 0xC.
AnyGCStress = GCStress3 | GCStressC // Disable when any GCStress is exercised.
}
}
26 changes: 26 additions & 0 deletions src/tests/Common/XUnitWrapperGenerator/TestPlatforms.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;

namespace Xunit
{
[Flags]
public enum TestPlatforms
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reflect on Microsoft.DotNet.XUnitExtensions in generator to avoid duplicating these types?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can, but using the Roslyn symbol APIs to reflect like that is not fast. I think eventually making this generator live in dotnet/arcade might be a better solution.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In future versions of the generator (v1 milestone of the new test infra), we need to support spitting out an XUnit-style results file as well as basic test filtering, so living in arcade would make more sense then.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be careful not to regress managed build time too significantly. While ultimately we'll optimize it by compiling tests as larger apps, it remains to be seen whether compiling tests in merged form makes sense for all pipelines - some of the GC stress pipelines come to mind where some tests are timing out already today even without merging - so that the "standalone" mode is likely to stay as a valid way of running tests at least for niche scenarios. If we pay the reflection startup cost just once per Roslyn generation of all the wrappers, it's probably just fine but paying the cost for every single test may be significant.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking in generator case, startup performance is not a major issue (e.g. we can discover types using reflection (or perhaps Roslyn symbol APIs) to collect required infos in a static constructor just once during the lifetime of compilation).

I don't like caching it like that due to the fact that it's technically not correct and can break some assumptions that Roslyn has about the lifetime of a generator (as Roslyn can throw them away or reuse them in many different ways including in compiler server scenarios)

{
Windows = 1,
Linux = 2,
OSX = 4,
FreeBSD = 8,
NetBSD = 16,
illumos= 32,
Solaris = 64,
iOS = 128,
tvOS = 256,
Android = 512,
Browser = 1024,
MacCatalyst = 2048,
AnyUnix = FreeBSD | Linux | NetBSD | OSX | illumos | Solaris | iOS | tvOS | MacCatalyst | Android | Browser,
Any = ~0
}
}
15 changes: 15 additions & 0 deletions src/tests/Common/XUnitWrapperGenerator/TestRuntimes.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;

namespace Xunit
safern marked this conversation as resolved.
Show resolved Hide resolved
{
[Flags]
public enum TestRuntimes
{
CoreCLR = 1,
Mono = 2,
Any = ~0
}
}
Loading