Update xUnit.net testing template to support xUnit.net v3

[DamianEdwards]

xUnit.net released their long-awaited v3. We need to decide how to support it in our Aspire xUnit.net testing project template. Some options:

1. Add support for selecting the version of xUnit.net to use as a template option, i.e. v2 or v3. The single template would cover both versions and we'd make v3 the default.
2. Add an additional template for xUnit.net v3 and update the name and description of the existing template to clearly indicate it's for v2
3. Update the existing template xUnit.net to use v3 and don't provide an option or separate template for xUnit.net v2

My thought is we go with option 1 and add a new option that allows the selection of the xUnit.net version to use. We can consider removing the v2 option at some time in the future.

@bradwilson interested in your thoughts. Seems there are going to be no bug fixes for v2 but as you point out some folks might still have valid reasons for using it until the ecosystem has reacted to the v3 release.

[DamianEdwards]

Another factor to consider is xUnit.net v3 supports the new Microsoft Test Platform but it's optional. Should this be an option presented in the template? What should the default be?

[bradwilson]

Seems there are going to be no bug fixes for v2 but as you point out some folks might still have valid reasons for using it until the ecosystem has reacted to the v3 release.

That is currently the plan, yes. I anticipate 2.9.3 is the final v2 release. It's hard to say for sure, but at this point the bar for show-stopper bug is incredibly high.

My thought is we go with option 1

I like option 1.

Another factor to consider is xUnit.net v3 supports the new Microsoft Test Platform but it's optional. Should this be an option presented in the template? What should the default be?

Something to consider, combining this with option 1, is that the option for M.T.P on/off is only valid if you're using v3. There is no plan to support M.T.P for v2 outside of an issue whose implementation is owned by the VSTest team, which they've gone silent on and I presume is actually dead at this point. I'm honestly not even sure it's doable the way they're hoping to do it.

I assume if you flip the M.T.P option, it would turn on both TestingPlatformDotnetTestSupport and UseMicrosoftTestingPlatformRunner. The former is what enables current dotnet test to use M.T.P, and the latter changes how dotnet run behaves with v3 test projects (giving you the M.T.P command line options when it's enabled, vs. the native xUnit.net command line options when it's disabled).

I don't really have a gut feeling for how many people want this option to be on, because I have no telemetry. I have gotten a couple issues from people who turned it on and then didn't understand that VSTest-specific options (like trying to specify a VSTest-style filter via dotnet test, or trying to use a .runsettings file) would no longer work. I have done my best to ensure as much of the "native" xUnit.net experience is available even via M.T.P (for example, supporting all of our report formats like CTRF or JUnit are available, ensuring our CI experience with TeamCity/AppVeyor/Azure DevOps still works, etc.). It really comes down to whether developers will intend to dotnet run their tests (rather than dotnet test them), and right now it feels like people don't really know enough about this execution mode to even know whether they should care about it (and whether they would prefer the new standardized-ish command line experience that comes with enabling M.T.P).

[bradwilson]

FYI there is an open issue regarding dotnet test experience in .NET 10 with M.T.P test projects, which presumably will deprecate the usage of TestingPlatformDotnetTestSupport: dotnet/sdk#45927

[DamianEdwards]

Thanks for your input @bradwilson. As well as option 1 RE the xUnit.net version, I'm leaning to adding an option to allow the user to select the testing platform but defaulting it to the traditional/xUnit option, so users have to opt-in to using Microsoft Test Platform. As you point out, that would only be applicable if xUnit v3 was selected. Do you think this should be a simple on/off option named something like "Use Microsoft Test Platform"? Or should it be a select with choices for "Microsoft Test Platform" and.... whatever we call not using that 😄?

[bradwilson]

You could simplify the choices by collapsing them into a tri-mode on the version, something like:

--xunitVersion v2
--xunitVersion v3
--xunitVersion v3mtp

[bradwilson]

FWIW, our templates just unfold into native mode with commented out stuff. I haven't considered whether I would want to add additional switches or not (first up would be --framework almost assuredly):

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <OutputType>Exe</OutputType>
    <RootNamespace>TestProject</RootNamespace>
    <TargetFramework>net8.0</TargetFramework>
    <!--
    To enable the Microsoft Testing Platform 'dotnet test' experience, add property:
      <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>

    To enable the Microsoft Testing Platform native command line experience, add property:
      <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>

    For more information on Microsoft Testing Platform support in xUnit.net, please visit:
      https://xunit.net/docs/getting-started/v3/microsoft-testing-platform
    -->
  </PropertyGroup>

  <ItemGroup>
    <Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>

  <ItemGroup>
    <Using Include="Xunit" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="MICROSOFT_NET_TEST_SDK_VERSION" />
    <PackageReference Include="xunit.v3" Version="PACKAGE_VERSION" />
    <PackageReference Include="xunit.runner.visualstudio" Version="XUNIT_VISUAL_STUDIO_RUNNER_VERSION" />
  </ItemGroup>

</Project>