dotnet test completely ignores command line parameters

[karolz-ms]

Describe the bug

Seems like the latest version of dotnet test command completely ignores invocation parameters

To Reproduce

1. Clone Aspire repository https://github.com/dotnet/aspire/
2. cd <repo root>\tests\Aspire.Hosting.Valkey.Tests
3. dotnet test --list-tests

Expected

Should only list tests, not run them

Actual

Tests are being run. I also tried other invocation parameters like --filter and they seem to be completely ineffective too.

Further technical details

• Win 11 fully patched
• dotnet CLI version 10.0.25.25910 @Commit: c22dcd0c7a78d095a94d20e59ec0271b9924c82c that came with 10.0.0-preview.4.25258.110 @Commit: c22dcd0c7a78d095a94d20e59ec0271b9924c82c that came with Visual Studio 18.0.0 Preview 1.0 [10629.154.main]

dotnet --info output

PS C:\code\aspire\tests\Aspire.Hosting.Valkey.Tests> dotnet --info
.NET SDK:
 Version:           9.0.300
 Commit:            15606fe0a8
 Workload version:  9.0.300-manifests.15545612
 MSBuild version:   17.14.5+edd3bbf37

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.26100
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\9.0.300\

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
  Version:      10.0.0-preview.4.25258.110
  Architecture: x64
  Commit:       c22dcd0c7a

.NET SDKs installed:
  8.0.410 [C:\Program Files\dotnet\sdk]
  9.0.106 [C:\Program Files\dotnet\sdk]
  9.0.300 [C:\Program Files\dotnet\sdk]
  10.0.100-preview.4.25258.110 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 10.0.0-preview.4.25258.110 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 10.0.0-preview.4.25258.110 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 9.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 10.0.0-preview.4.25258.110 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  C:\code\aspire\global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[karolz-ms]

Also, I tried on a couple different machines, same effect.

[Viir]

Hey @karolz-ms, thank you for sharing your experience!

I was able to reproduce this problem, and also see a workaround.
To list the tests in Aspire.Hosting.Valkey.Tests, you can use the following workaround:

dotnet  run  --  --list-tests

On commit 63fd1f44a8da9f3e62f1855739320bdd378b7cc2 output is as follows:

xUnit.net v3 Microsoft.Testing.Platform Runner v2.0.2+ccd27e9792 (64-bit .NET 8.0.16)

The following Tests are available:
Aspire.Hosting.Valkey.Tests.AddValkeyTests.AddValkeyContainerWithDefaultsAddsAnnotationMetadata
Aspire.Hosting.Valkey.Tests.AddValkeyTests.AddValkeyContainerAddsAnnotationMetadata
Aspire.Hosting.Valkey.Tests.AddValkeyTests.ValkeyCreatesConnectionStringWithDefaultPassword
Aspire.Hosting.Valkey.Tests.AddValkeyTests.ValkeyCreatesConnectionStringWithPassword
Aspire.Hosting.Valkey.Tests.AddValkeyTests.ValkeyCreatesConnectionStringWithPasswordAndPort
Aspire.Hosting.Valkey.Tests.AddValkeyTests.VerifyWithoutPasswordManifest
Aspire.Hosting.Valkey.Tests.AddValkeyTests.VerifyWithPasswordManifest
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataVolumeAddsVolumeAnnotation(isReadOnly: null)
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataVolumeAddsVolumeAnnotation(isReadOnly: True)
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataVolumeAddsVolumeAnnotation(isReadOnly: False)
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataBindMountAddsMountAnnotation(isReadOnly: null)
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataBindMountAddsMountAnnotation(isReadOnly: True)
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataBindMountAddsMountAnnotation(isReadOnly: False)
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataVolumeAddsPersistenceAnnotation
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataVolumeDoesNotAddPersistenceAnnotationIfIsReadOnly
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataBindMountAddsPersistenceAnnotation
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithDataBindMountDoesNotAddPersistenceAnnotationIfIsReadOnly
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithPersistenceReplacesPreviousAnnotationInstances
Aspire.Hosting.Valkey.Tests.AddValkeyTests.WithPersistenceAddsCommandLineArgsAnnotation
Aspire.Hosting.Valkey.Tests.AddValkeyTests.AddValkeyContainerWithPasswordAnnotationMetadata
Aspire.Hosting.Valkey.Tests.ValkeyFunctionalTests.VerifyValkeyResource
Aspire.Hosting.Valkey.Tests.ValkeyFunctionalTests.WithDataShouldPersistStateBetweenUsages(useVolume: True)
Aspire.Hosting.Valkey.Tests.ValkeyFunctionalTests.WithDataShouldPersistStateBetweenUsages(useVolume: False)
Aspire.Hosting.Valkey.Tests.ValkeyFunctionalTests.VerifyWaitForOnValkeyBlocksDependentResources
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.AddValkeyShouldThrowWhenBuilderIsNull
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.AddValkeyShouldThrowWhenNameIsNullOrEmpty(isNull: True)
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.AddValkeyShouldThrowWhenNameIsNullOrEmpty(isNull: False)
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.WithDataVolumeShouldThrowWhenBuilderIsNull
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.WithDataBindMountShouldThrowWhenBuilderIsNull
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.WithDataBindMountShouldThrowWhenSourceIsNullOrEmpty(isNull: True)
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.WithDataBindMountShouldThrowWhenSourceIsNullOrEmpty(isNull: False)
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.WithPersistenceBindMountShouldThrowWhenBuilderIsNull
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.CtorValkeyResourceShouldThrowWhenNameIsNullOrEmpty(isNull: True)
Aspire.Hosting.Valkey.Tests.ValkeyPublicApiTests.CtorValkeyResourceShouldThrowWhenNameIsNullOrEmpty(isNull: False)

To see more available options of the command line interface, use dotnet run -- --help

This issue might be in the scope of #45927

For more related discussions, see:

• Enable MTP tests via MSBuild property aspire#9361
• Revert to MTP and disable server capability as a workaround aspire#8833

[karolz-ms]

😲 thank you @Viir I would never have guessed this is a side effect of using the new test platform. Will try the workaround. Hope that #45927 gets some love soon.

@radical @danmoseley @joperezr FYI

[Youssef1313]

This behavior is expected and is well documented in https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-dotnet-test.

Now that Aspire is using .NET 10 SDK though, it should start moving to the new dotnet test experience we shipped in .NET 10 SDK. But there is nothing wrong on dotnet test side. Hence I'm closing this.