xunit.runner.visualstudio 2.8.1 fails to discover .Net 4.7.2 Unit Tests

[onur-ozguzel]

Hello,

For a long time, we were using xunit.runner.visualstudio 2.4.5. vstest.console.exe was able to detect both our .NET 6.0 and .NET Framework 4.7.2 unit tests with these parameters:

vstest.console.exe "C:\a\2\s<Full path of first test assembly>"
"C:\a\2\s<Full path of the second test assembly>"
"C:\a\2\s<Full path of the third test assembly>"
/TestCaseFilter:"Category=UnitTest"
/Settings:"C:\a_temp\lbxw3zg1gnt.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\a\2\s"
/Framework:.NETFramework,Version=v4.7.2

After we upgraded to xunit.runner.visualstudio 2.8.1, our .NET Framework unit test discoveries stopped working.

We tested all the new versions, the results were the same except 2.5.0. So discovery were not working correctly for 2.8.1 - 2.8.0 - 2.5.8 - 2.5.7 - 2.5.6 - 2.5.5 - 2.5.4 - 2.5.3 - 2.5.1.

The error message we were getting was:

No test matches the given testcase filter Category=UnitTest in "C:\a\2\s<Full path of first test assembly>"
No test matches the given testcase filter Category=UnitTest in "C:\a\2\s<Full path of second test assembly>"
No test matches the given testcase filter Category=UnitTest in "C:\a\2\s<Full path of third test assembly>"

For discovery to work again for .NET 4.7.2 Unit Tests, we had to change TestAdapterPath like below:

vstest.console.exe "C:\a\2\s<Full path of the first test assembly>"
C:\a\2\s<Full path of the second test assembly>"
C:\a\2\s<Full path of the third test assembly>"
/TestCaseFilter:"Category=UnitTest"
/Settings:"C:\a_temp\qfap23dg5er.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\a\2\s<Full folder path of one of the test assemblies>"
/Framework:.NETFramework,Version=v4.7.2

We have our project in C:\a\2\s folder. Meaning inside this folder, we have both .NET 6.0 and .NET Framework 4.7.2 Unit Tests and Test adapter assemblies. Our hunch is, after xunit.runner.visualstudio 2.4.5 and 2.5.0, vstest.console.exe started to find the wrong test adapter for .NET Framework 4.7.2 tests if we provide the base path, and because of that it failed to discover them.

In the newer versions of xunit.runner.visualstudio, can we provide the base path of the TestAdapterPath and make it work just like it was working in 2.4.5 and 2.5.0?

PS: For .NET 6.0 unit test, it is still able to discover tests by providing the base path to the TestAdapterPath like below:

vstest.console.exe "C:\a\2\s<Full path of first test assembly>"
"C:\a\2\s<Full path of the second test assembly>"
"C:\a\2\s<Full path of the third test assembly>"
/TestCaseFilter:"Category=UnitTest"
/Settings:"C:\a_temp\wizpx4ngr2k.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\a\2\s"
/Framework:.NETCoreApp,Version=v6.0

[bradwilson]

Our hunch is, after xunit.runner.visualstudio 2.4.5 and 2.5.0, vstest.console.exe started to find the wrong test adapter for .NET Framework 4.7.2 tests if we provide the base path, and because of that it failed to discover them.

I suspect your hunch is correct, in which case the bug is in VSTest and not with us, because we don't control that directory-based adapter discovery logic.

I don't see much that changed between 2.5.0 and 2.5.1 that looks "suspect"; it was mostly pulling newer versions of xunit.runner.utility from the core project. If the VSTest wants to try to narrow it down on our side, we offered a few pre-release 2.5.1 packages that might help them.