Honouring Test Explorer Filter to determine tests for FCC coverage runs

[KeithWalters]

Installed product versions

• Visual Studio: 2022 Professional
• This extension: 1.1.173

Description

I've been trying to find a way to reduce the number of tests that are run by FCC for its coverage reporting.

From what I can tell, the inclusion/exclusion available in FCC settings applies to what Code is instrumented, rather than what tests are run.

It would be quite helpful if the filtering that, as a user, I have applied in Test Explorer could be used (optionally) as the set of tests that FCC runs.

Steps to recreate

1. Create a solution containing a project with code, and a Test project with more than one test, maybe with different Traits
2. Build the solution
3. Using Test Explorer, filter out 1 or more tests (e.g. using a Trait)
4. Run all filtered tests in Test Explorer

Current behavior

All tests that are discovered across the whole solution are run by FCC in the background to establish coverage, including tests that were filtered out in Test Explorer.

Expected behavior

A setting/option to use the Test Explorer-configured filter to decide which tests to run for FCC purposes. When this setting is on, FCC runs coverage based only on that subset of tests.

Apologies in advance if this is a user education issue!

[tonyhallett]

Use the new ms code coverage.

[KeithWalters]

Hi Tony,

Apologies but I guess I'm missing a few steps in understanding your recommendation.

Does ms code coverage let me define which tests are run to produce coverage? If so, can you point me to how this is controlled?

I had a thought that it might be .runsettings related, but there doesn't seem to be formal support for defining a test filter in .runsettings - there are plenty of coverage filters, which I think just decide which assemblies, functions are reported, not which ones are used to run tests

[KeithWalters]

Done some more digging. With Run (Common):

• Enabled false
• RunMsCodeCoverage true

When I run a test in Test Explorer, it runs fine. Test output window shows:

Building Test Projects

========== Starting test run ==========

Executing:   -> [myTestCase]

========== Test run finished: 1 Tests (1 Passed, 0 Failed, 0 Skipped) run in 4 sec ==========

If I change Enabled to true and try and run the test again in Test Explorer:

Building Test Projects

Using automatically detected runsettings file(s). To learn more visit https://aka.ms/vs-runsettings.

========== Starting test run ==========

Failed to negotiate protocol, waiting for response timed out after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

Failed to negotiate protocol, waiting for response timed out after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

System.NullReferenceException: Object reference not set to an instance of an object.

   at Microsoft.VisualStudio.TestPlatform.Client.Execution.TestRunRequest.HandleTestRunComplete(TestRunCompleteEventArgs runCompleteArgs, TestRunChangedEventArgs lastChunkArgs, ICollection`1 runContextAttachments, ICollection`1 executorUris)

   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)

   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManagerWithDataCollection.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)

   at Microsoft.VisualStudio.TestPlatform.Client.Execution.TestRunRequest.ExecuteAsync()

   at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(IRequestData requestData, TestRunCriteria testRunCriteria, ITestRunEventsRegistrar testRunEventsRegistrar, TestPlatformOptions options)

   at Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.RunTests(TestRunRequestPayload testRunRequestPayload, ITestHostLauncher testHostLauncher, ITestRunEventsRegistrar testRunEventsRegistrar, ProtocolConfig protocolConfig)

   at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.<>c__DisplayClass23_0.<StartTestRun>b__0()

========== Test run aborted: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in < 1 ms ==========

It seems that FCC in mscodecoverage mode is affecting the ability to run the test in Test Explorer?

[tonyhallett]

Please can you share a solution that exhibits this behaviour

[tonyhallett]

It appears that this issue can arise with code coverage microsoft/vstest#1544 (comment)
Have you tried setting the environment variable ?

Perhaps this can help ? https://www.c-sharpcorner.com/article/how-to-resolve-issue-of-test-project-not-running-the-unit-tests-after-upgrade-to/

[tonyhallett]

Closing as I do not believe that the exception is FCC related and that the test explorer filter is honoured when using ms code coverage.

As for honouring the test explorer when not using ms code coverage. FCC uses reflection to obtain some Test Explorer information and from what I remember this information is not surfaced in the object structure that is available.
I may be wrong on this matter. Feel free to reflector / peek around if you believe it is still important and let me know if you do find this information.

[tonyhallett]

Reopening as

It should be possible for user to specify a logger in runsettings that will write the tests that were run and for that information to be used in the second ( coverage ) test run.

I was also hoping that it was not necessary to have a second test run if the coverlet data collector is specified in runsettings but all I seem to get is

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0" branch-rate="0" version="1.9" timestamp="1654622362" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
  <sources />
  <packages />
</coverage>

[tonyhallett]

Issue with coverlet data collector mentioned in previous comment is documented in coverlet-coverage/coverlet#1332 (comment)

So when that gets rectified or version 3.0.3 is used it will be possible to have a single run.

[tonyhallett]

@KeithWalters is Coverlet collecting coverage ?