Add code coverage support to Arcade SDK with VSTest integration

[Copilot]

Summary

This PR implements comprehensive code coverage support for Arcade SDK test projects using VSTest and coverlet.collector, addressing issue #1027.

Problem

The original issue highlighted that the current XUnit test runner implementation (dotnet exec $(TestRunnerPath)) prevents using packages like coverlet.msbuild for code coverage collection. Repositories using Arcade SDK had no built-in way to collect and report code coverage, making it difficult to measure test effectiveness and integrate with Azure DevOps pipelines.

Solution

This PR adds native code coverage support by leveraging:

1. VSTest runner - Already supported in Arcade SDK via UseVSTestRunner property
2. coverlet.collector - Cross-platform code coverage library that works with VSTest
3. Automatic configuration - Generates .runsettings files with coverage settings

Usage

Enable code coverage in your test project with just two properties:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <UseVSTestRunner>true</UseVSTestRunner>
    <CollectCoverage>true</CollectCoverage>
  </PropertyGroup>
</Project>

Then run tests normally:

./build.sh --test

Coverage reports are automatically generated in artifacts/TestResults/coverage/ in Cobertura format (configurable).

Features

Code Coverage Properties

All properties are optional with sensible defaults:

• CollectCoverage - Enable/disable coverage (default: false)
• CodeCoverageFormat - Output format: cobertura, opencover, lcov, json, or combinations (default: cobertura)
• CodeCoverageOutputDirectory - Where to save reports (default: $(ArtifactsTestResultsDir)coverage)
• CoverageDeterministic - Deterministic reports (default: true)
• CoverageInclude - Assemblies to include (glob patterns)
• CoverageExclude - Assemblies to exclude (glob patterns)
• CoverageIncludeByFile - Files to include (glob patterns)
• CoverageExcludeByFile - Files to exclude (glob patterns)
• CoverageExcludeByAttribute - Attributes to exclude

Advanced Example

<PropertyGroup>
  <UseVSTestRunner>true</UseVSTestRunner>
  <CollectCoverage>true</CollectCoverage>
  <CodeCoverageFormat>cobertura,opencover</CodeCoverageFormat>
  <CoverageExclude>[*.Tests]*;[xunit.*]*;[Moq]*</CoverageExclude>
  <CoverageExcludeByFile>**/*Designer.cs;**/Generated/*.cs</CoverageExcludeByFile>
  <CoverageExcludeByAttribute>Obsolete;GeneratedCode</CoverageExcludeByAttribute>
</PropertyGroup>

Azure DevOps Integration

The Cobertura format is natively supported by Azure DevOps. Add this task to your pipeline:

- task: PublishCodeCoverageResults@2
  inputs:
    summaryFileLocation: '$(Build.SourcesDirectory)/artifacts/TestResults/coverage/**/coverage.cobertura.xml'
    codecoverageTool: 'cobertura'
  displayName: 'Publish Code Coverage Results'

Technical Details

Implementation

1. Tests.props - Defines code coverage properties and automatically adds coverlet.collector package reference when coverage is enabled
2. Tests.targets - Passes CollectCoverage metadata to TestToRun items
3. VSTest.targets - Adds --collect:"XPlat Code Coverage" to the test command and generates .runsettings file with coverage configuration

Automatic .runsettings Generation

When CollectCoverage=true and no custom .runsettings file is specified, the build automatically generates one with:

• Coverage format configuration
• Output path configuration
• Include/exclude filters
• Deterministic report settings
• Other coverlet options

Backward Compatibility

• No breaking changes - Coverage is disabled by default
• Existing tests work unchanged - No modifications needed to existing test projects
• Optional feature - Projects can continue using XUnit console runner if desired

Documentation

Added comprehensive documentation:

• ArcadeSdk.md - Detailed property descriptions and examples
• CodeCoverageExample.md - Complete usage guide with Azure DevOps integration examples

Testing

• ✅ Full repository build succeeds with no errors or warnings
• ✅ Existing test projects compile and run without modification
• ✅ Coverage properties correctly default and can be overridden
• ✅ Generated .runsettings files have valid XML structure

Addresses Issue #1027

This implementation provides:

• ✅ Built-in targets/properties for code coverage report generation
• ✅ Works out of the box with Arcade SDK
• ✅ Produces Cobertura format that Azure DevOps can ingest by default
• ✅ Simple opt-in (no workarounds needed like with coverlet.msbuild)

Fixes #1027

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add Code Coverage targets</issue_title>
<issue_description>It would be nice to add targets/properties to allow code coverage report generation out of the box in Arcade. Bonus points if it produces a format that AZDO pipelines can ingest by default.

After fiddling with things a bit it seems that the dotnet exec $(TestRunnerPath) method of executing tests used by the Xunit target prevents using packages like coverlet.msbuild, so an implementation in Arcade or documentation of a simple way to DIY would be ideal.</issue_description>

Comments on the Issue (you are @copilot in this section)

@markwilkie Any updates @sharwell ? @ViktorHofer Relates to https://github.com//issues/2076. After enabling full code coverage in corefx (with codecov) I can help out here. @markwilkie Thanks @ViktorHofer @missymessa Hi @ViktorHofer, I see that this issue is a bit old, so checking into see if any requirements have changed on this? Thanks! @ViktorHofer VSTest now supports all the feature that we need. I recommend to use dotnet test's coverlet integration instead of invoking coverlet directly. The necessary work here would be:

1. Add the option to run on VSTest instead of xunit.console.
2. When using VSTest, add the option to collect code coverage: dotnet test --collect:"XPlat Code Coverage" (https://github.com/tonerdo/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application)

Optional:

1. Add a step to aggregate all reports together with ReportGenerator
2. Upload the report to AzDO and/or codecov.io.</comment_new>
   <comment_new>@ViktorHofer
   I'm currently not actively working on these things...</comment_new>
   <comment_new>@missymessa
   @ViktorHofer Thanks for the update! I've implemented something similar to your suggestion in our services' CI for code coverage, too, so we'll see how much of that we can leverage for this addition to Arcade. </comment_new>
   <comment_new>@Evangelink
   Happy to help providing something standard using Microsoft Code Coverage if we are interested. I have tried to create a PR on roslyn-analyzers and it seems that the changes required for coverage only is really small.

Adding

<TestRunnerAdditionalArguments>--collect:"Code Coverage;Format=Cobertura"</TestRunnerAdditionalArguments>
    <!-- TODO: On next arcade bump of Microsoft.CodeCoverage/Microsoft.NET.Test.Sdk we will have to update this line to be netstandard2.0 -->
    <!-- Because arcade is passing dlls directly to 'dotnet test' command, we need to manually provide path to coverage datacollector -->
    <TestRunnerAdditionalArguments>$(TestRunnerAdditionalArguments) --test-adapter-path $(NUGET_PACKAGES)/microsoft.codecoverage/$(MicrosoftNETTestSdkVersion)/build/netstandard1.0</TestRunnerAdditionalArguments>

in some props/target file seems to be enough. Obviously, it would be better to also provide a .runsettings.</comment_new>
<comment_new>@rmarinho
So is there an easy way to add this ? </comment_new>

Fixes #1946

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Evangelink]

Code should probably ensure that coverlet NuGet package is installed (or at least assume that some property linked to enabling this feature is linked to enabling coverlet).

Note that we probably should also support Microsoft Code Coverage and MTP.

cc @Youssef1313

[Youssef1313]

I think it's very possible to install coverlet.msbuild instead, and have that working both for VSTest and MTP.

winforms and wpf are both already using coverlet.msbuild with MTP IIRC. But coverlet.msbuild won't work with dotnet test + MTP. It will only work with Arcade.

[Youssef1313]

Maybe we should have a CodeCoverageTool MSBuild property that can either be left empty (no coverage), or be set to either coverlet or MS.CC.

Additionally, we could have CodeCoverageMode that can be MSBuild, PlatformExtension, or GlobalTool, taking into considerations that not all modes are supported in all scenarios, and unsupported scenarios need to be handled by producing an error. For example, there is no PlatformExtension for coverlet when using MTP.