Skip to content

Add code coverage support to Arcade SDK with VSTest integration #16206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Add code coverage support to Arcade SDK with VSTest integration #16206

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fd6abb4
Initial plan
Copilot Oct 10, 2025
76eecbb
Add code coverage support to Arcade SDK with VSTest integration
Copilot Oct 10, 2025
4703bcb
Fix runsettings generation to use ItemGroup for proper XML formatting
Copilot Oct 10, 2025
02a26bd
Add code coverage example documentation
Copilot Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions Documentation/ArcadeSdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1064,6 +1064,114 @@ To override the default Shared Framework version that is selected based on the t

Timeout to apply to an individual invocation of the test runner (e.g. `xunit.console.exe`) for a single configuration. Integer number of milliseconds.

### Code Coverage Properties

Arcade SDK supports code coverage collection when using the VSTest runner. To enable code coverage, set `UseVSTestRunner` to `true` and `CollectCoverage` to `true`.

#### `CollectCoverage` (bool)

Set to `true` to enable code coverage collection when running tests. Default is `false`.

```text
msbuild Project.UnitTests.csproj /t:Test /p:UseVSTestRunner=true /p:CollectCoverage=true
```

Or set it in your project file:

```xml
<PropertyGroup>
<UseVSTestRunner>true</UseVSTestRunner>
<CollectCoverage>true</CollectCoverage>
</PropertyGroup>
```

#### `CodeCoverageFormat` (string)

Specifies the output format for code coverage reports. Supported values: `cobertura`, `opencover`, `lcov`, `json`, or a combination like `cobertura,opencover`. Default is `cobertura`.

The Cobertura format is supported by Azure DevOps and can be published using the `PublishCodeCoverageResults` task.

#### `CodeCoverageOutputDirectory` (string)

Directory where code coverage reports will be generated. Default is `$(ArtifactsTestResultsDir)coverage`.

#### `CoverageDeterministic` (bool)

Enable or disable generation of deterministic coverage reports. Default is `true`.

#### `CoverageInclude` (string)

Semicolon-separated list of assembly patterns to include in code coverage. Uses glob patterns. Default is empty (includes all assemblies).

Example:
```xml
<CoverageInclude>[MyProject]*;[MyLibrary]*</CoverageInclude>
```

#### `CoverageExclude` (string)

Semicolon-separated list of assembly patterns to exclude from code coverage. Uses glob patterns. Default is empty.

Example:
```xml
<CoverageExclude>[*.Tests]*;[xunit.*]*</CoverageExclude>
```

#### `CoverageIncludeByFile` (string)

Semicolon-separated list of file path patterns to include in code coverage. Uses glob patterns. Default is empty.

#### `CoverageExcludeByFile` (string)

Semicolon-separated list of file path patterns to exclude from code coverage. Uses glob patterns. Default is empty.

Example:
```xml
<CoverageExcludeByFile>**/*Designer.cs;**/Generated/*.cs</CoverageExcludeByFile>
```

#### `CoverageExcludeByAttribute` (string)

Semicolon-separated list of attributes to exclude from code coverage. Default is empty.

Example:
```xml
<CoverageExcludeByAttribute>Obsolete;GeneratedCode;CompilerGenerated</CoverageExcludeByAttribute>
```

#### Example: Complete Code Coverage Configuration

```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<UseVSTestRunner>true</UseVSTestRunner>
<CollectCoverage>true</CollectCoverage>
<CodeCoverageFormat>cobertura</CodeCoverageFormat>
<CoverageExclude>[*.Tests]*;[xunit.*]*</CoverageExclude>
<CoverageExcludeByFile>**/*Designer.cs</CoverageExcludeByFile>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\MyProject\MyProject.csproj" />
</ItemGroup>
</Project>
```

Then run tests with:
```text
./build.sh --test
```

Code coverage reports will be generated in `artifacts/TestResults/coverage/` directory in Cobertura format, which can be published to Azure DevOps using the `PublishCodeCoverageResults` task in your pipeline:

```yaml
- task: PublishCodeCoverageResults@2
inputs:
summaryFileLocation: '$(Build.SourcesDirectory)/artifacts/TestResults/coverage/**/coverage.cobertura.xml'
codecoverageTool: 'cobertura'
```

### `GenerateResxSource` (bool)

When set to `true`, Arcade will generate a class source for all embedded .resx files.
Expand Down
108 changes: 108 additions & 0 deletions Documentation/CodeCoverageExample.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Code Coverage Example for Arcade SDK

This example demonstrates how to enable code coverage collection in test projects using Arcade SDK.

## Prerequisites

- Arcade SDK 11.0.0 or later
- Test project using XUnit

## Basic Configuration

Add the following properties to your test project file:

```xml
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>

<!-- Enable VSTest runner instead of XUnit console runner -->
<UseVSTestRunner>true</UseVSTestRunner>

<!-- Enable code coverage collection -->
<CollectCoverage>true</CollectCoverage>

<!-- Optional: Specify coverage format (default is cobertura) -->
<CodeCoverageFormat>cobertura</CodeCoverageFormat>
</PropertyGroup>
</Project>
```

## Running Tests with Coverage

Run tests using the build script:

```bash
./build.sh --test
```

Or using MSBuild directly:

```bash
dotnet build /t:Test /p:Configuration=Release
```

## Coverage Reports

Code coverage reports will be generated in the `artifacts/TestResults/coverage/` directory.

For Cobertura format, the report file will be named `coverage.cobertura.xml` and can be published to Azure DevOps.

## Azure DevOps Integration

Add the following task to your Azure Pipelines YAML to publish coverage results:

```yaml
- task: PublishCodeCoverageResults@2
inputs:
summaryFileLocation: '$(Build.SourcesDirectory)/artifacts/TestResults/coverage/**/coverage.cobertura.xml'
codecoverageTool: 'cobertura'
displayName: 'Publish Code Coverage Results'
```

## Advanced Configuration

### Filtering Coverage

Exclude specific assemblies or files from coverage:

```xml
<PropertyGroup>
<!-- Exclude test assemblies and external dependencies -->
<CoverageExclude>[*.Tests]*;[xunit.*]*;[Moq]*</CoverageExclude>

<!-- Exclude generated files -->
<CoverageExcludeByFile>**/*Designer.cs;**/Generated/*.cs</CoverageExcludeByFile>

<!-- Exclude obsolete code -->
<CoverageExcludeByAttribute>Obsolete;GeneratedCode;CompilerGenerated</CoverageExcludeByAttribute>
</PropertyGroup>
```

### Multiple Output Formats

Generate coverage in multiple formats:

```xml
<PropertyGroup>
<CodeCoverageFormat>cobertura,opencover,lcov</CodeCoverageFormat>
</PropertyGroup>
```

## Troubleshooting

### Coverage not collected

- Ensure `UseVSTestRunner` is set to `true`
- Verify `CollectCoverage` is set to `true`
- Check that `coverlet.collector` package is restored (should be automatic)

### Empty coverage report

- Make sure tests are actually running and passing
- Verify the test project has a reference to the code you want to cover

## See Also

- [Arcade SDK Documentation](../Documentation/ArcadeSdk.md)
- [Coverlet Documentation](https://github.com/coverlet-coverage/coverlet)
35 changes: 35 additions & 0 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/Tests.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,39 @@
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="$(MicrosoftNETTestSdkVersion)" IsImplicitlyDefined="true" />
</ItemGroup>

<!-- Code Coverage Support -->
<PropertyGroup Condition="'$(IsTestProject)' == 'true'">
<!-- Enable code coverage collection. Set to 'true' to collect code coverage when running tests. -->
<CollectCoverage Condition="'$(CollectCoverage)' == ''">false</CollectCoverage>

<!-- Code coverage output format. Supported values: 'cobertura', 'opencover', 'lcov', 'json', or combination like 'cobertura,opencover'. -->
<CodeCoverageFormat Condition="'$(CodeCoverageFormat)' == ''">cobertura</CodeCoverageFormat>

<!-- Directory where code coverage reports will be generated. -->
<CodeCoverageOutputDirectory Condition="'$(CodeCoverageOutputDirectory)' == ''">$(ArtifactsTestResultsDir)coverage</CodeCoverageOutputDirectory>

<!-- Enable/disable generation of deterministic coverage reports. -->
<CoverageDeterministic Condition="'$(CoverageDeterministic)' == ''">true</CoverageDeterministic>

<!-- Assemblies to include in code coverage (semicolon-separated patterns). Default is all assemblies. -->
<CoverageInclude Condition="'$(CoverageInclude)' == ''"></CoverageInclude>

<!-- Assemblies to exclude from code coverage (semicolon-separated patterns). -->
<CoverageExclude Condition="'$(CoverageExclude)' == ''"></CoverageExclude>

<!-- Files to include in code coverage (semicolon-separated patterns). -->
<CoverageIncludeByFile Condition="'$(CoverageIncludeByFile)' == ''"></CoverageIncludeByFile>

<!-- Files to exclude from code coverage (semicolon-separated patterns). -->
<CoverageExcludeByFile Condition="'$(CoverageExcludeByFile)' == ''"></CoverageExcludeByFile>

<!-- Attributes to exclude from code coverage (semicolon-separated). -->
<CoverageExcludeByAttribute Condition="'$(CoverageExcludeByAttribute)' == ''"></CoverageExcludeByAttribute>
</PropertyGroup>

<!-- Add coverlet.collector package when code coverage is enabled and using VSTest -->
<ItemGroup Condition="'$(IsTestProject)' == 'true' and '$(CollectCoverage)' == 'true' and '$(UseVSTestRunner)' == 'true'">
<PackageReference Include="coverlet.collector" IsImplicitlyDefined="true" PrivateAssets="all" IncludeAssets="runtime; build; native; contentfiles; analyzers; buildtransitive" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's very possible to install coverlet.msbuild instead, and have that working both for VSTest and MTP.

winforms and wpf are both already using coverlet.msbuild with MTP IIRC. But coverlet.msbuild won't work with dotnet test + MTP. It will only work with Arcade.

</ItemGroup>

</Project>
1 change: 1 addition & 0 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/Tests.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@
<TestRunnerAdditionalArguments>$(TestRunnerAdditionalArguments)</TestRunnerAdditionalArguments>
<RunArguments>$(RunArguments)</RunArguments>
<RunCommand>$(RunCommand)</RunCommand>
<CollectCoverage>$(CollectCoverage)</CollectCoverage>
</TestToRun>
</ItemGroup>
</Target>
Expand Down
72 changes: 72 additions & 0 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/VSTest.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@

<_TestRunnerCommand>&quot;$(DotNetTool)&quot; test $(_TestAssembly) --logger:"console%3Bverbosity=normal" --logger:"trx%3BLogFileName=$(_TestResultTrxFileName)" --logger:"html%3BLogFileName=$(_TestResultHtmlFileName)" "--ResultsDirectory:$(_TestResultDirectory)" "--Framework:%(TestToRun.TargetFrameworkIdentifier),Version=%(TestToRun.TargetFrameworkVersion)"</_TestRunnerCommand>
<_TestRunnerCommand Condition="'%(TestToRun.TestRunSettingsFile)' != ''">$(_TestRunnerCommand) "--settings:%(TestToRun.TestRunSettingsFile)"</_TestRunnerCommand>
<_TestRunnerCommand Condition="'%(TestToRun.CollectCoverage)' == 'true'">$(_TestRunnerCommand) --collect:"XPlat Code Coverage"</_TestRunnerCommand>
<_TestRunnerCommand Condition="'$(_TestRunnerAdditionalArguments)' != ''">$(_TestRunnerCommand) $(_TestRunnerAdditionalArguments)</_TestRunnerCommand>

<!--
Expand Down Expand Up @@ -81,10 +82,81 @@
</ItemGroup>
</Target>

<!-- Generate .runsettings file with code coverage configuration when code coverage is enabled -->
<Target Name="_GenerateCodeCoverageRunSettings"
BeforeTargets="_AddVSTestSpecificSettingsToInnerBuild"
Condition="'$(CollectCoverage)' == 'true' and '$(TestRunSettingsFile)' == ''">

<PropertyGroup>
<_GeneratedRunSettingsFile>$(ArtifactsObjDir)$(MSBuildProjectName).runsettings</_GeneratedRunSettingsFile>

<!-- Build Include filter -->
<_CoverageIncludeFilter Condition="'$(CoverageInclude)' != ''">$([System.String]::new('$(CoverageInclude)').Replace(';',','))</_CoverageIncludeFilter>

<!-- Build Exclude filter -->
<_CoverageExcludeFilter Condition="'$(CoverageExclude)' != ''">$([System.String]::new('$(CoverageExclude)').Replace(';',','))</_CoverageExcludeFilter>

<!-- Build IncludeByFile filter -->
<_CoverageIncludeByFileFilter Condition="'$(CoverageIncludeByFile)' != ''">$([System.String]::new('$(CoverageIncludeByFile)').Replace(';',','))</_CoverageIncludeByFileFilter>

<!-- Build ExcludeByFile filter -->
<_CoverageExcludeByFileFilter Condition="'$(CoverageExcludeByFile)' != ''">$([System.String]::new('$(CoverageExcludeByFile)').Replace(';',','))</_CoverageExcludeByFileFilter>

<!-- Build ExcludeByAttribute filter -->
<_CoverageExcludeByAttributeFilter Condition="'$(CoverageExcludeByAttribute)' != ''">$([System.String]::new('$(CoverageExcludeByAttribute)').Replace(';',','))</_CoverageExcludeByAttributeFilter>
</PropertyGroup>

<!-- Generate the runsettings file content dynamically -->
<ItemGroup>
<_RunSettingsLines Include="&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;" />
<_RunSettingsLines Include="&lt;RunSettings&gt;" />
<_RunSettingsLines Include=" &lt;DataCollectionRunSettings&gt;" />
<_RunSettingsLines Include=" &lt;DataCollectors&gt;" />
<_RunSettingsLines Include=" &lt;DataCollector friendlyName=&quot;XPlat code coverage&quot;&gt;" />
<_RunSettingsLines Include=" &lt;Configuration&gt;" />
<_RunSettingsLines Include=" &lt;Format&gt;$(_CodeCoverageFormat)&lt;/Format&gt;" />
<_RunSettingsLines Include=" &lt;OutputPath&gt;$(_CodeCoverageOutputPath)&lt;/OutputPath&gt;" />
<_RunSettingsLines Include=" &lt;DeterministicReport&gt;$(_CoverageDeterministic)&lt;/DeterministicReport&gt;" />
<_RunSettingsLines Include=" &lt;Include&gt;$(_CoverageIncludeFilter)&lt;/Include&gt;" Condition="'$(_CoverageIncludeFilter)' != ''" />
<_RunSettingsLines Include=" &lt;Exclude&gt;$(_CoverageExcludeFilter)&lt;/Exclude&gt;" Condition="'$(_CoverageExcludeFilter)' != ''" />
<_RunSettingsLines Include=" &lt;ExcludeByFile&gt;$(_CoverageExcludeByFileFilter)&lt;/ExcludeByFile&gt;" Condition="'$(_CoverageExcludeByFileFilter)' != ''" />
<_RunSettingsLines Include=" &lt;IncludeTestAssembly&gt;false&lt;/IncludeTestAssembly&gt;" />
<_RunSettingsLines Include=" &lt;ExcludeByAttribute&gt;$(_CoverageExcludeByAttributeFilter)&lt;/ExcludeByAttribute&gt;" Condition="'$(_CoverageExcludeByAttributeFilter)' != ''" />
<_RunSettingsLines Include=" &lt;SkipAutoProps&gt;true&lt;/SkipAutoProps&gt;" />
<_RunSettingsLines Include=" &lt;DoesNotReturnAttribute&gt;DoesNotReturnAttribute&lt;/DoesNotReturnAttribute&gt;" />
<_RunSettingsLines Include=" &lt;SingleHit&gt;false&lt;/SingleHit&gt;" />
<_RunSettingsLines Include=" &lt;UseSourceLink&gt;true&lt;/UseSourceLink&gt;" />
<_RunSettingsLines Include=" &lt;/Configuration&gt;" />
<_RunSettingsLines Include=" &lt;/DataCollector&gt;" />
<_RunSettingsLines Include=" &lt;/DataCollectors&gt;" />
<_RunSettingsLines Include=" &lt;/DataCollectionRunSettings&gt;" />
<_RunSettingsLines Include="&lt;/RunSettings&gt;" />
</ItemGroup>

<MakeDir Directories="$(ArtifactsObjDir)" />
<WriteLinesToFile File="$(_GeneratedRunSettingsFile)"
Lines="@(_RunSettingsLines)"
Overwrite="true"
WriteOnlyWhenDifferent="true" />

<PropertyGroup>
<TestRunSettingsFile>$(_GeneratedRunSettingsFile)</TestRunSettingsFile>
</PropertyGroup>

<ItemGroup>
<FileWrites Include="$(_GeneratedRunSettingsFile)" />
</ItemGroup>
</Target>

<!-- Set VSTest specific settings in a target so that the TestToRun item can read from it and customers can set it at any time during evaluation. -->
<Target Name="_AddVSTestSpecificSettingsToInnerBuild" BeforeTargets="_InnerGetTestsToRun">
<PropertyGroup>
<TestRunSettingsFile Condition="'$(TestRunSettingsFile)' == ''">$(VSTestRunSettingsFile)</TestRunSettingsFile>

<!-- Set code coverage output format and path for coverlet -->
<_CodeCoverageFormat Condition="'$(CollectCoverage)' == 'true'">$(CodeCoverageFormat)</_CodeCoverageFormat>
<_CodeCoverageOutputPath Condition="'$(CollectCoverage)' == 'true'">$(CodeCoverageOutputDirectory)</_CodeCoverageOutputPath>
<_CoverageDeterministic Condition="'$(CollectCoverage)' == 'true'">$(CoverageDeterministic)</_CoverageDeterministic>
</PropertyGroup>
</Target>

Expand Down
Loading