Skip to content

Commit ea35bff

Browse files
heathsannelo-msft
heaths
authored and
annelo-msft
committed
Enable code coverage (Azure#17067)
* Use VSTest integration for Coverlet Fixes Azure#15231. Relates to Azure#14427. * Do not ignore CompilerGeneratedAttribute See coverlet-coverage/coverlet#794 (comment) * Enable code coverage for PRs * Publish code coverage even for failed tests * Enable code coverage reports Resolves Azure#14427 by limiting reports to a single service directory (CIs) or per test project (dev environments). For CIs, full coverage reports of everything build will be uploaded. For dev environments, a summary HTML file will be output to test projects' TestResults directories. * Fix filefilters path for CI * Use absolute path for filefilters Same as dev support; reportgenerator documentation is not clear on when absolute or relative paths are required. * Use absolute path for reports * Update report directory to limit globbing * Disable code coverage of track 2 mgmt Opened Azure#17090 to track re-enabling once improved. * Generate report before uploading test results The exact same commands are working locally using the same versions of ReportGenerator, sans running the test publishing executable. There's also extra coverage files showing up that I'm wondering if it's responsible. * Temporarily upload all code coverage artifacts * Split between props and targets again At one point, this mostly worked. Going back to how I was split between props and targets before to see if that makes a difference. * Collect more information * Always define CoverletGetPathMap @clairernovotny recommended important some changes. These are working locally when simulating a CI, so 🤞. * Resolve PR feedback Fix extra (temporary) logging as well. * Use different variable to detect CI ContinuousIntegrationBuild wasn't defined for test projects. * Replace curly braces with Of in file names Fixes Azure#17164 * Removing extra logging * Renames files with curly braces Fixes Azure#17164 * Resolve PR feedback * Do not reformat HTML coverage report
1 parent ea6d9a8 commit ea35bff

File tree

53 files changed

+191
-59
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+191
-59
lines changed

‎.config/dotnet-tools.json‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"version": 1,
3+
"isRoot": true,
4+
"tools": {
5+
"dotnet-reportgenerator-globaltool": {
6+
"version": "4.8.0",
7+
"commands": [
8+
"reportgenerator"
9+
]
10+
}
11+
}
12+
}

‎CONTRIBUTING.md‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,8 +55,11 @@ Nuget package will be created in root directory under \artifacts\packages\Debug
5555
### Using the command line:
5656

5757
Run e.g. `msbuild eng\mgmt.proj /t:"Runtests" /p:Scope=Compute`
58-
In the above example _RunTests_ will build and run tests for Compute only or you can use command line CLI
59-
`dotnet test Compute\Microsoft.Azure.Management.Compute\tests\Microsoft.Azure.Management.Tests.csproj`
58+
In the above example _RunTests_ will build and run tests for Compute only or you can use command line CLI:
59+
60+
```bash
61+
dotnet test Compute\Microsoft.Azure.Management.Compute\tests\Microsoft.Azure.Management.Tests.csproj
62+
```
6063

6164
### Non-Windows command line build
6265

@@ -67,6 +70,22 @@ Now you can use the same command on non-windows as above for e.g. on Ubuntu you
6770
- `dotnet msbuild eng\mgmt.proj /t:CreateNugetPackage /p:scope=Compute`
6871
- `dotnet msbuild build.proj /t:Util /p:UtilityName=InstallPsModules`
6972

73+
### Code Coverage
74+
75+
If you want to enable code coverage reporting, on the command line pass `/p:CollectCoverage=true` like so:
76+
77+
```bash
78+
dotnet tool restore
79+
dotnet test /p:CollectCoverage=true
80+
```
81+
82+
On developers' machines, you can open `index.html` from within the `TestResults` directory in each of your test projects.
83+
Coverage reports can also be found in Azure Pipelines on the "Code Coverage" tab after a pull request validation build completes.
84+
All covered projects should have 70% or better test coverage.
85+
86+
By default, all _Azure.*_ libraries are covered, and any project that sets the `IsClientLibrary=true` MSBuild property.
87+
To exclude a project, set `ExcludeFromCodeCoverage=true` in the project's MSBuild properties before other targets are imported.
88+
7089
### Update build tools
7190

7291
Build tools are now downloaded as part of a nuget package under `root\restoredPackages\microsoft.internal.netsdkbuild.mgmt.tools`

‎eng/CodeCoverage.runsettings‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
<?xml version="1.0"?>
2+
<RunSettings>
3+
<DataCollectionRunSettings>
4+
<DataCollectors>
5+
<DataCollector friendlyName="XPlat Code Coverage" enabled="true">
6+
<Configuration>
7+
<Format>cobertura</Format>
8+
<ExcludeByAttribute>ExcludeFromCodeCoverageAttribute,GeneratedCodeAttribute,Obsolete</ExcludeByAttribute>
9+
<IncludeTestAssembly>false</IncludeTestAssembly>
10+
<SingleHit>false</SingleHit>
11+
<SkipAutoProps>true</SkipAutoProps>
12+
</Configuration>
13+
</DataCollector>
14+
<!-- Enable logging to diagnose test host failures -->
15+
<DataCollector friendlyName="blame" enabled="true" />
16+
</DataCollectors>
17+
</DataCollectionRunSettings>
18+
</RunSettings>

‎eng/CodeCoverage.targets‎

Lines changed: 64 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,70 @@
11
<Project>
2-
<ItemGroup Condition="'$(IsTestProject)' == 'true'">
3-
<PackageReference Include="coverlet.msbuild">
2+
<PropertyGroup>
3+
<_IsCodeCoverable Condition="'$(IsClientLibrary)' == 'true' and '$(IsMgmtClientLibrary)' != 'true'">true</_IsCodeCoverable>
4+
</PropertyGroup>
5+
6+
<PropertyGroup Condition="'$(CollectCoverage)' == 'true' and '$(_IsCodeCoverable)' == 'true' and '$(IsTestProject)' == 'true' and '$(ExcludeFromCodeCoverage)' != 'true'">
7+
<CodeCoverageDirectory Condition="'$(CodeCoverageDirectory)' == ''">$([System.IO.Path]::GetFullPath("$(MSBuildProjectDirectory)\.."))</CodeCoverageDirectory>
8+
<SkipCoverageReport Condition="'$(SkipCoverageReport)' == '' and ('$(ContinuousIntegrationBuild)' == 'true' or '$(TF_BUILD)' == 'true')">true</SkipCoverageReport>
9+
<VSTestCollect Condition="'$(VSTestCollect)' == ''">XPlat Code Coverage</VSTestCollect>
10+
<VSTestSetting Condition="'$(VSTestSetting)' == ''">$(MSBuildThisFileDirectory)CodeCoverage.runsettings</VSTestSetting>
11+
<_CollectCoverage>true</_CollectCoverage>
12+
<_TestResultsDirectory>$(MSBuildProjectDirectory)\TestResults</_TestResultsDirectory>
13+
</PropertyGroup>
14+
15+
<ItemGroup Condition="'$(_CollectCoverage)' == 'true'">
16+
<!--
17+
Use VSTest integration to work around test host crashes on larger collections:
18+
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md
19+
-->
20+
<PackageReference Include="coverlet.collector">
421
<PrivateAssets>all</PrivateAssets>
522
<IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
623
</PackageReference>
724
</ItemGroup>
825

9-
<!-- Allows Collection of Code Coverage for Deterministic Builds
10-
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/DeterministicBuild.md -->
26+
<!-- Clean up previous TestResults so reports are recent. -->
27+
<Target Name="CleanPreviousCodeCoverage"
28+
BeforeTargets="VSTest"
29+
Condition="'$(_CollectCoverage)' == 'true'">
30+
<RemoveDir Directories="$(_TestResultsDirectory)" />
31+
</Target>
32+
33+
<!-- Should be similar to what's in the pipelines, though generate a full HTML report. -->
34+
<Target Name="GenerateCodeCoverageReport"
35+
AfterTargets="VSTest"
36+
Condition="'$(_CollectCoverage)' == 'true' and '$(SkipCoverageReport)' != 'true'">
37+
<PropertyGroup>
38+
<CoverageReportCommandLine>dotnet tool run reportgenerator --</CoverageReportCommandLine>
39+
<CoverageReportCommandLine>$(CoverageReportCommandLine) "-reports:$(_TestResultsDirectory)\**\coverage.cobertura.xml"</CoverageReportCommandLine>
40+
<CoverageReportCommandLine>$(CoverageReportCommandLine) -reporttypes:Html</CoverageReportCommandLine>
41+
<CoverageReportCommandLine>$(CoverageReportCommandLine) "-targetdir:$(_TestResultsDirectory)"</CoverageReportCommandLine>
42+
<CoverageReportCommandLine>$(CoverageReportCommandLine) "-filefilters:+$(CodeCoverageDirectory)\**"</CoverageReportCommandLine>
43+
</PropertyGroup>
44+
<Exec Command="$(CoverageReportCommandLine)"
45+
IgnoreExitCode="true"
46+
StandardErrorImportance="high"
47+
StandardOutputImportance="low" />
48+
</Target>
49+
50+
<Target Name="_ValidateSourceFileNames"
51+
BeforeTargets="CoreBuild"
52+
Condition="'$(_IsCodeCoverable)' == 'true'">
53+
<ItemGroup>
54+
<!-- Prevent https://github.com/Azure/azure-sdk-for-net/issues/17164 from becoming an issue further in the build process -->
55+
<_ContainsCurlyBraces Include="@(Compile)" Condition="$([MSBuild]::ValueOrDefault('%(Directory)%(Filename)', '').Contains('{')) or $([MSBuild]::ValueOrDefault('%(Directory)%(Filename)', '').Contains('}'))" />
56+
</ItemGroup>
57+
<Error
58+
Text="File name '%(_ContainsCurlyBraces.FullPath)' cannot contain { or }; remove type parameters from the file name, or change {T} to OfT and disable SA1649 if the class has a non-generic counterpart (https://github.com/Azure/azure-sdk-for-net/issues/17164)."
59+
Condition="'@(_ContainsCurlyBraces)' != ''" />
60+
</Target>
61+
62+
<!--
63+
Allows Collection of Code Coverage for Deterministic Builds:
64+
https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/DeterministicBuild.md
65+
66+
This needs to be available in all projects.
67+
-->
1168
<ItemGroup>
1269
<SourceRoot Include="$(NuGetPackageRoot)" />
1370
</ItemGroup>
@@ -17,7 +74,8 @@
1774
Returns="@(_LocalTopLevelSourceRoot)"
1875
Condition="'$(DeterministicSourcePaths)' == 'true'">
1976
<ItemGroup>
20-
<_LocalTopLevelSourceRoot Include="@(SourceRoot)" Condition="'%(SourceRoot.NestedRoot)' == ''"/>
77+
<_LocalTopLevelSourceRoot Include="@(SourceRoot)"
78+
Condition="'%(SourceRoot.NestedRoot)' == ''"/>
2179
</ItemGroup>
2280
</Target>
23-
</Project>
81+
</Project>

‎eng/Directory.Build.Data.props‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,6 @@
5555
<GenerateAPIListing Condition="'$(IsShippingClientLibrary)' == 'true'">true</GenerateAPIListing>
5656
<UpdateSourceOnBuild Condition="'$(UpdateSourceOnBuild)' == ''">$(AZURE_DEV_UPDATESOURCESONBUILD)</UpdateSourceOnBuild>
5757
<PowerShellExe Condition="'$(PowerShellExe)' == ''">pwsh</PowerShellExe>
58-
<CoverletOutputFormat Condition="'$(CoverletOutputFormat)' == '' and '$(CollectCoverage)' == 'true'">cobertura</CoverletOutputFormat>
5958
<InheritDocEnabled>false</InheritDocEnabled>
6059
</PropertyGroup>
6160

@@ -97,7 +96,8 @@
9796
<PropertyGroup Condition="'$(IsTestProject)' == 'true' or '$(IsTestSupportProject)' == 'true' or '$(IsSamplesProject)' == 'true' or '$(IsPerfProject)' == 'true' or '$(IsStressProject)' == 'true'">
9897
<IsPackable>false</IsPackable>
9998
<RequiredTargetFrameworks>netcoreapp2.1;net5.0</RequiredTargetFrameworks>
100-
<RequiredTargetFrameworks Condition="'$(OS)' == 'Windows_NT'">netcoreapp2.1;net5.0;net461</RequiredTargetFrameworks>
99+
<!-- Also test net461 on Windows; it's listed first so that coverage reports are for netcoreapp2.1 (the "primary"). -->
100+
<RequiredTargetFrameworks Condition="'$(OS)' == 'Windows_NT'">net461;netcoreapp2.1;net5.0</RequiredTargetFrameworks>
101101
</PropertyGroup>
102102

103103
<Import Project="$(RepoRoot)/sdk/core/Azure.Core/src/Azure.Core.props" Condition="'$(IsMgmtClientLibrary)' == 'true'"/>

‎eng/Directory.Build.Data.targets‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,6 @@
7171
<None Condition="Exists('$(MSBuildProjectDirectory)/../README.md')" Include="$(MSBuildProjectDirectory)/../README.md" Pack="true" PackagePath=""/>
7272
</ItemGroup>
7373

74-
<!-- Collect Code Coverage -->
75-
<Import Condition="'$(CollectCoverage)' == 'true'" Project="$(MSBuildThisFileDirectory)\CodeCoverage.targets" />
76-
7774
<!-- Add StyleCop Analyzers -->
7875
<ItemGroup Condition="'$(EnableStyleCopAnalyzers)' == 'true'" >
7976
<PackageReference Include="StyleCop.Analyzers">
@@ -85,7 +82,7 @@
8582
</AdditionalFiles>
8683
</ItemGroup>
8784

88-
<!-- Enable SourceLink -->
85+
<!-- Enable SourceLink -->
8986
<ItemGroup>
9087
<PackageReference Include="Microsoft.SourceLink.GitHub" PrivateAssets="All" />
9188
</ItemGroup>
@@ -105,6 +102,8 @@
105102

106103
<Import Project="ApiListing.targets" />
107104

105+
<Import Project="CodeCoverage.targets" />
106+
108107
<Import Project="CodeGeneration.targets" Condition="'$(TemporaryUsePreviousGeneratorVersion)' == 'true'" />
109108

110109
<Import Project="TestFramework.targets" Condition="'$(IsTestProject)' == 'true'"/>

‎eng/Packages.Data.props‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@
99
<ItemGroup>
1010
<PackageReference Update="ApprovalTests" Version="3.0.22" />
1111
<PackageReference Update="ApprovalUtilities" Version="3.0.22" />
12-
<PackageReference Update="AutoRest.CSharp.V3" Version="1.0.0-alpha.20201123.1" />
12+
<PackageReference Update="AutoRest.CSharp.V3" Version="1.0.0-alpha.20201123.2" />
1313
<PackageReference Update="Azure.AI.FormRecognizer" Version="3.0.0" />
14-
<PackageReference Update="Azure.AI.TextAnalytics" Version="5.0.0" />
14+
<PackageReference Update="Azure.AI.TextAnalytics" Version="5.0.0" />
1515
<PackageReference Update="Azure.Data.AppConfiguration" Version="1.0.0" />
1616
<PackageReference Update="Azure.Core" Version="1.6.0" />
1717
<PackageReference Update="Azure.Core.Amqp" Version="1.0.0" />
@@ -32,7 +32,7 @@
3232
<PackageReference Update="Azure.Storage.Blobs.ChangeFeed" Version="12.0.0-preview.1" />
3333
<PackageReference Update="BenchmarkDotNet" Version="0.11.5" />
3434
<PackageReference Update="Castle.Core" Version="4.4.0" />
35-
<PackageReference Update="coverlet.msbuild" Version="2.9.0" />
35+
<PackageReference Update="coverlet.collector" Version="1.3.0" />
3636
<PackageReference Update="FluentAssertions" Version="5.10.3" />
3737
<PackageReference Update="FsCheck.Xunit" Version="2.14.0" />
3838
<PackageReference Update="Microsoft.Azure.Amqp" Version="2.4.8" />
@@ -69,7 +69,7 @@
6969
<PackageReference Update="Microsoft.IdentityModel.Clients.ActiveDirectory" Version="4.5.1" />
7070
<PackageReference Update="Microsoft.Identity.Client" Version="4.22.0" />
7171
<PackageReference Update="Microsoft.Identity.Client.Extensions.Msal" Version="2.16.5" />
72-
<PackageReference Update="Microsoft.NET.Test.Sdk" Version="16.1.0" />
72+
<PackageReference Update="Microsoft.NET.Test.Sdk" Version="16.8.0" />
7373
<PackageReference Update="Microsoft.NETCore.Platforms" Version="2.2.1" />
7474
<PackageReference Update="Microsoft.NETFramework.ReferenceAssemblies" Version="1.0.0" />
7575
<PackageReference Update="Microsoft.Rest.ClientRuntime.Azure.Authentication" Version="[2.4.0]" />
@@ -180,9 +180,9 @@
180180
<PackageReference Update="Microsoft.Extensions.PlatformAbstractions" Version="1.1.0" />
181181

182182
<PackageReference Update="Microsoft.Extensions.DependencyInjection" Version="2.1.0" />
183-
183+
184184
<PackageReference Update="Microsoft.Azure.Devices.Client" Version="1.27.0" />
185-
185+
186186
<!-- Mgmt sdk packages-->
187187
<PackageReference Update="Azure.ResourceManager.Resources" Version="1.0.0-preview.2" />
188188
<PackageReference Update="Azure.ResourceManager.Compute" Version="1.0.0-preview.2" />

‎eng/pipelines/templates/jobs/archetype-sdk-client.yml‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,8 @@ jobs:
110110
condition: and(succeededOrFailed(), ne(variables['Skip.Test'], true))
111111
variables:
112112
- template: ../variables/globals.yml
113+
- name: disable.coverage.autogenerate
114+
value: true
113115
strategy:
114116
maxParallel: $[ variables['MaxParallelTestJobs'] ]
115117
matrix:
@@ -119,6 +121,7 @@ jobs:
119121
Windows_NetCoreApp:
120122
OSVmImage: "windows-2019"
121123
TestTargetFramework: netcoreapp2.1
124+
CollectCoverage: true
122125
Windows_NetCoreApp_ProjectReferences:
123126
OSVmImage: "windows-2019"
124127
TestTargetFramework: netcoreapp2.1
@@ -149,7 +152,8 @@ jobs:
149152
--logger "trx;LogFileName=$(TestTargetFramework).trx" --logger:"console;verbosity=normal"
150153
/p:ServiceDirectory=${{parameters.ServiceToTest}}
151154
/p:IncludeSrc=false /p:IncludeSamples=false /p:IncludePerf=false /p:IncludeStress=false
152-
/p:Configuration=$(BuildConfiguration) $(ConvertToProjectReferenceOption) /p:CollectCoverage=$(CollectCoverage)
155+
/p:Configuration=$(BuildConfiguration) $(ConvertToProjectReferenceOption)
156+
/p:CollectCoverage=$(CollectCoverage) /p:CodeCoverageDirectory=${{parameters.ServiceDirectory}}
153157
displayName: "Build & Test ($(TestTargetFramework))"
154158
env:
155159
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1
@@ -163,15 +167,18 @@ jobs:
163167
testResultsFormat: "VSTest"
164168
mergeTestResults: true
165169
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
166-
condition: and(succeeded(), eq(variables['CollectCoverage'], 'true'))
167-
displayName: ReportGenerator
170+
condition: and(succeededOrFailed(), eq(variables['CollectCoverage'], 'true'))
171+
displayName: Generate Code Coverage Reports
168172
inputs:
169-
reports: '**/*coverage.netcoreapp2.1.cobertura.xml'
170-
targetdir: '$(Build.SourcesDirectory)'
171-
reporttypes: Cobertura
173+
reports: $(Build.SourcesDirectory)\sdk\${{parameters.ServiceDirectory}}\**\coverage.cobertura.xml
174+
targetdir: $(Build.ArtifactStagingDirectory)\coverage
175+
reporttypes: Cobertura;HtmlInline_AzurePipelines
176+
filefilters: +$(Build.SourcesDirectory)\sdk\${{parameters.ServiceDirectory}}\**
177+
verbosity: Verbose
172178
- task: PublishCodeCoverageResults@1
173-
condition: and(succeeded(), eq(variables['CollectCoverage'], 'true'))
174-
displayName: 'Publish code coverage report'
179+
condition: and(succeededOrFailed(), eq(variables['CollectCoverage'], 'true'))
180+
displayName: Publish Code Coverage Reports
175181
inputs:
176-
codeCoverageTool: 'Cobertura'
177-
summaryFileLocation: 'Cobertura.xml'
182+
codeCoverageTool: Cobertura
183+
summaryFileLocation: $(Build.ArtifactStagingDirectory)\coverage\Cobertura.xml
184+
reportDirectory: $(Build.ArtifactStagingDirectory)\coverage

‎eng/service.proj‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@
99
<IncludeStress Condition="'$(IncludeStress)' == ''">true</IncludeStress>
1010
<IncludeSamplesApplications Condition="'$(IncludeSamplesApplications)' == ''">true</IncludeSamplesApplications>
1111
<IncludeSamplesApplications Condition="'$(ServiceDirectory)' != '*' or '$(IncludeSamples)' == 'false'">false</IncludeSamplesApplications>
12+
<TraversalGlobalProperties>
13+
CodeCoverageDirectory=$([System.IO.Path]::GetFullPath("$(CodeCoverageDirectory)", "$(MSBuildThisFileDirectory)..\sdk"));
14+
</TraversalGlobalProperties>
1215
</PropertyGroup>
1316

1417
<ItemGroup>

‎sdk/appconfiguration/Azure.Data.AppConfiguration/src/Azure.Data.AppConfiguration.csproj‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
<Compile Include="$(AzureCoreSharedSources)ConditionalRequestOptionsExtensions.cs" />
3131
<Compile Include="$(AzureCoreSharedSources)DiagnosticScope.cs" />
3232
<Compile Include="$(AzureCoreSharedSources)HashCodeBuilder.cs" />
33-
<Compile Include="$(AzureCoreSharedSources)NoBodyResponse{T}.cs" />
33+
<Compile Include="$(AzureCoreSharedSources)NoBodyResponseOfT.cs" />
3434
<Compile Include="$(AzureCoreSharedSources)PageResponseEnumerator.cs" />
3535
<Compile Include="$(AzureCoreSharedSources)DiagnosticScopeFactory.cs" />
3636
<Compile Include="$(AzureCoreSharedSources)TaskExtensions.cs" />

0 commit comments

Comments
 (0)