-
Notifications
You must be signed in to change notification settings - Fork 325
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support non-binary format for Code Coverage #2874
Comments
Cobertura support is added to code coverage. It is available in the following version. https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.1.0-preview-20211118-03 To get cobertura report, add
Available format options:
|
Stable 17.1.0 packages will be released in February. |
@jakubch1 Thanks! |
Hi @fhnaseer & @jakubch1, I currently use version 17.1.0 of microsoft.codecoverage package which targets net6.0 Test project and I have updated the format as xml and cobertura in runsettings and it generated both formats xml file, but looking those file did gave only very minimal data and I am not sure how to use this data for coverage of whats actually covered, could you let us know, is this expected coverage report or do I need to use preview versions of the packages? |
@Manikandanmani on what OS you collect code coverage? |
@jakubch1 I am trying to generate this on windows 10. |
@Manikandanmani could you please add to you command line: |
@jakubch1, I have attached all the log files that were generated using this command to run test in powershell, But if we specify two formats like
vstestcobertura01.host.22-03-15_21-55-54_07155_9.txt |
Please verify on your computer that below files exists:
|
Of these 4 file paths, first 2 exists, rest 2 doesn't exists, could you share the absolute file path for the ones below, do I need to locate it under here
|
I think those files are only part of Enterprise edition. Please use this package to run your tests: |
You can specify only one output format. |
I added this package https://www.nuget.org/packages/Microsoft.TestPlatform/17.1.0, and ran the tests and it generated the coverage xml file same as old, and I also tried by adding the test adapter path of coverlet.collector xml files generated after adding package https://www.nuget.org/packages/Microsoft.TestPlatform/17.1.0 |
Can you please share logs? Also if you generate .coverage file, is it also empty? |
@fhnaseer, I have shared the logs above for collect coverage with cobertura and vstest, yes the generated .coverage file seems empty when I used VsTest.Console.exe from VS2022 program files, since I can't view the .coverage file with vs2022 community I converted with CodeCoverage.exe and it was empty. I have attached the diag logs generated for visual studio coverage, codecoveragexml file. vstestcobertura01.host.22-03-17_10-43-31_62605_9.txt Generated .CoverageXml file from .Coverage, I couldn't able to attach the coveragexml , xml files here. |
It seems like you are using vstest.console.exe from Visual Studio (Community Edition). In the datacollector.txt Please use the one from nuget package. Also you can try |
@fhnaseer, I thought this would work directly when using with vstest.console.exe from Visual Studio (Community Edition) and by referencing microsoft.codecoverage nuget package in the test project, do you guys able to generate cobertura xml file directly by pointing to vstest.console.exe comes with Visual Studio 2022[community/enterprise]? or the only way to generate cobertura xml file is to target to vstest.console.exe from microsoft.testplatform.17.1.0 nuget package ? kindly let me know am I missing anything here? |
If you have Visual Studio Enterprise then you can generate code coverage either using UI commands inside VS or by running vstest.console.exe. So if you have community edition, there are two ways to generate coverage report
|
If I have Visual Studio Enterprise Can I able to generate cobertura coverage xml directly vstest.console.exe, without specifying the vstest from microsoft.testplatform.17.1.0 package? |
Yes. You need to specify runsettings file when Analyzing Code Coverage. |
Thanks @fhnaseer, I could able to generate cobertura xml as coverage file when I used enterprise Vstest.console.exe |
@fhnaseer, could you let us know how the coverage file will be generated for test reruns if its failed at the first time and passed at the next run? |
Foreach test run a separate coverage file will be generated. Coverage report is not dependent on test results. So test fail or pass coverage file will be generated. Coverage reports will be generated in "Test Results" folder in your solution. |
And does this feature supports only Visual studio version 17.0/17.1 ? |
Support of cobertura was added in 17.1.0. |
@fhnaseer, could you able to publish this xml file in the azure devops build pipelines after the test ran using the vstest task version-2, In the vstest task after the tests are completed before starting the next build step the vstest tasks deletes the TestResults folder, and the ResultsDirectory tag in the runsettings file is also overridden to the $(Agent.TempDirectory)\TestResults in the build, and does generating coverage xml works on vs2019 enterprise latest version? |
Support for cobertura and xml file was not supported in 17.0 or earlier versions. For azure devops and code coverage you can follow the following article. https://docs.microsoft.com/en-us/azure/devops/pipelines/test/review-code-coverage-results?view=azure-devops |
Hi @fhnaseer, When the code coverage file format is either cobertura xml or coverage xml, In vstest version-2 task when code coverage checkbox is enabled or disabled, it wipes the test results directory after the test execution completed but it doesn't shows the coverage xml as the build artifact like the .coverage file generated as a artifact. 1.https://docs.microsoft.com/en-us/rest/api/azure/devops/testresults/testlogstoreendpoint/get-test-log-store-endpoint-details-for-run-log?view=azure-devops-rest-6.0 |
@Manikandanmani you can combine
If you want it inside the artifacts to download you can combine |
Hey @jakubch1, is there a non-preview version of this available yet? |
Yes:
|
@MarcoRossignoli , the vstest task will delete the test results directory specified in the task options, so the only way I found is through the api's. |
I think that if the result folder points to a staging directory will be preserved between tasks https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#build-variables-devops-services |
I use the classic version,If you ran the pipeline in debug mode you could able to see the logs to deleting test results attachments, I hope I haven't missed anything. |
What you mean for classic version? |
Azure devops build pipeline template type |
@MarcoRossignoli, can we able to differentiate coverage files been generated for test reran, at present we couldn't able to.
|
@Manikandanmani can you show your yaml tasks it's not clear to me what you mean or if you've private information you can open a ticket here https://developercommunity.visualstudio.com/ |
steps:
|
Thanks to microsoft/vstest#2874 Cobertura reports can be obtained faster without coverlet.
* Use vstest Cobertura support Thanks to microsoft/vstest#2874 Cobertura reports can be obtained faster without coverlet. * Update Kubectl.Tests.csproj * Update KubernetesClient.Classic.Tests.csproj * Update KubernetesClient.Tests.csproj * Update buildtest.yaml * Update AbstractKubernetes.cs.template * Update IBasicKubernetes.cs.template * Update IOperations.cs.template * Update Model.cs.template * Add filter * revert
Description
Support non-binary format for dynamic code coverage. Code coverage is supported on ubuntu and alpine but users can't open coverage report on this platforms as .coverage files can be opened only in VS.
Created based on feedback from users on #981
The text was updated successfully, but these errors were encountered: