Support python coverage

[ergoithz]

The python tool coverage is much more common and mainstream than pytest-cov (which requires non-standard pytest runner) see both pytest-cov stats and coverage stats

This means this plugin is not useful for projects not willing to use the non-standard pytest.

Please consider adding support for coverage.

[ryanluker]

@ergoithz thanks for submitting an issue! do you know what coverage file type coverage outputs? depending on that it might already be supported 🤔

[ergoithz]

Example file with line coverage:

!coverage.py: This is a private format, don't read it directly!{"lines":{"/absolute/path/to/file.py":[3,4,5,6,7,8,9,10]}}

Example file with branch coverage:

!coverage.py: This is a private format, don't read it directly!{"arcs":{"/absolute/path/to/file.py":[[-4,4],[4,5],[5,6],[6,7],[7,8],[8,9],[9,10],[10,11],[11,-4]]

This are to be found on project root with name .coverage.

[SimonKagstrom]

My project kcov, https://github.com/SimonKagstrom/kcov, supports collecting Python code coverage and outputting it in a format which coverage-gutters understands. I use it for C/C++ code, but since the output is the same regardless of language, I don't see why the Python support shouldn't work.

Caveat: kcov only runs on OSX/Linux/FreeBSD

[ergoithz]

My project kcov, https://github.com/SimonKagstrom/kcov, supports collecting Python code coverage and outputting it in a format which coverage-gutters understands. I use it for C/C++ code, but since the output is the same regardless of language, I don't see why the Python support shouldn't work.

Caveat: kcov only runs on OSX/Linux/FreeBSD

No branch coverage though.

[ryanluker]

@ergoithz Looking at the coverage.py [0] it does seem doable to make another parser [1] (similar to the other ones created so far, clover, cobertura, etc). We would consume the .coverage file and transition the branch and line coverage information to the lcov format that the extension understands.

I am not sure I would be able to get to it before the next major release (3.0.0) but if someone wants to take the first steps and find a format specification for .coverage files that would help greatly 😄 . I couldn't find something similar to lcov's after a brief search... [2]

Resources

0: https://github.com/nedbat/coveragepy/
1: https://github.com/ryanluker/vscode-coverage-gutters/blob/master/src/files/coverageparser.ts#L38-L52
2: nedbat/coveragepy#587 (comment)

[drettie]

The python coverage module already has a command to convert the .coverage file to html and cobertura compliant xml which then can be parse with this extension.

Here are the commands i run -

coverage run --branch -m unittest discover
coverage xml -o cov.xml
coverage html

I was quite confused getting this to work at first as the extension by default only searches for files named cov.xml whereas the default output filename from the coverage xml command is coverage.xml.

If for some reason you can't rename the xml file there is also an option to change the default xml file in the extension settings.

@ryanluker might be worth adding coverage.xml as a default somewhere ?

[ryanluker]

@drettie Thanks for the input! Good to know there is a workaround for now with regards to the usual python coverage generation flow (still planning on adding support for this in 3.0.0).

Interesting mention around the default file name 🤔 I vaguely remember setting it to cov.xml for some reason as it might have been the default for php coverage generation for example.

One way to improve this would be to add multiple default coverage file names for the same type of coverage (can already be done via the default settings https://github.com/ryanluker/vscode-coverage-gutters/blob/v2.4.0/package.json#L137-L145). This could be apart of the work set out in this ticket or if someone wishes to quickly add it to the package.json file that would be great as well 😄.

[Josverl]

it was a matter of minutes to configure pytest to instruct coverage.py to also create an xml file bya adding --cov-report xml:cov.xml

[pytest]
junit_family=legacy

addopts=-v -sv --cov --cov-report xml:cov.xml --cov-report html

[ssbarnea]

@Josverl your comment makes no sense as is about pytest-cov plugin and not coverage.py it.

Plugin used the second but the ticket is about coverage.py itself, especially as the plugin use is discouraged.

[Josverl]

I stand corrected on that this is indeed configuring the plugin ,
but AFAIKT that plugin in itself then uses coverage,py , the config works in minutes , and I do not need to change my normal workflow to make coverage gutters work.

[ssbarnea]

What I am trying to explain here is that many users of coverage.py already have complex workflows, ones that likely upload the files to remove servers or use integration with third party services that expect specific filenames and locations. Most people will rely on default settings, so vscode-coverage-gutters should do the same and ensure that is able to detect coverage files that use the default layout, without requiring user to do any modification.

This is quite important because I really doubt any random repository owner would accept a pull-request that reconfigures the way coverage files are managed just to please one extension of vscode. Likely many of them are not using vscode at all. The real value of this extension comes into place when "it just works"... as long you are following standard patterns.

[Josverl]

but then why this discussion if all one needs is to run the command coverage json to enable this ?
and I personally see a 100% of the user of a vscode extension use VSCode , so imho maintainers of a VSCode extension should care about VSCode integration. but fully agree that the ecosystem is larger than VSCode alone.

anyway , just trying to help out , not flame intended , but i'm bailing out of this conversation.

good luck ✌🏻

[ryanluker]

@Josverl @ssbarnea Thanks for the spirited discussion, hopefully calmer heads have prevailed in the end.
Just to give some input from myself, the plan is still to eventually provide support to coverage in milestone 3.0.0, but that is still a bit away and development on the extension is purely on a volunteer basis making it hard to give an expected timeline for this ticket.

We will continue to support what is laid out in the example python project (using the older setup) that is covered via integrations test to make sure that flow still can be used by python devs. Once the new coverage parser is developed, we will add this as a new option for python devs, while still supporting the older flow in the meantime as well.

[ssbarnea]

Thanks a lot! That is a great way to deal with this. Doing an ack on the problem and expressing the willingness to address it is just perfect.

[ergoithz]

starting from coverage 6.3, lcov file generation is now supported via its new 'lcov' command, no extra dependencies required

[ryanluker]

@ergoithz Thanks for the heads-up! I think this ticket can be closed shortly but I will try to get in a small example using coverage 6.3 as an example for folks to use.

[MarximusMaximus]

fwiw, pytest-cov doesn't (yet? pytest-dev/pytest-cov#536) support lcov.xml generation, so it may be worth parsing the coverage.py coverage.xml format; primarily b/c calling pytest+pytest-cov from vscode still only generates the coverage.xml

[ryanluker]

@MarximusMaximus Thanks for the extra info, looks like there might have been some action on this front pytest-dev/pytest-cov#536 (comment) . That others above could leverage.