Test coverage support

[caseyduquettesc]

Via slack:

In this case, I got pytest-style tests working using https://github.com/caseyduquettesc/rules_python_pytest. It works, and I can even set up a github actions module to list the test results; all happy. So of course I'm greedy and want to use it for coverage too 🙂 I know the "approved" answer for Python coverage (the "modified version of coverage.py") but passing the "--cov" and "--cov-report=xml" gives me the nice test output Coverage XML written to file coverage.xml.
...of course that file is nowhere to be found; bazel-out obligingly has an appropriate test.xml (because of the argument --junit-xml=$$XML_OUTPUT_FILE to the pytest rule). And the file probably is being generated but somehow not being exported, and I'm new enough to bazel to not quite understand what's happening (or not).
Where should I start looking?

Related issues:
bazelbuild/rules_python#43
bazelbuild/bazel#10660

[caseyduquettesc]

Code coverage didn't even occur to me when writing rules_pytest. Seems like something that should be doable in theory. I've used pytest-cov before. Should be able to hook that up to pytest and get the report for bazel.

[basilnsage]

I'm interested in taking a stab at this. Any thoughts what a nice change to the rule signature might be? I'm not sure if

• exposing a new boolean parameter cover and using default pytest-cov arguments
  or
• exposing pytest-cov arguments to the caller
  or something else... is better.

[caseyduquettesc]

I need to read up on how bazel exposes it's running coverage before offering up an opinion. I think the thing we need to keep in mind is that this is going to be generating coverage reports for individual targets and we need bazel to merge them. The mechanism, I don't have an understanding of yet.

Purely from a user interface standpoint, I would expect bazel coverage //:py_test to generate the coverage data for target :py_test. This means turning coverage on or off implicitly for the user and not through an attribute on the rule.

I don't have a great answer to how to customize the coverage options off the top of my head.

bazelbuild/bazel#10660 (comment) seems like a really good starting point, though a little outdated.

According to pytest-dev/pytest-cov#536, coverage.py gained lcov support (https://coverage.readthedocs.io/en/6.4.2/cmd.html#cmd-lcov), so coverage-lconv shouldn't be necessary.

Current documentation for enabling test coverage, though looks hacky to me - https://bazel.build/docs/coverage#python