Run coverage on travis #4605

stsewd · 2018-09-05T14:55:43Z

Our coverage is good, and I would love to maintain it like that. The coverage report is 0% on management commands (not sure if we care to have tests for that)

What change? Almost nothing, devs can still run normal tests, as usual. The only thing is that Travis will fail if the coverage goes down 79% (our current coverage).

We can increase the coverage later, but keeping it >=79% is a good first step.

If you want to check the coverage locally, you'll need to run tox like this tox -e py36,coverage

stsewd · 2018-09-05T14:56:45Z

tox.ini

 whitelist_externals = echo
 commands =
-    py.test --disable-pytest-warnings \
-        --cov-report=term --cov-report=html --cov-config {toxinidir}/.coveragerc --cov=.  {posargs}
+    coverage report --show-missing --fail-under=79


79 is our current percent

Well, looks like it is on 80% on travis, this is for the search tests I guess

humitos

I like this.

On the other hand, going so strict could bring problems, as usual. Maybe we can see how it goes and downgrade the limit later.

I think it will be better if we execute the coverage command as a separate job under travis. That way, we can immediately know if it fails because of the coverage or because of a test.

Also,

I think we can exclude the rtd_tests and tests/ and tests.py files so the output is not that verbose.
- why there are test files saying that some lines are missing? aren't those tests being executed? why? are they marked as @Skip
analytics/vendor can be excluded also. It's not our own code
*/management/commands can be excluded
settings can be excluded also
wsgi.py also

I love this PR, but I'd like to request these changes to make it more accurate.

stsewd · 2018-09-05T15:34:19Z

why there are test files saying that some lines are missing? aren't those tests being executed? why? are they marked as @Skip

Those aren't being executed at the moment but are unskiped in other PRs.

stsewd · 2018-09-05T15:41:09Z

I think it will be better if we execute the coverage command as a separate job under travis. That way, we can immediately know if it fails because of the coverage or because of a test.

Not sure about that, It will slow down the CI even more :/

stsewd · 2018-09-05T15:42:18Z

Also, we could easily see the tox report to see which command fail, tests or coverage.

stsewd · 2018-09-05T16:16:07Z

tox.ini

 whitelist_externals = echo
 commands =
-    py.test --disable-pytest-warnings \
-        --cov-report=term --cov-report=html --cov-config {toxinidir}/.coveragerc --cov=.  {posargs}
+    coverage report --show-missing --fail-under=76


And with the search tests we are at 78%, should I increase the coverage?

ericholscher · 2018-10-17T11:21:07Z

I'm not convinced this is the best path forward with coverage. I don't think we really want to fail tests on it, since that will rarely happen, but instead make the lack of coverage more visible.

I like what Sphinx does on their PR's -- sphinx-doc/sphinx#5549 (comment) -- I wonder if it would be more worthwhile to integrate something like that, so that we have better data around coverage.

stsewd · 2018-10-17T14:40:12Z

Yeah, integrating with codecov looks good, and it uses coverage so we can integrate that easily here. I'll update this PR to integrate with codecov

humitos · 2018-10-17T15:50:09Z

codecov is good, but it's too verbose. Is this comment added on every commit to the PR? If so, I think we will end up with many comments from this bot on PRs that will be impossible to follow.

I think we should tweak codecov to behave in a way that doesn't make hard to read all the comments of the PR. Maybe only the line

Merging #5549 into 1.8 will decrease coverage by <.01%.

with the link to the codecov.io page is enough.

stsewd · 2018-10-17T15:55:37Z

@humitos what I saw, it just post one comment and then updates that comment on each commit, and yeah, it's very verbose, should be a way of a more simple report.

stsewd · 2018-10-19T06:11:35Z

Here we go https://codecov.io/gh/rtfd/readthedocs.org/ the request for the bot is sent, I guess someone with admin access needs to accept the invitation 🤷‍♂️

stsewd · 2018-10-19T06:13:40Z

People at home can still see the reports, with tox -e py36 and tox -e coverage or just tox -e py36,coverage

stsewd · 2018-10-19T06:22:30Z

Also, I think we are good with the defaults for the configuration, for the comment we only have this

comment:
  layout: "header, diff"

https://docs.codecov.io/docs/codecov-yaml#section-default-yaml

ericholscher · 2018-10-19T08:07:14Z

I poked around on codecov but didn't see what confirmation I needed. Is it to install the bot on GH? Seems like that is optional, so I'm not 100% sure what's required.

ericholscher · 2018-10-19T08:10:28Z

Ah, found the request in email -- approved. 👍

ericholscher

I like this approach better. Haven't seen the output from the bot, but this looks like a good way to do it.

I'm a little worried that coverage will slow down our test runs, but we can cross that bridge if we hit it.

codecov · 2018-10-19T16:36:28Z

Codecov Report

❗ No coverage uploaded for pull request base (master@df85fef). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master   #4605   +/-   ##
========================================
  Coverage          ?   75.3%           
========================================
  Files             ?     158           
  Lines             ?   10045           
  Branches          ?    1266           
========================================
  Hits              ?    7564           
  Misses            ?    2145           
  Partials          ?     336

stsewd · 2018-10-19T16:40:48Z

I retriggered the job, looks like we need to set up a less verbose comment for the bot p:

I'm a little worried that coverage will slow down our test runs

I didn't see any big difference locally, I think we are good.

stsewd · 2018-10-19T17:17:47Z

The bot is ready, @humitos are we ok with the bot's comment? If so, I'll merge this PR.

humitos

Looks good to me!

We can tweak the bot config if we found that it's still too verbose. For now, it looks good, though.

The header layout was removed from the comment but it's still there 😕

humitos

Changes look good.

The header layout was removed from the comment but it's still there, though.

We can tweak the configuration later if we find this too annoying.

humitos

Changes look good.

The header layout was removed from the comment but it's still there, though.

We can tweak the configuration later if we find this too annoying.

humitos

Changes look good.

The header layout was removed from the comment but it's still there, though.

We can tweak the configuration later if we find this too annoying.

humitos

Changes look good.

The header layout was removed from the comment but it's still there, though.

We can tweak the configuration later if we find this too annoying.

humitos · 2018-10-22T12:01:41Z

Haha! If it's not clear, I've trying to approve this PR --GitHub was working terrible :)

ericholscher · 2018-10-22T19:03:01Z

🎉

stsewd requested a review from a team September 5, 2018 14:56

stsewd commented Sep 5, 2018

View reviewed changes

humitos requested changes Sep 5, 2018

View reviewed changes

stsewd force-pushed the run-coverage branch from fc99f3e to d85807f Compare September 5, 2018 16:06

stsewd commented Sep 5, 2018

View reviewed changes

agjohnson added this to the Development improvements milestone Sep 5, 2018

agjohnson added the Improvement Minor improvement to code label Sep 5, 2018

stsewd added 3 commits October 19, 2018 00:00

Run coverage on travis

12d7b00

Don't report some files on coverage

1bacd1e

Update description

4ee8260

stsewd force-pushed the run-coverage branch from 3385fc2 to 4ee8260 Compare October 19, 2018 05:00

stsewd added 3 commits October 19, 2018 00:19

More setttings

85257dd

Use pytest-cov

fd62e94

Run codecov

6a2720c

ericholscher approved these changes Oct 19, 2018

View reviewed changes

Less verbose comment for bot

326d621

humitos approved these changes Oct 22, 2018

View reviewed changes

stsewd merged commit a4291d1 into readthedocs:master Oct 22, 2018

stsewd deleted the run-coverage branch October 22, 2018 15:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run coverage on travis #4605

Run coverage on travis #4605

stsewd commented Sep 5, 2018

stsewd Sep 5, 2018

stsewd Sep 5, 2018

humitos left a comment

stsewd commented Sep 5, 2018

stsewd commented Sep 5, 2018

stsewd commented Sep 5, 2018

stsewd Sep 5, 2018

ericholscher commented Oct 17, 2018

stsewd commented Oct 17, 2018

humitos commented Oct 17, 2018 •

edited

Loading

stsewd commented Oct 17, 2018

stsewd commented Oct 19, 2018

stsewd commented Oct 19, 2018 •

edited

Loading

stsewd commented Oct 19, 2018

ericholscher commented Oct 19, 2018

ericholscher commented Oct 19, 2018

ericholscher left a comment

codecov bot commented Oct 19, 2018 •

edited

Loading

stsewd commented Oct 19, 2018

stsewd commented Oct 19, 2018

humitos left a comment

humitos left a comment

humitos left a comment

humitos left a comment

humitos left a comment

humitos commented Oct 22, 2018

ericholscher commented Oct 22, 2018

Run coverage on travis #4605

Run coverage on travis #4605

Conversation

stsewd commented Sep 5, 2018

stsewd Sep 5, 2018

Choose a reason for hiding this comment

stsewd Sep 5, 2018

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

stsewd commented Sep 5, 2018

stsewd commented Sep 5, 2018

stsewd commented Sep 5, 2018

stsewd Sep 5, 2018

Choose a reason for hiding this comment

ericholscher commented Oct 17, 2018

stsewd commented Oct 17, 2018

humitos commented Oct 17, 2018 • edited Loading

stsewd commented Oct 17, 2018

stsewd commented Oct 19, 2018

stsewd commented Oct 19, 2018 • edited Loading

stsewd commented Oct 19, 2018

ericholscher commented Oct 19, 2018

ericholscher commented Oct 19, 2018

ericholscher left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 19, 2018 • edited Loading

Codecov Report

stsewd commented Oct 19, 2018

stsewd commented Oct 19, 2018

humitos left a comment

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

humitos commented Oct 22, 2018

ericholscher commented Oct 22, 2018

humitos commented Oct 17, 2018 •

edited

Loading

stsewd commented Oct 19, 2018 •

edited

Loading

codecov bot commented Oct 19, 2018 •

edited

Loading