Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage not calculated correctly if coverage report is uploaded without files being present #1707

Open
undsoft opened this issue Nov 29, 2024 · 0 comments
Assignees
Labels
Medium Medium Priority Issues (to be fixed or re-evaluated in 3 months

Comments

@undsoft
Copy link

undsoft commented Nov 29, 2024

We have recently updated our CI workflow to use sharding in Jest to speed up the tests. The flow is something like this:

  • Job 1: Checkout, install and build.
  • Then job 2 and 3 in parallel: checkout, install and run tests. Each job runs 50% of the tests. Coverage then gets uploaded as an artifact via actions/upload-artifact.
  • Job 4: download coverage with actions/download-artifact, merge into a single json and report to Codecov via codecov/codecov-action.

The problem was that job 4 didn't checkout the repo (because it didn't really need those files).
When codecov-action was used without the repo being present, even though the coverage file was uploaded, it always said All modified and coverable lines are covered by tests ✅ and all changes were reported as indirect changes.

File paths were incorrect (extra webui) and clicking on the file resulted in an error:

2024-11-29.10.45.32.mov

Example of a bad job:
https://github.com/truenas/webui/actions/runs/12073223216/job/33669795147?pr=11114

After I added repo checkout as a step, everything started to work correctly:
https://github.com/truenas/webui/actions/runs/12082374244/job/33694460349?pr=11114

You can see my previous CI configuration and fixes in the associated PR. There were no important changes in how coverage file is generated.

While this is a configuration error, I feel like there could be something to improve on Codecov's end:

  • It could have given me an error on upload if it really needed files to be present.
  • We have patch coverage target set up. It would have been better if it started failing on all jobs. Instead it was always green for a month, before we spotted that something was wrong.
@thomasrockhu-codecov thomasrockhu-codecov self-assigned this Dec 2, 2024
@thomasrockhu-codecov thomasrockhu-codecov added the Medium Medium Priority Issues (to be fixed or re-evaluated in 3 months label Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Medium Medium Priority Issues (to be fixed or re-evaluated in 3 months
Projects
None yet
Development

No branches or pull requests

2 participants