Provider v8 does not show test coverage

[aleksi-magner]

Describe the bug

The @vitest/coverage-v8 and @vitest/coverage-c8 packages are expected to provide identical coverage and test results. But that doesn't happen.

Package @vitest/coverage-v8 does not work correctly in terms of coverage for sure. Package @vitest/coverage-c8 is more accurate - sees all uncovered and skipped areas in Vue templates, takes into account missing test files and overall coverage.
Other things being equal settings, package @vitest/coverage-v8 simply always passes the tests and never outputs missing lines or coverage problems.

Reproduction

A simplified example is in the repository.

Install dependencies: yarn
Running tests: yarn unit

We see a report:

 RUN  v0.32.0 /home/user/projects/vitest-reproduction/tests
      Coverage enabled with v8

 ✓ unit/UI/Icon.spec.js (11)
 ✓ unit/UI/Img.spec.js (5)
 ✓ unit/layouts/BottomNavigation.spec.js (3)
 ✓ unit/UI/Badge.spec.js (7)
 ✓ unit/views/AnyPage.spec.js (4)
 ✓ unit/layouts/TopBar.spec.js (1)
 ✓ unit/router/routes.spec.js (9)
 ✓ unit/App.spec.js (1)
 ✓ unit/layouts/MainLayout.spec.js (1)
 ✓ unit/views/InitError.spec.js (1)
 ✓ unit/stores.spec.js (2)
 ✓ unit/UI/Loader.spec.js (1)
 ✓ unit/services/helpers.spec.js (1)
 ↓ unit/main.spec.js (2) [skipped]

 Test Files  13 passed | 1 skipped (14)
      Tests  47 passed | 2 skipped (49)
   Start at  16:01:38
   Duration  4.23s (transform 170ms, setup 380ms, collect 2.21s, tests 695ms, environment 2.54s, prepare 1.46s)

 % Coverage report from v8
----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |       0 |        0 |       0 |       0 |                   
----------|---------|----------|---------|---------|-------------------
Done in 5.24s.

We go into the config and change the coverage provider from v8 to c8. Run the tests again and see the following report:

 RUN  v0.32.0 /home/user/projects/vitest-reproduction/tests
      Coverage enabled with c8

 ✓ unit/UI/Badge.spec.js (7)
 ✓ unit/UI/Icon.spec.js (11)
 ✓ unit/UI/Img.spec.js (5)
 ✓ unit/router/routes.spec.js (9)
 ✓ unit/App.spec.js (1)
 ✓ unit/layouts/BottomNavigation.spec.js (3)
 ✓ unit/layouts/TopBar.spec.js (1)
 ✓ unit/views/AnyPage.spec.js (4)
 ✓ unit/layouts/MainLayout.spec.js (1)
 ✓ unit/UI/Loader.spec.js (1)
 ✓ unit/stores.spec.js (2)
 ✓ unit/views/InitError.spec.js (1)
 ✓ unit/services/helpers.spec.js (1)
 ↓ unit/main.spec.js (2) [skipped]

 Test Files  13 passed | 1 skipped (14)
      Tests  47 passed | 2 skipped (49)
   Start at  16:02:56
   Duration  4.62s (transform 163ms, setup 325ms, collect 2.32s, tests 975ms, environment 2.45s, prepare 1.69s)

 % Coverage report from c8
-----------------------|---------|----------|---------|---------|-------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------------------|---------|----------|---------|---------|-------------------
All files              |   96.37 |    96.42 |   97.67 |   96.37 |                   
 src                   |   75.23 |    84.61 |    87.5 |   75.23 |                   
  App.vue              |     100 |    83.33 |     100 |     100 | 2                 
  main.ts              |       0 |        0 |       0 |       0 | 1-26              
  stores.ts            |     100 |      100 |     100 |     100 |                   
 src/UI                |     100 |      100 |     100 |     100 |                   
  Badge.vue            |     100 |      100 |     100 |     100 |                   
  Icon.vue             |     100 |      100 |     100 |     100 |                   
  Img.vue              |     100 |      100 |     100 |     100 |                   
  Loader.vue           |     100 |      100 |     100 |     100 |                   
 src/layouts           |     100 |    92.85 |     100 |     100 |                   
  BottomNavigation.vue |     100 |      100 |     100 |     100 |                   
  MainLayout.vue       |     100 |      100 |     100 |     100 |                   
  TopBar.vue           |     100 |       75 |     100 |     100 | 46                
 src/router            |     100 |      100 |     100 |     100 |                   
  index.ts             |     100 |      100 |     100 |     100 |                   
  routes.ts            |     100 |      100 |     100 |     100 |                   
 src/services          |     100 |      100 |     100 |     100 |                   
  helpers.js           |     100 |      100 |     100 |     100 |                   
 src/views             |     100 |      100 |     100 |     100 |                   
  AnyPage.vue          |     100 |      100 |     100 |     100 |                   
  InitError.vue        |     100 |      100 |     100 |     100 |                   
-----------------------|---------|----------|---------|---------|-------------------
ERROR: Coverage for lines (96.37%) does not meet global threshold (100%)
ERROR: Coverage for functions (97.67%) does not meet global threshold (100%)
ERROR: Coverage for branches (96.42%) does not meet global threshold (100%)
ERROR: Coverage for statements (96.37%) does not meet global threshold (100%)
error Command failed with exit code 1.

Conclusion: at the moment v8 does not show coverage in principle

System Info

System:
    OS: Linux 6.3 Manjaro Linux
    CPU: (4) x64 Intel(R) Core(TM) i3-8100T CPU @ 3.10GHz
    Memory: 3.41 GB / 11.04 GB
    Container: Yes
    Shell: 5.1.16 - /bin/bash

  Binaries:
    Node: 18.15.0 - /usr/bin/node
    Yarn: 1.22.19 - ~/.node_modules_global/bin/yarn
    npm: 9.6.6 - ~/.node_modules_global/bin/npm

  npmPackages:
    @vitejs/plugin-vue: ^4.2.3 => 4.2.3 
    @vitest/coverage-c8: ^0.32.0 => 0.32.0 
    @vitest/coverage-v8: ^0.32.0 => 0.32.0 
    vite: ^4.3.9 => 4.3.9 
    vitest: ^0.32.0 => 0.32.0

Used Package Manager

yarn

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[AriPerkkio]

The configuration file sets root as test, but your source files are located outside that directory. https://github.com/aleksi-magner/vitest-c8-vs-v8/blob/ee3912e70af678bee58d3b48f7dbf95a3f1930a0/vite.config.ts#L41

The @vitest/coverage-v8 (and also @vitest/coverage-istanbul) do not see files outside this root. When I remove the root and change the setupFiles to include that previous root value, the coverage is 100% identical as with @vitest/coverage-c8.

The reason why this happens on c8 is that it does not pass cwd option to test-exclude. This is something that they should do as we've earlier seen that it can cause other issues (#1902 (comment)). For example nyc does this out of the box.

[aleksi-magner]

That is, the current behavior of v8 is correct and there is no error? If so, then I'll just fix my config under the root directory of the project.
Or will it still make some changes so that you can specify the source directory?

[AriPerkkio]

The current behaviour is correct and no code changes to Vitest are needed.

To summarize, when you have following configuration:

test: {
  root: './tests/',
  coverage: {
    include: ['**/src/**'],

This will include all files in root that match the coverage.include -> ./tests/**/src/**.