node:test coverage reporter unexpected behaviour with mock modules

[edge33]

Version

v22.11.0

Platform

Darwin 23.6.0 Darwin Kernel Version 23.6.0:

Subsystem

No response

What steps will reproduce the bug?

I have noticed some unexpected behavior in the coverage report when running tests that involve module mocking.

Minimal repo that exposes the issue: https://github.com/edge33/node-test-issue-repro/tree/main

Consider the following module (sum.js):

// console.log('imported')

const data = { type: 'object' }

// console.log(data)

export const sum = (a, b) => a + b

export const getData = () => data

And the following module (module.js), which imports sum.js:

import { getData, sum } from './sum.js'

export const theModuleSum = (a, b) => sum(a, b)

export const theModuleGetData = () => getData()

Now let’s create a test file for the modules just described (theModule.test.js):

import { describe, it, mock } from "node:test";

describe('module test with mock', async () => {

    mock.module('./sum.js', {
        namedExports: {
            sum: (a, b) => 1,
            getData: () => ({})
        }   
    });

    const { theModuleSum, theModuleGetData } = await import('./module.js');

    it('tests correct thing', (t) => {
        t.assert.deepStrictEqual(theModuleSum(1, 2), 1)
        t.assert.deepStrictEqual(theModuleGetData(), {})
    });
});

This file tests the module.js file, but mocks out sum.js.

If you run the tests with npm run test or:

node --disable-warning=ExperimentalWarning      --experimental-test-module-mocks      --disable-warning=MaxListenersExceededWarning      --test      --experimental-test-coverage      --test-reporter=spec

This is the output I get:

▶ module test with mock
  ✔ tests correct thing (0.640542ms)
✔ module test with mock (1.056959ms)
ℹ tests 1
ℹ suites 1
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 63.019584
ℹ start of coverage report
ℹ ---------------------------------------------------------------------
ℹ file                 | line % | branch % | funcs % | uncovered lines
ℹ ---------------------------------------------------------------------
ℹ src                  |        |          |         | 
ℹ  module.js           | 100.00 |   100.00 |  100.00 | 
ℹ  sum.js              | 100.00 |    50.00 |  100.00 | 
ℹ  theModule.test.js   | 100.00 |   100.00 |  100.00 | 
ℹ ---------------------------------------------------------------------
ℹ all files            | 100.00 |    90.00 |  100.00 | 
ℹ ---------------------------------------------------------------------

Notice that sum.js coverage is reported as 50%, and no uncovered lines are listed.
The expected behavior here would be to have the uncovered lines actually reported.

Another thing to notice is that the sum.js module is mocked out — hence, I would expect zero coverage for this module, as it is not being tested.

Let’s compare this to how it is printed by c8, using:

c8 node --test         --disable-warning=ExperimentalWarning         --experimental-test-module-mocks         --disable-warning=MaxListenersExceededWarning

▶ module test with mock
  ✔ tests correct thing (0.605792ms)
✔ module test with mock (0.9895ms)
ℹ tests 1
ℹ suites 1
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 62.134916
-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------|---------|----------|---------|---------|-------------------
All files  |     100 |       80 |     100 |     100 |                   
 module.js |     100 |      100 |     100 |     100 |                   
 sum.js    |     100 |       50 |     100 |     100 | 9                 
-----------|---------|----------|---------|---------|-------------------

c8 reports line 9 of sum.js as uncovered, but that line has only an export statement.

Now, if we remove the commented line from sum.js:

const data = { type: 'object' }

export const sum = (a, b) => a + b

export const getData = () => data

Node Test Report:

▶ module test with mock
  ✔ tests correct thing (0.625375ms)
✔ module test with mock (1.020292ms)
ℹ tests 1
ℹ suites 1
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 61.755959
ℹ start of coverage report
ℹ ---------------------------------------------------------------------
ℹ file                 | line % | branch % | funcs % | uncovered lines
ℹ ---------------------------------------------------------------------
ℹ src                  |        |          |         | 
ℹ  module.js           | 100.00 |   100.00 |  100.00 | 
ℹ  sum.js              | 100.00 |   100.00 |  100.00 | 
ℹ  theModule.test.js   | 100.00 |   100.00 |  100.00 | 
ℹ ---------------------------------------------------------------------
ℹ all files            | 100.00 |   100.00 |  100.00 | 
ℹ ---------------------------------------------------------------------

c8 Report:

▶ module test with mock
  ✔ tests correct thing (0.60725ms)
✔ module test with mock (0.989166ms)
ℹ tests 1
ℹ suites 1
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 62.5515
-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------|---------|----------|---------|---------|-------------------
All files  |     100 |      100 |     100 |     100 |                   
 module.js |     100 |      100 |     100 |     100 |                   
 sum.js    |     100 |      100 |     100 |     100 |                   
-----------|---------|----------|---------|---------|-------------------

In this case, there are no unexpected lines reported, but 100% coverage is still shown for an unused module.

---

How often does it reproduce? Is there a required condition?

systematically

What is the expected behavior? Why is that the expected behavior?

Coverage should be computed only when a module is imported directly, and the covered lines should be reported correctly.

In the case of mocked modules, they should not be considered part of the covered code.

What do you see instead?

Unexpected lines are reported as uncovered, and 100% coverage is shown for modules that are not imported (mocked).

Additional information

No response