Merging coverage files doesn't produce accurate cumulative result

[stahloss]

I execute the following nyc commands, after generating individual coverage reports on the same codebase using different test executions run by Jest and Cypress.

nyc merge coverage-jest combined/coverage-jest.json
nyc merge coverage-cypress combined/coverage-cypress.json
nyc report combined --reporter html --reporter text

Expected Behavior

I'd expect this to show a correct cumulative coverage report of .ts and .html files.

Observed Behavior

1. Coverage of .html files is not shown in merged report, while they are show in individial report.
2. Coverage of some .ts files is shown correctly, but for others a Cannot read property 'locations' of undefined TypeError is shown.

Troubleshooting steps

• still occurring when I put cache: false in my nyc config

Environment Information

  System:
    OS: Windows 10 10.0.18363
    CPU: (8) x64 Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
    Memory: 7.97 GB / 31.88 GB
  Binaries:
    Node: 13.6.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.0 - C:\Program Files\nodejs\yarn.CMD
    npm: 6.13.4 - C:\Program Files\nodejs\npm.CMD
  npmPackages:
    istanbul-instrumenter-loader: 1.2.0 => 1.2.0
    istanbul-lib-coverage: 1.2.1 => 1.2.1
    nyc: ^15.0.1 => 15.0.1
    source-map-support: ^0.5.16 => 0.5.16
    ts-node: ^7.0.1 => 7.0.1
    typescript: ~3.7.4 => 3.7.5

For Cypress coverage I use above dependencies as well as @cypress/code-coverage:^3.0.2.
I'm using jest: 24.1.0 to produce unit test coverage.

[coreyfarrell]

This is difficult for us to support as you are not using nyc to generate any coverage. From the limited information you shared I see some version conflicts:

• istanbul-instrument-loader is using a really old version of istanbul-lib-instrument.
• jest 24 uses the previous version of all istanbul libraries
• @cypress/code-coverage@3 and nyc@15 both use the latest versions of istanbul libraries

As for your command, nyc report combined --reporter html --reporter text is incorrect. You need nyc report -t combined --reporter html --reporter text.

Further you have not linked a repository as required so we do not know how you are even running the tests, what kind of configurations you are using, etc.

[stahloss]

Thanks for your response. Yes, Jest has the istanbul instrumenter built in, so I tried to align other dependencies to that, because before the downgrade I got this error: Invalid file coverage object, missing keys, found:data. @cypress/code-coverage doesn't instrument by itself, so I can control that as opposed to Jest.

You're right the command I posted is wrong, but I used the correct one, so it's a typo.

Yes, sorry for that, I was hoping the information I supplies would be enough, because my production setup if fairly complex with a mix of nx with angular, cypress, jest and nyc. It would take some time to duplicate that setup.

I've tried to bump jest to the latest version (25.3.0) but wasn't able to merge.
From what you said I understand that it should work if I align the libraries correctly and I see that @jest/[email protected] is using [email protected] so if I align that with my cypress instrumentation, it should work.

I'll try to make it work with [email protected], if the problems persists I'll make a demo project or else I'll close the issue.

EDIT
I've tried with the following dependencies:

  npmPackages:
    istanbul-instrumenter-loader: 3.0.1 => 3.0.1
    istanbul-lib-coverage: 3.0.0 => 3.0.0
    nyc: ^15.0.1 => 15.0.1
    source-map-support: ^0.5.16 => 0.5.16
    ts-node: ^7.0.1 => 7.0.1
    typescript: ~3.7.4 => 3.7.5

And [email protected]. This resulted in Invalid file coverage object, missing keys, found:data. Jest's coverage-final.json indeed contains data properties. Does this relate to #1226?

I've replaced istanbul-instrumenter-loader with babel-plugin-istanbul, so the outdated istanbul-lib-instrument is gone. The coverage files still cannot be merged with the one Jest produces.

[lirantal]

@D0rmouse the reason you're still getting this error Invalid file coverage object, missing keys, found:data is because you aren't producing and merging the correct reporter file.

Make sure these happen:

1. Specify the json reporter. If you're using jest it's something like this for its config:

"coverageReporters": [
      "lcov",
      "text",
      "json"
    ]

The important part being that there's a json entry there in the array

2. Then after you run the tests you'll have a coverage-final.json created in the coverage dir. That's the file you need to copy to the combined directory along with the other file of the same name from a different directory. So both of these files should exist in a directory to combine them (give them different names)

3. Then run the nyc merge and report commands

Hope it helps 🤞
Goodluck!

[stahloss]

@lirantal I do have a json entry in my coverageReporters array. I got json from the start it's just not merging correctly because the formats don't match.

[lirantal]

@D0rmouse can you confirm that you have that file called coverage-final.json created for both of your coverage report runs? Copy them to a directory, and then run the merge on that when they are both there alone with no other files.

[stahloss]

@lirantal Yes I can, the output of the Jest test is simply different than my Cypress coverage output.
I created an issue in the Jest repo as well. Can't reproduce using a minimal setup unfortunately.
jestjs/jest#9808

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[quisido]

I'm experiencing this same problem.

My coverage results look like so:

cypress/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "1": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": 3 }
      },
      "2": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "3": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "4": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "5": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "6": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 47 },
          "end": { "line": 9, "column": 1 }
        },
        "line": 3
      },
      "1": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": 3 }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": 3 }
          },
          { "start": {}, "end": {} }
        ],
        "line": 5
      },
      "1": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 6, "1": 4, "2": 2, "3": 4, "4": 2, "5": 2, "6": 2 },
    "f": { "0": 4, "1": 2 },
    "b": { "0": [0, 4], "1": [2] },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "81128de9166cac88dff871fa724f973dcd80167a"
  }
}

jest/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "1": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "2": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "3": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "4": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "5": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 1, "1": 1, "2": 1, "3": 1, "4": 0, "5": 1 },
    "f": { "0": 1 },
    "b": { "0": [1] }
  }
}

The expected outcome would be path/to/file.ts having 100% coverage, since the gaps on one test are covered by the other test. However, instead, what you see is the gaps being reported:

combined/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "1": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": 3 }
      },
      "2": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "3": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "4": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "5": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "6": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 47 },
          "end": { "line": 9, "column": 1 }
        },
        "line": 3
      },
      "1": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": 3 }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": 3 }
          },
          { "start": {}, "end": {} }
        ],
        "line": 5
      },
      "1": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 7, "1": 4, "2": 3, "3": 4, "4": 3, "5": 3, "6": 3 },
    "f": { "0": 4, "1": 3 },
    "b": { "0": [0, 4], "1": [3] },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "81128de9166cac88dff871fa724f973dcd80167a"
  }
}

The "coverage" merge is really just concatenated instead of merging.

It appears that: line 1 characters 1-10 uncovered + line 1 characters 1-5 covered + line 1 characters 5-10 covered = line 1 characters 1-10 uncovered (wrong) + line 1 characters 1-10 covered (right).

The uncovered characters are still shown as uncovered even though their character-span is covered by a different part of the map. Just guessing.

Here is a nice visual LCOV report for the above JSON:

Cypress (fully covered except for 1 branch):

Jest (specifically covering the above branch):

Combined report (expected to be fully covered):

Any advice to unblock this? I don't want to have to write a JSON merger manually. 😓

Maybe this is just a weird issue with TypeScript.

Below, how are imports and exports not covered? The export statement isn't covered, but the inside of the function somehow is. The inside references an import, but the import isn't covered. Nonsensical. 😵

[nishantchapa]

I am facing the same issue as the above user @CharlesStover
Any update on how to fix this issue?

[quisido]

My issue ended up simply being that the version of Jest needed to be downgraded to 26. No idea why 27 is broken. The instrumentation of Jest 27 is simply off and doesn't match the one generated by react-scripts when running the local application in dev mode for Cypress.

[nishantchapa]

I am trying to integrate this to existing application which is using Jest 27.0.6 and can't downgrade to lower version.
Any other way to fix this?

[quisido]

I spent a long time trying to fix it, and was not able to solve it with Jest 27. A downgrade to 26 resolved it like magic. This may need to be a bug report for the Jest team, as it appears 27 is not instrumenting coverage correctly, producing incorrect results for TypeScript files (this occurs with either ts-jest or Babel for transpilation). Things like "import type" should not be showing as uncovered, ever.

If you want to open a ticket with the Jest team, I think that's the right solution, and you could link to my screenshots and data in this issue as a reference for them.

[jogelin]

I have the same issue just by merging multiple reports from various cypress runs in parallel. The merge is just taking the last report base on the order:

coverage-final-A.json

{
  "global.component.ts": {
    // ...same value in both files
    "s": {
      "0": 0,
      "1": 0,
      "2": 0,
      "3": 0,
      "4": 0,
      "5": 0,
      "6": 0,
      "7": 0,
      "8": 0,
      "9": 0,
      "10": 0,
      "11": 0,
      "12": 0,
      "13": 0,
      "14": 0,
      "15": 0,
      "16": 0,
      "17": 0,
      "18": 0,
      "19": 0,
      "20": 64,
      "21": 0
    },
    "f": { "0": 0 },
    "b": { "0": [0, 0] }
  }
}

coverage-final-B.json

{
  "global.component.ts": {
    // ...same value in both files
    "s": {
      "0": 77,
      "1": 77,
      "2": 5151,
      "3": 77,
      "4": 77,
      "5": 5151,
      "6": 77,
      "7": 77,
      "8": 5151,
      "9": 77,
      "10": 77,
      "11": 5151,
      "12": 77,
      "13": 77,
      "14": 5151,
      "15": 77,
      "16": 77,
      "17": 77,
      "18": 77,
      "19": 77,
      "20": 154,
      "21": 77
    },
    "f": { "0": 77 },
    "b": { "0": [77, 77] }
}

In this case, only the result of coverage-final-A.json will be taken, so without coverage.
If I rename coverage-final-A.json to coverage-final-Z.json, then the order is different and the result of coverage-final-B.jsonwill be taken.

Any ideas? Should rewrite a merge myself?

[jogelin]

@D0rmouse did you finally find a good configuration to combine multiple coverage reports?

I have the same type of stack with Nx + Cypress + Storybook + Jest, generating reports separately works fine but when I want to merge, it is not good.

It makes sense because each tool are using different versions for each generation.

[mmisty]

Hi all,
Maybe it will help someone:
I had the same problem (merging of cypress and jest coverage wasn't right) and I got that fixed after adding coverageProvider into jest config:
coverageProvider: 'v8',

[sscaff1]

Same issue. Any updates. Screenshots below:

Cypress Coverage - correct:

Jest Coverage - correct:

Combined Coverage - wrong:

I'm using nyc merge and nyc report just like the original reporter of this issue. The final JSON just doesn't appear correct.

@mmisty I tried your suggestion and it didn't work. I end up with this instead:

[mmisty]

@sscaff1, hi Steven
After sometime I ended up with this cypress-template.

Key points as I remember:

• I created integration/plugins/ts-preprocessor.ts to correctly instrument the code
• added nyc.config.js
• And created merging script(partly based on @cypress/code-coverage) - .scripts/merge.js:

node ./.scripts/merge.js --cypress reports/coverage-cypress --jest reports/coverage-jest --out reports/coverage-temp --report reports/coverage-full

Jest

Cypress

Merged:

[janow25]

In my case, I fixed this issue by using the babel coverage provider in my Jest config coverageProvider: 'babel',. In my babel config I added a enviroment variable to check if the Code should be instrumented or not.

const shouldInstrumentCode = 'INSTRUMENT_CODE' in process.env

//console.log('shouldInstrumentCode', shouldInstrumentCode)



module.exports = {

    presets: ["next/babel"],

    plugins: shouldInstrumentCode? ["istanbul"]: []

  };

To run this in instrument mode, add a script like this to your package.json "dev:test": "set INSTRUMENT_CODE=1 && next dev",

This finally provides accurate coverage results for me.

[filipomar]

I was facing the same issue as OP mentioned, as well as of other people here in the thread.

My issue was a wrong merged report based on:

• cypress e2e tests on parcel (See setup)
• jest unit tests of react typescript components transformed with ts-jest

Snooping around at the code/produced reports, I found that:

1. Jest and Cypress emitted wildy different report lines/statements and so on (2x as many lines if using v8)
2. The mergeProp on instambul-lib-coverage was not made to handle reports that disagreed on what lines/statements/functions/branches did or did not exist

So the actual problem was getting one or the other inline with their counterpart.

First I tried to force my npm dependencies to all be the same, assuming this was the issue (different versions of istanbul, which I did have installed):

Example package.json:

    "overrides": {
        "@cypress/code-coverage": {
            "istanbul-lib-coverage": "3.2.0"
        },
        "nyc": {
            "istanbul-lib-instrument": "5.2.1"
        }

But that was a dud, as the problem lied with ts-jest transformation strategy, and swapped that out and now I am using babel:

module.exports = {
    presets: [['@babel/preset-env', { targets: { node: 'current' } }], ['@babel/preset-react', { runtime: 'automatic' }], '@babel/preset-typescript'],
}

That seems to have finally aligned the stars, and there a difference of about 300 lines, which I suspect is some other config I still have to tweak.

[KissBalazs]

I'm encountering the same problem. When I try to merge two coverage-final.json files, the output is erroneous.

I have posted a Stack Overflow question with the details here .

[snake-py]

I am having the same issue. I add both coverage files I generated:

This one was generated using vitest coverage-final.json
This one was generated using the storybook addon coverage-storybook.json

Here are my configs for vitest

    coverage: {
      all: true,
      clean: false,
      exclude: [
      ...
      ],
      provider: 'istanbul',
      reporter: ['json', 'html']
    }
  },
// my command 
"test:coverage": "vitest --run --coverage",

For storybook I used:

addons: [
 {
      name: '@storybook/addon-coverage',
      options: {
        istanbul: {
          exclude: [
....
          ]
        }
      }
    },
    ]
    "test-storybook:coverage": "test-storybook --coverage --json --coverageDirectory=./coverage",

At the end I use nyc to merge the output files:

"test-all:coverage": "nyc merge coverage coverage/merged/coverage.json && nyc report -t coverage/merged --report-dir coverage/merged --reporter=html --reporter=cobertura",

In my merged coverage report I can then see:

However, when looking at only the unit test report, I can see that these lines are being hitted.

Okay I digged a little bit and it seems that the storybook addon and vitest generate slightly different line and column for the statements.

For example in one component vitest said that there is a stmt on line 38 start at col 18 and ends on the same line.

            "2": {
                "start": {
                    "line": 38,
                    "column": 18
                },
                "end": {
                    "line": 38,
                    "column": null
                }
            },

However, storybook coverage report says the statement is on line 38 in column 2:

      "3": {
        "start": {
          "line": 38,
          "column": 2
        },
        "end": {
          "line": 38,
          "column": null
        }
      },

So now the question becomes how can I make sure that both instrumentalize the code the same way?

[ruler501]

This happens with vitest 1.2.1, cypress 13.2.0, and @cypress/code-coverage 3.12.19 as well

[godlanbo]

I had the same problem, and when I merged the Cypress and Jest test reports, the result was obviously wrong.

Is there any solution?

[technikhil314]

I am also facing same issue. I am trying to migrate from jest to vitest and some tests are written so bad that I have to keep them in jest for time being but when I try to merge the coverage reports from jest and vitest it just generates wrong combined numbers.

[sk33wiff]

I can see what nyc version cypress code coverage plugin uses. How can I check what inbuilt nyc version a given jest release uses?

UPDATE: after making sure nyc versions (jest, cypress/code-coverage) match, I can confirm this issue is still happening.
I have a file that gets full 100% coverage from jest tests, but less from cypress tests. When using nyc merge, strangely the last overrides the first.

[snake-py]

@sk33wiff https://stackoverflow.com/questions/25997519/how-to-view-the-dependency-tree-of-a-given-npm-module