Failure in code coverage job

[mhdawson]

• Version: master
• Platform: linux
• Subsystem: coverage

Coverage job has failed the last 2 runs:
https://ci.nodejs.org/job/node-test-commit-linux-coverage/nodes=benchmark/800/console

23:23:00 match failed
23:23:00 line=0
23:23:00 expect=^punycode\.js\:42$
23:23:00 actual=punycode.js:3166
23:23:00 not ok 2402 message/core_line_numbers
23:23:00   ---
23:23:00   duration_ms: 0.608
23:23:00   severity: fail
23:23:00   exitcode: 1
23:23:00   stack: |-
23:23:00     punycode.js:3166
23:23:00       throw new RangeError(errors[type]);
23:23:00       ^
23:23:00     
23:23:00     RangeError: Invalid input
23:23:00         at error (punycode.js:3166:9)
23:23:00         at Object.decode (punycode.js:3475:9)
23:23:00         at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/test/message/core_line_numbers.js:11:10)
23:23:00         at Module._compile (internal/modules/cjs/loader.js:10552:30)
23:23:00         at Object.Module._extensions..js (internal/modules/cjs/loader.js:10577:10)
23:23:00         at Module.load (internal/modules/cjs/loader.js:10348:32)
23:23:00         at tryModuleLoad (internal/modules/cjs/loader.js:10227:12)
23:23:00         at Function.Module._load (internal/modules/cjs/loader.js:10215:3)
23:23:00         at Function.Module.runMain (internal/modules/cjs/loader.js:10671:12)
23:23:00         at startup (internal/bootstrap/node.js:7565:27)
23:23:00   ...

[mhdawson]

Looks like it has been failing in previous runs as well, but those were reported as green. Not sure why.

[mhdawson]

I see, failure is not related to the test but problems at the end of the job

23:23:34 make[1]: *** [test-ci] Error 1
23:23:34 make[1]: Leaving directory `/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark'
23:23:34 make: [coverage-test] Error 2 (ignored)
23:23:34 mv lib lib__
23:23:34 mv lib_ lib
23:23:34 mkdir -p coverage .cov_tmp
23:23:34 ./node ./node_modules/.bin/nyc merge 'out/Release/.coverage' \
23:23:34 		.cov_tmp/libcov.json
23:29:13 coverage files in out/Release/.coverage merged into .cov_tmp/libcov.json
23:29:13 (cd lib && ../node ../node_modules/.bin/nyc report \
23:29:13 		--temp-directory "/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/.cov_tmp" \
23:29:13 		--report-dir "/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/coverage")
23:29:14 /home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/node_modules/yargs/yargs.js:1133
23:29:14       else throw err
23:29:14            ^
23:29:14 
23:29:14 Error: ENOENT: no such file or directory, scandir '/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/lib/.nyc_output'
23:29:14     at Object.readdirSync (fs.js:22200:3)
23:29:14     at NYC.eachReport (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/index.js:525:31)
23:29:14     at NYC.getCoverageMapFromAllCoverageFiles (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/index.js:424:8)
23:29:14     at NYC.report (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/index.js:442:18)
23:29:14     at Object.exports.handler (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/lib/commands/report.js:48:7)
23:29:14     at Object.runCommand (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/node_modules/yargs/lib/command.js:235:44)
23:29:14     at Object.parseArgs [as _parseArgs] (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/node_modules/yargs/yargs.js:1046:30)
23:29:14     at Object.parse (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/node_modules/yargs/yargs.js:551:25)
23:29:14     at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/node_modules/nyc/bin/nyc.js:24:35)
23:29:14     at Module._compile (internal/modules/cjs/loader.js:10552:30)
23:29:14 make: *** [coverage-test] Error 1
23:29:14 Build step 'Execute shell' marked build as failure
23:29:14 Sending e-mails to: [email protected] [email protected]
23:29:15 Notifying upstream projects of job completion
23:29:15 Finished: FAILURE

[mhdawson]

Running locally to see if the issue recreates there as well.

[mhdawson]

Same failure locally.

[mhdawson]

Don't see any recent changes in Makefile, guessing it might be a change in nyc

[mhdawson]

Looking at npm does not look like nyc was updated recently

[mhdawson]

Going to build with commit for last passing run to rule out any external changes.

[mhdawson]

If I did it correctly still seemed to fail after checking out version that passed in last good coverage job.

[mhdawson]

Confirmed that the jobs seem to delete the dependencies each run and install new vesrions of nyc etc. On the other hand using an older version seems to get further but still does not seem to work properly.

[mhdawson]

This change in nyc seems relevant as it looks like it affects the call that is failing.

istanbuljs/nyc@ccf42df

[mhdawson]

So looking at nyc it seems like there were a number of changes that would have broken our testing. I had tried some earlier versions (ex 12.X) and they did not seem to resolve the problem for me. It also looks like the last release was over a month ago so this falls into the category of I'm not sure how it was working, but currently testing new options to our nyc commands to get it working again.

[mhdawson]

PR submitted to fix yesterday and have requested fasttrack.