Coverage as part of regressions tests

[mhdawson]

@bcoe and myself have been working on:

• allow us to run coverage and have failing tests not block generation of results
• add support for a coverage level check (ex 90%) that will cause a failure @bcoe is working on that
• add job to main PR regression test that will run coverage to make sure new failing tests are
  not introduced
• Update makefile so that we don't use '-' for the coverage target. At this point we will fail is
  some other problem occurs but not if one of the known tests that affects coverage fails.

The question is what machines do we think we can target the coverage job (https://ci.nodejs.org/view/All/job/node-test-commit-coverage/) and have it run for every PR without adding undue load? I know it runs on ubuntu and targetted the last run on unbutu16 but I'm not sure if that will be a bottleneck if it runs for every PR. I am a bit concerned since there are only 2 of them. The job only takes ~7 minutes now but that might still double the load on those machines.

[rvagg]

ubuntu1604_sharedlibs_x64 for an isolated job, it'll choose one from (currently) 20 Docker containers.

[mhdawson]

@rvagg thanks will try that out.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[bcoe]

👋 I wonder if we should consider dusting this off, I feel like we've been on an upward swing with coverage without enforcing it?

[mhdawson]

I think we have been pretty successful, and have not really had any major regressions. I'd be happy to close this.