circleci: publish test reports

[alxndrsn]

No description provided.

[matthew-white]

I haven't looked deeply into this, but the idea of seeing test insights in CircleCI seems potentially useful! It sounds like it could help us identify intermittent test failures and slow tests.

[alxndrsn]

Example:

• working: https://app.circleci.com/insights/github/alxndrsn/odk-central-backend/workflows/workflow/tests?branch=circle-ci-test-reports
• current master (not working): https://app.circleci.com/insights/github/getodk/central-backend/workflows/workflow/tests?branch=master

[alxndrsn]

I've rebased this onto latest master. Some additional thoughts:

• the current test output location seems esoteric (~/junit/)
• it's a shame that we don't get the standard text output as well as the JUnit XML output; not supported directly by mocha, but maybe possible with e.g.
  • https://www.npmjs.com/package/mocha-multi
  • Multiple reporters support (continuation of #930) mochajs/mocha#1360 (comment)
  • https://www.npmjs.com/package/mocha-multi-reporters
  • https://stackoverflow.com/questions/37965049/mocha-multi-reporter-with-junit

---

@matthew-white would this be acceptable as-is? Is it worth trying to address points raised above?

[matthew-white]

the current test output location seems esoteric (~/junit/)

That seems OK to me. 🤷

it's a shame that we don't get the standard text output as well as the JUnit XML output

I do think that that has the potential to make things harder to debug. With the current output, you can scroll up to a failed test to check whether there was any logging or an unhandled promise rejection before the test failure. But with the JUnit output, it looks like you just see the test failure and its associated stack trace. If there's a reasonable way to output both, that seems ideal. It's unfortunate though that Mocha doesn't support that out of the box. 😭

[sadiqkhoja]

I am interesting in the recent feature of CircleCI "rerunning failed tests", for that we would need this step to be:

      - run: 
          name: Run tests
          command: |
            mkdir test-results
            TEST_FILES=$(circleci tests glob "test/**/*.js")
            echo "$TEST_FILES" | circleci tests run --command="xargs npx mocha --exit --reporter mocha-junit-reporter --reporter-options mochaFile=./test-results/junit.xml"
      - store_test_results:
          path: test-results

[alxndrsn]

👍 that could be a good next step

[sadiqkhoja]

How about using mocha-multi-reporters with mocha-junit-reporter? I am happy without text output as well

[alxndrsn]

How about using mocha-multi-reporters with mocha-junit-reporter? I am happy without text output as well

I've implemented this directly - it turns out it's quite simple:

const mocha = require('mocha');
const MochaJunitReporter = require('mocha-junit-reporter');

module.exports = function CiMochaReporter(runner) {
  new mocha.reporters.Spec(runner);
  new MochaJunitReporter(runner);
};

[alxndrsn]

@matthew-white @sadiqkhoja this is ready for review again.

• test results are now stored in ./junit-reports/
• normal text output and junit XML output are both included

Example failure at:

• logs: https://app.circleci.com/pipelines/gh/alxndrsn/odk-central-backend/1052/workflows/98e85503-79a0-4699-aaaa-875d493a9f03/jobs/976/steps
• CircleCI test integration: https://app.circleci.com/pipelines/gh/alxndrsn/odk-central-backend/1052/workflows/98e85503-79a0-4699-aaaa-875d493a9f03/jobs/976/tests

[sadiqkhoja]

Looks great to me