GitHub Test Annotations

.changeset/ninety-pillows-move.md

@@ -0,0 +1,7 @@
+---
+"skuba": minor
+---
+
+test: Add GitHub check run annotations
+
+`skuba test` can now automatically annotate GitHub commits when you [propagate CI environment variables and a GitHub API token](https://github.com/seek-oss/skuba/blob/master/docs/deep-dives/github.md#github-annotations). These annotations also appear inline with code under the “Files changed” tab in pull requests.

.changeset/unlucky-walls-drop.md

@@ -0,0 +1,5 @@
+---
+"skuba": patch
+---
+
+Jest.mergePreset: Allow `displayName` and `projects`

docs/cli/lint.md

@@ -65,7 +65,7 @@ you can limit this with the `--serial` flag.
 `skuba lint` can automatically emit annotations in CI.
 
 - [Buildkite annotations] are enabled when Buildkite environment variables and the `buildkite-agent` binary are present.
-- [GitHub annotations] are enabled when Buildkite and GitHub environment variables are present.
+- [GitHub annotations] are enabled when CI and GitHub environment variables are present.
 
 [`skuba format`]: #skuba-format
 [buildkite annotations]: ../deep-dives/buildkite.md#buildkite-annotations

docs/cli/test.md

@@ -22,4 +22,21 @@ Arguments are passed through to the Jest CLI:
 skuba test --coverage path/to/file.test.ts
 ```
 
+### Annotations
+
+`skuba test` can automatically emit annotations in CI.
+
+- [Buildkite annotations] are planned in future.
+- [GitHub annotations] are enabled when CI and GitHub environment variables are present.
+
+GitHub check runs are created with a default title of `skuba/test`.
+You can further qualify this by providing a [displayName] in your Jest config;
+for example, the display name `integration` will result in the title `skuba/test (integration)`.
+
+See our [Jest guide] for a more detailed configuration breakdown.
+
+[displayname]: https://jestjs.io/docs/configuration#displayname-string-object
+[github annotations]: ../deep-dives/github.md#github-annotations
 [jest]: https://jestjs.io
+[jest guide]: ../deep-dives/jest.md
+[projects]: https://jestjs.io/docs/configuration#projects-arraystring--projectconfig

docs/deep-dives/github.md

@@ -14,7 +14,7 @@ This topic details GitHub integration features baked into **skuba**.
 
 ## GitHub annotations
 
-**skuba** can annotate the first 50 issues detected by [`skuba lint`] via the [GitHub Checks API].
+**skuba** can annotate the first 50 issues detected by [`skuba lint`] and [`skuba test`] via the [GitHub Checks API].
 
 This can be enabled by propagating Buildkite environment variables and a GitHub API token (at SEEK, this token can be configured through Build Agency).
 For example, with the Docker plugin:
@@ -62,8 +62,6 @@ propagate the following environment variables to achieve the same effect:
 - `GITHUB_RUN_NUMBER`
 - `GITHUB_TOKEN`
 
-This feature is also planned for [`skuba test`] in future.
-
 **skuba**'s development API includes a [GitHub.createCheckRun] function.
 You can use this to create your own check runs from other JavaScript code running in your CI workflow.
 

docs/deep-dives/jest.md

@@ -0,0 +1,78 @@
+---
+parent: Deep dives
+---
+
+# Jest
+
+---
+
+## Configuring a single test suite
+
+In the typical case, your project has a single command to run all of its tests:
+
+```console
+skuba test --coverage
+```
+
+This works well out-of-the-box with the [annotation] features built in to [`skuba test`].
+
+---
+
+## Configuring multiple test suites
+
+If you have multiple test suites in your project,
+you may have multiple corresponding config files and commands:
+
+- `skuba test --config jest.config.ts`
+- `skuba test --config jest.config.int.ts`
+
+In this scenario, declare unique [displayName]s so that [`skuba test`] can differentiate between the test suites when annotating your builds.
+
+For example, if you set the following display names:
+
+```typescript
+// jest.config.ts
+import { Jest } from 'skuba';
+
+export default Jest.mergePreset({
+  displayName: 'unit',
+  testPathIgnorePatterns: ['\\.int\\.test\\.ts'],
+});
+```
+
+```typescript
+// jest.config.int.ts
+import { Jest } from 'skuba';
+
+export default Jest.mergePreset({
+  displayName: 'integration',
+  testPathIgnorePatterns: ['\\.unit\\.test\\.ts'],
+});
+```
+
+**skuba** will generate two GitHub check runs titled `skuba/test (unit)` and `skuba/test (integration)` respectively.
+
+Alternatively, you can declare multiple [projects] in a single config file:
+
+```typescript
+// jest.config.ts
+import { Jest } from 'skuba';
+
+export default Jest.mergePreset({
+  projects: [
+    Jest.mergePreset({
+      displayName: 'unit',
+      testPathIgnorePatterns: ['\\.int\\.test\\.ts'],
+    }),
+    Jest.mergePreset({
+      displayName: 'integration',
+      testPathIgnorePatterns: ['\\.unit\\.test\\.ts'],
+    }),
+  ],
+});
+```
+
+[`skuba test`]: ../cli/test.md
+[annotation]: ../cli/test.md#annotations
+[displayname]: https://jestjs.io/docs/configuration#displayname-string-object
+[projects]: https://jestjs.io/docs/configuration#projects-arraystring--projectconfig

jest-preset.js

@@ -44,4 +44,5 @@ module.exports = {
     '/node_modules.*/',
     '<rootDir>/(coverage|dist|lib|tmp).*/',
   ],
+  reporters: ['default', require.resolve('./lib/cli/test/reporters/github')],
 };

jest.config.int.ts

@@ -1,6 +1,7 @@
 import { Jest } from './src';
 
 export default Jest.mergePreset({
+  displayName: 'integration',
   setupFiles: ['<rootDir>/jest.setup.ts'],
   testPathIgnorePatterns: ['<rootDir>/template/', '/test\\.ts'],
   watchPathIgnorePatterns: [

package.json

@@ -41,6 +41,7 @@
   "devDependencies": {
     "@changesets/cli": "2.18.0",
     "@changesets/get-github-info": "0.5.0",
+    "@jest/reporters": "27.3.1",
     "@types/concurrently": "6.3.0",
     "@types/ejs": "3.1.0",
     "@types/express": "4.17.13",

src/api/jest/index.ts

@@ -10,9 +10,11 @@ type Props = Pick<
   | 'collectCoverageOnlyFrom'
   | 'coveragePathIgnorePatterns'
   | 'coverageThreshold'
+  | 'displayName'
   | 'globals'
   | 'globalSetup'
   | 'globalTeardown'
+  | 'projects'
   | 'setupFiles'
   | 'setupFilesAfterEnv'
   | 'snapshotSerializers'

src/cli/lint/annotate/github/index.ts

@@ -28,7 +28,6 @@ export const createGitHubAnnotations = async (
   ];
 
   const isOk = eslint.ok && prettier.ok && tscOk;
-  const conclusion = isOk ? 'success' : 'failure';
 
   const summary = isOk
     ? '`skuba lint` passed.'
@@ -40,7 +39,7 @@ export const createGitHubAnnotations = async (
     name: 'skuba/lint',
     summary,
     annotations,
-    conclusion,
+    conclusion: isOk ? 'success' : 'failure',
     title: `${build} ${isOk ? 'passed' : 'failed'}`,
   });
 };

src/cli/test.ts → src/cli/test/index.ts

@@ -2,9 +2,7 @@ import { run } from 'jest';
 
 export const test = () => {
   // This is usually set in `jest-cli`'s binary wrapper
-  if (process.env.NODE_ENV === undefined) {
-    process.env.NODE_ENV = 'test';
-  }
+  process.env.NODE_ENV ??= 'test';
 
   const argv = process.argv.slice(2);
 

src/cli/test/reporters/github/annotations.test.ts

@@ -0,0 +1,153 @@
+import type { SerializableError, TestResult } from '@jest/test-result';
+
+import { createAnnotations } from './annotations';
+
+jest.spyOn(process, 'cwd').mockReturnValue('/workdir/skuba');
+
+it('should create annotations from Jest test results', () => {
+  const testResult = {
+    leaks: false,
+    numFailingTests: 1,
+    numPassingTests: 0,
+    numPendingTests: 0,
+    numTodoTests: 0,
+    openHandles: [],
+    perfStats: {
+      end: 1636247736632,
+      runtime: 2646,
+      slow: false,
+      start: 1636247733986,
+    },
+    skipped: false,
+    snapshot: {
+      added: 0,
+      fileDeleted: false,
+      matched: 0,
+      unchecked: 0,
+      uncheckedKeys: [],
+      unmatched: 0,
+      updated: 0,
+    },
+    testFilePath: '/workdir/skuba/src/test.test.ts',
+    testResults: [
+      {
+        ancestorTitles: [],
+        duration: 3,
+        failureDetails: [
+          {
+            matcherResult: {
+              actual: 'b',
+              expected: 'a',
+              message:
+                '\u001b[2mexpect(\u001b[22m\u001b[31mreceived\u001b[39m\u001b[2m).\u001b[22mtoBe\u001b[2m(\u001b[22m\u001b[32mexpected\u001b[39m\u001b[2m) // Object.is equality\u001b[22m\n\nExpected: \u001b[32m"a"\u001b[39m\nReceived: \u001b[31m"b"\u001b[39m',
+              name: 'toBe',
+              pass: false,
+            },
+          },
+        ],
+        failureMessages: [
+          'Error: \u001b[2mexpect(\u001b[22m\u001b[31mreceived\u001b[39m\u001b[2m).\u001b[22mtoBe\u001b[2m(\u001b[22m\u001b[32mexpected\u001b[39m\u001b[2m) // Object.is equality\u001b[22m\n\nExpected: \u001b[32m"a"\u001b[39m\nReceived: \u001b[31m"b"\u001b[39m\n    at Object.<anonymous> (/workdir/skuba/src/test.test.ts:2:15)\n    at Promise.then.completed (/workdir/skuba/node_modules/jest-circus/build/utils.js:390:28)\n    at new Promise (<anonymous>)\n    at callAsyncCircusFn (/workdir/skuba/node_modules/jest-circus/build/utils.js:315:10)\n    at _callCircusTest (/workdir/skuba/node_modules/jest-circus/build/run.js:218:40)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at _runTest (/workdir/skuba/node_modules/jest-circus/build/run.js:155:3)\n    at _runTestsForDescribeBlock (/workdir/skuba/node_modules/jest-circus/build/run.js:66:9)\n    at run (/workdir/skuba/node_modules/jest-circus/build/run.js:25:3)\n    at runAndTransformResultsToJestFormat (/workdir/skuba/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:167:21)',
+        ],
+        fullName: 'should output a',
+        invocations: 1,
+        location: null,
+        numPassingAsserts: 0,
+        status: 'failed',
+        title: 'should output a',
+      },
+    ],
+    failureMessage:
+      "\u001b[1m\u001b[31m  \u001b[1m● \u001b[22m\u001b[1mshould output a\u001b[39m\u001b[22m\n\n    \u001b[2mexpect(\u001b[22m\u001b[31mreceived\u001b[39m\u001b[2m).\u001b[22mtoBe\u001b[2m(\u001b[22m\u001b[32mexpected\u001b[39m\u001b[2m) // Object.is equality\u001b[22m\n\n    Expected: \u001b[32m\"a\"\u001b[39m\n    Received: \u001b[31m\"b\"\u001b[39m\n\u001b[2m\u001b[22m\n\u001b[2m    \u001b[0m \u001b[90m 1 |\u001b[39m it(\u001b[32m'should output a'\u001b[39m\u001b[33m,\u001b[39m () \u001b[33m=>\u001b[39m {\u001b[0m\u001b[22m\n\u001b[2m    \u001b[0m\u001b[31m\u001b[1m>\u001b[22m\u001b[2m\u001b[39m\u001b[90m 2 |\u001b[39m   expect(\u001b[32m'b'\u001b[39m)\u001b[33m.\u001b[39mtoBe(\u001b[32m'a'\u001b[39m)\u001b[33m;\u001b[39m\u001b[0m\u001b[22m\n\u001b[2m    \u001b[0m \u001b[90m   |\u001b[39m               \u001b[31m\u001b[1m^\u001b[22m\u001b[2m\u001b[39m\u001b[0m\u001b[22m\n\u001b[2m    \u001b[0m \u001b[90m 3 |\u001b[39m })\u001b[33m;\u001b[39m\u001b[0m\u001b[22m\n\u001b[2m    \u001b[0m \u001b[90m 4 |\u001b[39m\u001b[0m\u001b[22m\n\u001b[2m\u001b[22m\n\u001b[2m      \u001b[2mat Object.<anonymous> (\u001b[22m\u001b[2m\u001b[0m\u001b[36msrc/test.test.ts\u001b[39m\u001b[0m\u001b[2m:2:15)\u001b[22m\u001b[2m\u001b[22m\n",
+  } as TestResult;
+
+  const annotations = createAnnotations([testResult]);
+  expect(annotations).toStrictEqual([
+    {
+      annotation_level: 'failure',
+      path: 'src/test.test.ts',
+      start_line: 2,
+      end_line: 2,
+      start_column: 15,
+      end_column: 15,
+      message:
+        'Error: \x1B[2mexpect(\x1B[22m\x1B[31mreceived\x1B[39m\x1B[2m).\x1B[22mtoBe\x1B[2m(\x1B[22m\x1B[32mexpected\x1B[39m\x1B[2m) // Object.is equality\x1B[22m\n' +
+        '\n' +
+        'Expected: \x1B[32m"a"\x1B[39m\n' +
+        'Received: \x1B[31m"b"\x1B[39m\n' +
+        '    at Object.<anonymous> (/workdir/skuba/src/test.test.ts:2:15)\n' +
+        '    at Promise.then.completed (/workdir/skuba/node_modules/jest-circus/build/utils.js:390:28)\n' +
+        '    at new Promise (<anonymous>)\n' +
+        '    at callAsyncCircusFn (/workdir/skuba/node_modules/jest-circus/build/utils.js:315:10)\n' +
+        '    at _callCircusTest (/workdir/skuba/node_modules/jest-circus/build/run.js:218:40)\n' +
+        '    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n' +
+        '    at _runTest (/workdir/skuba/node_modules/jest-circus/build/run.js:155:3)\n' +
+        '    at _runTestsForDescribeBlock (/workdir/skuba/node_modules/jest-circus/build/run.js:66:9)\n' +
+        '    at run (/workdir/skuba/node_modules/jest-circus/build/run.js:25:3)\n' +
+        '    at runAndTransformResultsToJestFormat (/workdir/skuba/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:167:21)',
+      title: 'Jest',
+    },
+  ]);
+});
+
+it('should create annotations from Jest exec errors', () => {
+  const error = {
+    message:
+      "\x1B[96msrc/test.ts\x1B[0m:\x1B[93m1\x1B[0m:\x1B[93m1\x1B[0m - \x1B[91merror\x1B[0m\x1B[90m TS6133: \x1B[0m'a' is declared but its value is never read.\n" +
+      '\n' +
+      "\x1B[7m1\x1B[0m import { a } from 'b';\n" +
+      '\x1B[7m \x1B[0m \x1B[91m~~~~~~~~~~~~~~~~~~~~~~\x1B[0m\n' +
+      "\x1B[96msrc/test.ts\x1B[0m:\x1B[93m1\x1B[0m:\x1B[93m19\x1B[0m - \x1B[91merror\x1B[0m\x1B[90m TS2307: \x1B[0mCannot find module 'b' or its corresponding type declarations.\n" +
+      '\n' +
+      "\x1B[7m1\x1B[0m import { a } from 'b';\n" +
+      '\x1B[7m \x1B[0m \x1B[91m                  ~~~\x1B[0m',
+  } as SerializableError;
+  const testResult = {
+    failureMessage:
+      "  \u001b[1m● \u001b[22mTest suite failed to run\n\n    \u001b[96msrc/test.ts\u001b[0m:\u001b[93m1\u001b[0m:\u001b[93m1\u001b[0m - \u001b[91merror\u001b[0m\u001b[90m TS6133: \u001b[0m'a' is declared but its value is never read.\n\n    \u001b[7m1\u001b[0m import { a } from 'b';\n    \u001b[7m \u001b[0m \u001b[91m~~~~~~~~~~~~~~~~~~~~~~\u001b[0m\n    \u001b[96msrc/test.ts\u001b[0m:\u001b[93m1\u001b[0m:\u001b[93m19\u001b[0m - \u001b[91merror\u001b[0m\u001b[90m TS2307: \u001b[0mCannot find module 'b' or its corresponding type declarations.\n\n    \u001b[7m1\u001b[0m import { a } from 'b';\n    \u001b[7m \u001b[0m \u001b[91m                  ~~~\u001b[0m\n",
+    leaks: false,
+    numFailingTests: 0,
+    numPassingTests: 0,
+    numPendingTests: 0,
+    numTodoTests: 0,
+    openHandles: [],
+    perfStats: {
+      end: 0,
+      runtime: 0,
+      slow: false,
+      start: 0,
+    },
+    skipped: false,
+    snapshot: {
+      added: 0,
+      fileDeleted: false,
+      matched: 0,
+      unchecked: 0,
+      uncheckedKeys: [],
+      unmatched: 0,
+      updated: 0,
+    },
+    testExecError: error,
+    testFilePath: '/workdir/skuba/src/test.test.ts',
+    testResults: [],
+  } as TestResult;
+
+  const annotations = createAnnotations([testResult]);
+  expect(annotations).toStrictEqual([
+    {
+      annotation_level: 'failure',
+      path: 'src/test.test.ts',
+      start_line: 1,
+      end_line: 1,
+      message:
+        "\x1B[96msrc/test.ts\x1B[0m:\x1B[93m1\x1B[0m:\x1B[93m1\x1B[0m - \x1B[91merror\x1B[0m\x1B[90m TS6133: \x1B[0m'a' is declared but its value is never read.\n" +
+        '\n' +
+        "\x1B[7m1\x1B[0m import { a } from 'b';\n" +
+        '\x1B[7m \x1B[0m \x1B[91m~~~~~~~~~~~~~~~~~~~~~~\x1B[0m\n' +
+        "\x1B[96msrc/test.ts\x1B[0m:\x1B[93m1\x1B[0m:\x1B[93m19\x1B[0m - \x1B[91merror\x1B[0m\x1B[90m TS2307: \x1B[0mCannot find module 'b' or its corresponding type declarations.\n" +
+        '\n' +
+        "\x1B[7m1\x1B[0m import { a } from 'b';\n" +
+        '\x1B[7m \x1B[0m \x1B[91m                  ~~~\x1B[0m',
+      title: 'Jest',
+    },
+  ]);
+});