Run processes serially on Buildkite (#343)

.changeset/purple-points-wash.md

@@ -0,0 +1,7 @@
+---
+'skuba': patch
+---
+
+**build-package, lint:** Run serially on Buildkite
+
+These commands now run their underlying processes serially when the `BUILDKITE` environment variable is set. This reduces the chance of resource exhaustion on smaller instance sizes but slows down builds.

docs/buildkite.md

@@ -0,0 +1,63 @@
+# Buildkite
+
+## My agent exits with status -1
+
+**Scenario:**
+you're running some skuba commands like `skuba lint`,
+and observe the following error message on a Buildkite step:
+
+> Exited with status -1 (process killed or agent lost; see the timeline tab for more information)
+
+Navigating to the Timeline tab reveals this:
+
+> **Dispatcher Cancelled Job** (+Xm)
+>
+> |                          |                             |
+> | :----------------------- | :-------------------------- |
+> | **Last Agent Heartbeat** | Yesterday at 0:00:00.000 PM |
+> | **Command Exit Status**  | `-1 (Agent Lost)`           |
+
+**Explanation:**
+This implies that the step(s) exhausted the Buildkite agent's resources.
+The agent may be tied up running a particularly compute-intensive step.
+
+**Options:**
+
+1. Propagate the `BUILDKITE` environment variable to `skuba build-package` and `skuba lint` steps.
+   This will cause them to run their underlying processes serially,
+   reducing the chance of resource exhaustion.
+
+   With the [Docker Buildkite plugin],
+   you can achieve this in a couple ways:
+
+   ```yaml
+   steps:
+     - plugins:
+         - docker#v3.7.0:
+             # Option 1a: List environment variable explicitly
+             environment:
+               - BUILDKITE
+             # Option 1b: Propagate all pipeline variables
+             propagate-environment: true
+   ```
+
+   With Docker Compose,
+   you can add the variable to your [Compose file]:
+
+   ```yaml
+   services:
+     app:
+       environment:
+         - BUILDKITE
+   ```
+
+1. Reduce the number of agents that run on each instance.
+
+   At SEEK, this can be configured through Build Agency.
+
+1. Increase the instance size.
+
+   At SEEK, this can be configured through Build Agency.
+
+[compose file]: https://docs.docker.com/compose/compose-file
+[docker buildkite plugin]: https://github.com/buildkite-plugins/docker-buildkite-plugin

src/cli/lint.ts

@@ -13,16 +13,16 @@ const externalLint = () =>
       name: 'ESLint',
       prefixColor: 'magenta',
     },
-    {
-      command: 'prettier --check .',
-      name: 'Prettier',
-      prefixColor: 'cyan',
-    },
     {
       command: 'tsc --noEmit',
       name: 'tsc',
       prefixColor: 'blue',
     },
+    {
+      command: 'prettier --check .',
+      name: 'Prettier',
+      prefixColor: 'cyan',
+    },
   ]);
 
 const noSkubaTemplateJs = async () => {

src/utils/exec.ts

@@ -122,6 +122,10 @@ export const execConcurrently = async (commands: ExecConcurrentlyCommand[]) => {
         name: name.padEnd(maxNameLength),
         prefixColor,
       })),
+      {
+        // Run serially on Buildkite, where we often use puny agents.
+        maxProcesses: process.env.BUILDKITE ? 1 : 0,
+      },
     );
   } catch (err: unknown) {
     const result = ConcurrentlyErrors.validate(err);