github-runner: "no space left on device"

[adrian-gierakowski]

this is more of a question than a bug report:

Occasionally I would get the above error in during gh action job execution. Initially I though it was due to running out of disk space, as I didn't have automatic nix.gc configured, however it didn't make sense since I still had tens of GBs of disk space left.

Recently I've been working on adding a job which writes a lot of data to disk (over 10GB) and when executing on smaller VM it always fails with this error. I had an idea to monitor the file system with watch df -h while the job was running and realised that it was tmpfs mount at /run that was running out of space.

It seems that that by defaults all files written to WorkingDirectory by the runner are written to the above tmpfs, which size is limited to 25% of available RAM.

Is there an easy and secure way to allow jobs to use more disk space?

I could simply bump boot.runSize but then I could run out of memory since I don't have swap enabled.

Thanks!