nim compiler is cached on runner-node-02-linux-01-eu-hel1

[siddarthkay]

As reported by @tersec https://github.com/status-im/nimbus-eth2/actions/runs/17697747838/job/50299340420 failed and the logs show

Building: Nim compiler
Nim Compiler Version 2.2.1 [Linux: amd64]
Compiled at 2025-09-13
Copyright (c) 2006-2025 by Andreas Rumpf

git hash: 12347eae74dbaa0e4dd99691e3a4509e8fe8b265
active boot switches: -d:release

This should not have happened and these were the attempts made in the past to fix caching :

• 6ee404c
• 3d74066

This could be a problem with nimbus build system itself which provides this script to build nim : https://github.com/status-im/nimbus-build-system/blob/master/scripts/build_nim.sh

[tersec]

• https://github.com/status-im/nimbus-eth2/actions/runs/17738731583/job/50407060891?pr=7502

[tersec]

• https://github.com/status-im/nimbus-eth2/actions/runs/17739302257/job/50409035632

[siddarthkay]

Checking the kind of projects built on this GH runner

linux-01% ls -la /github-runner/github-runner-node-02/workspace
total 64
drwxr-xr-x 13 github github 4096 Sep 15 19:50 .
drwxr-xr-x 24 github github 4096 Sep 15 23:43 ..
drwxr-xr-x  3 github github 4096 Sep 15 19:50 _actions
drwxr-xr-x  3 github github 4096 Oct 20  2024 nimbus-eth2
drwxr-xr-x  3 github github 4096 Jan 28  2025 nim-codex
drwxr-xr-x  3 github github 4096 Mar 13  2025 nomos
drwxr-xr-x  3 github github 4096 Oct 19  2024 nomos-node
drwxr-xr-x  6 github github 4096 Jan 28  2025 _PipelineMapping
drwxr-xr-x  2 github github 4096 Sep 15 19:53 _temp
drwxr-xr-x  6 github github 4096 May 17 19:37 _tool
drwxr-xr-x  4 github github 4096 Aug 16 07:12 _update
-rw-r--r--  1 github github 9815 Aug 16 07:12 _update.sh
drwxr-xr-x  3 github github 4096 May 17 19:31 waku-go-bindings
drwxr-xr-x  3 github github 4096 Dec  2  2024 waku-interop-tests

Took it offline for now to investigate [nimbus-build-system/blob/master/scripts/build_nim.sh](https://github.com/status-im/nimbus-build-system/blob/master/scripts/build_nim.sh)

linux-01% sudo systemctl stop io.status.github-runnner-node-02.service

[siddarthkay]

This case is easily reproducible on the host

github@linux-01.he-eu-hel1.ci.github:/github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2$ 
make -j8 NIM_COMMIT=upstream/version-2-2 ARCH_OVERRIDE=x64 QUICK_AND_DIRTY_COMPILER=1 update
 m vendor/nimbus-build-system
^---top level
 M scripts/build_nim.sh
 M vendor/Nim
^---vendor/nimbus-build-system
Submodule path 'vendor/nimbus-build-system/vendor/Nim': 
checked out 'f7145dd26efeeeb6eeae6fff649db244d81b212d'
github@linux-01.he-eu-hel1.ci.github:/github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2$ 
./env.sh nim --version
Nim Compiler Version 2.2.1 [Linux: amd64]
Compiled at 2025-09-15
Copyright (c) 2006-2025 by Andreas Rumpf

git hash: 12347eae74dbaa0e4dd99691e3a4509e8fe8b265
active boot switches: -d:release

[tersec]

V=1 might illuminate things.

[siddarthkay]

I noticed vendor/Nim to be in a wierd state on the host

manually fixed it with :

cd /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nimbus-build-system
git submodule update --init vendor/Nim
git fetch upstream version-2-2

and then we see the correct commit being picked up after building

github@linux-01.he-eu-hel1.ci.github:/github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2$ ./env.sh nim --version
Nim Compiler Version 2.2.5 [Linux: amd64]
Compiled at 2025-09-16
Copyright (c) 2006-2025 by Andreas Rumpf

git hash: 4c7ddcd79a0e29126b9cdfcf13b3e9a9930cbecb
active boot switches: -d:release

its weird, perhaps

nimbus-eth2/.github/workflows/ci.yml

Line 82 in da9305a

run: rm -rf "$GITHUB_WORKSPACE"/*

is not clearing everything inside vendor.

[siddarthkay]

I have started the github runner now and lets wait to see if this workspace issue happens again or not since this issue is only localised to this host.

linux-01% sudo systemctl restart io.status.github-runnner-node-02.service
linux-01% sudo systemctl status io.status.github-runnner-node-02.service
● io.status.github-runnner-node-02.service - Github Action Self Hosted runner for https://github.com/enterprises/ift
     Loaded: loaded (/etc/systemd/system/io.status.github-runnner-node-02.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2025-09-16 09:02:24 UTC; 9s ago
       Docs: https://github.com/status-im/infra-role-github_action_runner;https://docs.github.com/en/actions/hosting-your-own-runners/
   Main PID: 2965843 (runsvc.sh)
      Tasks: 36 (limit: 76980)
     Memory: 27.1G
        CPU: 884ms
     CGroup: /system.slice/io.status.github-runnner-node-02.service
             ├─ 886832 /bin/bash /github-runner/github-runner-node-02/.actions-runner1/run-helper.sh
             ├─ 886836 /github-runner/github-runner-node-02/.actions-runner1/bin/Runner.Listener run
             ├─2409043 /bin/bash ./run.sh
             ├─2965843 /bin/bash /github-runner/github-runner-node-02/runsvc.sh
             ├─2965845 ./externals/node20/bin/node ./bin/RunnerService.js
             ├─2965852 /github-runner/github-runner-node-02/bin/Runner.Listener run --startuptype service
             └─3741655 gpg-agent --homedir /github-runner/github-runner-node-02/.gnupg --use-standard-socket --daemon

Sep 16 09:02:24 linux-01.he-eu-hel1.ci.github systemd[1]: Started Github Action Self Hosted runner for https://github.com/enterprises/ift.
Sep 16 09:02:24 linux-01.he-eu-hel1.ci.github runsvc.sh[2965843]: .path=/usr/local/bin:/usr/bin:/bin:/snap/bin:/nix/var/nix/profiles/default/bin
Sep 16 09:02:24 linux-01.he-eu-hel1.ci.github runsvc.sh[2965845]: Starting Runner listener with startup type: service
Sep 16 09:02:24 linux-01.he-eu-hel1.ci.github runsvc.sh[2965845]: Started listener process, pid: 2965852
Sep 16 09:02:24 linux-01.he-eu-hel1.ci.github runsvc.sh[2965845]: Started running service
Sep 16 09:02:25 linux-01.he-eu-hel1.ci.github runsvc.sh[2965845]: √ Connected to GitHub
Sep 16 09:02:26 linux-01.he-eu-hel1.ci.github runsvc.sh[2965845]: Current runner version: '2.328.0'
Sep 16 09:02:26 linux-01.he-eu-hel1.ci.github runsvc.sh[2965845]: 2025-09-16 09:02:26Z: Listening for Jobs

[tersec]

I noticed vendor/Nim to be in a wierd state on the host

manually fixed it with :

cd /github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2/vendor/nimbus-build-system
git submodule update --init vendor/Nim
git fetch upstream version-2-2

and then we see the correct commit being picked up after building

github@linux-01.he-eu-hel1.ci.github:/github-runner/github-runner-node-02/workspace/nimbus-eth2/nimbus-eth2$ ./env.sh nim --version
Nim Compiler Version 2.2.5 [Linux: amd64]
Compiled at 2025-09-16
Copyright (c) 2006-2025 by Andreas Rumpf

git hash: 4c7ddcd79a0e29126b9cdfcf13b3e9a9930cbecb
active boot switches: -d:release

its weird, perhaps

nimbus-eth2/.github/workflows/ci.yml

Line 82 in da9305a
run: rm -rf "$GITHUB_WORKSPACE"/*

is not clearing everything inside vendor.

So it's reusing a vendor/Nim directory from run to run?

[siddarthkay]

@tersec : I would assume that for some reason vendor/Nim was not getting cleaned up, but I'm not 100% sure of this hypothesis.. I'll check again with a dummy PR targeted at runner-node-02-linux-01-eu-hel1 to verify.

[siddarthkay]

So it's reusing a vendor/Nim directory from run to run?

Not right now: https://github.com/status-im/nimbus-eth2/actions/runs/17996558341/job/51197141059#step:4:14
But could have been a stale workspace 2 weeks ago which was not getting cleaned up automatically.

I checked and CI seems to be currently picking up correct nim version on the problematic runner runner-node-02-linux-01-eu-hel1

Building: Nim compiler
Nim Compiler Version 2.2.5 [Linux: amd64]
Compiled at 2025-09-25
Copyright (c) 2006-2025 by Andreas Rumpf

git hash: 4c7ddcd79a0e29126b9cdfcf13b3e9a9930cbecb
active boot switches: -d:release