java.io.IOException: Cannot run program "git"

[richardlau]

Looks like builds on test-equinix-ubuntu2204-x64-2 (one of the jenkins-workspace machines) are failing:
e.g.
https://ci.nodejs.org/job/node-test-commit-linux-containered/43835/console

08:02:11 java.io.IOException: error=0, Failed to exec spawn helper: pid: 524586, exit value: 1
08:02:11 	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
...
08:02:14 Caused by: hudson.plugins.git.GitException: Error performing git command: git init /home/iojs/build/workspace/node-test-commit-linux-containered
08:02:14 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2862)
08:02:14 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2766)
08:02:14 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2761)
08:02:14 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:2051)
08:02:14 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1071)
08:02:14 	... 12 more
08:02:14 Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/iojs/build/workspace/node-test-commit-linux-containered"): error=0, Failed to exec spawn helper: pid: 524597, exit value: 1
08:02:14 	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
...

I'll take the machine offline in Jenkins for investigation.

[richardlau]

According to logs, there was an automatically installed Java update today:

root@test-equinix-ubuntu2204-x64-2:~# cat /var/log/apt/history.log

Start-Date: 2024-06-05  06:40:53
Commandline: /usr/bin/unattended-upgrade
Upgrade: libarchive13:amd64 (3.6.0-1ubuntu1, 3.6.0-1ubuntu1.1)
End-Date: 2024-06-05  06:40:53

Start-Date: 2024-06-06  06:26:13
Commandline: /usr/bin/unattended-upgrade
Upgrade: openjdk-17-jre-headless:amd64 (17.0.10+7-1~22.04.1, 17.0.11+9-1~22.04.1)
End-Date: 2024-06-06  06:26:14
root@test-equinix-ubuntu2204-x64-2:~#

Will try restarting the agent.

[richardlau]

Just checked and the other jenkins-workspace machines are still running 17.0.10+7-Ubuntu-122.04.1.

[richardlau]

I think it's just a matter of time before the other machines pick up the Java update.
e.g. test-equinix-ubuntu2204-x64-1 will probably pick it up in a couple of hours when this timer triggers:

root@test-equinix-ubuntu2204-x64-1:~# systemctl list-timers apt-daily.timer
NEXT                        LEFT          LAST                        PASSED  UNIT            ACTIVATES
Thu 2024-06-06 14:17:57 UTC 2h 43min left Wed 2024-06-05 23:44:06 UTC 11h ago apt-daily.timer apt-daily.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.
root@test-equinix-ubuntu2204-x64-1:~#

[richardlau]

Agent on test-equinix-ubuntu2204-x64-2 has been restarted and the builds it has picked up look okay (they got through the git checkout).

I'll leave this open in case we need to fix up the other workspace machine later this afternoon.

[richardlau]

ah test-digitalocean-ubuntu2204-x64-2 as well:
https://ci.nodejs.org/job/node-test-commit-linux/58294/nodes=ubuntu2204-64/console

[richardlau]

Potentially all of these ubuntu2204 hosts could be affected:

• infra-digitalocean-ubuntu2204-x64-1 (ci)
• test-digitalocean-ubuntu2204-x64-1
• test-digitalocean-ubuntu2204-x64-2
• test-digitalocean-ubuntu2204_docker-x64-1
• test-digitalocean-ubuntu2204_docker-x64-2
• test-equinix-ubuntu2204-x64-1
• test-equinix-ubuntu2204-x64-2
• test-hetzner-ubuntu2204-x64-1
• test-hetzner-ubuntu2204-x64-2
• test-ibm-ubuntu2204-x64-1
• test-ibm-ubuntu2204-x64-2
• test-ibm-ubuntu2204-x64-3
• test-ibm-ubuntu2204_docker-x64-1

Checkboxes denote if the machines are running Java 17.0.11+9-Ubuntu-122.04.1 and have had their agent restarted.
I'll proactively update the quiet ones (via Ansible) and restart their agents.

[richardlau]

The docker hosts are not running Jenkins agents directly -- the agents are in the containers. I reran the docker-host playbook on all three test-*-ubuntu2204_docker-x64* machines, but they didn't rebuild any containers (presumably because the Dockerfile was not changed) so the containers are still running the older Java 17.0.10 (even after restarting them).

[targos]

Should we disable auto-update of system packages?

[targos]

For the Docker agents, unless there's a critical security vulnerability in Java, I think we can keep them at the old version?

[richardlau]

For the Docker agents, unless there's a critical security vulnerability in Java, I think we can keep them at the old version?

I think so.

[richardlau]

Should we disable auto-update of system packages?

That's a good question. On the one hand it would mean no unexpected surprises but the onus is then back on us to keep the systems reasonably up-to-date. Maybe we should schedule regular (monthly?) maintenance windows for applying routine package updates?

@nodejs/build-infra thoughts?

[richardlau]

ci-release is affected (since 8 June).
e.g. https://ci-release.nodejs.org/job/iojs+release/10252/

Again, looks to be the JDK update:

root@infra-ibm-ubuntu1804-x64-1:~# tail /var/log/apt/history.log

Start-Date: 2024-06-08  07:00:13
Commandline: /usr/bin/unattended-upgrade
Upgrade: openjdk-8-jdk:amd64 (8u402-ga-2ubuntu1~20.04, 8u412-ga-1~20.04.1), openjdk-8-jre:amd64 (8u402-ga-2ubuntu1~20.04, 8u412-ga-1~20.04.1), openjdk-8-jdk-headless:amd64 (8u402-ga-2ubuntu1~20.04, 8u412-ga-1~20.04.1), openjdk-8-jre-headless:amd64 (8u402-ga-2ubuntu1~20.04, 8u412-ga-1~20.04.1)
End-Date: 2024-06-08  07:00:19

Start-Date: 2024-06-08  07:00:22
Commandline: /usr/bin/unattended-upgrade
Upgrade: openjdk-17-jre-headless:amd64 (17.0.10+7-1~20.04.1, 17.0.11+9-1~20.04.2)
End-Date: 2024-06-08  07:00:27
root@infra-ibm-ubuntu1804-x64-1:~#

I've just restarted the jenkins service on the machine.
Rebuilding the most recent nightly (which failed): https://ci-release.nodejs.org/job/iojs+release/10260/