node-test-commit is broken

[aduh95]

It looks like all jobs are failing is less than a second, starting with https://ci.nodejs.org/job/node-test-commit/75761/

00:00:00.379 Honoring refspec on initial clone
00:00:00.423 Cloning repository [email protected]:nodejs/node.git
00:00:00.423  > git init /home/iojs/build/workspace/node-test-commit # timeout=10
00:00:00.427 ERROR: Error cloning remote repo 'origin'
00:00:00.427 hudson.plugins.git.GitException: Could not init /home/iojs/build/workspace/node-test-commit

[targos]

/cc @nodejs/build-infra

I'm in a meeting the whole day and cannot investigate this.

[mhdawson]

@ryanaslett is this something you could take a look at?

[ryanaslett]

I did some preliminary investigation into this and it appears as though all three of our jenkins workspace machines can no longer connect to github.

All the clone urls are [email protected], and result in

[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

So either the url's changed in the build and they used to be https urls, or something changed on the workspace machines to remove the private keys.

[mhdawson]

Earlier issue where we had trouble with connectivity from jenkins to GitHub, but this looks different- #3254

[mhdawson]

Being looking at the keys/setup with Ryan to try to understand what might have changed, have followed some threads but no answer yet.

[mhdawson]

Found this issue which looks like it might be the same issue after looking at the error more closely and some googling - #3614

Looks like packages were updated yesterday (by looking at /var/log/dpkg.log) on test-ibm-ubuntu2204-x64-3 were some of the failures were being seen.

Possible that it ran/failed so fast that all jobs were running failing there.

Have restarted the jenkins agent and job seems to be running now. Looks ike it was just that that git just failing to run from java running the agent after the system updates like was the case reported earlier in #3614.

Restarted the agent on the 3 workspace machines that I see being associated with the node-test-commit job.

[mhdawson]

This job seems to be running ok - https://ci.nodejs.org/job/node-test-commit/75781/

Will leave this open until we confirm that we see jobs running ok across the different workspace machines.

[ljharb]

still seeing the same issue on https://ci.nodejs.org/job/node-test-pull-request/63546/

[aduh95]

The issue for node-test-commit is resolved, but there are still other jobs impacted (node-test-commit-linux and node-test-commit-custom-suites-freestyle (test-worker))

[targos]

I updated and rebooted all 4 ubuntu2204 hosts

[aduh95]

That seems to have fixed it, https://ci.nodejs.org/job/node-test-pull-request/63552/ doesn’t have the issue. Closing now, thanks everyone.

[ljharb]

fwiw, https://ci.nodejs.org/job/node-test-pull-request/63557/ still has the same issue.

[aduh95]

fwiw, https://ci.nodejs.org/job/node-test-pull-request/63557/ still has the same issue.

No it does not

[ljharb]

hmm - maybe i linked to the wrong thing. https://ci.nodejs.org/job/node-test-commit/75799/console ?

[aduh95]

I don't see any anywhere in those logs anything related to a connection problem with github.com, AFAICT it's working fine

[ljharb]

alrighty, i'll dig into it, maybe it's an actual failure. sorry for the noise :-)

[richardlau]

The two Hetzner hosted machines also needed to have their agents restarted. I've just done so. (node-test-commit-v8-linux and node-daily-coverity were still broken this morning prior to restarting the agent.)

FWIW AFAICT there was no connectivity issue with GitHub and this was another case of automatic updates on Ubuntu updating the Java Runtime Environment which breaks being able to run git/bash (others?) until the Jenkins agent is restarted. The last time this happened it was suggested disabling automatic updates -- I've pulled that discussion into its own issue: #3962

I did some preliminary investigation into this and it appears as though all three of our jenkins workspace machines can no longer connect to github.

@ryanaslett @mhdawson FYI we don't store the keys for GitHub access on the machines -- instead we have Jenkins configured to inject a secret into the builds to allow fetches.