node-test-commit-v8-linux fails to pull V8 deps

[targos]

See https://ci.nodejs.org/job/node-test-commit-v8-linux/4903/nodes=benchmark-ubuntu1604-intel-64,v8test=v8test/console

There are often errors like this:

16:55:25 [0:05:35] fatal: unable to access 'https://chromium.googlesource.com/external/github.com/google/googletest.git/': Could not resolve host: chromium.googlesource.com

[richardlau]

This only appears to be occurring on the benchmark machine.

I'm confused how this happens as the job successfully does git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git depot_tools earlier on so the host was resolvable then.

There's also this is that job log

15:55:25 [0:01:18] Cloning into './v8/third_party/googletest/_gclient_src_Y_ZTgA'...
15:55:25 [0:01:25] remote: Sending approximately 14.58 MiB ...        
15:55:25 [0:01:25] remote: Counting objects: 1           
15:55:25 [0:01:25] remote: Counting objects: 15, done        
15:55:25 [0:01:27] Receiving objects:   0% (1/25036)   
15:55:25 [0:01:34] Receiving objects:   0% (176/25036), 67.99 KiB | 9.00 KiB/s   
15:55:25 [0:01:39] Receiving objects:   2% (619/25036), 195.95 KiB | 16.00 KiB/s   
15:55:25 [0:01:43] Receiving objects:   3% (835/25036), 259.93 KiB | 16.00 KiB/s   
15:55:25 [0:01:47] Receiving objects:   4% (1063/25036), 323.91 KiB | 16.00 KiB/s   
15:55:25 [0:02:52] Receiving objects:   5% (1312/25036), 387.89 KiB | 4.00 KiB/s   
15:55:25 [0:05:23] error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
15:55:25 [0:05:23] fatal: The remote end hung up unexpectedly
15:55:25 [0:05:23] fatal: early EOF
15:55:25 [0:05:23] fatal: index-pack failed

I wiped out the workspace and tried again and got a similar error albeit with a different dependency repository (so we can rule out at least a bad workspace)
https://ci.nodejs.org/job/node-test-commit-v8-linux/4904/nodes=benchmark-ubuntu1604-intel-64,v8test=v8test/console

17:06:46 [0:00:52] Cloning into './v8/buildtools/third_party/libc++abi/_gclient_trunk_qW77kh'...
17:06:46 [0:00:53] Receiving objects:   0% (1/6887)   
17:06:46 [0:01:26] Receiving objects:   7% (536/6887), 259.93 KiB | 7.00 KiB/s   
17:06:46 [0:01:26] Receiving objects:  10% (689/6887), 259.93 KiB | 7.00 KiB/s   
17:06:46 [0:06:20] error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
17:06:46 [0:06:20] fatal: The remote end hung up unexpectedly
17:06:46 [0:06:20] fatal: early EOF
17:06:46 [0:06:20] fatal: index-pack failed
17:06:46 Traceback (most recent call last):

Unless anyone has other ideas I'm tempted to try updating this machine to at least Ubuntu 18.04 tomorrow. We've been putting off updating it for a while. From a risk POV in the event that the update goes wrong we can run the V8 builds on the x64 RHEL8 CI machines but it's slower (less CPUs).

[richardlau]

FTR https://ci.nodejs.org/job/node-test-commit-v8-linux/4894/ passed but it was the only one in the last fortnight to do so.

[richardlau]

Upgrade won't happen today due to other infra issues.

FTR, neither of the two builds today had the deps fetching problem:

• https://ci.nodejs.org/job/node-test-commit-v8-linux/4905/ passed.
• https://ci.nodejs.org/job/node-test-commit-v8-linux/4906/ built but failed the perf test.

Although all of the builds yesterday did fail with the deps fetching problem.

[richardlau]

I'm upgrading the machine now. Proving to be a bit tricky 😞. Machine was configured to not look for new releases (so kept saying no new release was available to upgrade to). And it seems the upgrade command doesn't like python3 being controlled by update-alternatives.

[richardlau]

Upgraded the benchmark machine used for V8 builds to Ubuntu 18.04. I'm hesitant to go higher due to available versions of python and unversioned python (since I think the older V8 versions in Node.js 14 may still require python to be python 2).

Test builds:

• v14.x: https://ci.nodejs.org/job/node-test-commit-v8-linux/4913/
• v16.x:
• v18.x:
• main: https://ci.nodejs.org/job/node-test-commit-v8-linux/4915/nodes=benchmark-ubuntu1804-intel-64,v8test=v8test/

[richardlau]

Upgrade doesn't appear to have fixed the issue 😞
https://ci.nodejs.org/job/node-test-commit-v8-linux/4913/nodes=benchmark-ubuntu1804-intel-64,v8test=v8test/console

7:31:02 [0:03:48] error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
17:31:02 [0:03:48] fatal: The remote end hung up unexpectedly
17:31:02 [0:03:48] fatal: early EOF
17:31:02 [0:03:48] fatal: index-pack failed

[sxa]

Two things to try:

• Are you able to replicate it from the command line outside jenkins? (i.e. is it something environmental when run via jenkins?)
• If "yes" to the first, can you use a [email protected]:... URL to do the operation and does that fail?

[richardlau]

Two things to try:

• Are you able to replicate it from the command line outside jenkins? (i.e. is it something environmental when run via jenkins?)

hmm. I just ran two builds manually on the machine and they succeded 😕. I don't have time to do more manual runs tonight.

• If "yes" to the first, can you use a [email protected]:... URL to do the operation and does that fail?

No, because these are fetching deps from chromium.googlesource.com.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[richardlau]

The Nearform machines are now retired from the CI. This was only seen there and has not occurred on the Hetzner replacements.