x/build/dashboard: linux-mips64le-mengzhuo repeatedly timing out while writing snapshot

[dmitshur]

Almost all recent commits don't have a build result for this builder yet, as seen on build.golang.org:

From coordinator logs:

$ kubectl logs coordinator-deployment-86f6c447fc-qkkhc | grep 'failed to write snapshot to GCS'
2022/04/08 14:28:29 [build linux-mips64le-mengzhuo 3e7ffb862f550c38ce0611b970a4dce10a01226e]: failed to write snapshot to GCS after copying 6622422 bytes: context deadline exceeded
2022/04/08 14:41:16 [build linux-mips64le-mengzhuo 3e387528e54971d6009fe8833dcab6fc08737e04]: failed to write snapshot to GCS after copying 5668283 bytes: context deadline exceeded
2022/04/08 14:54:19 [build linux-mips64le-mengzhuo 3e7ffb862f550c38ce0611b970a4dce10a01226e]: failed to write snapshot to GCS after copying 7044318 bytes: context deadline exceeded
2022/04/08 15:06:47 [build linux-mips64le-mengzhuo 3e387528e54971d6009fe8833dcab6fc08737e04]: failed to write snapshot to GCS after copying 5897663 bytes: context deadline exceeded
2022/04/08 15:19:49 [build linux-mips64le-mengzhuo 3e387528e54971d6009fe8833dcab6fc08737e04]: failed to write snapshot to GCS after copying 5864896 bytes: context deadline exceeded
2022/04/08 15:33:23 [build linux-mips64le-mengzhuo 3e387528e54971d6009fe8833dcab6fc08737e04]: failed to write snapshot to GCS after copying 5930430 bytes: context deadline exceeded

The write snapshot timeout at this moment is 5 minutes, and this builder is running into that timeout repeatedly while only writing approximately 5-7 MB. A typical snapshot is around 150 MB, so at that speed it'd need more than 2 hours.

Is this an unexpected issue with the builder and/or its internet uplink speed, or should the builder be configured with SkipSnapshot: true? I see that it already has FlakyNet: true set. @mengzhuo, are you able to take a look and advise how you'd like to proceed? Thanks.

CC @golang/release.

[gopherbot]

Change https://go.dev/cl/398697 mentions this issue: cmd/coordinator: triple writeSnapshot timeout for reverse builders

[dmitshur]

Based on the comment in CL 399034, I see a possible explanation is if the builder is slow on compression, not the upload, so the 2 hour estimate might be way off.

If so, we can see the effect that CL 398697 makes (it's deployed now).

[dmitshur]

@mengzhuo Result after making the timeout 15 minutes instead of 5:

2022/04/09 00:10:30 [build linux-mips64le-mengzhuo 0f0c89243044a5a5de142e51da3a98f082fd3771]: failed to write snapshot to GCS after copying 18973578 bytes: context deadline exceeded

It was able to transfer an additional ~10 MB in the 10 extra minutes. It seems even 20 minutes would not be nearly enough.

Edit: A newer run completed in 9 minutes:

2022-04-09T00:58:24Z finish_write_snapshot_to_gcs after 8m51.1s

[mengzhuo]

@dmitshur Thank you and this detail update. I've changed the proxy and it works.

[mengzhuo]

Since I transfer all MIPS owner to @Rongronggg9 , we can close this issue now.