cmd/go: retry failed fetches

[chai2010]

$ go version
go version go1.11.1 darwin/amd64

package main

import (
	_ "github.com/google/go-cloud/wire/cmd/wire"
)

func main() {
	//
}

go build -o a.out
go: finding github.com/google/go-cloud/wire/cmd/wire latest
go: finding github.com/google/go-cloud/wire/cmd latest
go: finding github.com/google/go-cloud/wire latest
go: finding google.golang.org/api v0.0.0-20180606215403-8e9de5a6de6d
go: google.golang.org/[email protected]: git fetch -f https://code.googlesource.com/google-api-go-client refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/chai/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        error: RPC failed; curl 18 transfer closed with outstanding read data remaining
        fatal: The remote end hung up unexpectedly
        fatal: early EOF
        fatal: index-pack failed
go: error loading module requirements

[mvdan]

@chai2010 it looks like git had an issue fetching data from the remote repository. Have you tried running the same command again?

@bcmills perhaps we should try to capture network flakiness errors from HTTP/VCS and report a more user friendly error, such as "perhaps try again in a few minutes".

[chai2010]

$ HTTPS_PROXY=socks5://127.0.0.1:2080 go build -o a.out
go: finding github.com/google/go-cloud/wire/cmd/wire latest
go: finding github.com/google/go-cloud/wire/cmd latest
go: finding github.com/google/go-cloud/wire latest
go: finding google.golang.org/api v0.0.0-20180606215403-8e9de5a6de6d
go: google.golang.org/[email protected]: git fetch -f https://code.googlesource.com/google-api-go-client refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/chai/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        error: RPC failed; curl 18 transfer closed with outstanding read data remaining
        fatal: The remote end hung up unexpectedly
        fatal: early EOF
        fatal: index-pack failed
go: error loading module requirements

[thepudds]

@gopherbot, please add label modules

[mrxliu]

感觉是并行下载外加网速不好导致的.

[mvdan]

Please write in English.

[agnivade]

In any case, this is indeed a network issue. I was able to reproduce this once out of 3 tries by just running the plain git command (git fetch -f https://code.googlesource.com/google-api-go-client ).

``` $git fetch -f https://code.googlesource.com/google-api-go-client remote: Sending approximately 87.36 MiB ... remote: Counting objects: 45, done remote: Finding sources: 100% (71244/71244) error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received. fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed 14:13:31-agniva-~/play/gg$rm -rf .git/ 14:13:48-agniva-~/play/gg$git init Initialized empty Git repository in /home/agniva/play/gg/.git/ 14:13:53-agniva-~/play/gg$git fetch -f https://code.googlesource.com/google-api-go-client remote: Sending approximately 87.36 MiB ... remote: Counting objects: 45, done remote: Finding sources: 100% (71244/71244) remote: Total 71244 (delta 44173), reused 71141 (delta 44173) Receiving objects: 100% (71244/71244), 86.49 MiB | 999.00 KiB/s, done. Resolving deltas: 100% (44173/44173), done. From https://code.googlesource.com/google-api-go-client * branch HEAD -> FETCH_HEAD 14:15:46-agniva-~/play/gg$rm -rf .git/ 14:15:51-agniva-~/play/gg$git init Initialized empty Git repository in /home/agniva/play/gg/.git/ 14:15:52-agniva-~/play/gg$git fetch -f https://code.googlesource.com/google-api-go-client remote: Sending approximately 87.36 MiB ... remote: Counting objects: 45, done remote: Finding sources: 100% (71244/71244) remote: Total 71244 (delta 44173), reused 71141 (delta 44173) Receiving objects: 100% (71244/71244), 86.49 MiB | 1.10 MiB/s, done. Resolving deltas: 100% (44173/44173), done. From https://code.googlesource.com/google-api-go-client * branch HEAD -> FETCH_HEAD ```

@chai2010 - Could you run the git command manually as shown above and see if you get any error ? If yes, then it's a network issue. If not, then try to run go build once more and see if you get the same error.

[smallnest]

I have the same issue:

workspace git clone https://code.googlesource.com/google-api-go-client
Cloning into 'google-api-go-client'...
remote: Sending approximately 139.63 MiB ...
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

I have modified the git buffer:

git config --global http.postBuffer 524288000

[northfun]

@mvdan Is there any solutions, such as copy the git repository directly to the mod directory?
I always get this problem,under $GOPATH I can download the zip file from git then do some other work to solve it, but with go mod I have no idea.
Thanks~

[bcmills]

@northfun, this issue is about retrying past temporary failures, not permanent ones.

If you are unable to download from some origin in general, you may need to set up a proxy for that module. (See https://tip.golang.org/cmd/go/#hdr-Module_proxy_protocol.)

[northfun]

@bcmills ,thanks for your advice~
I solved it using GOPROXY in local file system finally, it cost me some time (- -|||)

[horacimacias]

@northfun would you mind sharing a bit more information on what you did? I'm also hitting this same issue

[rsc]

It's unclear how to decide what is worth retrying and what is not, especially as it interacts with GOPROXY list fallback as well as executing VCS commands (the example above is git itself failing). Let's at least move this off to Go 1.14. It may be that there's nothing to do anyway, but certainly not for Go 1.13.

[engin]

go: finding github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72
go: golang.org/x/[email protected]: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/b44680b3c3708a854d4c89f55aedda0b223beb8d9e30fba969cefb5bd9c1e843: exit status 128:
	fatal: the remote end hung up unexpectedly
	fatal: early EOF
	fatal: index-pack failed
go: golang.org/x/[email protected]: unknown revision 11955173bddd
go: golang.org/x/[email protected]: unknown revision 07fd8470d635
go: golang.org/x/[email protected]: unknown revision bf090417da8b
go: golang.org/x/[email protected]: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/5b03666c2d7b526129bad48c5cea095aad8b83badc1daa202e7b0279e3a5d861: exit status 128:
	fatal: the remote end hung up unexpectedly
	fatal: early EOF
	fatal: index-pack failed
go: error loading module requirements

for me it was a kind of same problem; using GOPROXY=https://proxy.golang.org fixed it

[rahmatrhd]

Similar issue here, but solved using GOPROXY=https://proxy.golang.org as @engin suggested 👍

btw, this is my go mod download error log:

...
go: finding github.com/go-playground/locales v0.12.1
go: github.com/go-sql-driver/[email protected]: unknown revision v1.4.1
go: github.com/cznic/[email protected]: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/bdb87437caea4326c185626ae164c5d8ab96681b040cbd9dd4b9a3552547e1e3: exit status 128:
	fatal: unable to access 'https://github.com/cznic/fileutil/': Failed to connect to github.com port 443: Connection timed out
go: github.com/onsi/[email protected]: unknown revision v1.4.3
go: finding golang.org/x/tools v0.0.0-20190328211700-ab21143f2384
go: finding google.golang.org/api v0.3.2
...

[linzhp]

Would GOPROXY=https://proxy.internal.com,https://proxy.internal.com effectively retry failed fetches for a second time?

[jayconrod]

@linzhp No, but in Go 1.15, GOPROXY=https://proxy.internal.com|https://proxy.internal.com would do that (using a | separator instead of ,). #37367 has the details on that.

[thempatel]

This is still a problem in 2022, would it be possible to prioritize this body of work? Retrying module fetches would solve ~6% of our CI failures which is a very large number of jobs for us.

[eduardo-mior]

In full 2022, year of the world cup, and go mod download does not have retry :(.

My company's CI/CD suffers a lot from the lack of repetition. Several dependencies are hosted on unstable servers. When building the program or running the tests, there is instability on the server and the dependency cannot be downloaded and the pipeline breaks.

If the "go mod download" had retries, it would solve our problem.

[linzhp]

At Uber, we do two things to workaround this issue:

1. setting GOPROXY=https://proxy.internal.com|https://proxy.internal.com after Go 1.15, which essentially retries. You can let it retry as many times as you want this way.
2. Populating GOPATH/pkg/mod/cache periodically, create an archive out of it and upload that to a blob storage close to the CI cluster. Then we download it to all CI nodes and extract it to either GOPATH/pkg/mod/cache or GOMODCACHE/cache before starting builds. It's much faster than downloading one by one during the build. Less downloads mean less retries.

[thempatel]

@eduardo-mior similar to @linzhp we at FullStory also use a module cache directory that we pre-populate in our CI jobs. You can build a retry loop in your flavor of shell (e.g. https://unix.stackexchange.com/a/82610) and wrap go mod download inside of it. For us, the process of archiving and inflating all of our dependencies is slower than simply re-downloading them all so we prefer that route but could be a good avenue for you.

[eduardo-mior]

@linzhp Your idea with GOPROXY is really good, but it doesn't work for private modules :(

@thempatel It seems to be a good idea, I had already tried retries with the SSH command and with the GIT command, but I had not thought about it with the GO MOD DOWNLOAD command. I will try, thanks.

[eduardo-mior]

@thempatel Thanks for the idea.

I created this script that uses as a base go mod download to download the dependency and if it fails try again.

The script tries to download the dependency 5 times with an interval of 20 seconds.

#!/bin/bash
go mod download $@;
status=$?;
attempt=1;
while [ $status -ne 0 ] && [ $attempt -le 5 ]; do
    sleep 20;
    (( attempt++ ));
    go mod download $@;
    status=$?;
done;