cmd/go: fetching dependencies can be very aggressive when going via an HTTP proxy

[errordeveloper]

What version of Go are you using (go version)?

$ go version
go version go1.12.8 windows/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

windows/amd64

What did you do?

1. Set http_proxy environment variable
2. git clone https://github.com/weaveworks/eksctl && cd eksctl
3. go build ./cmd/eksctl

What did you expect to see?

All dependencies get downloaded and binary gets built. If errors occur, there are a few retries with a back-off.

What did you see instead?

Some dependencies get downloaded, some fail to download with 407 error in git fetch. Binary is not built. I had to rerun go build multiple times before it succeeded.

[andybons]

@bcmills @jayconrod

[errordeveloper]

It's kind of to do with having to download the internet, and once you hit a proxy with all the request for git object, it chokes up. One option would be to have retries, as I suggested above, another could be to provide a flag that allows user to set a pause interval between requests.

[errordeveloper]

I figured the title of the issue was too prescriptive, I renamed it to reflect what the issue is really about.

[heschi]

Does using a GOPROXY, e.g. GOPROXY=https://proxy.golang.org, improve things? It should be less demanding of an HTTP proxy than Git. That doesn't answer the question of whether the go command should retry though.

[bcmills]

Retries are #28194.

That said, I'm not sure that there is anything we should do on the go command side if the problem is that the git commands overwhelm the proxy. For HTTP requests initiated on the go command side we rely on whatever connection pooling and limiting the net/http package provides by default, but I assume that individual git commands probably don't share proxy connections.

Ideally, your HTTP proxy should be robust enough to just not read from the incoming socket when it doesn't have the capacity to spare, rather than returning spurious 407s for some of the requests.

[bcmills]

That said, I've been thinking about moving the parallelism limits from the various par.Work calls down to the leaf packages where we actually need to limit things. If we do that, perhaps we can use a more restrictive semaphore for git invocations than we use for package fetches in general.

[gopherbot]

Change https://golang.org/cl/220080 mentions this issue: design/36460: add design for lazy module loading