Cannot talk to GitHub

[vinayak-kukreja]

Hey, could you please take a look at this issue? Looks like its not just for the YAML packages.

go-yaml/yaml#887

[a-hilaly]

We're experiencing the same issue here https://prow.ack.aws.dev/view/s3/ack-prow-logs/pr-logs/pull/aws-controllers-k8s_rds-controller/109/rds-unit-test/1558538624394334208

[SVilgelm]

I see same issue:

go: gopkg.in/[email protected]: unrecognized import path "gopkg.in/check.v1" (parse https://gopkg.in/check.v1?go-get=1: no go-import meta tags ())
go: gopkg.in/[email protected]: unrecognized import path "gopkg.in/yaml.v2" (parse https://gopkg.in/yaml.v2?go-get=1: no go-import meta tags ())

[blackknight467]

same issue here:

[gopkg.in/[email protected]:](http://gopkg.in/[email protected]) unrecognized import path "[gopkg.in/yaml.v3](http://gopkg.in/yaml.v3)": reading [https://gopkg.in/yaml.v3?go-get=1:](https://gopkg.in/yaml.v3?go-get=1) 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get [https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack:](https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack) net/http: request canceled (Client.Timeout exceeded while awaiting headers)
The command '/bin/sh -c go mod download' returned a non-zero code: 1

[mahendrabishnoi2]

I'm also facing same issue.

package gopkg.in/gorp.v2: unrecognized import path "gopkg.in/gorp.v2": reading https://gopkg.in/gorp.v2?go-get=1: 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-gorp/gorp.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[guillermo]

I am having the same issue right now. Any hints about how this issue was solved back in the time?

[mdmourao]

I am having the same issue right now. I think the problem is related to Github but the status is showing all Operational
https://stackoverflow.com/questions/74290916/cannot-obtain-refs-from-github-github-down-status-not-showing-any-issues

[EvanBoyle]

Also hitting this

[metal3d]

And the problems is repeated fro days now... impossible to work on my apps.

A “replace” directive didn't work for me... That's a very serious problem.

Maybe I can make it work on local machine and change /etc/hosts to fake the server...

[johnpekcan]

Having this same issue for multiple days

/go/pkg/mod/github.com/spf13/[email protected]/internal/encoding/ini/codec.go:9:2: unrecognized import path "gopkg.in/ini.v1": reading https://gopkg.in/ini.v1?go-get=1: 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-ini/ini.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/go/pkg/mod/github.com/spf13/[email protected]/internal/encoding/yaml/yaml3.go:6:8: unrecognized import path "gopkg.in/yaml.v3": reading https://gopkg.in/yaml.v3?go-get=1: 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[johnpekcan]

lots of comments in #63 as well

[metal3d]

I managed some things by forking repo and changed references to not use gopkg.in - but here, with Fyne.io, there are too many problems to solve.

Really, the gopkg.in idea is good, but if the service doesn't respond, that a SPOF for thouthands of projects.

[vwbusguy]

Really, the gopkg.in idea is good, but if the service doesn't respond, that a SPOF for thouthands of projects.

Totally agree. I wonder if it's feasible to backup these repos/versions on a gitea or a CDN and failover to it. This just ate up my afternoon debugging why the go installs were failing at different points trying to resolve deps while GitHub was having issues (again).

[mdmourao]

This is a critical problem i wasn't able to deploy my workload for hours. But i think the problem is related to the way this tool communicates with github. Its weird we don't hear anything from the developers. We real need to find a work around for this.

[niemeyer]

I've been off for two weeks now on intense company wide events that happen twice a year, and we still have the Ubuntu Summit starting tomorrow. That's why you haven't heard from me. That said, it's for exactly this kind of reason that this service is under the supervision of a 24x7 team at Canonical, and not me personally. I'll reach out to them.

[niemeyer]

Here is some info from the team:

We’ve had some issues one of our transit providers affecting gopkg and a number of other services. We’ve weighted/prepended them out for now so traffic is going via our other providers links.

[sivag73]

@niemeyer , I am seeing this issue again now.
Pls let me know if you a new issue needs to be created for this one.

[niemeyer]

Indeed, there was an issue today:

https://status.canonical.com/

[niemeyer]

And again it was an infrastructure-wide issue, so probably the same link misbehaving again. 😕

[sanghvisagar]

@niemeyer Getting the same issue again
go: gopkg.in/natefinch/[email protected]: unrecognized import path "gopkg.in/natefinch/lumberjack.v2": https fetch: Get "https://gopkg.in/natefinch/lumberjack.v2?go-get=1": dial tcp 185.125.188.113:443: i/o timeout

[zekenautilus]

just saw this today:

go: gopkg.in/DataDog/[email protected]: unrecognized import path "gopkg.in/DataDog/dd-trace-go.v1": reading https://gopkg.in/DataDog/dd-trace-go.v1?go-get=1: 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/DataDog/dd-trace-go.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[siddhanta-rath]

@niemeyer this is happening again today:

/.go/pkg/mod/github.com/spf13/[email protected]/internal/encoding/ini/codec.go:9:2: unrecognized import path "gopkg.in/ini.v1": reading https://gopkg.in/ini.v1?go-get=1: 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-ini/ini.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/.go/pkg/mod/github.com/spf13/[email protected]/internal/encoding/yaml/yaml3.go:6:8: unrecognized import path "gopkg.in/yaml.v3": reading https://gopkg.in/yaml.v3?go-get=1: 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[alecholmez]

Yup I see the same thing:

remote: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
	fatal: unable to access 'https://gopkg.in/yaml.v3/': The requested URL returned error: 502

EDIT:

It's fixed for me now

[zdandoh]

Seeing the same right now, not fixed yet.

[Mansh05]

Same thing for me as well.

[niemeyer]

Sorry for the trouble. I'm in touch with our IS team and apparently it's a connectivity issue again. If that's confirmed, I've suggested moving the service out of this provider to give gopkg.in a break as this is getting shameful. 😕

[hiyunfengzhao]

anyone still seeing the issue? I still get server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-check/check.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[Mansh05]

@niemeyer When would this be resolved in your opinion. Not pushing you but rather need to inform my teammates. Thanks

[jimmyraywv]

So, is there any option to get around gopkg.in?

[adamdavis40208]

I had a successful build, if it gives anyone hope.

[pelletier197]

Think this might be here again. Am I the only one?

[mdmourao]

Its down again: https://status.canonical.com/
Actually was down 3 times today.

[siddhanta-rath]

Today I am facing the issue again: the build is keep on falling even if gopkg.in came up after the major incident happened on today.

/.go/pkg/mod/github.com/spf13/[email protected]/internal/encoding/ini/codec.go:9:2: unrecognized import path "gopkg.in/ini.v1": reading https://gopkg.in/ini.v1?go-get=1: 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-ini/ini.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
/.go/pkg/mod/github.com/spf13/[email protected]/internal/encoding/yaml/yaml3.go:6:8: unrecognized import path "gopkg.in/yaml.v3": reading https://gopkg.in/yaml.v3?go-get=1: 502 Bad Gateway
	server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[curtismenmuir]

+1 seeing this issue today:

remote: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: write tcp 10.131.9.188:60820->140.82.121.3:443: write: broken pipe
fatal: unable to access 'https://gopkg.in/yaml.v2/': The requested URL returned error: 502

remote: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-playground/colors.git/info/refs?service=git-upload-pack: write tcp 10.131.9.188:60820->140.82.121.3:443: write: broken 
fatal: unable to access 'https://gopkg.in/go-playground/colors.v1/': The requested URL returned error: 502

[harshit777]

Seeing the same issue as @curtismenmuir

remote: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/natefinch/lumberjack.git/info/refs?service=git-upload-pack: write tcp 10.131.9.188:60820->140.82.121.3:443: write: broken pipe fatal: unable to access 'https://gopkg.in/natefinch/lumberjack.v2/': The requested URL returned error: 502

remote: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: write tcp 10.131.9.188:60820->140.82.121.3:443: write: broken pipe fatal: unable to access 'https://gopkg.in/yaml.v2/': The requested URL returned error: 502

[curtismenmuir]

@niemeyer is this something you could help us with?

[j-maltese]

Reporting the same issue as @curtismenmuir and @harshit777 on our pipelines this morning - issue is intermittent (sometimes fails, sometimes succeeds):

unrecognized import path "gopkg.in/yaml.v2": reading https://gopkg.in/yaml.v2?go-get=1: 502 Bad Gateway server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: write tcp 10.131.9.188:60820->140.82.121.3:443: write: broken pipe

[harshit777]

Gopkg was affected today : Issue

[jalilalchy]

Still appears to be ongoing. @niemeyer

[mahsaelasticpath]

Having the same issue : @niemeyer
go: gopkg.in/DataDog/[email protected] requires
gopkg.in/[email protected]: unrecognized import path "gopkg.in/yaml.v2": reading https://gopkg.in/yaml.v2?go-get=1: 502 Bad Gateway
server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: write tcp 10.131.9.188:60820->140.82.121.3:443: write: broken pipe

[curtismenmuir]

Work around for this: https://github.com/orgs/community/discussions/56932#discussioncomment-6069091

We got this working by setting the GOPROXY Env Var to https://proxy.golang.org,direct :)

EG add GOPROXY=https://proxy.golang.org,direct to Jenkinsfile env vars

[Tulsishah]

I am getting this issue, my GOPROXYEnv Var to https://proxy.golang.org,direct as default. Is there any fix for this?

[opohorel]

Running into the same issue today with
GOPROXY="direct"

[rjcoupe]

Hitting this issue too, GOPROXY=https://proxy.golang.org,direct set.

[j-maltese]

Having similar issues:
remote: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: write: broken pipe
fatal: unable to access 'https://gopkg.in/yaml.v2/': The requested URL returned error: 502
package gopkg.in/yaml.v2: exit status 128

[curtismenmuir]

Fixed this previously by setting GOPROXY Env Var to https://proxy.golang.org,direct

We have not seen this issue since.

[rweir]

I'm still seeing this intermittently from my Github Actions, even with the GOPROXY set. I note that Canonical status says there was a brief outage of about 14 hours ago - presumably this is related.

[suntong]

I'm getting this from my Github Actions just now. Thanks for the confirmation @rweir.

[niemeyer]

We still see issues from time to time. Some of them are just either side (Canonical or GitHub) having connectivity issues, but there's also some sort of longstanding issue which seems related to problematic caching of idle connections after some other event. The code on gopkg itself is quite simple, though, so unless I'm missing something obvious it doesn't feel like this should persist the way it does. In either case, I've added a small attempt to workaround the issue by detecting this situation and dropping the idle connections. This has been deployed today, together with updated dependencies to bring them to recent releases. Let's see what happens next.