Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle lost connections to the datastore #23

Open
BondAnthony opened this issue Sep 1, 2019 · 0 comments
Open

Handle lost connections to the datastore #23

BondAnthony opened this issue Sep 1, 2019 · 0 comments
Assignees
Labels
kind/bug Something isn't working

Comments

@BondAnthony
Copy link
Contributor

I know when using etcd as a backing datastore a panic can occur when wirey is unable to communicate to etcd. Wirey should handle the timeout correctly and retry after a period of time.

panic: Unexpected error from context packet: context deadline exceeded

goroutine 1 [running]:
github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/transport.ContextErr(0xa48240, 0xd05620, 0xd05620, 0x0, 0xc42002e088)
	/home/fntlnz/go/src/github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/transport/transport.go:468 +0x2ae
github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/transport.(*Stream).Header(0xc42024c1c0, 0x0, 0x0, 0x1d)
	/home/fntlnz/go/src/github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/transport/transport.go:223 +0x175
github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc.recvResponse(0xa4dcc0, 0xd058a8, 0x0, 0x0, 0xa4f780, 0xc42017dfe0, 0x101, 0x0, 0x0, 0x0, ...)
	/home/fntlnz/go/src/github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/call.go:54 +0x32
github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc.Invoke(0x7f41f890e050, 0xc420219620, 0x9e5494, 0x16, 0x984b80, 0xc4202337c0, 0x9915a0, 0xc4202560c0, 0xc420198090, 0x0, ...)
	/home/fntlnz/go/src/github.com/coreos/etcd/Godeps/_workspace/src/google.golang.org/grpc/call.go:181 +0x4d2
github.com/coreos/etcd/etcdserver/etcdserverpb.(*kVClient).Range(0xc42000eae0, 0x7f41f890e050, 0xc420219620, 0xc4202337c0, 0x0, 0x0, 0x0, 0xc42021c9a0, 0xc4201414e0, 0x75f51d)
	/home/fntlnz/go/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:1217 +0xd2
github.com/coreos/etcd/clientv3.(*kv).do(0xc420256090, 0x7f41f890e050, 0xc420219620, 0x1, 0xc4202541f0, 0xa, 0x10, 0xc420254200, 0xa, 0xa, ...)
	/home/fntlnz/go/src/github.com/coreos/etcd/clientv3/kv.go:138 +0x68c
github.com/coreos/etcd/clientv3.(*kv).Get(0xc420256090, 0x7f41f890e050, 0xc420219620, 0xc4202541e0, 0xa, 0xc42000eae8, 0x1, 0x1, 0xcec040, 0x100000001, ...)
	/home/fntlnz/go/src/github.com/coreos/etcd/clientv3/kv.go:88 +0xfc
github.com/influxdata/wirey/backend.(*EtcdBackend).GetPeers(0xc42000e510, 0x9d9f93, 0x3, 0xc42022fd01, 0x40, 0x4, 0x0, 0x0)
	/home/fntlnz/go/src/github.com/influxdata/wirey/backend/etcd.go:52 +0x267
github.com/influxdata/wirey/backend.(*Interface).Connect(0xc4200d7480, 0xc42000e510, 0x9d9f93)
	/home/fntlnz/go/src/github.com/influxdata/wirey/backend/plumber.go:190 +0x10b
main.glob..func1(0xcde9e0, 0xc42006a660, 0x0, 0x6)
	/home/fntlnz/go/src/github.com/influxdata/wirey/cmd/wirey/root.go:57 +0x43c
github.com/spf13/cobra.(*Command).execute(0xcde9e0, 0xc42001e080, 0x6, 0x6, 0xcde9e0, 0xc42001e080)
	/home/fntlnz/go/src/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/spf13/cobra.(*Command).ExecuteC(0xcde9e0, 0xc42000e488, 0x0, 0x1)
	/home/fntlnz/go/src/github.com/spf13/cobra/command.go:852 +0x30a
github.com/spf13/cobra.(*Command).Execute(0xcde9e0, 0xc420074058, 0x0)
	/home/fntlnz/go/src/github.com/spf13/cobra/command.go:800 +0x2b
main.Execute()
	/home/fntlnz/go/src/github.com/influxdata/wirey/cmd/wirey/root.go:99 +0x2d
main.main()
	/home/fntlnz/go/src/github.com/influxdata/wirey/cmd/wirey/main.go:4 +0x20
@BondAnthony BondAnthony self-assigned this Sep 1, 2019
@BondAnthony BondAnthony added the kind/bug Something isn't working label Sep 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant