Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mvcc: check null before set FillPercent not to panic #9119

Merged
merged 1 commit into from
Jan 8, 2018

Conversation

yudai
Copy link
Contributor

@yudai yudai commented Jan 8, 2018

Since CreateBucketIfNotExists() can return nil when it gets an error,
accessing FilePercent must be done after a nil check, not to cause
a panic.

We observed a panic during a defrag like below with etcd v3.2.10 under heavy load (larger IO wait). It seems it's due to a nil check failure. As the code is till there in the master, I'm sending a patch to fix the issue. Probably backporting to v3.2.x is required.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x8c76a7]

goroutine 905788 [running]:
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend.defragdb(0xc4401b81e0, 0xc44617a000, 0x2710, 0x0, 0x0)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend/backend.go:373 +0x297
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend.(*backend).defrag(0xc431512480, 0x0, 0x0)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend/backend.go:309 +0x206
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend.(*backend).Defrag(0xc431512480, 0x14b77c0, 0xc431512480)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend/backend.go:275 +0x2b
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*maintenanceServer).Defragment(0xc43bb680e0, 0x7f3918a11e98, 0xc457b4fcb0, 0x1520a10, 0x0, 0xc4bc21f810, 0x3)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/maintenance.go:69 +0x8c
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.(*authMaintenanceServer).Defragment(0xc426286120, 0x7f3918a11e98, 0xc457b4fcb0, 0x1520a10, 0xc426286120, 0xf8dc92, 0xa)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/maintenance.go:169 +0x9e
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Maintenance_Defragment_Handler.func1(0x7f3918a11e98, 0xc457b4fcb0, 0xeb0a40, 0x1520a10, 0xc469ed2c80, 0xc4bc21f801, 0xc4bc21f8b8, 0x4ec4e3)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3823 +0x86
github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.UnaryServerInterceptor(0x7f3918a11e98, 0xc457b4fcb0, 0xeb0a40, 0x1520a10, 0xc45f6d7820, 0xc45f6d7840, 0x40ff68, 0x20, 0xe4b5e0, 0x1)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server.go:29 +0xd2
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1(0x7f3918a11e98, 0xc457b4fcb0, 0xeb0a40, 0x1520a10, 0xc45f6d7820, 0xc45f6d7840, 0x0, 0xc4bc21f9d8, 0x40ff68, 0x50)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/api/v3rpc/interceptor.go:57 +0xc3
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb._Maintenance_Defragment_Handler(0xe77340, 0xc426286120, 0x7f3918a11e98, 0xc457b4fcb0, 0xc469ed2c30, 0xc42c106170, 0x0, 0x0, 0xc441efd7c8, 0xc430be0280)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:3825 +0x177
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc4437ee000, 0x14b73a0, 0xc442c85340, 0xc46b2ada00, 0xc452c089f0, 0x14f4750, 0x0, 0x0, 0x0)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:843 +0xc41
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4437ee000, 0x14b73a0, 0xc442c85340, 0xc46b2ada00, 0x0)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:1040 +0x15a6
github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc43d4ad440, 0xc4437ee000, 0x14b73a0, 0xc442c85340, 0xc46b2ada00)
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:589 +0xa9
created by github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
__/home/gyuho/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/google.golang.org/grpc/server.go:590 +0xa1

@gyuho
Copy link
Contributor

gyuho commented Jan 8, 2018

lgtm. Thanks! /cc @xiang90

@xiang90
Copy link
Contributor

xiang90 commented Jan 8, 2018

lgtm

@yudai yudai changed the title mvcc: check null before set FilePercent not to panic mvcc: check null before set FillPercent not to panic Jan 8, 2018
Since CreateBucketIfNotExists() can return nil when it gets an error,
accessing FillPercent must be done after a nil check, not to cause
a panic.
@gyuho gyuho merged commit a1d3456 into etcd-io:master Jan 8, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

3 participants