Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release Blocker] t.Block slice index crash #1388

Closed
jarifibrahim opened this issue Jun 30, 2020 · 1 comment
Closed

[Release Blocker] t.Block slice index crash #1388

jarifibrahim opened this issue Jun 30, 2020 · 1 comment
Labels
area/crash This issue causes a panic or some other of exception that causes a crash. help wanted Please help us fix this! kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) releaseblocker status/accepted We accept to investigate or work on it.

Comments

@jarifibrahim
Copy link
Contributor

The crash was seen on commit ef28ef3 .
I haven't been able to reproduce the issue but the reuse of byte slices while decompressing aadda9a might be causing this. There was a bug in the reuse code which was fixed by 21735af .

panic: runtime error: slice bounds out of range [-782712:]

goroutine 768 [running]:
github.com/dgraph-io/badger/v2/table.(*Table).block(0xc04879bc00, 0x3a55, 0x1e0000001e0001, 0x3a57, 0x3a56)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/table.go:477 +0xac7
github.com/dgraph-io/badger/v2/table.(*Iterator).seekHelper(0xc18f0bc9c0, 0x3a55, 0xc18e249c00, 0x1e, 0x3e)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/iterator.go:235 +0x40
github.com/dgraph-io/badger/v2/table.(*Iterator).seekFrom(0xc18f0bc9c0, 0xc18e249c00, 0x1e, 0x3e, 0x0)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/iterator.go:271 +0x13a
github.com/dgraph-io/badger/v2/table.(*Iterator).seek(...)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/iterator.go:287
github.com/dgraph-io/badger/v2/table.(*Iterator).Seek(0xc18f0bc9c0, 0xc18e249c00, 0x1e, 0x3e)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/table/iterator.go:397 +0x93
github.com/dgraph-io/badger/v2.(*levelHandler).get(0xc023bb4000, 0xc18e249c00, 0x1e, 0x3e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/level_handler.go:275 +0x230
github.com/dgraph-io/badger/v2.(*levelsController).get(0xc00016e310, 0xc18e249c00, 0x1e, 0x3e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/levels.go:988 +0xcf
github.com/dgraph-io/badger/v2.(*DB).get(0xc023e61400, 0xc18e249c00, 0x1e, 0x3e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:638 +0x33f
github.com/dgraph-io/badger/v2.(*valueLog).rewrite.func1(0xc18e249c00, 0x1e, 0x3e, 0xc18e249c1e, 0x20, 0x20, 0x4, 0x0, 0x40, 0x0, ...)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:517 +0x114
github.com/dgraph-io/badger/v2.(*valueLog).rewrite.func2(0xc18e249c00, 0x1e, 0x3e, 0xc18e249c1e, 0x20, 0x20, 0x4, 0x0, 0x40, 0x0, ...)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:625 +0x44
github.com/dgraph-io/badger/v2.(*valueLog).iterate(0xc023e615d0, 0xc02256fe60, 0x0, 0xc2303f9a20, 0x1, 0x1, 0x65ec059c)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:489 +0x5ab
github.com/dgraph-io/badger/v2.(*valueLog).rewrite(0xc023e615d0, 0xc02256fe60, 0x1b81960, 0xc00038b520, 0x4, 0x4)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:624 +0x34a
github.com/dgraph-io/badger/v2.(*valueLog).doRunGC(0xc023e615d0, 0xc02256fe60, 0x3fe6666666666666, 0x1b81960, 0xc00038b520, 0x0, 0x0)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1757 +0xae7
github.com/dgraph-io/badger/v2.(*valueLog).runGC(0xc023e615d0, 0x3fe6666666666666, 0x24000003dd, 0x160100f, 0x0, 0x0)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/value.go:1797 +0x2c3
github.com/dgraph-io/badger/v2.(*DB).RunValueLogGC(0xc023e61400, 0x3fe6666666666666, 0x0, 0x0)
	/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:1192 +0x1e3
github.com/dgraph-io/dgraph/x.RunVlogGC.func1()
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:894 +0x4a
github.com/dgraph-io/dgraph/x.RunVlogGC(0xc023e61400, 0xc0793442c0)
	/ext-go/1/src/github.com/dgraph-io/dgraph/x/x.go:910 +0x219
created by github.com/dgraph-io/dgraph/worker.(*ServerState).initStorage
	/ext-go/1/src/github.com/dgraph-io/dgraph/worker/server_state.go:171 +0x7fb

The crash is originating from this line
https://github.com/dgraph-io/badger/blob/ef28ef36b5923f12ffe3a1702bdfa6b479db6637/table/table.go#L477
@jarifibrahim jarifibrahim changed the title t.Block slice index crash [Release Blocker] t.Block slice index crash Jun 30, 2020
@jarifibrahim jarifibrahim added area/crash This issue causes a panic or some other of exception that causes a crash. kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) help wanted Please help us fix this! labels Jun 30, 2020
This was referenced Jun 30, 2020
Merged
Closed
@lgalatin lgalatin added the status/accepted We accept to investigate or work on it. label Jul 1, 2020
This was referenced Jul 10, 2020
Closed
Merged
Merged
jarifibrahim pushed a commit that referenced this issue Jul 10, 2020
Revert "add assert to check integer overflow for table size (#1402)" (#…
e0d058c 
…1406)

This reverts commit 7f4e4b5. 
This revert is necessary because b.sz won't exist after builder changes are
reverted and the assert is no longer valid.

The commit 7f4e4b5 is being reverted because we have seen some crashes
which could be caused by these changes. We haven't been able to reproduce the
crashes yet.

Related to #1389, #1388, #1387
Also see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
jarifibrahim pushed a commit that referenced this issue Jul 10, 2020
Revert "fix: Fix race condition in block.incRef (#1337)" (#1407)
63d9309 
This reverts commit 21735af.

The commit is reverted because we have seen some crashes which
could be caused by this change. We haven't been able to reproduce
the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
@jarifibrahim jarifibrahim mentioned this issue Jul 10, 2020
Merged
jarifibrahim pushed a commit that referenced this issue Jul 10, 2020
Revert "Buffer pool for decompression (#1308)" (#1408)
800305e 
This reverts commit aadda9a.

This commit is being reverted because we have seen some crashes
which could be caused by it. We haven't been able to reproduce the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
@jarifibrahim jarifibrahim mentioned this issue Jul 10, 2020
Merged
jarifibrahim pushed a commit that referenced this issue Jul 10, 2020
Revert "Compress/Encrypt Blocks in the background (#1227)" (#1409)
6001230 
This reverts commit b13b927.
This commit is being reverted because we have seen some crashes which
could be caused by it. We haven't been able to reproduce the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602

This commit had some conflicts. See PR description for details.
@jarifibrahim
Copy link
Contributor Author

I wasn't able to reproduce this issue after multiple attempts. I'm closing this as non-reproducible.

jarifibrahim pushed a commit that referenced this issue Oct 2, 2020
Revert "add assert to check integer overflow for table size (#1402)" (#…
5eb5c41 
…1406)

This reverts commit 7f4e4b5. 
This revert is necessary because b.sz won't exist after builder changes are
reverted and the assert is no longer valid.

The commit 7f4e4b5 is being reverted because we have seen some crashes
which could be caused by these changes. We haven't been able to reproduce the
crashes yet.

Related to #1389, #1388, #1387
Also see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
jarifibrahim pushed a commit that referenced this issue Oct 2, 2020
Revert "fix: Fix race condition in block.incRef (#1337)" (#1407)
268db59 
This reverts commit 21735af.

The commit is reverted because we have seen some crashes which
could be caused by this change. We haven't been able to reproduce
the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
jarifibrahim pushed a commit that referenced this issue Oct 2, 2020
Revert "Buffer pool for decompression (#1308)" (#1408)
0cff38a 
This reverts commit aadda9a.

This commit is being reverted because we have seen some crashes
which could be caused by it. We haven't been able to reproduce the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602
jarifibrahim pushed a commit that referenced this issue Oct 2, 2020
Revert "Compress/Encrypt Blocks in the background (#1227)" (#1409)
f198dc7 
This reverts commit b13b927.
This commit is being reverted because we have seen some crashes which
could be caused by it. We haven't been able to reproduce the crashes yet.

Related to #1389, #1388, #1387
Also, see https://discuss.dgraph.io/t/current-state-of-badger-crashes/7602

This commit had some conflicts. See PR description for details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/crash This issue causes a panic or some other of exception that causes a crash. help wanted Please help us fix this! kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) releaseblocker status/accepted We accept to investigate or work on it.
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jarifibrahim @lgalatin