Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compress/Encrypt Blocks in background #1227

Merged
merged 36 commits into from
Mar 12, 2020
Merged

Compress/Encrypt Blocks in background #1227

Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
ee65856
Doesn't work
Jan 31, 2020
72d1932
Everything works but has race
Feb 11, 2020
ebbb23e
Everything works but super slow
Feb 11, 2020
62cda51
Everything works but slower than master
Feb 12, 2020
7230da4
Reuse encode buffer
Feb 12, 2020
277b0af
Reuse pool for decompression and []byte in encode
Feb 12, 2020
15ae690
fix benchmark
Feb 12, 2020
93963ad
performance issues. One go routine faster than multiple go routines
Feb 12, 2020
37d0c0a
Dont use go routines for no compression/encryption
Feb 21, 2020
5795be5
All table tests work
Feb 21, 2020
b86d1b1
Reduce memory consumption
Feb 21, 2020
01f304e
fix test
Feb 21, 2020
da3c4c4
Cleanup
Feb 24, 2020
269bdbd
cleanup go.mod/sum
Feb 24, 2020
1f99b13
fixup
Feb 24, 2020
c141ebf
Merge branch 'master' into ibrahim/replace-b.buf
Feb 24, 2020
bd0ae87
Use locks to protect buf copy
Mar 2, 2020
d807f05
Use locks to protect buf copy
Mar 2, 2020
dece005
Merge branch 'ibrahim/replace-b.buf' of https://github.com/dgraph-io/…
Mar 2, 2020
30a4c31
Remove unused key from bblock
Mar 2, 2020
833bae4
Fix benchmark
Mar 2, 2020
7061c81
Fix slicepool usage
Mar 2, 2020
2fecb9d
Fix slicepool usage
Mar 2, 2020
8014669
fix review comments
Mar 3, 2020
95dffd2
Use dst to store final table
Mar 3, 2020
0a506f0
Reduce size of blockchan
Mar 3, 2020
f869acc
Disable all stream writer tests
Mar 3, 2020
34af954
Revert "Disable all stream writer tests"
Mar 4, 2020
ac6d2c0
Revert "fix review comments"
Mar 4, 2020
58185c6
Merge branch 'master' into ibrahim/replace-b.buf
Mar 4, 2020
19d9717
Revert "Reduce size of blockchan"
Mar 5, 2020
d1b850f
Revert "Use dst to store final table"
Mar 5, 2020
c6cd35f
Revert "Revert "Reduce size of blockchan""
Mar 5, 2020
10e0c2b
Revert "Revert "fix review comments""
Mar 5, 2020
6f2a0fc
Use dst instead of b.buf
Mar 6, 2020
1b4a04f
Cleanup
Mar 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions db2_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -669,16 +669,13 @@ func TestL0GCBug(t *testing.T) {
// Simulate a crash by not closing db1 but releasing the locks.
if db1.dirLockGuard != nil {
require.NoError(t, db1.dirLockGuard.release())
db1.dirLockGuard = nil
}
if db1.valueDirGuard != nil {
require.NoError(t, db1.valueDirGuard.release())
db1.valueDirGuard = nil
}
for _, f := range db1.vlog.filesMap {
require.NoError(t, f.fd.Close())
}
require.NoError(t, db1.registry.Close())
require.NoError(t, db1.lc.close())
require.NoError(t, db1.manifest.close())
require.NoError(t, db1.Close())

db2, err := Open(opts)
require.NoError(t, err)
Expand Down
1 change: 1 addition & 0 deletions options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ func DefaultOptions(path string) Options {

func buildTableOptions(opt Options) table.Options {
return table.Options{
TableSize: uint64(opt.MaxTableSize),
BlockSize: opt.BlockSize,
BloomFalsePositive: opt.BloomFalsePositive,
LoadingMode: opt.TableLoadingMode,
Expand Down
Loading