Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for mmap-backed CDB reader (yielding a x35 improvement) #11

Open
cipriancraciun opened this issue Dec 26, 2021 · 0 comments

Comments

@cipriancraciun
Copy link

Currently the CDB reader uses io.ReaderAt, that for each read operation will issue a read system call (or equivalent). An alternative would be to use mmap-ed memory to access the backing file.

In my experiments (see the patch below) by using mmap yields a 35 times performance improvement:

BenchmarkGet-4          13902710              2571 ns/op              28 B/op          2 allocs/op
BenchmarkGetMmap-4      484713384               73.96 ns/op            0 B/op          0 allocs/op

The following are a few patches that provide such support. If accepted I can submit a pull request:

  • cipriancraciun@6c948e5 -- adds support of using a []byte buffer that backs the CDB reader; such a []byte buffer can be either the result of a mmap or just reading the entire file in memory;
  • cipriancraciun@0b1bb73 -- this adds support for a custom io.Close to be called on cdb.Close; (it will be used to munmap the memory if needed;)
  • cipriancraciun@924564b -- this adds the actual mmap support;

The only semantic change is the following:

  • the []byte value returned as result of Get should not be written to by the application, as it's actually a slice from the mmap-ed file;
  • the []byte value can't be used after the cdb.Close was called;
  • if this is not acceptable, and one still wants to keep the current (undocumented) semantic, one could clone that slice (thus incurring an extra alloc, which in my patch doesn't happen);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant