Skip to content

[kvdb-rocksdb] Add benchmark for point lookups#275

Merged
dvdplm merged 7 commits into
masterfrom
dp-add-rocksdb-benchmark
Dec 10, 2019
Merged

[kvdb-rocksdb] Add benchmark for point lookups#275
dvdplm merged 7 commits into
masterfrom
dp-add-rocksdb-benchmark

Conversation

@dvdplm
Copy link
Copy Markdown
Contributor

@dvdplm dvdplm commented Nov 29, 2019

This is far from perfect but might be a start of something. The tricky part of benchmarking databases is to setup the data set properly. This PR doesn't do anything particularly fancy but it's worth mentioning the basics:

  • each benchmark run adds 1M keys to the DB (single tx, so if you're low on memory be patient) ;
  • empty/small DBs are much faster obviously so the first few benchmark runs are not very indicative of anything; I think at least 3-8M keys are needed before the values start to actually mean something;
  • interestingly though, during my testing, performance seem to plateau and after a while, and adding more data doesn't seem to slow any of the tests down by much
  • the benchmarks all count the number of allocations; this is why they use iter_custom() rather than iter().

Benchmark results:

get key                 time:   [2.5151 us 2.5616 us 2.6140 us]
                        change: [-4.3238% -1.1310% +2.0720%] (p = 0.50 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

[get key] total: iterations=4016151, allocations=6880994; allocations per iter=1.71

get key by prefix       time:   [15.011 us 15.252 us 15.511 us]
                        change: [-1.5796% +2.0998% +5.6256%] (p = 0.25 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe

[get key by prefix] total: iterations=580293, allocations=1160586; allocations per iter=2.00

iterate over 1k keys    time:   [430.52 us 433.30 us 436.16 us]
                        change: [-11.873% -3.7270% +2.5027%] (p = 0.44 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

[iterate over 1k keys] total: iterations=23341, allocations=46705341; allocations per iter=2.00

single key from iterator
                        time:   [2.8204 us 2.8454 us 2.8730 us]
                        change: [-6.8037% -4.5874% -2.2547%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

[single key from iterator] total: iterations=2785775, allocations=5571550; allocations per iter=2.00

@dvdplm dvdplm self-assigned this Nov 29, 2019
@dvdplm dvdplm marked this pull request as ready for review November 29, 2019 17:21
Comment thread kvdb-rocksdb/benches/bench_read_perf.rs
Copy link
Copy Markdown
Contributor

@ordian ordian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good as a starting point.

@dvdplm
Copy link
Copy Markdown
Contributor Author

dvdplm commented Dec 10, 2019

ping @niklasad1

Comment thread kvdb-rocksdb/benches/bench_read_perf.rs Outdated
Comment thread kvdb-rocksdb/benches/bench_read_perf.rs
Comment thread kvdb-rocksdb/benches/bench_read_perf.rs
Comment thread kvdb-rocksdb/benches/bench_read_perf.rs
Comment thread kvdb-rocksdb/benches/bench_read_perf.rs
Comment thread kvdb-rocksdb/benches/bench_read_perf.rs Outdated
@dvdplm dvdplm merged commit a6058e9 into master Dec 10, 2019
@dvdplm dvdplm deleted the dp-add-rocksdb-benchmark branch December 10, 2019 18:54
dvdplm added a commit that referenced this pull request Dec 17, 2019
* master:
  Fix a typo in error rlp error message (#283)
  [kvdb*] Make column type u32 instead of Option<u32> (#278)
  [kvdb-rocksdb] Add benchmark for point lookups (#275)
  Compile triehash for no_std (#280)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants