Add benchmark for RndState and add non-null tracers in benchmark
#2030
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KtorZ
added
the
RESOLVING ISSUE
rvl
reviewed
Aug 12, 2020
lib/core/test/bench/db/Main.hs
Outdated
| import qualified Data.ByteString as BS | ||
| import qualified Data.ByteString.Char8 as B8 | ||
| import qualified Data.Map.Strict as Map | ||
|
|
||
| main :: IO () | ||
| main = withUtf8Encoding $ do | ||
| main = withUtf8Encoding $ withLogging $ \trace -> do | ||
| let tr = trMessageText $ filterTraceSeverity Error trace |
There was a problem hiding this comment.
Could we use Cardano.Wallet.Shelley.setupTracers to get a tracer? That would be more real.
There was a problem hiding this comment.
Then we would have to relocate the benchmark code. In particular, the benchmarks are also still using the JormungandrKey implementations and not the Shelley ones for addresses.
I originally used stuff from cardano-cli, but it was a sort of circular dependency since cardano-cli depends on cardano-core and the benchmark is part of cardano-core. It was quite annoying to compile because on any change, instead of having to simply recompile the benchmark code, it had to re-compile the entire cardano-wallet-core :|
I also did some re-structuring of the benchmark fixtures, so that the
creation of benchmark data happens outside of the actual benchmarks.
Also, I reviewed some of the work bench to use numbers that reflects a
bit more reality (we actually always store checkpoints one by one, so
the time it takes to store N checkpoints isn't quite interesting).
Latest result on my machine:
- NVMe SSD Controller SM981/PM981
- Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
- 256KiB L1 cache
- 1MiB L2 cache
- 8MiB L3 cache
- 32GiB System Memory
- 16GiB SODIMM DDR4 Synchronous 2400 MHz (0,4 ns)
- 16GiB SODIMM DDR4 Synchronous 2400 MHz (0,4 ns)
```
benchmarking UTxO (Write)/1 CP x 0 UTxO
time 3.182 ms (2.943 ms .. 3.493 ms)
0.943 R² (0.902 R² .. 0.975 R²)
mean 3.228 ms (3.071 ms .. 3.439 ms)
std dev 433.0 μs (329.2 μs .. 563.3 μs)
variance introduced by outliers: 69% (severely inflated)
benchmarking UTxO (Write)/1 CP x 10 UTxO
time 3.583 ms (3.407 ms .. 3.821 ms)
0.970 R² (0.943 R² .. 0.988 R²)
mean 3.602 ms (3.446 ms .. 3.839 ms)
std dev 463.5 μs (307.3 μs .. 695.7 μs)
variance introduced by outliers: 65% (severely inflated)
benchmarking UTxO (Write)/1 CP x 100 UTxO
time 6.134 ms (5.834 ms .. 6.451 ms)
0.986 R² (0.970 R² .. 0.994 R²)
mean 6.002 ms (5.849 ms .. 6.173 ms)
std dev 392.8 μs (310.1 μs .. 569.3 μs)
variance introduced by outliers: 29% (moderately inflated)
benchmarking UTxO (Write)/1 CP x 1000 UTxO
time 29.62 ms (28.15 ms .. 31.61 ms)
0.990 R² (0.976 R² .. 0.998 R²)
mean 29.83 ms (29.04 ms .. 30.76 ms)
std dev 1.768 ms (1.312 ms .. 2.395 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking UTxO (Write)/1 CP x 10000 UTxO
time 271.5 ms (258.9 ms .. 285.0 ms)
0.998 R² (0.996 R² .. 1.000 R²)
mean 268.5 ms (258.0 ms .. 274.1 ms)
std dev 9.334 ms (3.659 ms .. 12.75 ms)
variance introduced by outliers: 16% (moderately inflated)
benchmarking UTxO (Write)/1 CP x 100000 UTxO
time 2.844 s (2.552 s .. 3.177 s)
0.998 R² (0.994 R² .. 1.000 R²)
mean 2.768 s (2.691 s .. 2.811 s)
std dev 73.91 ms (17.62 ms .. 98.78 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking UTxO (Write)/10 CP x 1000 UTxO
time 316.2 ms (222.6 ms .. 417.3 ms)
0.981 R² (0.979 R² .. 1.000 R²)
mean 316.9 ms (292.7 ms .. 341.0 ms)
std dev 30.48 ms (14.55 ms .. 38.38 ms)
variance introduced by outliers: 22% (moderately inflated)
benchmarking UTxO (Write)/100 CP x 1000 UTxO
time 2.565 s (2.239 s .. 2.766 s)
0.998 R² (0.996 R² .. 1.000 R²)
mean 3.037 s (2.804 s .. 3.226 s)
std dev 261.6 ms (100.5 ms .. 357.2 ms)
variance introduced by outliers: 22% (moderately inflated)
benchmarking UTxO (Read)/1 CP x 0 UTxO
time 5.069 ms (4.804 ms .. 5.363 ms)
0.967 R² (0.923 R² .. 0.993 R²)
mean 4.829 ms (4.619 ms .. 5.102 ms)
std dev 533.6 μs (333.9 μs .. 780.3 μs)
variance introduced by outliers: 54% (severely inflated)
benchmarking UTxO (Read)/1 CP x 10 UTxO
time 4.009 ms (3.720 ms .. 4.293 ms)
0.969 R² (0.931 R² .. 0.991 R²)
mean 4.577 ms (4.387 ms .. 4.897 ms)
std dev 552.9 μs (331.6 μs .. 889.7 μs)
variance introduced by outliers: 58% (severely inflated)
benchmarking UTxO (Read)/1 CP x 100 UTxO
time 5.092 ms (4.932 ms .. 5.252 ms)
0.995 R² (0.991 R² .. 0.998 R²)
mean 5.137 ms (5.054 ms .. 5.201 ms)
std dev 158.1 μs (120.2 μs .. 222.4 μs)
benchmarking UTxO (Read)/1 CP x 1000 UTxO
time 19.79 ms (18.20 ms .. 22.09 ms)
0.953 R² (0.878 R² .. 0.990 R²)
mean 19.86 ms (18.82 ms .. 21.49 ms)
std dev 2.510 ms (1.607 ms .. 4.136 ms)
variance introduced by outliers: 48% (moderately inflated)
benchmarking UTxO (Read)/1 CP x 10000 UTxO
time 161.8 ms (105.9 ms .. 180.0 ms)
0.986 R² (0.937 R² .. 1.000 R²)
mean 151.8 ms (138.0 ms .. 159.9 ms)
std dev 13.51 ms (5.826 ms .. 19.63 ms)
variance introduced by outliers: 18% (moderately inflated)
benchmarking UTxO (Read)/1 CP x 100000 UTxO
time 1.428 s (1.369 s .. 1.555 s)
0.999 R² (0.998 R² .. 1.000 R²)
mean 1.381 s (1.368 s .. 1.406 s)
std dev 23.90 ms (130.2 μs .. 28.16 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking SeqState/1 CP x 10 addr
time 3.379 ms (3.170 ms .. 3.572 ms)
0.976 R² (0.954 R² .. 0.991 R²)
mean 3.311 ms (3.195 ms .. 3.430 ms)
std dev 284.2 μs (212.5 μs .. 396.9 μs)
variance introduced by outliers: 47% (moderately inflated)
benchmarking SeqState/1 CP x 100 addr
time 5.443 ms (5.138 ms .. 5.713 ms)
0.979 R² (0.956 R² .. 0.992 R²)
mean 5.466 ms (5.308 ms .. 5.672 ms)
std dev 427.8 μs (291.6 μs .. 592.0 μs)
variance introduced by outliers: 39% (moderately inflated)
benchmarking SeqState/1 CP x 1000 addr
time 26.29 ms (25.03 ms .. 27.87 ms)
0.993 R² (0.986 R² .. 0.999 R²)
mean 24.41 ms (23.76 ms .. 25.15 ms)
std dev 1.436 ms (1.024 ms .. 2.145 ms)
variance introduced by outliers: 18% (moderately inflated)
benchmarking SeqState/1 CP x 10000 addr
time 217.6 ms (143.9 ms .. 283.6 ms)
0.965 R² (0.948 R² .. 1.000 R²)
mean 249.6 ms (231.4 ms .. 266.7 ms)
std dev 22.71 ms (13.88 ms .. 31.51 ms)
variance introduced by outliers: 18% (moderately inflated)
benchmarking SeqState/1 CP x 100000 addr
time 2.629 s (2.361 s .. 2.815 s)
0.999 R² (0.996 R² .. 1.000 R²)
mean 2.487 s (2.314 s .. 2.552 s)
std dev 120.2 ms (4.664 ms .. 149.2 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking SeqState/10 CP x 1000 addr
time 49.44 ms (44.08 ms .. 54.40 ms)
0.975 R² (0.946 R² .. 0.997 R²)
mean 51.44 ms (49.24 ms .. 54.66 ms)
std dev 4.991 ms (2.424 ms .. 6.888 ms)
variance introduced by outliers: 32% (moderately inflated)
benchmarking SeqState/100 CP x 1000 addr
time 295.9 ms (261.5 ms .. 343.9 ms)
0.992 R² (0.981 R² .. 1.000 R²)
mean 287.5 ms (280.0 ms .. 295.1 ms)
std dev 10.26 ms (5.842 ms .. 14.28 ms)
variance introduced by outliers: 16% (moderately inflated)
benchmarking RndState/1 CP x 10 addr
time 2.544 ms (2.375 ms .. 2.710 ms)
0.963 R² (0.929 R² .. 0.983 R²)
mean 2.614 ms (2.515 ms .. 2.778 ms)
std dev 309.7 μs (192.6 μs .. 439.4 μs)
variance introduced by outliers: 66% (severely inflated)
benchmarking RndState/1 CP x 100 addr
time 3.527 ms (3.271 ms .. 3.754 ms)
0.963 R² (0.931 R² .. 0.987 R²)
mean 3.505 ms (3.340 ms .. 3.797 ms)
std dev 548.9 μs (308.4 μs .. 1.013 ms)
variance introduced by outliers: 77% (severely inflated)
benchmarking RndState/1 CP x 1000 addr
time 8.955 ms (8.512 ms .. 9.397 ms)
0.987 R² (0.977 R² .. 0.994 R²)
mean 8.904 ms (8.689 ms .. 9.127 ms)
std dev 502.1 μs (427.8 μs .. 606.0 μs)
variance introduced by outliers: 25% (moderately inflated)
benchmarking RndState/1 CP x 10000 addr
time 59.16 ms (54.70 ms .. 61.69 ms)
0.988 R² (0.963 R² .. 0.997 R²)
mean 60.60 ms (58.74 ms .. 63.37 ms)
std dev 3.753 ms (2.618 ms .. 4.804 ms)
variance introduced by outliers: 18% (moderately inflated)
benchmarking RndState/1 CP x 100000 addr
time 570.5 ms (549.2 ms .. 602.6 ms)
1.000 R² (0.999 R² .. 1.000 R²)
mean 585.9 ms (576.4 ms .. 598.2 ms)
std dev 13.13 ms (3.653 ms .. 17.76 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking RndState/10 CP x 1000 addr
time 561.2 ms (494.1 ms .. 708.5 ms)
0.992 R² (0.984 R² .. 1.000 R²)
mean 513.5 ms (500.1 ms .. 539.3 ms)
std dev 25.48 ms (685.7 μs .. 30.17 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking RndState/100 CP x 1000 addr
time 52.21 s (48.98 s .. 54.44 s)
1.000 R² (0.999 R² .. 1.000 R²)
mean 51.66 s (51.02 s .. 52.21 s)
std dev 797.5 ms (352.6 ms .. 1.031 s)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Write)/1 w/ 1i + 1o [1..10]
time 2.018 ms (1.821 ms .. 2.215 ms)
0.926 R² (0.865 R² .. 0.967 R²)
mean 2.526 ms (2.375 ms .. 2.704 ms)
std dev 415.1 μs (339.3 μs .. 510.2 μs)
variance introduced by outliers: 77% (severely inflated)
benchmarking TxHistory (Write)/10 w/ 1i + 1o [1..10]
time 2.959 ms (2.738 ms .. 3.186 ms)
0.961 R² (0.932 R² .. 0.982 R²)
mean 3.299 ms (3.147 ms .. 3.471 ms)
std dev 400.0 μs (301.7 μs .. 564.0 μs)
variance introduced by outliers: 63% (severely inflated)
benchmarking TxHistory (Write)/10 w/ 10i + 10o [1..10]
time 7.899 ms (7.527 ms .. 8.293 ms)
0.989 R² (0.981 R² .. 0.995 R²)
mean 7.345 ms (7.138 ms .. 7.553 ms)
std dev 511.7 μs (425.2 μs .. 624.8 μs)
variance introduced by outliers: 31% (moderately inflated)
benchmarking TxHistory (Write)/10 w/ 50i + 100o [1..10]
time 53.62 ms (47.93 ms .. 57.53 ms)
0.978 R² (0.931 R² .. 0.997 R²)
mean 54.73 ms (51.41 ms .. 59.62 ms)
std dev 7.023 ms (3.806 ms .. 10.75 ms)
variance introduced by outliers: 44% (moderately inflated)
benchmarking TxHistory (Write)/10 w/ 255i + 255o [1..100]
time 166.2 ms (145.8 ms .. 180.3 ms)
0.992 R² (0.983 R² .. 1.000 R²)
mean 151.8 ms (145.0 ms .. 157.7 ms)
std dev 9.344 ms (6.262 ms .. 13.87 ms)
variance introduced by outliers: 13% (moderately inflated)
benchmarking TxHistory (Write)/100 w/ 10i + 10o [1..100]
time 65.11 ms (56.94 ms .. 76.31 ms)
0.969 R² (0.915 R² .. 0.998 R²)
mean 65.39 ms (62.09 ms .. 69.57 ms)
std dev 6.323 ms (4.347 ms .. 8.712 ms)
variance introduced by outliers: 28% (moderately inflated)
benchmarking TxHistory (Write)/100 w/ 50i + 100o [1..100]
time 546.7 ms (387.3 ms .. 704.4 ms)
0.989 R² (0.960 R² .. 1.000 R²)
mean 526.1 ms (507.0 ms .. 558.2 ms)
std dev 30.39 ms (7.684 ms .. 38.90 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Write)/100 w/ 255i + 255o [1..100]
time 1.731 s (1.481 s .. 1.897 s)
0.998 R² (0.992 R² .. 1.000 R²)
mean 1.621 s (1.522 s .. 1.677 s)
std dev 96.13 ms (29.17 ms .. 130.9 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Write)/1000 w/ 10i + 10o [1..1000]
time 554.0 ms (512.9 ms .. 583.7 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 559.0 ms (552.8 ms .. 566.8 ms)
std dev 8.040 ms (3.100 ms .. 11.02 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Write)/1000 w/ 50i + 100o [1..1000]
time 4.592 s (4.014 s .. 4.937 s)
0.998 R² (0.995 R² .. 1.000 R²)
mean 4.842 s (4.663 s .. 5.131 s)
std dev 283.6 ms (3.899 ms .. 360.3 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Write)/1000 w/ 255i + 255o [1..1000]
time 14.42 s (13.98 s .. 15.46 s)
0.999 R² (0.999 R² .. 1.000 R²)
mean 14.10 s (14.00 s .. 14.28 s)
std dev 177.2 ms (5.722 ms .. 212.1 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Write)/10000 w/ 10i + 10o [1..10000]
time 5.793 s (5.746 s .. 5.837 s)
1.000 R² (1.000 R² .. 1.000 R²)
mean 5.814 s (5.800 s .. 5.838 s)
std dev 22.72 ms (3.755 ms .. 29.17 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Read)/1000 [1..100] DESC - *
time 3.111 s (2.894 s .. 3.342 s)
0.999 R² (NaN R² .. 1.000 R²)
mean 3.351 s (3.230 s .. 3.447 s)
std dev 121.3 ms (59.69 ms .. 160.2 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Read)/1000 [1..100] ASC - *
time 3.182 s (2.844 s .. 3.351 s)
0.999 R² (0.997 R² .. 1.000 R²)
mean 3.371 s (3.275 s .. 3.427 s)
std dev 93.88 ms (27.85 ms .. 127.6 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Read)/1000 [1..1000] DESC - *
time 3.518 s (3.237 s .. 3.736 s)
0.999 R² (0.999 R² .. 1.000 R²)
mean 3.458 s (3.389 s .. 3.493 s)
std dev 66.33 ms (32.05 ms .. 80.19 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Read)/1000 [1..100] DESC pending *
time 955.7 ms (712.4 ms .. 1.289 s)
0.982 R² (0.976 R² .. 1.000 R²)
mean 829.7 ms (779.1 ms .. 897.1 ms)
std dev 68.89 ms (23.29 ms .. 93.21 ms)
variance introduced by outliers: 21% (moderately inflated)
benchmarking TxHistory (Read)/1000 [1..100] DESC - 40..60
time 135.4 ms (94.69 ms .. 166.4 ms)
0.986 R² (0.980 R² .. 1.000 R²)
mean 154.6 ms (140.7 ms .. 160.1 ms)
std dev 9.667 ms (1.847 ms .. 12.70 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Read)/1000 [1..10000] DESC - 42..1337
time 3.182 s (2.610 s .. 4.009 s)
0.993 R² (0.978 R² .. 1.000 R²)
mean 3.286 s (3.125 s .. 3.388 s)
std dev 167.7 ms (87.87 ms .. 236.0 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking TxHistory (Read)/10000 [1..100] DESC - 40..60
time 21.48 s (17.82 s .. 23.75 s)
0.997 R² (0.990 R² .. 1.000 R²)
mean 19.03 s (17.94 s .. 20.13 s)
std dev 1.290 s (1.083 s .. 1.455 s)
variance introduced by outliers: 20% (moderately inflated)
benchmarking TxHistory (Read)/10000 [1..10000] DESC - 42..1337
time 6.274 s (6.142 s .. 6.343 s)
1.000 R² (1.000 R² .. 1.000 R²)
mean 6.246 s (6.205 s .. 6.262 s)
std dev 29.77 ms (459.4 μs .. 36.66 ms)
variance introduced by outliers: 19% (moderately inflated)
--
Database disk space usage tests for UTxO
File size /100 CP x 0 UTxO
bench22158-7.db 4 KB
bench22158-7.db-shm 32 KB
bench22158-7.db-wal 249 KB
bench22158-7.db (closed) 192 KB
File size /1000 CP x 0 UTxO
bench22158-8.db 4 KB
bench22158-8.db-shm 32 KB
bench22158-8.db-wal 555 KB
bench22158-8.db (closed) 496 KB
File size /10 CP x 10 UTxO
bench22158-9.db 4 KB
bench22158-9.db-shm 32 KB
bench22158-9.db-wal 257 KB
bench22158-9.db (closed) 192 KB
File size /100 CP x 10 UTxO
bench22158-10.db 4 KB
bench22158-10.db-shm 32 KB
bench22158-10.db-wal 639 KB
bench22158-10.db (closed) 572 KB
File size /1000 CP x 10 UTxO
bench22158-11.db 4 MB
bench22158-11.db-shm 32 KB
bench22158-11.db-wal 4 MB
bench22158-11.db (closed) 4 MB
File size /10 CP x 100 UTxO
bench22158-12.db 4 KB
bench22158-12.db-shm 32 KB
bench22158-12.db-wal 599 KB
bench22158-12.db (closed) 532 KB
File size /100 CP x 100 UTxO
bench22158-13.db 3 MB
bench22158-13.db-shm 32 KB
bench22158-13.db-wal 4 MB
bench22158-13.db (closed) 3 MB
File size /1000 CP x 100 UTxO
bench22158-14.db 37 MB
bench22158-14.db-shm 96 KB
bench22158-14.db-wal 38 MB
bench22158-14.db (closed) 37 MB
File size /10 CP x 1000 UTxO
bench22158-15.db 3 MB
bench22158-15.db-shm 32 KB
bench22158-15.db-wal 3 MB
bench22158-15.db (closed) 3 MB
File size /100 CP x 1000 UTxO
bench22158-16.db 37 MB
bench22158-16.db-shm 96 KB
bench22158-16.db-wal 37 MB
bench22158-16.db (closed) 37 MB
File size /1000 CP x 1000 UTxO
bench22158-17.db 378 MB
bench22158-17.db-shm 768 KB
bench22158-17.db-wal 380 MB
bench22158-17.db (closed) 378 MB
Database disk space usage tests for TxHistory
File size /100 w/ 10i + 20o
bench22158-18.db 4 KB
bench22158-18.db-shm 32 KB
bench22158-18.db-wal 1 MB
bench22158-18.db (closed) 1 MB
File size /1000 w/ 10i + 20o
bench22158-19.db 9 MB
bench22158-19.db-shm 32 KB
bench22158-19.db-wal 9 MB
bench22158-19.db (closed) 9 MB
File size /10000 w/ 10i + 20o
bench22158-20.db 92 MB
bench22158-20.db-shm 192 KB
bench22158-20.db-wal 92 MB
bench22158-20.db (closed) 92 MB
File size /100000 w/ 10i + 20o
bench22158-21.db 920 MB
bench22158-21.db-shm 1 MB
bench22158-21.db-wal 926 MB
bench22158-21.db (closed) 920 MB
File size /100 w/ 50i + 100o
bench22158-22.db 4 MB
bench22158-22.db-shm 32 KB
bench22158-22.db-wal 4 MB
bench22158-22.db (closed) 4 MB
File size /1000 w/ 50i + 100o
bench22158-23.db 47 MB
bench22158-23.db-shm 96 KB
bench22158-23.db-wal 47 MB
bench22158-23.db (closed) 47 MB
File size /10000 w/ 50i + 100o
bench22158-24.db 471 MB
bench22158-24.db-shm 960 KB
bench22158-24.db-wal 474 MB
bench22158-24.db (closed) 471 MB
File size /100000 w/ 50i + 100o
bench22158-25.db 4 GB
bench22158-25.db-shm 9 MB
bench22158-25.db-wal 4 GB
bench22158-25.db (closed) 4 GB
Benchmark db: FINISH
Success! Waiting for next file change.
```
…gandr delegation addresses...
Benchmark should somewhat reflect the production setup, which has a non-null logger enabled. In principle, logging should not impact the performances of the application, but this commit shows that it currently does, and a lot!
This has a dramatic impact on performances. Here are some comparisons of the database benchmarks with and without this fix, as well a comparison with the original benchmark behavior when using a `nullTracer`. benchmark | before | after | nullTracer --- | --- | --- | --- UTxO (Write)/1 CP x 100000 UTxO | 3.839s | 2.238s | 2.225s UTxO (Read)/1 CP x 100000 UTxO | 1.074s | 0.998s | 1.072s SeqState/1 CP x 100000 addr | 4.281s | 1.887s | 2.020s RndState/1 CP x 100000 addr x 100000 | 3.996s | 0.886s | 0.849s TxHistory (Write)/10000 w/ 10i + 10o | 8.032s | 5.793s | 5.587s TxHistory (Read)/10000 | 17.08s | 16.11s | 16.47s
KtorZ
force-pushed
the
KtorZ/slow-restore-investigation
branch
from
f4b62a2 to
dff0168
Compare
rvl
approved these changes
Aug 13, 2020
|
bors r+ |
iohk-bors bot
added a commit
that referenced
this pull request
Aug 13, 2020
2030: Add benchmark for `RndState` and add non-null tracers in benchmark r=KtorZ a=KtorZ # Issue Number <!-- Put here a reference to the issue this PR relates to and which requirements it tackles --> #1997 # Overview <!-- Detail in a few bullet points the work accomplished in this PR --> - 36bfac4 📍 **extend benchmark with RndState and byron address storage** I also did some re-structuring of the benchmark fixtures, so that the creation of benchmark data happens outside of the actual benchmarks. Also, I reviewed some of the work bench to use numbers that reflects a bit more reality (we actually always store checkpoints one by one, so the time it takes to store N checkpoints isn't quite interesting). <details> <summary>see full results</summary> - NVMe SSD Controller SM981/PM981 - Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz - 256KiB L1 cache - 1MiB L2 cache - 8MiB L3 cache - 32GiB System Memory - 16GiB SODIMM DDR4 Synchronous 2400 MHz (0,4 ns) - 16GiB SODIMM DDR4 Synchronous 2400 MHz (0,4 ns) ``` benchmarking UTxO (Write)/1 CP x 0 UTxO time 3.182 ms (2.943 ms .. 3.493 ms) 0.943 R² (0.902 R² .. 0.975 R²) mean 3.228 ms (3.071 ms .. 3.439 ms) std dev 433.0 μs (329.2 μs .. 563.3 μs) variance introduced by outliers: 69% (severely inflated) benchmarking UTxO (Write)/1 CP x 10 UTxO time 3.583 ms (3.407 ms .. 3.821 ms) 0.970 R² (0.943 R² .. 0.988 R²) mean 3.602 ms (3.446 ms .. 3.839 ms) std dev 463.5 μs (307.3 μs .. 695.7 μs) variance introduced by outliers: 65% (severely inflated) benchmarking UTxO (Write)/1 CP x 100 UTxO time 6.134 ms (5.834 ms .. 6.451 ms) 0.986 R² (0.970 R² .. 0.994 R²) mean 6.002 ms (5.849 ms .. 6.173 ms) std dev 392.8 μs (310.1 μs .. 569.3 μs) variance introduced by outliers: 29% (moderately inflated) benchmarking UTxO (Write)/1 CP x 1000 UTxO time 29.62 ms (28.15 ms .. 31.61 ms) 0.990 R² (0.976 R² .. 0.998 R²) mean 29.83 ms (29.04 ms .. 30.76 ms) std dev 1.768 ms (1.312 ms .. 2.395 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking UTxO (Write)/1 CP x 10000 UTxO time 271.5 ms (258.9 ms .. 285.0 ms) 0.998 R² (0.996 R² .. 1.000 R²) mean 268.5 ms (258.0 ms .. 274.1 ms) std dev 9.334 ms (3.659 ms .. 12.75 ms) variance introduced by outliers: 16% (moderately inflated) benchmarking UTxO (Write)/1 CP x 100000 UTxO time 2.844 s (2.552 s .. 3.177 s) 0.998 R² (0.994 R² .. 1.000 R²) mean 2.768 s (2.691 s .. 2.811 s) std dev 73.91 ms (17.62 ms .. 98.78 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking UTxO (Write)/10 CP x 1000 UTxO time 316.2 ms (222.6 ms .. 417.3 ms) 0.981 R² (0.979 R² .. 1.000 R²) mean 316.9 ms (292.7 ms .. 341.0 ms) std dev 30.48 ms (14.55 ms .. 38.38 ms) variance introduced by outliers: 22% (moderately inflated) benchmarking UTxO (Write)/100 CP x 1000 UTxO time 2.565 s (2.239 s .. 2.766 s) 0.998 R² (0.996 R² .. 1.000 R²) mean 3.037 s (2.804 s .. 3.226 s) std dev 261.6 ms (100.5 ms .. 357.2 ms) variance introduced by outliers: 22% (moderately inflated) benchmarking UTxO (Read)/1 CP x 0 UTxO time 5.069 ms (4.804 ms .. 5.363 ms) 0.967 R² (0.923 R² .. 0.993 R²) mean 4.829 ms (4.619 ms .. 5.102 ms) std dev 533.6 μs (333.9 μs .. 780.3 μs) variance introduced by outliers: 54% (severely inflated) benchmarking UTxO (Read)/1 CP x 10 UTxO time 4.009 ms (3.720 ms .. 4.293 ms) 0.969 R² (0.931 R² .. 0.991 R²) mean 4.577 ms (4.387 ms .. 4.897 ms) std dev 552.9 μs (331.6 μs .. 889.7 μs) variance introduced by outliers: 58% (severely inflated) benchmarking UTxO (Read)/1 CP x 100 UTxO time 5.092 ms (4.932 ms .. 5.252 ms) 0.995 R² (0.991 R² .. 0.998 R²) mean 5.137 ms (5.054 ms .. 5.201 ms) std dev 158.1 μs (120.2 μs .. 222.4 μs) benchmarking UTxO (Read)/1 CP x 1000 UTxO time 19.79 ms (18.20 ms .. 22.09 ms) 0.953 R² (0.878 R² .. 0.990 R²) mean 19.86 ms (18.82 ms .. 21.49 ms) std dev 2.510 ms (1.607 ms .. 4.136 ms) variance introduced by outliers: 48% (moderately inflated) benchmarking UTxO (Read)/1 CP x 10000 UTxO time 161.8 ms (105.9 ms .. 180.0 ms) 0.986 R² (0.937 R² .. 1.000 R²) mean 151.8 ms (138.0 ms .. 159.9 ms) std dev 13.51 ms (5.826 ms .. 19.63 ms) variance introduced by outliers: 18% (moderately inflated) benchmarking UTxO (Read)/1 CP x 100000 UTxO time 1.428 s (1.369 s .. 1.555 s) 0.999 R² (0.998 R² .. 1.000 R²) mean 1.381 s (1.368 s .. 1.406 s) std dev 23.90 ms (130.2 μs .. 28.16 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking SeqState/1 CP x 10 addr time 3.379 ms (3.170 ms .. 3.572 ms) 0.976 R² (0.954 R² .. 0.991 R²) mean 3.311 ms (3.195 ms .. 3.430 ms) std dev 284.2 μs (212.5 μs .. 396.9 μs) variance introduced by outliers: 47% (moderately inflated) benchmarking SeqState/1 CP x 100 addr time 5.443 ms (5.138 ms .. 5.713 ms) 0.979 R² (0.956 R² .. 0.992 R²) mean 5.466 ms (5.308 ms .. 5.672 ms) std dev 427.8 μs (291.6 μs .. 592.0 μs) variance introduced by outliers: 39% (moderately inflated) benchmarking SeqState/1 CP x 1000 addr time 26.29 ms (25.03 ms .. 27.87 ms) 0.993 R² (0.986 R² .. 0.999 R²) mean 24.41 ms (23.76 ms .. 25.15 ms) std dev 1.436 ms (1.024 ms .. 2.145 ms) variance introduced by outliers: 18% (moderately inflated) benchmarking SeqState/1 CP x 10000 addr time 217.6 ms (143.9 ms .. 283.6 ms) 0.965 R² (0.948 R² .. 1.000 R²) mean 249.6 ms (231.4 ms .. 266.7 ms) std dev 22.71 ms (13.88 ms .. 31.51 ms) variance introduced by outliers: 18% (moderately inflated) benchmarking SeqState/1 CP x 100000 addr time 2.629 s (2.361 s .. 2.815 s) 0.999 R² (0.996 R² .. 1.000 R²) mean 2.487 s (2.314 s .. 2.552 s) std dev 120.2 ms (4.664 ms .. 149.2 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking SeqState/10 CP x 1000 addr time 49.44 ms (44.08 ms .. 54.40 ms) 0.975 R² (0.946 R² .. 0.997 R²) mean 51.44 ms (49.24 ms .. 54.66 ms) std dev 4.991 ms (2.424 ms .. 6.888 ms) variance introduced by outliers: 32% (moderately inflated) benchmarking SeqState/100 CP x 1000 addr time 295.9 ms (261.5 ms .. 343.9 ms) 0.992 R² (0.981 R² .. 1.000 R²) mean 287.5 ms (280.0 ms .. 295.1 ms) std dev 10.26 ms (5.842 ms .. 14.28 ms) variance introduced by outliers: 16% (moderately inflated) benchmarking RndState/1 CP x 10 addr time 2.544 ms (2.375 ms .. 2.710 ms) 0.963 R² (0.929 R² .. 0.983 R²) mean 2.614 ms (2.515 ms .. 2.778 ms) std dev 309.7 μs (192.6 μs .. 439.4 μs) variance introduced by outliers: 66% (severely inflated) benchmarking RndState/1 CP x 100 addr time 3.527 ms (3.271 ms .. 3.754 ms) 0.963 R² (0.931 R² .. 0.987 R²) mean 3.505 ms (3.340 ms .. 3.797 ms) std dev 548.9 μs (308.4 μs .. 1.013 ms) variance introduced by outliers: 77% (severely inflated) benchmarking RndState/1 CP x 1000 addr time 8.955 ms (8.512 ms .. 9.397 ms) 0.987 R² (0.977 R² .. 0.994 R²) mean 8.904 ms (8.689 ms .. 9.127 ms) std dev 502.1 μs (427.8 μs .. 606.0 μs) variance introduced by outliers: 25% (moderately inflated) benchmarking RndState/1 CP x 10000 addr time 59.16 ms (54.70 ms .. 61.69 ms) 0.988 R² (0.963 R² .. 0.997 R²) mean 60.60 ms (58.74 ms .. 63.37 ms) std dev 3.753 ms (2.618 ms .. 4.804 ms) variance introduced by outliers: 18% (moderately inflated) benchmarking RndState/1 CP x 100000 addr time 570.5 ms (549.2 ms .. 602.6 ms) 1.000 R² (0.999 R² .. 1.000 R²) mean 585.9 ms (576.4 ms .. 598.2 ms) std dev 13.13 ms (3.653 ms .. 17.76 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking RndState/10 CP x 1000 addr time 561.2 ms (494.1 ms .. 708.5 ms) 0.992 R² (0.984 R² .. 1.000 R²) mean 513.5 ms (500.1 ms .. 539.3 ms) std dev 25.48 ms (685.7 μs .. 30.17 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking RndState/100 CP x 1000 addr time 52.21 s (48.98 s .. 54.44 s) 1.000 R² (0.999 R² .. 1.000 R²) mean 51.66 s (51.02 s .. 52.21 s) std dev 797.5 ms (352.6 ms .. 1.031 s) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/1 w/ 1i + 1o [1..10] time 2.018 ms (1.821 ms .. 2.215 ms) 0.926 R² (0.865 R² .. 0.967 R²) mean 2.526 ms (2.375 ms .. 2.704 ms) std dev 415.1 μs (339.3 μs .. 510.2 μs) variance introduced by outliers: 77% (severely inflated) benchmarking TxHistory (Write)/10 w/ 1i + 1o [1..10] time 2.959 ms (2.738 ms .. 3.186 ms) 0.961 R² (0.932 R² .. 0.982 R²) mean 3.299 ms (3.147 ms .. 3.471 ms) std dev 400.0 μs (301.7 μs .. 564.0 μs) variance introduced by outliers: 63% (severely inflated) benchmarking TxHistory (Write)/10 w/ 10i + 10o [1..10] time 7.899 ms (7.527 ms .. 8.293 ms) 0.989 R² (0.981 R² .. 0.995 R²) mean 7.345 ms (7.138 ms .. 7.553 ms) std dev 511.7 μs (425.2 μs .. 624.8 μs) variance introduced by outliers: 31% (moderately inflated) benchmarking TxHistory (Write)/10 w/ 50i + 100o [1..10] time 53.62 ms (47.93 ms .. 57.53 ms) 0.978 R² (0.931 R² .. 0.997 R²) mean 54.73 ms (51.41 ms .. 59.62 ms) std dev 7.023 ms (3.806 ms .. 10.75 ms) variance introduced by outliers: 44% (moderately inflated) benchmarking TxHistory (Write)/10 w/ 255i + 255o [1..100] time 166.2 ms (145.8 ms .. 180.3 ms) 0.992 R² (0.983 R² .. 1.000 R²) mean 151.8 ms (145.0 ms .. 157.7 ms) std dev 9.344 ms (6.262 ms .. 13.87 ms) variance introduced by outliers: 13% (moderately inflated) benchmarking TxHistory (Write)/100 w/ 10i + 10o [1..100] time 65.11 ms (56.94 ms .. 76.31 ms) 0.969 R² (0.915 R² .. 0.998 R²) mean 65.39 ms (62.09 ms .. 69.57 ms) std dev 6.323 ms (4.347 ms .. 8.712 ms) variance introduced by outliers: 28% (moderately inflated) benchmarking TxHistory (Write)/100 w/ 50i + 100o [1..100] time 546.7 ms (387.3 ms .. 704.4 ms) 0.989 R² (0.960 R² .. 1.000 R²) mean 526.1 ms (507.0 ms .. 558.2 ms) std dev 30.39 ms (7.684 ms .. 38.90 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/100 w/ 255i + 255o [1..100] time 1.731 s (1.481 s .. 1.897 s) 0.998 R² (0.992 R² .. 1.000 R²) mean 1.621 s (1.522 s .. 1.677 s) std dev 96.13 ms (29.17 ms .. 130.9 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/1000 w/ 10i + 10o [1..1000] time 554.0 ms (512.9 ms .. 583.7 ms) 0.999 R² (0.998 R² .. 1.000 R²) mean 559.0 ms (552.8 ms .. 566.8 ms) std dev 8.040 ms (3.100 ms .. 11.02 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/1000 w/ 50i + 100o [1..1000] time 4.592 s (4.014 s .. 4.937 s) 0.998 R² (0.995 R² .. 1.000 R²) mean 4.842 s (4.663 s .. 5.131 s) std dev 283.6 ms (3.899 ms .. 360.3 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/1000 w/ 255i + 255o [1..1000] time 14.42 s (13.98 s .. 15.46 s) 0.999 R² (0.999 R² .. 1.000 R²) mean 14.10 s (14.00 s .. 14.28 s) std dev 177.2 ms (5.722 ms .. 212.1 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Write)/10000 w/ 10i + 10o [1..10000] time 5.793 s (5.746 s .. 5.837 s) 1.000 R² (1.000 R² .. 1.000 R²) mean 5.814 s (5.800 s .. 5.838 s) std dev 22.72 ms (3.755 ms .. 29.17 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..100] DESC - * time 3.111 s (2.894 s .. 3.342 s) 0.999 R² (NaN R² .. 1.000 R²) mean 3.351 s (3.230 s .. 3.447 s) std dev 121.3 ms (59.69 ms .. 160.2 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..100] ASC - * time 3.182 s (2.844 s .. 3.351 s) 0.999 R² (0.997 R² .. 1.000 R²) mean 3.371 s (3.275 s .. 3.427 s) std dev 93.88 ms (27.85 ms .. 127.6 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..1000] DESC - * time 3.518 s (3.237 s .. 3.736 s) 0.999 R² (0.999 R² .. 1.000 R²) mean 3.458 s (3.389 s .. 3.493 s) std dev 66.33 ms (32.05 ms .. 80.19 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..100] DESC pending * time 955.7 ms (712.4 ms .. 1.289 s) 0.982 R² (0.976 R² .. 1.000 R²) mean 829.7 ms (779.1 ms .. 897.1 ms) std dev 68.89 ms (23.29 ms .. 93.21 ms) variance introduced by outliers: 21% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..100] DESC - 40..60 time 135.4 ms (94.69 ms .. 166.4 ms) 0.986 R² (0.980 R² .. 1.000 R²) mean 154.6 ms (140.7 ms .. 160.1 ms) std dev 9.667 ms (1.847 ms .. 12.70 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/1000 [1..10000] DESC - 42..1337 time 3.182 s (2.610 s .. 4.009 s) 0.993 R² (0.978 R² .. 1.000 R²) mean 3.286 s (3.125 s .. 3.388 s) std dev 167.7 ms (87.87 ms .. 236.0 ms) variance introduced by outliers: 19% (moderately inflated) benchmarking TxHistory (Read)/10000 [1..100] DESC - 40..60 time 21.48 s (17.82 s .. 23.75 s) 0.997 R² (0.990 R² .. 1.000 R²) mean 19.03 s (17.94 s .. 20.13 s) std dev 1.290 s (1.083 s .. 1.455 s) variance introduced by outliers: 20% (moderately inflated) benchmarking TxHistory (Read)/10000 [1..10000] DESC - 42..1337 time 6.274 s (6.142 s .. 6.343 s) 1.000 R² (1.000 R² .. 1.000 R²) mean 6.246 s (6.205 s .. 6.262 s) std dev 29.77 ms (459.4 μs .. 36.66 ms) variance introduced by outliers: 19% (moderately inflated) -- Database disk space usage tests for UTxO File size /100 CP x 0 UTxO bench22158-7.db 4 KB bench22158-7.db-shm 32 KB bench22158-7.db-wal 249 KB bench22158-7.db (closed) 192 KB File size /1000 CP x 0 UTxO bench22158-8.db 4 KB bench22158-8.db-shm 32 KB bench22158-8.db-wal 555 KB bench22158-8.db (closed) 496 KB File size /10 CP x 10 UTxO bench22158-9.db 4 KB bench22158-9.db-shm 32 KB bench22158-9.db-wal 257 KB bench22158-9.db (closed) 192 KB File size /100 CP x 10 UTxO bench22158-10.db 4 KB bench22158-10.db-shm 32 KB bench22158-10.db-wal 639 KB bench22158-10.db (closed) 572 KB File size /1000 CP x 10 UTxO bench22158-11.db 4 MB bench22158-11.db-shm 32 KB bench22158-11.db-wal 4 MB bench22158-11.db (closed) 4 MB File size /10 CP x 100 UTxO bench22158-12.db 4 KB bench22158-12.db-shm 32 KB bench22158-12.db-wal 599 KB bench22158-12.db (closed) 532 KB File size /100 CP x 100 UTxO bench22158-13.db 3 MB bench22158-13.db-shm 32 KB bench22158-13.db-wal 4 MB bench22158-13.db (closed) 3 MB File size /1000 CP x 100 UTxO bench22158-14.db 37 MB bench22158-14.db-shm 96 KB bench22158-14.db-wal 38 MB bench22158-14.db (closed) 37 MB File size /10 CP x 1000 UTxO bench22158-15.db 3 MB bench22158-15.db-shm 32 KB bench22158-15.db-wal 3 MB bench22158-15.db (closed) 3 MB File size /100 CP x 1000 UTxO bench22158-16.db 37 MB bench22158-16.db-shm 96 KB bench22158-16.db-wal 37 MB bench22158-16.db (closed) 37 MB File size /1000 CP x 1000 UTxO bench22158-17.db 378 MB bench22158-17.db-shm 768 KB bench22158-17.db-wal 380 MB bench22158-17.db (closed) 378 MB Database disk space usage tests for TxHistory File size /100 w/ 10i + 20o bench22158-18.db 4 KB bench22158-18.db-shm 32 KB bench22158-18.db-wal 1 MB bench22158-18.db (closed) 1 MB File size /1000 w/ 10i + 20o bench22158-19.db 9 MB bench22158-19.db-shm 32 KB bench22158-19.db-wal 9 MB bench22158-19.db (closed) 9 MB File size /10000 w/ 10i + 20o bench22158-20.db 92 MB bench22158-20.db-shm 192 KB bench22158-20.db-wal 92 MB bench22158-20.db (closed) 92 MB File size /100000 w/ 10i + 20o bench22158-21.db 920 MB bench22158-21.db-shm 1 MB bench22158-21.db-wal 926 MB bench22158-21.db (closed) 920 MB File size /100 w/ 50i + 100o bench22158-22.db 4 MB bench22158-22.db-shm 32 KB bench22158-22.db-wal 4 MB bench22158-22.db (closed) 4 MB File size /1000 w/ 50i + 100o bench22158-23.db 47 MB bench22158-23.db-shm 96 KB bench22158-23.db-wal 47 MB bench22158-23.db (closed) 47 MB File size /10000 w/ 50i + 100o bench22158-24.db 471 MB bench22158-24.db-shm 960 KB bench22158-24.db-wal 474 MB bench22158-24.db (closed) 471 MB File size /100000 w/ 50i + 100o bench22158-25.db 4 GB bench22158-25.db-shm 9 MB bench22158-25.db-wal 4 GB bench22158-25.db (closed) 4 GB Benchmark db: FINISH Success! Waiting for next file change. ``` </details> - 3b6234f 📍 **generate Byron addresses for benchmark on RndState (instead of Jörmungandr delegation addresses...** - 33bcee0 📍 **enable logging in benchmarks, with minimum severity at Error.** Benchmark should somewhat reflect the production setup, which has a non-null logger enabled. In principle, logging should not impact the performances of the application, but this commit shows that it currently does, and a lot! - dff0168 📍 **benchmarks: filter logs before creating a LogObject** This has a dramatic impact on performances. Here are some comparisons of the database benchmarks with and without this fix, as well a comparison with the original benchmark behavior when using a `nullTracer`. benchmark | before | after | nullTracer --- | --- | --- | --- UTxO (Write)/1 CP x 100000 UTxO | 3.839s | 2.238s | 2.225s UTxO (Read)/1 CP x 100000 UTxO | 1.074s | 0.998s | 1.072s SeqState/1 CP x 100000 addr | 4.281s | 1.887s | 2.020s RndState/1 CP x 100000 addr x 100000 | 3.996s | 0.886s | 0.849s TxHistory (Write)/10000 w/ 10i + 10o | 8.032s | 5.793s | 5.587s TxHistory (Read)/10000 | 17.08s | 16.11s | 16.47s # Comments <!-- Additional comments or screenshots to attach if any --> There's an upcoming commit that fixes the severity filtering as per #2028 , I am however re-running the benchmark without the fix first to have a clear comparison of results before and after the fix. <!-- Don't forget to: ✓ Self-review your changes to make sure nothing unexpected slipped through ✓ Assign yourself to the PR ✓ Assign one or several reviewer(s) ✓ Once created, link this PR to its corresponding ticket ✓ Assign the PR to a corresponding milestone ✓ Acknowledge any changes required to the Wiki --> Co-authored-by: KtorZ <[email protected]> Co-authored-by: IOHK <[email protected]>
Build failed |
|
OOM unit tests failed, reported failure in the spreadsheet. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue Number
#1997
Overview
36bfac4
📍 extend benchmark with RndState and byron address storage
I also did some re-structuring of the benchmark fixtures, so that the
creation of benchmark data happens outside of the actual benchmarks.
Also, I reviewed some of the work bench to use numbers that reflects a
bit more reality (we actually always store checkpoints one by one, so
the time it takes to store N checkpoints isn't quite interesting).
see full results
3b6234f
📍 generate Byron addresses for benchmark on RndState (instead of Jörmungandr delegation addresses...
33bcee0
📍 enable logging in benchmarks, with minimum severity at Error.
Benchmark should somewhat reflect the production setup, which has a
non-null logger enabled. In principle, logging should not impact the
performances of the application, but this commit shows that it
currently does, and a lot!
dff0168
📍 benchmarks: filter logs before creating a LogObject
This has a dramatic impact on performances. Here are some comparisons
of the database benchmarks with and without this fix, as well a
comparison with the original benchmark behavior when using a
nullTracer.Comments
There's an upcoming commit that fixes the severity filtering as per #2028 , I am however re-running the benchmark without the fix first to have a clear comparison of results before and after the fix.