perf: Upgrade compress package #1850
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* zstd version bump
zstd is not set by default even when cgo is enabled.
Add a Builder type in skiplist package which can be used to insert sorted keys efficiently. Add a test and benchmark for it.
This change makes the skiplist grow for the case of sorted skiplist builder. The normal skiplist still cannot grow. Note: The growing skiplist is not thread safe. Co-authored-by: Ahsan Barkati <[email protected]>
…ermodeinc#1696) In Dgraph, we already use Raft write-ahead log. Also, when we commit transactions, we update tens of thousands of keys in one go. To optimize this write path, this PR introduces a way to directly hand over Skiplist to Badger, short circuiting Badger's Value Log and WAL. This feature allows Dgraph to generate Skiplists while processing mutations and just hand them over to Badger during commits. It also accepts a callback which can be run when Skiplist is written to disk. This is useful for determining when to create a snapshot in Dgraph.
…isher (hypermodeinc#1697) When a skip-list is handed over to badger we should also send the entries in skiplist to the publisher so that all the subscribers get notified.
This PR adds DropPrefixNonBlocking and DropPrefixBlocking API that can be used to logically delete the data for specified prefixes. DropPrefix now makes decision based on badger option AllowStopTheWorld whose default is to use DropPrefixBlocking. With DropPrefixNonBlocking the data would not be cleared from the LSM tree immediately. It would be deleted eventually through compactions. Co-authored-by: Rohan Prasad <[email protected]>
Add benchmark tool for picktable benchmarking.
hypermodeinc#1700) This PR adds FullCopy option in Stream. This allows sending the table entirely to the writer. If this option is set to true we directly copy over the tables from the last 2 levels. This option increases the stream speed while also lowering the memory consumption on the DB that is streaming the KVs. For 71GB, compressed and encrypted DB we observed 3x improvement in speed. The DB contained ~65GB in the last 2 levels while remaining in the above levels. To use this option, the following options should be set in Stream. stream.KeyToList = nil stream.ChooseKey = nil stream.SinceTs = 0 db.managedTxns = true If we use stream writer for receiving the KVs, the encryption mode has to be the same in sender and receiver. This will restrict db.StreamDB() to use the same encryption mode in both input and output DB. Added TODO for allowing different encryption modes.
…odeinc#1706) Use https://github.com/klauspost/compress ZSTD compression when CGO is not enabled. Related to hypermodeinc#1383
…deinc#1705) immudb has its own store since version 0.9
Remove "GitHub issues" reference. (we use discuss now)
…c#1709) Remove Datadog's ZSTD that requires CGO Make Klauspost's ZSTD as default
…#1722) This PR adds support for stream writing incrementally to the DB. Adds an API: StreamWriter.PrepareIncremental Co-authored-by: Manish R Jain <[email protected]>
…modeinc#1723) While doing an incremental stream write, we should look at the first level on which there is no data. Earlier, due to a bug we were writing to a level that already has some tables.
I propose this simple fix for detecting conflicts in managed mode. Addresses https://discuss.dgraph.io/t/fatal-error-when-writing-conflicting-keys-in-managed-mode/14784. When a write conflict exists for a managed DB, an internal assert can fail. This occurs because a detected conflict is indicated with commitTs of 0, but handling the error is skipped for managed DB instances. Rather than conflate conflict detection with a timestamp of 0, it can be indicated with another return value from hasConflict.
…permodeinc#1721) With the introduction of SinceTs, a bug was introduced hypermodeinc#1653 that skips the pending entries. The default value of SinceTs is zero. And for the transaction made at readTs 0, the pending entries have version set to 0. So they were also getting skipped.
* According to CHANGELOG.md#removed-apis. `TableLoadingMode` option is removed. fixup the memory usage part of doc. * Remove `Options.ValueLogLoadingMode` part.
## Problem Linter was not running due to Go config issue. See issue here hypermodeinc#1810. ## Solution We refactor the setup for the linter. See relevant discussion here golangci/golangci-lint#1920. Go setup for linter is now on parity with the badger test CI workflow.
## Problem Errcheck linter is failing. ## Solution On [Dgraph](https://github.com/dgraph-io/dgraph/blob/main/.golangci.yml) and [Ristretto](https://github.com/dgraph-io/ristretto/blob/main/.golangci.yml) repositories, neither is running errcheck. We temporarily disable errcheck here to bring this repository to parity. We also do some cleanup in the Readme.
## Problem We are not reporting test coverage. ## Solution We show test coverage with coveralls.
## Problem Previously we were running a nightly Badger bank test (4 hours). ## Solution We bring back the nightly badger bank test.
## Problem Resolving more lint errors. ## Solution We fix some basic gosimple and gofmt lint errors. We raise the lll line length to 120 (as in Dgraph repo).
## Problem Lint tests were failing. ## Solution We comment out varcheck and gosec. We will resolve these later.
## Problem gas linter was running gosec, which we will resolve later. ## Solution We comment out gas linter.
## Problem CI jobs are not running when updates are made to release branch. ## Solution Run all CI jobs when PR's are opened against release branch.
This PR adds CD steps for Badger releases. Artifacts (badger binary and checksum) will be uploaded automatically to Github. Final step will be to add artifacts to release. This reflects the process we already have in place for Dgraph. Badger build flags were taken from the [Dgraph release script](https://github.com/dgraph-io/dgraph/blob/main/contrib/release.sh). We add a Makefile to streamline the build process. (cherry picked from commit 11c81e3) ## Remark PR is duplicate (cherry-pick) because we have two branches running in parallel (main and release/v3.2103).
Latest runner tag now uses ubuntu-22.04. We pin to ubuntu 20.04.
## Problem Currently we only deploy amd64 badger CLI tool builds. We would like arm64 builds too. ## Solution Use an arm64 self-hosted runner to build arm64 badger CLI tool.
…inc#1837) Currently [appveyor tests](https://ci.appveyor.com/project/manishrjain/badger/builds/42502297) are failing in multiple places on Windows. Original PR hypermodeinc#1775 <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/dgraph-io/badger/1775) <!-- Reviewable:end --> <!-- Change Github PR Title Your title must be in the following format: - `topic(Area): Feature` - `Topic` must be one of `build|ci|docs|feat|fix|perf|refactor|chore|test` Sample Titles: - `feat(Enterprise)`: Backups can now get credentials from IAM - `fix(Query)`: Skipping floats that cannot be Marshalled in JSON - `perf: [Breaking]` json encoding is now 35% faster if SIMD is present - `chore`: all chores/tests will be excluded from the CHANGELOG --> ## Problem <!-- Please add a description with these things: 1. Explain the problem by providing a good description. 2. If it fixes any GitHub issues, say "Fixes #GitHubIssue". 3. If it corresponds to a Jira issue, say "Fixes DGRAPH-###". 4. If this is a breaking change, please prefix `[Breaking]` in the title. In the description, please put a note with exactly who these changes are breaking for. --> ## Solution <!-- Please add a description with these things: 1. Explain the solution to make it easier to review the PR. 2. Make it easier for the reviewer by describing complex sections with comments. --> Co-authored-by: Alexey Ivanov <[email protected]> Co-authored-by: Joshua Goldstein <[email protected]>
## Problem We don't need an `apt upgrade`* step here. this is an overkill - plus it's causing some dependency issue on the environment layer. it's also causing a bunch of tests to fail in the installation step - https://github.com/dgraph-io/badger/actions .. we don't have a consistent baseline mostly because of the failing environment setup step. we should have a tight pinning on our environments. operations like `upgrade` etc. can induce failures suddenly based on upstream changes. issue - https://github.com/dgraph-io/badger/actions/runs/3909870547/jobs/6681427503 - https://github.com/dgraph-io/badger/actions?query=event%3Aschedule ## Solution rm upgrade*
klauspost
requested review from
akon-dey,
billprovince,
joshua-goldstein and
skrdgraph
as code owners
|
|
Run CI checks against release branch.
Some README changes have only gone into the master branch. We bring these changes into main. hypermodeinc#1705 hypermodeinc#1781 hypermodeinc#1782 hypermodeinc#1789
Upgrade for faster/better compression: klauspost/compress@v1.12.3...v1.15.15 Use 's2' in Snappy compatible mode for faster/better Snappy (de)compression: https://github.com/klauspost/compress/tree/master/s2#blocks Should improve compression speed, and `MaxEncodedLen` is considerably less. From `n = 32 + n + n/6` to `n + ((log2(n)+7)/7)` (max 10 bytes) This change is both forward and backwards compatible, meaning output will be decompressable by older versions as well. Benchmarks. Most are very noisy and dominated by allocs. ``` benchmark old ns/op new ns/op delta BenchmarkDbGrowth-32 2387593900 2366353800 -0.89% BenchmarkIteratePrefixSingleKey/Key_lookups-32 89209 87230 -2.22% BenchmarkReadWrite/0.1,0064-32 1827 1821 -0.33% BenchmarkReadWrite/0.2,0064-32 1830 1819 -0.60% BenchmarkReadWrite/0.5,0064-32 1873 1820 -2.83% BenchmarkReadWrite/1.0,0064-32 1881 1879 -0.11% BenchmarkReadWrite/0.1,0128-32 1892 1891 -0.05% BenchmarkReadWrite/0.2,0128-32 1920 1895 -1.30% BenchmarkReadWrite/0.5,0128-32 1942 1917 -1.29% BenchmarkReadWrite/1.0,0128-32 2001 1997 -0.20% BenchmarkReadWrite/0.1,0256-32 2039 2067 +1.37% BenchmarkReadWrite/0.2,0256-32 2065 2063 -0.10% BenchmarkReadWrite/0.5,0256-32 2106 2120 +0.66% BenchmarkReadWrite/1.0,0256-32 2211 2270 +2.67% BenchmarkReadWrite/0.1,0512-32 2292 2320 +1.22% BenchmarkReadWrite/0.2,0512-32 2329 2368 +1.67% BenchmarkReadWrite/0.5,0512-32 2450 2449 -0.04% BenchmarkReadWrite/1.0,0512-32 2589 2625 +1.39% BenchmarkReadWrite/0.1,1024-32 2837 2886 +1.73% BenchmarkReadWrite/0.2,1024-32 2960 3001 +1.39% BenchmarkReadWrite/0.5,1024-32 3162 3186 +0.76% BenchmarkReadWrite/1.0,1024-32 3513 3572 +1.68% BenchmarkReadWrite/0.1,2048-32 3857 3911 +1.40% BenchmarkReadWrite/0.2,2048-32 4036 4077 +1.02% BenchmarkReadWrite/0.5,2048-32 4484 4561 +1.72% BenchmarkReadWrite/1.0,2048-32 5193 5203 +0.19% BenchmarkReadWrite/0.1,4096-32 5814 5862 +0.83% BenchmarkReadWrite/0.2,4096-32 6139 6109 -0.49% BenchmarkReadWrite/0.5,4096-32 7136 6972 -2.30% BenchmarkReadWrite/1.0,4096-32 8391 8434 +0.51% BenchmarkReadWrite/0.1,8192-32 9412 9479 +0.71% BenchmarkReadWrite/0.2,8192-32 10088 10118 +0.30% BenchmarkReadWrite/0.5,8192-32 11580 11715 +1.17% BenchmarkReadWrite/1.0,8192-32 14446 14516 +0.48% BenchmarkReadWrite/0.1,16384-32 16798 16754 -0.26% BenchmarkReadWrite/0.2,16384-32 17645 17687 +0.24% BenchmarkReadWrite/0.5,16384-32 21139 21396 +1.22% BenchmarkReadWrite/1.0,16384-32 26156 27147 +3.79% BenchmarkReadWrite/frac_0-32 194 190 -1.81% BenchmarkReadWrite/frac_1-32 168 169 +0.24% BenchmarkReadWrite/frac_2-32 171 160 -6.71% BenchmarkReadWrite/frac_3-32 183 152 -17.18% BenchmarkReadWrite/frac_4-32 131 129 -1.75% BenchmarkReadWrite/frac_5-32 122 122 +0.08% BenchmarkReadWrite/frac_6-32 117 114 -2.40% BenchmarkReadWrite/frac_7-32 97.7 93.0 -4.76% BenchmarkReadWrite/frac_8-32 79.2 68.0 -14.15% BenchmarkReadWrite/frac_9-32 54.6 59.3 +8.57% BenchmarkReadWrite/frac_10-32 6.57 8.01 +21.89% BenchmarkReadWriteMap/frac_0-32 363 369 +1.65% BenchmarkReadWriteMap/frac_1-32 344 343 -0.32% BenchmarkReadWriteMap/frac_2-32 334 338 +1.32% BenchmarkReadWriteMap/frac_3-32 345 341 -1.19% BenchmarkReadWriteMap/frac_4-32 346 350 +1.36% BenchmarkReadWriteMap/frac_5-32 354 360 +1.78% BenchmarkReadWriteMap/frac_6-32 361 357 -1.11% BenchmarkReadWriteMap/frac_7-32 376 374 -0.43% BenchmarkReadWriteMap/frac_8-32 366 364 -0.46% BenchmarkReadWriteMap/frac_9-32 312 310 -0.80% BenchmarkReadWriteMap/frac_10-32 40.1 40.0 -0.32% BenchmarkWrite-32 209 199 -4.83% BenchmarkSortedWrites/builder-32 58.4 62.1 +6.41% BenchmarkSortedWrites/skiplist-32 354 348 -1.84% BenchmarkSortedWrites/buffer-32 81.5 81.5 +0.09% BenchmarkBuilder/no_compression-32 150415843 145178786 -3.48% BenchmarkBuilder/encryption-32 202647000 201539767 -0.55% BenchmarkBuilder/snappy_compression/level_1-32 177067050 171329850 -3.24% BenchmarkBuilder/zstd_compression/level_1-32 178846817 172781467 -3.39% BenchmarkBuilder/zstd_compression/level_3-32 178101100 171115667 -3.92% BenchmarkBuilder/zstd_compression/level_15-32 179374867 174848183 -2.52% BenchmarkRead-32 658641400 483333167 -26.62% BenchmarkReadAndBuild-32 1624537000 1457130300 -10.30% BenchmarkReadMerged-32 1102582000 894604200 -18.86% BenchmarkChecksum/CRC_1024-32 68.6 68.5 -0.12% BenchmarkChecksum/xxHash64_1024-32 69.5 69.8 +0.42% BenchmarkChecksum/SHA256_1024-32 2175 2181 +0.28% BenchmarkChecksum/CRC_2048-32 128 128 +0.00% BenchmarkChecksum/xxHash64_2048-32 131 132 +0.69% BenchmarkChecksum/SHA256_2048-32 4159 4164 +0.12% BenchmarkChecksum/CRC_4096-32 250 251 +0.48% BenchmarkChecksum/xxHash64_4096-32 265 262 -1.39% BenchmarkChecksum/SHA256_4096-32 8135 8112 -0.28% BenchmarkChecksum/CRC_8192-32 499 498 -0.28% BenchmarkChecksum/xxHash64_8192-32 517 514 -0.56% BenchmarkChecksum/SHA256_8192-32 16050 16076 +0.16% BenchmarkChecksum/CRC_16384-32 986 984 -0.21% BenchmarkChecksum/xxHash64_16384-32 1016 1013 -0.30% BenchmarkChecksum/SHA256_16384-32 31963 31920 -0.13% BenchmarkChecksum/CRC_32768-32 1949 1950 +0.05% BenchmarkChecksum/xxHash64_32768-32 2016 2024 +0.40% BenchmarkChecksum/SHA256_32768-32 63541 63775 +0.37% BenchmarkChecksum/CRC_65536-32 3901 3902 +0.03% BenchmarkChecksum/xxHash64_65536-32 4028 4015 -0.32% BenchmarkChecksum/SHA256_65536-32 126903 129392 +1.96% BenchmarkChecksum/CRC_131072-32 7760 7976 +2.78% BenchmarkChecksum/xxHash64_131072-32 8030 8035 +0.06% BenchmarkChecksum/SHA256_131072-32 254342 259402 +1.99% BenchmarkChecksum/CRC_262144-32 15666 15760 +0.60% BenchmarkChecksum/xxHash64_262144-32 16080 16047 -0.21% BenchmarkChecksum/SHA256_262144-32 507193 515186 +1.58% BenchmarkChecksum/CRC_1048576-32 61978 63675 +2.74% BenchmarkChecksum/xxHash64_1048576-32 64210 64711 +0.78% BenchmarkChecksum/SHA256_1048576-32 2029004 2085432 +2.78% BenchmarkRandomRead-32 25890 18761 -27.54% BenchmarkBuffer/bytes-buffer-32 590 592 +0.32% BenchmarkBuffer/page-buffer/page-size-1024-32 164 162 -1.40% benchmark old MB/s new MB/s speedup BenchmarkBuilder/no_compression-32 553.13 573.09 1.04x BenchmarkBuilder/encryption-32 410.57 412.82 1.01x BenchmarkBuilder/snappy_compression/level_1-32 469.88 485.61 1.03x BenchmarkBuilder/zstd_compression/level_1-32 465.20 481.53 1.04x BenchmarkBuilder/zstd_compression/level_3-32 467.15 486.22 1.04x BenchmarkBuilder/zstd_compression/level_15-32 463.83 475.84 1.03x ```
joshua-goldstein
force-pushed
the
update-compression
branch
from
55c088d to
9dca0bf
Compare
|
Hi @klauspost, thanks for the PR. If you are still interested in getting this in, please cherry pick the change on release/v4.0 branch, that is the branch we are working on right now. Thanks |
|
Replaced by #1880 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Solution
Upgrade for faster/better compression: klauspost/compress@v1.12.3...v1.15.15
Use 's2' in Snappy compatible mode for faster/better Snappy (de)compression:
https://github.com/klauspost/compress/tree/master/s2#blocks
Should improve compression speed, and
MaxEncodedLenis considerably less. Fromn = 32 + n + n/6ton + ((log2(n)+7)/7)(max 10 bytes)Use zstd header to determine decompressed size.
This change is both forward and backwards compatible, meaning output will be decompressable by older versions as well.
Benchmarks. Most are very noisy and dominated by allocs.