Add compression level option #2171
Conversation
There was a problem hiding this comment.
@Beihao-Zhou Great!
Just one small issue, the rest is LGTM.
| @@ -190,6 +190,7 @@ Config::Config() { | |||
| {"rocksdb.compression", false, | |||
| new EnumField<rocksdb::CompressionType>(&rocks_db.compression, compression_types, | |||
| rocksdb::CompressionType::kNoCompression)}, | |||
| {"rocksdb.compression_level", true, new IntField(&rocks_db.compression_level, 32767, INT_MIN, INT_MAX)}, | |||
There was a problem hiding this comment.
🤔why compression_level can be dynamically changed?
There was a problem hiding this comment.
Seems like RocksDB just hardcode this value 32767 into their compression level logic.
exposed level option const: https://github.com/facebook/rocksdb/blob/096fb9b67d19a9a180e7c906b4a0cdb2b2d0c1f6/include/rocksdb/compression_type.h#L53
Zlib: https://github.com/facebook/rocksdb/blob/096fb9b67d19a9a180e7c906b4a0cdb2b2d0c1f6/util/compression.h#L805
Zstd: https://github.com/facebook/rocksdb/blob/096fb9b67d19a9a180e7c906b4a0cdb2b2d0c1f6/util/compression.h#L183
There was a problem hiding this comment.
aha, my bad. I mean why it's different from compression, which could be changed dynamically.
This looks ok to me
There was a problem hiding this comment.
Oh gotcha, nwnw, I did have the same question while I was working on it.
So when we change the compression type, it doesn't require re-encoding of the existing data because each block or SST file includes metadata specifying the compression algorithm used for that particular block or file. This allows RocksDB to understand how to decompress the data regardless of the global compression settings at any given time.
However, compression level, unlike compression types, which are recorded and recognized on a per-block or per-file basis, the specific level detail isn't stored with each block or file. Therefore, if we want to change it, it will require re-encoding everything, which is a huge overhead. I think that's the reason why RocksDB doesn't allow us to dynamically SetOption() for it.
Ref: https://github.com/facebook/rocksdb/wiki/Compression (The first line explains a bit about the granularity of encoding block)
There was a problem hiding this comment.
Rocksdb don't have this interface is fair enough, however, I don't think changing the level need re-encoding. And decompress block also doesn't need the level
There was a problem hiding this comment.
Besides, maybe refer to facebook/rocksdb#6615 helps.
There was a problem hiding this comment.
I see I see, thank you very much for correcting me and finding the reference!!
|
close: #2143