Skip to content

Commit

Permalink
Update upperBound ratio when guessing the required decompression buff…
Browse files Browse the repository at this point in the history
…er size

We noticed that for one of our services we sometimes have a lot of allocations caused by the `ioutil.ReadAll` call in the `Decompress` method.

After investigation those are coming from buffers whose decompressed size is greater than 10x the input size. This is quite wasteful because it means that we allocate twice for those buffers.

Once in this branch: https://github.com/DataDog/zstd/blob/869dae002e5efb372a0b09cd7d99390ca2089cc1/zstd.go#L143

And another time here once we realise that the buffer we previously allocated wasn't big enough: https://github.com/DataDog/zstd/blob/869dae002e5efb372a0b09cd7d99390ca2089cc1/zstd.go#L154

This technically doesn't fully solve the problem but it should now happen less often. I can also make this upper bound configurable if this change sounds too scary
  • Loading branch information
sfluor committed Jun 26, 2024
1 parent 869dae0 commit e75a26a
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions zstd.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ func cCompressBound(srcSize int) int {
// decompressSizeHint tries to give a hint on how much of the output buffer size we should have
// based on zstd frame descriptors. To prevent DOS from maliciously-created payloads, limit the size
func decompressSizeHint(src []byte) int {
// 1 MB or 10x input size
upperBound := 10 * len(src)
// 1 MB or 50x input size
upperBound := 50 * len(src)
if upperBound < decompressSizeBufferLimit {
upperBound = decompressSizeBufferLimit
}
Expand Down

0 comments on commit e75a26a

Please sign in to comment.