Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create leaner encoder interface for fast levels. #70

Closed
wants to merge 3 commits into from

Conversation

klauspost
Copy link
Owner

@klauspost klauspost commented Oct 30, 2016

Separate fast and "standard" compression into a shared interface.

This mainly makes a difference when encoder is used without "Reset".
With PR applied:

file    reset   out level   files   insize  outsize millis  mb/s
sites   false   gzkp    -2  21920   192640560   127352920   2703    67.95
sites   false   gzkp    1   21920   192640560   66222720    3771    48.71
sites   false   gzkp    2   21920   192640560   65271280    4326    42.46
sites   false   gzkp    3   21920   192640560   64362680    4839    37.96
sites   false   gzkp    4   21920   192640560   63061440    4808    38.20
sites   false   gzkp    5   21920   192640560   59535120    8405    21.86
sites   false   gzkp    6   21920   192640560   58412040    8691    21.14
sites   false   gzkp    7   21920   192640560   57069720    9542    19.25
sites   false   gzkp    8   21920   192640560   56384720    10511   17.48
sites   false   gzkp    9   21920   192640560   56011840    13240   13.88

Without:

file    reset   out level   files   insize  outsize millis  mb/s
sites   false   gzkp    -2  21920   192640560   127352920   6235    29.46
sites   false   gzkp    1   21920   192640560   66222720    7167    25.63
sites   false   gzkp    2   21920   192640560   65271280    8067    22.77
sites   false   gzkp    3   21920   192640560   64362680    8670    21.19
sites   false   gzkp    4   21920   192640560   63061440    8659    21.22
sites   false   gzkp    5   21920   192640560   59535120    8441    21.76
sites   false   gzkp    6   21920   192640560   58412040    8705    21.10
sites   false   gzkp    7   21920   192640560   57069720    9652    19.03
sites   false   gzkp    8   21920   192640560   56384720    10480   17.53
sites   false   gzkp    9   21920   192640560   56011840    12964   14.17

This is ~5% faster on standard encoding and considerably more on fast encodes.

Also removes AMD64 specific encoding for now.
Separate fast and "standard" compression into a shared interface.

This mainly makes a difference when encoder is used without "Reset".
@klauspost
Copy link
Owner Author

klauspost commented Nov 6, 2016

=== RUN   TestFile10
panic: runtime error: slice bounds out of range

goroutine 186 [running]:
panic(0x545820, 0xc042004060)
        c:/go/src/runtime/panic.go:500 +0x1af
github.com/klauspost/compress/flate.(*compressor).fillWindow(0xc043c48000, 0xc044df88d6, 0x4000, 0x572a)
        d:/dev/gopath/src/github.com/klauspost/compress/flate/deflate.go:236 +0x2d0
github.com/klauspost/compress/flate.(*Writer).ResetDict(0xc0438ea030, 0x6074c0, 0xc042070070, 0xc044df88d6, 0x4000, 0x572a)
        d:/dev/gopath/src/github.com/klauspost/compress/flate/deflate.go:860 +0xd0
github.com/klauspost/pgzip.compressBlock(0xc0438fc000, 0x0, 0x4c4b4, 0xc044df88d6, 0x4000, 0x572a, 0x0, 0x0, 0x0, 0x0, ...)
        d:/dev/gopath/src/github.com/klauspost/pgzip/gzip.go:402 +0x1bb
created by github.com/klauspost/pgzip.(*Writer).compressCurrent
        d:/dev/gopath/src/github.com/klauspost/pgzip/gzip.go:243 +0x377
exit status 2
FAIL    github.com/klauspost/pgzip      0.084s

@klauspost
Copy link
Owner Author

Replaced by #107

@klauspost klauspost closed this Jun 1, 2019
@klauspost klauspost deleted the leaner-fast-encoder branch March 7, 2020 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant