Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-threaded compression for gzip and snap using gzp #348

Merged
merged 4 commits into from
Jan 31, 2023

Conversation

figsoda
Copy link
Member

@figsoda figsoda commented Jan 31, 2023

Some benchmark results (compression):

gzip:
about 10 times the speed for a 12M text file
about 13 times the speed for a lot of regular sized files (tar.gz)
about 10% faster for a lot of small files (tar.gz)

snappy:
about 3 times the speed for a 12M text file
about 5 times the speed for a lot of regular sized files (tar.sz)
about 20% slower for a lot of small files (tar.sz)

no difference in size of compressed files, +0.2M in binary size in release mode

Copy link
Member

@marcospb19 marcospb19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both seem to be good additions, this should probably go to the changelog.

@figsoda
Copy link
Member Author

figsoda commented Jan 31, 2023

forgot about that

Copy link
Member

@marcospb19 marcospb19 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I'm a little bit afraid of how much we can trust this library for a new release 😅 I think I'll politely ask the author.

EDIT: nvm, it seems like this has been used by sccache.

@marcospb19 marcospb19 merged commit bef3e9e into ouch-org:main Jan 31, 2023
@marcospb19 marcospb19 deleted the gzp branch January 31, 2023 02:32
@vrmiguel
Copy link
Member

amazing PR 🥺

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants