Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use at most 8 threads for the xz stream #98

Merged
merged 1 commit into from
Aug 28, 2019
Merged

Use at most 8 threads for the xz stream #98

merged 1 commit into from
Aug 28, 2019

Commits on Aug 28, 2019

  1. Use at most 8 threads for the xz stream 

    At preset 6, xz2 uses about 173MB of memory per thread. This adds up
quickly -- e.g. over 8GB of memory on a 48-CPU machine. If you happen to
try this in a 32-bit build, you'll get `LZMA_MEM_ERROR`.

We can limit this to a heuristic maximum number of threads to avoid
using so much memory, like xz's [`04_compress_easy_mt` example].

    // The number 8 is arbitrarily chosen and may be too low or
    // high depending on the compression preset and the computer
    // being used.

[`04_compress_easy_mt` example]: https://github.com/xz-mirror/xz/blob/de1f47b2b40e960b7bc3acba754f66dd19705921/doc/examples/04_compress_easy_mt.c#L71
    @cuviper
    cuviper committed Aug 28, 2019
    Configuration menu
    Copy the full SHA
    3d7273c View commit details
    Browse the repository at this point in the history