Use libarchive for all decompression (except brotli)

[yorickvP]

Libarchive, introduced in an earlier PR, can be used for arbitrary compression and decompression, as well as tar files. This handles support for (de)compression with gzip, bzip2, lzip, xz, lzma, zstd or compress. The main new feature here is zstd support.

The thing conspicuously missing here is brotli. This is because brotli does not have a magic header, so it cannot be automatically detected. We will still have to work around this, all other tests pass.

Fixes #2255.

Todo

• brotli handling
• more parallelism where possible
• bzip2-compressed logs
• drop dependency on bzip2, xz, zlib
• add option for compression levels

cc:

@tomberek @puckipedia

[tomberek]

Exciting change. How are you building and testing zstd? I presume this will come with some release-common.nix and configure.ac changes.

[yorickvP]

libarchive already links statically with zstd (and falls back to zstd command line binary when it's not). No further changes are required. It might be good to test zstd specifically, if we plan on using it in global binary caches.

[tomberek]

Yes, while testing zstd it seems to fallback and then not find the command line binary (I specifically uninstalled it to check this).

[domenkozar]

https://guix.gnu.org/blog/2019/substitutes-are-now-available-as-lzip/

[domenkozar]

@yorickvP what's missing here?

[domenkozar]

Talking to @yorickvP this needs brolti fallback to the old behavior since there's no magic byte header.

[yorickvP]

Okay, this code now passes all tests.

[domenkozar]

Once serokell#4 is merged, this PR will have tests and can be reviewed.

[domenkozar]

@edolstra I'd really like this merged, could you review this PR?

[edolstra]

Thanks, merged!

[domenkozar]

Wohooo 🥳

[dtzWill]

I'm looking at my PR for adding compression level option-- which is mentioned as possibly on the TODO, thank you -- and was wondering if you happen to have looked into that at all? I've yet to read through these changes or look into how libarchive works yet so not sure if there were issues there or just wasn't a priority / mess plumbing things or whatever :).

Thanks regardless, have a good one!

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/switch-cache-nixos-org-to-zstd-to-fix-slow-nixos-updates-nix-downloads/23961/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/switch-cache-nixos-org-to-zstd-to-fix-slow-nixos-updates-nix-downloads/23961/6

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/switch-cache-nixos-org-to-zstd-to-fix-slow-nixos-updates-nix-downloads/23961/48