Reduce rar allocations

[adamhathcock]

This pull request introduces several improvements and optimizations to memory management and stream initialization in the RAR decompression logic. The main focus is on reducing allocations by leveraging ArrayPool<byte>.Shared for buffer reuse, ensuring proper disposal of resources, and refining the initialization logic for streams to prevent redundant work. Additionally, there are structural changes to the internal BLAKE2SP hash implementation for clarity and efficiency.

Memory Management and Buffer Pooling:

• Replaced direct buffer allocations with ArrayPool<byte>.Shared.Rent() in multiple places, including BitInput, FragmentedWindow, and unpacking logic, to reduce GC pressure and improve performance. Corresponding Return() calls were added to ensure buffers are released back to the pool. [1] [2] [3] [4]

• Updated EnsureCapacity utility in Unpack.cs to use pooled arrays and properly return old arrays to the pool, further optimizing memory usage.

Resource Disposal:

• Added Dispose implementations to classes such as BitInput and updated RarArchive.Dispose() to ensure all resources, including pooled buffers and unpackers, are properly cleaned up. [1] [2] [3]

Stream Initialization and Read Logic:

• Modified both sync and async RarStream initialization to be idempotent, preventing redundant unpacking work. Also, ensured that zero-length reads return immediately. [1] [2] [3] [4]

• Optimized async stream creation methods (CreateAsync) in RarCrcStream and RarBLAKE2spStream to return immediately wrapped ValueTask instead of awaiting initialization, simplifying usage and reducing overhead. [1] [2]

BLAKE2SP Hash Structure Refactoring:

• Refactored BLAKE2SP to use individual fields for each leaf (S0–S7) and a Root node instead of an array, with a helper method for access. Updated all usages accordingly for clarity and potential performance benefit. [1] [2] [3] [4] [5] [6]

Miscellaneous:

• Introduced filter pooling in unpacking logic to further reduce allocations during decompression.

These changes collectively improve the performance, reliability, and maintainability of the RAR decompression codebase by addressing memory efficiency and resource management.

[Copilot]

Pull request overview

This PR reduces allocations during RAR decompression by adopting ArrayPool<byte>.Shared for sliding-window/fragmented-window/filter-scratch/temporary buffers, pooling UnpackFilter objects, making BitInput/Unpack (V2017) disposable so pooled buffers are returned, and making RarStream initialization idempotent so reader stream factories (RarCrcStream/RarBLAKE2spStream) can defer initialization until the first nonzero read. It also refactors BLAKE2SP to use individual leaf fields plus a Root instead of an allocated array, pre-sizes some VM lists, hoists IsStandardFilter signatures into a static array, switches several async stream reads to Memory-based overloads on modern TFMs, and disposes the V2017 unpacker from RarArchive/RarReader.

Changes:

• Pool buffers and reset/dispose them: BitInput.InBuf, Window, FragmentedWindow.Mem, filter src/dst memory, UnstoreFileAsync temp buffer, RarStream.ReadAsync(Memory<byte>) fast path.
• Make RarStream.Initialize/InitializeAsync idempotent and lazy, return early on zero-count reads, and turn RarCrcStream/RarBLAKE2spStream async factories into synchronous ValueTask wrappers.
• Refactor BLAKE2sp leaves to S0..S7/Root with a GetLeaf(index) switch, pool UnpackFilter instances with RentFilter/ReturnFilter/InitFilters, dispose V2017 unpacker from RarArchive/RarReader, and downgrade Microsoft.NET.ILLink.Tasks lock entries.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/SharpCompress/Utility.Async.cs	Use Memory-based ReadAsync on netstandard2.1+/netcoreapp2.1+ in ReadExactAsync/ReadFullyAsync paths.
src/SharpCompress/Readers/Rar/RarReader.cs	Dispose lazily-created UnpackV2017 on reader disposal.
src/SharpCompress/packages.lock.json	Downgrades ILLink.Tasks lock entries for net8.0/net10.0.
src/SharpCompress/Compressors/Rar/VM/VMPreparedProgram.cs	Pre-size Commands/AltCommands/GlobalData lists.
src/SharpCompress/Compressors/Rar/VM/RarVM.cs	Hoist standard filter signatures into a static array.
src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack50_cpp.cs	Pool UnpackFilter via Rent/Return and reset filters in InitFilters.
src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack50_async.cs	Use filter pool in async unpack-5 path.
src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.unpack_cpp.cs	Add Dispose, return old Window to pool, stackalloc decode helper arrays.
src/SharpCompress/Compressors/Rar/UnpackV2017/Unpack.cs	Pool UnstoreFileAsync buffer; rework EnsureCapacity to use ArrayPool.
src/SharpCompress/Compressors/Rar/UnpackV2017/unpack_hpp.cs	Add FilterPool list.
src/SharpCompress/Compressors/Rar/UnpackV2017/FragmentedWindow.unpack50frag_cpp.cs	Make Reset public, pool fragmented window blocks.
src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_hpp.cs	Make BitInput IDisposable.
src/SharpCompress/Compressors/Rar/UnpackV2017/BitInput.getbits_cpp.cs	Rent InBuf from pool and return it in Dispose.
src/SharpCompress/Compressors/Rar/UnpackV1/Unpack.Async.cs	Pool UnstoreFileAsync buffer in V1 async path.
src/SharpCompress/Compressors/Rar/RarStream.cs	Idempotent Initialize; zero-count Read returns immediately and triggers lazy init.
src/SharpCompress/Compressors/Rar/RarStream.Async.cs	Idempotent InitializeAsync; zero-count short-circuit; array-backed ReadAsync(Memory<byte>) fast path.
src/SharpCompress/Compressors/Rar/RarCrcStream.cs	Drop eager Initialize() in Create.
src/SharpCompress/Compressors/Rar/RarCrcStream.Async.cs	CreateAsync returns wrapped ValueTask without awaiting init.
src/SharpCompress/Compressors/Rar/RarBLAKE2spStream.cs	Refactor BLAKE2SP to individual leaves + Root and GetLeaf; drop eager init in Create.
src/SharpCompress/Compressors/Rar/RarBLAKE2spStream.Async.cs	CreateAsync returns wrapped ValueTask without awaiting init.
src/SharpCompress/Archives/Rar/RarArchive.cs	Dispose lazily-created UnpackV2017 on archive disposal.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.