Reduce LZMA Allocations

[adamhathcock]

This pull request introduces significant memory optimizations across the LZMA/LZMA2 compression code by switching from direct array allocations to using ArrayPool for buffer management. Additionally, it updates resource cleanup logic to ensure pooled buffers are properly returned, and refactors the LZMA2 encoder stream for efficiency and clarity. The solution file is also updated to include a new test project.

Memory pooling and buffer management improvements:

• Replaced direct allocations of large arrays with pooled buffers using ArrayPool in LzBinTree, LzInWindow, and Lzma2EncoderStream, and ensured all buffers are returned to the pool during disposal to reduce GC pressure and improve performance. (src/SharpCompress/Compressors/LZMA/LZ/LzBinTree.cs, src/SharpCompress/Compressors/LZMA/LZ/LzInWindow.cs, src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs, [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]
• Updated async and sync chunk writing in Lzma2EncoderStream to use pooled buffers for headers and compressed data, with proper cleanup in finally blocks. (src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs, src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.csL344-R382)

Resource management and disposal:

• Implemented or improved Dispose methods in relevant classes to ensure all pooled resources are released and streams are properly closed. (src/SharpCompress/Compressors/LZMA/LZ/LzBinTree.cs, src/SharpCompress/Compressors/LZMA/LZ/LzInWindow.cs, src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs, [1] [2] [3] [4]

LZMA2 encoder refactoring:

• Refactored Lzma2EncoderStream to reuse a single Encoder instance, cache properties, and streamline chunk compression logic for improved efficiency and maintainability. (src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs, [1] [2] [3] [4]

Test infrastructure:

• Added the new SharpCompress.AotSmoke test project to the solution and updated configuration sections accordingly. (SharpCompress.sln, [1] [2] [3]

Async I/O improvements:

• Updated async header reading and writing to use pooled buffers with correct cleanup. (src/SharpCompress/Compressors/LZMA/LZipStream.Async.cs, [1] [2] src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs, [3]

These changes collectively improve memory efficiency, reduce allocations, and make the codebase more robust in high-throughput or resource-constrained environments.

[Copilot]

Pull request overview

This PR reduces GC pressure in LZMA/LZMA2 encode/decode paths by switching to ArrayPool for several hot buffers (match finder buffers, LZMA2 chunk buffer, async header buffers, single-byte range coder buffers), making the corresponding classes disposable, and refactoring Lzma2EncoderStream to reuse a single Encoder instance and cached properties. It also wires a new SharpCompress.AotSmoke test project into the solution and makes the Performance program async.

Changes:

• Use ArrayPool (and stackalloc/Stream.Null) in LzInWindow, LzBinTree, LzmaStream (async header), LZipStream.Async, RangeCoder (single-byte buffers), and Lzma2EncoderStream, with matching Dispose cleanup.
• Refactor Lzma2EncoderStream to reuse one Encoder, cache Properties and the LZMA props byte, and pass (Buffer, Length) tuples to avoid extra array copies; add Encoder.Dispose and an internal Span-based SetDecoderProperties.
• Add SharpCompress.AotSmoke project to SharpCompress.sln and convert Performance/Program.cs to async Task Main, switching the profiler sample to SevenZipBenchmarks.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
SharpCompress.sln	Adds AotSmoke project entries and config rows.
src/SharpCompress/packages.lock.json	Adjusts ILLink.Tasks versions (downgraded).
src/SharpCompress/Compressors/LZMA/LZ/LzBinTree.cs	Rents/returns _son/_hash via ArrayPool and overrides Dispose.
src/SharpCompress/Compressors/LZMA/LZ/LzInWindow.cs	Pools _bufferBase and implements IDisposable.
src/SharpCompress/Compressors/LZMA/LzmaEncoder.cs	Implements IDisposable to dispose the match finder.
src/SharpCompress/Compressors/LZMA/LzmaDecoder.cs	Adds ReadOnlySpan<byte> overload of SetDecoderProperties.
src/SharpCompress/Compressors/LZMA/LzmaStream.cs	Adds pooled async header buffer and disposes encoder/buffer.
src/SharpCompress/Compressors/LZMA/LzmaStream.Async.cs	Uses the pooled 6-byte header buffer for chunk header reads.
src/SharpCompress/Compressors/LZMA/LZipStream.Async.cs	Uses ArrayPool for the LZIP header with try/finally.
src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs	Reuses one Encoder, pools buffers, passes lengths explicitly, refactors writes.
src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.cs	Adds _singleByteBuffer field; removes dead comments.
src/SharpCompress/Compressors/LZMA/RangeCoder/RangeCoder.Async.cs	Reuses a single-byte buffer across async reads/writes.
tests/SharpCompress.Performance/Program.cs	Switches Main to async; samples SevenZipBenchmarks for profiling.

Comments suppressed due to low confidence (1)

src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs:203

• If FlushChunk or _output.WriteByte(0x00) throws, the rented _buffer and _encoder's internal pooled buffers are never released, leaking ArrayPool buffers. Consider wrapping the cleanup (_encoder.Dispose() and ArrayPool<byte>.Shared.Return(_buffer)) in a try/finally so the pooled resources are always returned even on failure during the final flush. The same issue applies to DisposeAsync below.

    protected override void Dispose(bool disposing)
    {
        if (disposing && !_isDisposed)
        {
            _isDisposed = true;

            // Flush remaining buffered data
            if (_bufferPosition > 0)
            {
                FlushChunk();
            }

            // Write LZMA2 end marker
            _output.WriteByte(0x00);
            _encoder.Dispose();
            ArrayPool<byte>.Shared.Return(_buffer);
        }
        base.Dispose(disposing);
    }

    private static readonly byte[] _endMarker = [0x00];

#if !LEGACY_DOTNET || NETSTANDARD2_1
    public override async ValueTask DisposeAsync()
#else
    public async ValueTask DisposeAsync()
#endif
    {
        if (!_isDisposed)
        {
            _isDisposed = true;

            if (_bufferPosition > 0)
            {
                await FlushChunkAsync().ConfigureAwait(false);
            }
#if !LEGACY_DOTNET || NETSTANDARD2_1

            await _output.WriteAsync(new Memory<byte>(_endMarker)).ConfigureAwait(false);
#else
            await _output.WriteAsync(_endMarker, 0, 1).ConfigureAwait(false);
#endif

            _encoder.Dispose();
            ArrayPool<byte>.Shared.Return(_buffer);
        }

#if !LEGACY_DOTNET || NETSTANDARD2_1
        await base.DisposeAsync().ConfigureAwait(false);
#endif
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/SharpCompress/Compressors/LZMA/Lzma2EncoderStream.cs:338

• This comment still refers to a “fresh LZMA encoder” per chunk, but the code now reuses a single encoder instance. Either re-instantiate the encoder per chunk or (preferably) adjust the comment to describe the actual reset behavior.

        // Each chunk is compressed independently with a fresh LZMA encoder,
        // so we must use 0xE0 (full reset: dictionary + state + properties) every time.
        // The decoder uses outWindow.Total for literal context and posState;
        // 0xE0 triggers outWindow.Reset() which zeros Total, matching the encoder's
        // assumption that position starts at 0 for each chunk.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.