Skip to content

BZip2: add opt-in tolerateTruncatedStream to decode a footerless/partial stream#1358

Merged
adamhathcock merged 1 commit into
adamhathcock:masterfrom
fiddyschmitt:tolerate-truncated-bzip2-stream
Jun 23, 2026
Merged

BZip2: add opt-in tolerateTruncatedStream to decode a footerless/partial stream#1358
adamhathcock merged 1 commit into
adamhathcock:masterfrom
fiddyschmitt:tolerate-truncated-bzip2-stream

Conversation

@fiddyschmitt

Copy link
Copy Markdown
Contributor

Hi Adam,

Over at clonezilla-util I use SharpCompress to make a seekable stream out of a bzip-2 stream (see Bzip2StreamSeekable)

That worked in 0.37 but not in later releases due to the way EOF is handled. This PR enables that function once more, in an opt-in fashion.

Thanks!
Fidel

Summary

Adds an opt-in tolerateTruncatedStream parameter (default false) to BZip2Stream.Create /
BZip2Stream.CreateAsync, threaded through to CBZip2InputStream. When enabled, the decoder accepts a
bzip2 stream that has no trailing footer — e.g. a truncated stream, or a sub-range of blocks
extracted for random access — instead of throwing.

Default behaviour is unchanged.

Motivation

Decoding a partial bzip2 stream isn't currently possible. After each block the decoder probes for the
next block/footer header, so the moment the input runs out it throws
ArchiveOperationException("BZip2 compressed file ends unexpectedly"). This makes it impossible to feed
the decoder a slice of blocks (a common technique for random access into large bzip2 files) or any stream
captured without its footer. This brings bzip2 in line with the more forgiving end-of-input handling
elsewhere in SharpCompress, for callers that explicitly opt in.

What it does

When tolerateTruncatedStream is true:

  1. End-of-input at a block boundary ends the stream instead of throwing. This is scoped to the point
    where a block/stream header is read (tracked by expectingBlockStart); EOF in the middle of a block,
    the block CRC, or the Huffman tables is still reported as an error, so genuinely corrupt /
    truncated-mid-block input still fails.
  2. The whole-stream combined CRC is not verified. A partial decode's running combined CRC can't match
    the value stored in the footer, so that check is skipped. Per-block CRCs are still enforced.

The flag defaults to false (no behaviour change), and the change is applied symmetrically to the sync
and async read paths. One internal caller (BZip2CompressionProvider) was updated for the new optional
parameter.

Tests

Added to BZip2StreamTests:

  • a footerless, header-only stream decodes to empty with the flag and throws without it;
  • a real block followed by end-of-input at the next block boundary decodes with the flag and throws
    without it;
  • a corrupted whole-stream combined CRC is tolerated with the flag and fatal without it;
  • a complete, well-formed stream still round-trips with the flag set (no regression).

All existing BZip2 tests continue to pass.

…ial stream

Add a `tolerateTruncatedStream` option (default false) to `BZip2Stream.Create`/`CreateAsync`,
threaded through to `CBZip2InputStream`. When enabled, the decoder accepts a stream that has no
trailing footer - e.g. a truncated stream, or a sub-range of blocks extracted for random access:

- An end-of-input reached while reading a block/footer header (a true block boundary, tracked by
  `expectingBlockStart`) is treated as a normal end of stream instead of throwing
  `ArchiveOperationException("BZip2 compressed file ends unexpectedly")`. EOF in the middle of a
  block, the block CRC, or the Huffman tables still throws.
- The whole-stream combined CRC in the footer is not verified, since a partial decode's running
  combined CRC won't match the stored whole-stream value. Per-block CRCs are still enforced.

Default behaviour is unchanged (the flag defaults to false) and the change is applied symmetrically
to the sync and async read paths.

Tests (BZip2StreamTests):
- a footerless header-only stream decodes to empty with the flag and throws without it;
- a real block followed by end-of-input at the next block boundary decodes with the flag and throws
  without it;
- a corrupted whole-stream combined CRC is tolerated with the flag and fatal without it;
- a complete, well-formed stream still round-trips with the flag set.
All existing BZip2 tests continue to pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@adamhathcock

Copy link
Copy Markdown
Owner

sounds good. I might add a config object. Thanks!

@adamhathcock
adamhathcock merged commit 0e20a10 into adamhathcock:master Jun 23, 2026
3 checks passed
@fiddyschmitt
fiddyschmitt deleted the tolerate-truncated-bzip2-stream branch July 8, 2026 12:11
fiddyschmitt added a commit to fiddyschmitt/clonezilla-util that referenced this pull request Jul 10, 2026
… the xz perf regression)

The 06-23 local package (0.49.1-localbzip2patch) was accidentally built from the fork's default
branch = upstream MASTER (0.49.1 + ~3.5 weeks of dev commits), one of which regressed XZ decode
~1.7x in sparse/high-compression regions. Dense-region decode is unaffected, which is what let a
head-only benchmark wrongly exonerate the package; the regression cost ~60-70 min on the 10h suite's
Xz drive-image listing (a ~2 TB mostly-null decode).

Null-region micro-bench (8 GB-of-zeros xz, MB/s): SharpCompress 0.37 = 275; official 0.49.1 = 283;
old master-based pack = 159; NEW tag-based pack = 283. The new 0.49.1-localbzip2patch2 is the
official 0.49.1 source plus only the (upstream-merged) bzip2 tolerateTruncatedStream patch -
adamhathcock/sharpcompress#1358, merged 2026-06-23, not yet in any release.

Full-suite validation: Xz listing 248-265 -> 192.8 min, matching the pre-migration baseline binary
(194) on the same machine; 74/74 green incl. all bzip2 tests.

Once upstream ships a release containing PR #1358: switch both PackageReferences to the official
version and delete nuget.config + the local-nuget feed.

PERFORMANCE_PLAN.md documents the full regression investigation (Experiments 1-4 + resolution).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EzNgdgRVgqwy2gcCFfjPY8
This was referenced Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants