BZip2: add opt-in tolerateTruncatedStream to decode a footerless/partial stream#1358
Merged
adamhathcock merged 1 commit intoJun 23, 2026
Conversation
…ial stream
Add a `tolerateTruncatedStream` option (default false) to `BZip2Stream.Create`/`CreateAsync`,
threaded through to `CBZip2InputStream`. When enabled, the decoder accepts a stream that has no
trailing footer - e.g. a truncated stream, or a sub-range of blocks extracted for random access:
- An end-of-input reached while reading a block/footer header (a true block boundary, tracked by
`expectingBlockStart`) is treated as a normal end of stream instead of throwing
`ArchiveOperationException("BZip2 compressed file ends unexpectedly")`. EOF in the middle of a
block, the block CRC, or the Huffman tables still throws.
- The whole-stream combined CRC in the footer is not verified, since a partial decode's running
combined CRC won't match the stored whole-stream value. Per-block CRCs are still enforced.
Default behaviour is unchanged (the flag defaults to false) and the change is applied symmetrically
to the sync and async read paths.
Tests (BZip2StreamTests):
- a footerless header-only stream decodes to empty with the flag and throws without it;
- a real block followed by end-of-input at the next block boundary decodes with the flag and throws
without it;
- a corrupted whole-stream combined CRC is tolerated with the flag and fatal without it;
- a complete, well-formed stream still round-trips with the flag set.
All existing BZip2 tests continue to pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
|
sounds good. I might add a config object. Thanks! |
fiddyschmitt
added a commit
to fiddyschmitt/clonezilla-util
that referenced
this pull request
Jul 10, 2026
… the xz perf regression) The 06-23 local package (0.49.1-localbzip2patch) was accidentally built from the fork's default branch = upstream MASTER (0.49.1 + ~3.5 weeks of dev commits), one of which regressed XZ decode ~1.7x in sparse/high-compression regions. Dense-region decode is unaffected, which is what let a head-only benchmark wrongly exonerate the package; the regression cost ~60-70 min on the 10h suite's Xz drive-image listing (a ~2 TB mostly-null decode). Null-region micro-bench (8 GB-of-zeros xz, MB/s): SharpCompress 0.37 = 275; official 0.49.1 = 283; old master-based pack = 159; NEW tag-based pack = 283. The new 0.49.1-localbzip2patch2 is the official 0.49.1 source plus only the (upstream-merged) bzip2 tolerateTruncatedStream patch - adamhathcock/sharpcompress#1358, merged 2026-06-23, not yet in any release. Full-suite validation: Xz listing 248-265 -> 192.8 min, matching the pre-migration baseline binary (194) on the same machine; 74/74 green incl. all bzip2 tests. Once upstream ships a release containing PR #1358: switch both PackageReferences to the official version and delete nuget.config + the local-nuget feed. PERFORMANCE_PLAN.md documents the full regression investigation (Experiments 1-4 + resolution). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EzNgdgRVgqwy2gcCFfjPY8
This was referenced Jul 13, 2026
Merged
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Adam,
Over at clonezilla-util I use SharpCompress to make a seekable stream out of a bzip-2 stream (see Bzip2StreamSeekable)
That worked in 0.37 but not in later releases due to the way EOF is handled. This PR enables that function once more, in an opt-in fashion.
Thanks!
Fidel
Summary
Adds an opt-in
tolerateTruncatedStreamparameter (defaultfalse) toBZip2Stream.Create/BZip2Stream.CreateAsync, threaded through toCBZip2InputStream. When enabled, the decoder accepts abzip2 stream that has no trailing footer — e.g. a truncated stream, or a sub-range of blocks
extracted for random access — instead of throwing.
Default behaviour is unchanged.
Motivation
Decoding a partial bzip2 stream isn't currently possible. After each block the decoder probes for the
next block/footer header, so the moment the input runs out it throws
ArchiveOperationException("BZip2 compressed file ends unexpectedly"). This makes it impossible to feedthe decoder a slice of blocks (a common technique for random access into large bzip2 files) or any stream
captured without its footer. This brings bzip2 in line with the more forgiving end-of-input handling
elsewhere in SharpCompress, for callers that explicitly opt in.
What it does
When
tolerateTruncatedStreamistrue:where a block/stream header is read (tracked by
expectingBlockStart); EOF in the middle of a block,the block CRC, or the Huffman tables is still reported as an error, so genuinely corrupt /
truncated-mid-block input still fails.
the value stored in the footer, so that check is skipped. Per-block CRCs are still enforced.
The flag defaults to
false(no behaviour change), and the change is applied symmetrically to the syncand async read paths. One internal caller (
BZip2CompressionProvider) was updated for the new optionalparameter.
Tests
Added to
BZip2StreamTests:without it;
All existing BZip2 tests continue to pass.