Skip to content

Latest commit

 

History

History
112 lines (72 loc) · 2.92 KB

nbdkit-xz-filter.pod

File metadata and controls

112 lines (72 loc) · 2.92 KB
Raw

NAME

nbdkit-xz-filter - nbdkit xz filter

SYNOPSIS

nbdkit --filter=xz file FILENAME.xz
nbdkit --filter=xz curl https://example.com/FILENAME.xz

DESCRIPTION

nbdkit-xz-filter is a filter for nbdkit(1) which uncompresses the underlying plugin on the fly. The filter only supports read-only connections.

Getting best random access performance from xz

xz(1) files are split into streams and blocks. Most xz files contain only one stream which contains one or more blocks. You can find out how many streams and blocks are in an xz file by doing:

$ xz --list winxp.img.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1       1  2,100.0 MiB  6,144.0 MiB  0.342  CRC64   winxp.img.xz
   ↑↑↑     ↑↑↑
 streams  blocks

xz files are seekable on block boundaries only. Seeking is done by seeking directly to the lower block boundary, then uncompressing data until the precise byte is reached. As uncompressing xz data is slow, to get best random access performance, you must prepare your xz files with many small blocks.

Use the --block-size option with a small-ish block size. For example this is the same image compressed with a 16 MB block size:

$ xz --best --block-size=16777216 winxp.img
            ─────────────────────
$ xz --list winxp.img.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1     384  2,120.1 MiB  6,144.0 MiB  0.345  CRC64   winxp.img.xz
          ↑↑↑
         blocks

This file can be accessed randomly. At most 16 MB will have to be uncompressed to seek to any byte.

As you would expect, xz cannot compress as efficiently when using a smaller block size. The space penalty in the above example is < 1% of the compressed file size.

PARAMETERS

xz-max-block=SIZE

The maximum block size that the filter will read. The filter will refuse to read xz files that contain any block larger than this size.

See the discussion above about creating xz files with small block sizes in order to reduce memory usage and increase performance.

This parameter is optional. If not specified it defaults to 512M.

xz-max-depth=N

Maximum number of blocks stored in the LRU block cache.

This parameter is optional. If not specified it defaults to 8.

The filter may allocate up to maximum block size in file × maxdepth bytes of memory per connection.

FILES

$filterdir/nbdkit-xz-filter.so

The filter.

Use nbdkit --dump-config to find the location of $filterdir.

VERSION

nbdkit-xz-filter first appeared in nbdkit 1.10. It is derived from an earlier plugin that first appeared in nbdkit 1.0.

SEE ALSO

nbdkit(1), nbdkit-filter(3), nbdkit-curl-plugin(1), nbdkit-file-plugin(1), xz(1).

AUTHORS

Richard W.M. Jones

COPYRIGHT

Copyright Red Hat

POD ERRORS

Hey! The above document had some coding errors, which are explained below:

Around line 28:

Non-ASCII character seen before =encoding in '↑↑↑'. Assuming UTF-8