Making chained compression an extension

[jakirkham]

Sorry if I'm rehashing things. Asked some colleagues to take a look at the spec and provide feedback. This is basically just me transcribing their thoughts. Also if I missed things or made errors, would encourage them to hop in and correct me. 🙂

One of the things that they found somewhat concerning about the existing Zarr implementation was support of chained compression. Namely that one could do things like apply GZIP and then LZ4. It would be preferable to just allow one form of compression by default. Allowing general purpose chaining like this complicates implementation and likely adds little value. Maybe this is already the common case?

That said, they did note that other storage specs (like Parquet or ORC) may have a data packing step prior to compression (like RLE or dictionary coding). This seems to still make sense to support, but it would be preferable to have only one optional compressor follow an optional data packing step.

Thoughts?

[alimanfoo]

Thanks @jakirkham for posting this, very useful.

The original motivation for filters came from HDF5. In addition to compression filters, there are also several filters that are intended to be used prior to compression, e.g., scale-offset filter, shuffle (or bitshuffle) filter, and checksum (fletcher32) filter.

When implementing Zarr Python, given that all filters could be wrapped with a simple common interface, it was straightforward to support a sequence of filters for each array, where there could be any number of filters in any order. However, N5 does not support this, only a single compressor, and I imagine it might be non-trivial to implement in other settings.

Personally I would be happy to pare back the Zarr core protocol to only support a single compressor codec, then define a protocol extension for adding a sequence of filters to be applied prior to compression, if that would make implementation of the core protocol simpler.

[jakirkham]

Yep, this makes sense to me.

Having access to some of the pre-compression filters definitely makes sense. Though maybe these can be separated out as something different from a compressor.

Good point. It would be good to get thoughts from @axtimwalde and @constantinpape on this.

At least from our perspective, it would be fine to have these in the core spec, but it is worth delineating that these are something different from compressors and that only one compressor can be used for a dataset.

[alimanfoo]

Thanks John. Out of interest, do you know the specific technical reason why allowing two compressors to be chained would be difficult? I can see why it probably doesn't make much sense from a user perspective, because chaining two compressors probably won't add much value. But that's usage advice. From a technical standpoint, if all codecs expose the same API, why should the spec state any restrictions on which codecs are used in what order? I.e., is the technical difficulty to do with supporting more than one codec (regardless of what they are) or is it specifically a difficulty with chaining two compression codecs?

…

On Mon, 10 Jun 2019, 16:44 jakirkham, ***@***.***> wrote: Yep, this makes sense to me. Having access to some of the pre-compression filters definitely makes sense. Though maybe these can be separated out as something different from a compressor. Good point. It would be good to get thoughts from @axtimwalde <https://github.com/axtimwalde> and @constantinpape <https://github.com/constantinpape> on this. At least from our perspective, it would be fine to have these in the core spec, but it is worth delineating that these are something different from compressors and that only one compressor can be used for a dataset. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#38?email_source=notifications&email_token=AAFLYQQDV2VAXXY4YOAGLNDPZZZFZA5CNFSM4HVYXHW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXKIERY#issuecomment-500466247>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFLYQWB7G7RWFLO72J7WPDPZZZFZANCNFSM4HVYXHWQ> .

[jakirkham]

I think it is mainly just more work to implement and maintain if others want to follow the spec. Given there isn't a whole lot of value provided by doing this work, it seems easier to just rule it out as non-spec conforming on the outset.

[constantinpape]

To clarify, how does chained compression work in the current zarr spec?
Would this work by having additional compressors via filters?

[jakirkham]

Today that is how it works in Zarr, yes. Though here I'm proposing we don't do that going forward.

[constantinpape]

Today that is how it works in Zarr, yes. Though here I'm proposing we don't do that going forward.

I see. Happy to give my thoughts on this. Unfortunately, I didn't have time lately to follow the general spec discussion very closely, so I am not quite sure what you have discussed for the compression / filter mechanism so far.

Today that is how it works in Zarr, yes. Though here I'm proposing we don't do that going forward.

Could you clarify this a bit. Are you proposing to drop the filters mechanism completely and to only have compressors? Or are you proposing to keep filters, but to not allow compressors to be used as filters?

[jakirkham]

I'm proposing having an optional pre-compression step (e.g. run-length encoding) followed by an optional compression step (e.g. lz4). Thinking there should be at most one compression step, but could be uncompressed. Haven't thought much on how many pre-compression steps makes sense. Probably only one, but could be skipped as well.

[alimanfoo]

On Mon, 10 Jun 2019 at 23:31, jakirkham ***@***.***> wrote: I'm proposing having an optional pre-compression step (e.g. run-length encoding) followed by an optional compression step (e.g. lz4). Thinking there should be at most one compression step, but could be uncompressed. Haven't thought much on how many pre-compression steps makes sense. Probably only one, but could be skipped as well.

Just to flesh this out a bit, I think we are proposing that the v3.0 core protocol would only support a single compression codec, which could be null (no compression). We would then also define a protocol extension (via a separate spec) which would support the addition of pre-compression filters, which would be a sequence of zero or more codecs to be applied in sequence prior to the compression codec. Implementations would then have an easier job of implementing the core protocol, and could decide whether or not to also implement the filters extension. How does that sound? (I'll try and work this into the v3.0 dev branch soon so we can see how it would look in practice.)

[constantinpape]

How does that sound? (I'll try and work this into the v3.0 dev branch soon so we can see how it would look in practice.)

Yes, this sounds very reasonable to me.

[meggart]

I actually liked the idea of separating filters and compressor because they are IMO different things. As I understand it a compressor takes a stream of raw bytes as an input and just compresses this down without caring what is inside. In contrast a filter operates on the data, that means in order to do a RLE you have to know how to interpret the raw bytes as integers and to apply the filter.

I know that is is nowhere explicitly mentioned in the current specs but was rather my interpretation of the specs. It might, however be important when defining interfaces for compressors to decide if a compressor actually knows how to interpret the byte stream it should operate on.

So one suggestion would be to allow only a single compressor, because using multiple does not make sense and then allow defining filters as a spec extension with the difference that filters know about the data type and shape of the input while compressors generally don't.

[alimanfoo]

I actually liked the idea of separating filters and compressor because they are IMO different things. As I understand it a compressor takes a stream of raw bytes as an input and just compresses this down without caring what is inside. In contrast a filter operates on the data, that means in order to do a RLE you have to know how to interpret the raw bytes as integers and to apply the filter.

I know that is is nowhere explicitly mentioned in the current specs but was rather my interpretation of the specs. It might, however be important when defining interfaces for compressors to decide if a compressor actually knows how to interpret the byte stream it should operate on.

So one suggestion would be to allow only a single compressor, because using multiple does not make sense and then allow defining filters as a spec extension with the difference that filters know about the data type and shape of the input while compressors generally don't.

This is interesting and gets to the heart of a fairly tricky issue, which the current zarr python implementation & v2 spec fudges somewhat.

Just for comparison, the way the current zarr python implementation works is that all codecs must implement a simple interface which assumes (mostly, although see caveat below) that only raw bytes (buffers) are passed through when encoding or decoding. However, some codecs do need to know the data type, and so this must be explicitly provided as part of the codec configuration.

E.g., if you want to use the FixedScaleOffset codec, you have to explicitly provide the input data type, e.g.:

>>> codec = numcodecs.FixedScaleOffset(offset=1000, scale=10, dtype='f8', astype='u1')

I.e., it is up to the user to ensure that the codec is configured with the correct data type. If multiple codecs are being chained, then each has to be configured so that input and output data types match. There is no checking to ensure that has been done.

There is a small caveat to the above. It is true that almost all of the compression codecs in numcodecs take an arbitrary sequence of bytes as input, i.e., do not need to know anything about data type or shape. However, the Blosc codec does under some circumstances need to know the item size of the input data, because it uses this information to configure it's internal shuffle filter if activated. The way it can get this is because the codec API is actually built around the Python buffer interface, which does pass through some information about the buffer's data type. So although almost all of the codecs currently implemented in numcodecs ignore this and treat the input buffer as just a plain sequence of bytes, the Blosc codec does pull out the item size and use it.

So I guess the question is, what is the codec interface? Is it the same for compressors and filters, or not? And in either case, does it just pass through a sequence of bytes, or does it pass through a sequence of bytes plus some other information like data type, item size, etc.?

[jakirkham]

Generally your proposal above sounds fine to me Alistair. I'm not totally sure about chaining pre-compression filters, but there could be use cases here. So am happy to revisit later.

As a small note about compressors, endianness is likely something people will want to track as well.

[alimanfoo]

Just to say I've done some editing on the v3.0 core protocol in the development branch to implement the suggestion to limit the core protocol to just a compressor codec. More info in this comment.