Add Zstandard

[mkitti]

Add Zstandard dependency

		<dependency>
			<groupId>org.janelia</groupId>
			<artifactId>n5-zstandard</artifactId>
			<version>1.0.1</version>
		</dependency>

[bogovicj]

Turns out the unit tests are not quite thorough enough :-/

@mkitti could you please merge https://github.com/saalfeldlab/n5-zarr/commits/zstandard/ into your branch?

I'm also running into an issue when using zarr-python to read data written by n5-zarr

example

Write the data:

final String root = "...";
final N5Writer zarr = new N5Factory().openWriter( root );
final String dset = "simple-zst";
ArrayImg<UnsignedByteType, ByteArray> img = ArrayImgs.unsignedBytes(new byte[]{0,1,2,3,4,5,6,7,8,9,10,11}, 12);
N5Utils.save(img, zarr, dset, new int[]{12}, new ZstandardCompression());

Read the data:

import zarr
root = zarr.open('zstd-test.zarr')
arr = root['n5-test/simple-zst']
arr[:]

results in this error

RuntimeError                              Traceback (most recent call last)
Cell In[3], line 4
      2 root = zarr.open('zstd-test.zarr')
      3 arr = root['n5-test/simple-zst']
----> 4 arr[:]

File ~/.local/lib/python3.10/site-packages/zarr/core.py:844, in Array.__getitem__(self, selection)
    842     result = self.get_orthogonal_selection(pure_selection, fields=fields)
    843 else:
--> 844     result = self.get_basic_selection(pure_selection, fields=fields)
    845 return result

File ~/.local/lib/python3.10/site-packages/zarr/core.py:970, in Array.get_basic_selection(self, selection, out, fields)
    968     return self._get_basic_selection_zd(selection=selection, out=out, fields=fields)
    969 else:
--> 970     return self._get_basic_selection_nd(selection=selection, out=out, fields=fields)

File ~/.local/lib/python3.10/site-packages/zarr/core.py:1012, in Array._get_basic_selection_nd(self, selection, out, fields)
   1006 def _get_basic_selection_nd(self, selection, out=None, fields=None):
   1007     # implementation of basic selection for array with at least one dimension
   1008 
   1009     # setup indexer
   1010     indexer = BasicIndexer(selection, self)
-> 1012     return self._get_selection(indexer=indexer, out=out, fields=fields)

File ~/.local/lib/python3.10/site-packages/zarr/core.py:1388, in Array._get_selection(self, indexer, out, fields)
   1385 if math.prod(out_shape) > 0:
   1386     # allow storage to get multiple items at once
   1387     lchunk_coords, lchunk_selection, lout_selection = zip(*indexer)
-> 1388     self._chunk_getitems(
   1389         lchunk_coords,
   1390         lchunk_selection,
   1391         out,
   1392         lout_selection,
   1393         drop_axes=indexer.drop_axes,
   1394         fields=fields,
   1395     )
   1396 if out.shape:
   1397     return out

File ~/.local/lib/python3.10/site-packages/zarr/core.py:2228, in Array._chunk_getitems(self, lchunk_coords, lchunk_selection, out, lout_selection, drop_axes, fields)
   2226 for ckey, chunk_select, out_select in zip(ckeys, lchunk_selection, lout_selection):
   2227     if ckey in cdatas:
-> 2228         self._process_chunk(
   2229             out,
   2230             cdatas[ckey],
   2231             chunk_select,
   2232             drop_axes,
   2233             out_is_ndarray,
   2234             fields,
   2235             out_select,
   2236             partial_read_decode=partial_read_decode,
   2237         )
   2238     else:
   2239         # check exception type
   2240         if self._fill_value is not None:

File ~/.local/lib/python3.10/site-packages/zarr/core.py:2098, in Array._process_chunk(self, out, cdata, chunk_selection, drop_axes, out_is_ndarray, fields, out_selection, partial_read_decode)
   2096     if isinstance(cdata, PartialReadBuffer):
   2097         cdata = cdata.read_full()
-> 2098     self._compressor.decode(cdata, dest)
   2099 else:
   2100     if isinstance(cdata, UncompressedPartialReadBufferV3):

File numcodecs/zstd.pyx:219, in numcodecs.zstd.Zstd.decode()

File numcodecs/zstd.pyx:153, in numcodecs.zstd.decompress()

RuntimeError: Zstd decompression error: invalid input data

[mkitti]

Where does N5Factory().openWriter( root ) come from? I don't see that method in N5Utils?

[mkitti]

This seems to be a bug in zarr-developers/numcodecs. There they use the C function ZSTD_getDecompressedSize:

https://github.com/zarr-developers/numcodecs/blob/366318f3b82403fe56db5ae647f8747e7a4aaf38/numcodecs/zstd.pyx#L151C21-L153

According to the Zstandard manual, that routine is now deprecated. One issue with it is that it returns 0 if the result is empty, unknown, or if an error has occurred.

https://facebook.github.io/zstd/zstd_manual.html

The numcodecs bug is that they assume that a value of 0 means error. In this case, it actually means unknown. I know it means unknown since I used the function ZSTD_getFrameContentSize and that returns 0xffffffffffffffff or ZSTD_CONTENTSIZE_UNKNOWN.

[mkitti]

Here's what my current test class looks like:

package org.janelia.saalfeldlab.n5.zarr;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

import org.janelia.saalfeldlab.n5.N5Writer;
import org.janelia.saalfeldlab.n5.imglib2.N5Utils;
import org.janelia.scicomp.n5.zstandard.ZstandardCompression;
import org.junit.Test;

import com.github.luben.zstd.Zstd;

import net.imglib2.img.array.ArrayImg;
import net.imglib2.img.array.ArrayImgs;
import net.imglib2.img.basictypeaccess.array.ByteArray;
import net.imglib2.type.numeric.integer.UnsignedByteType;

public class ZstandardTest {

	@Test
	public void testZstandard() throws IOException {
		final String root = "/home/mkitti/eclipse-workspace/n5-zarr/test.zarr";
		final N5Writer zarr = new N5ZarrWriter(root);
		final String dset = "simple-zst";
		final byte[] bytes = new byte[1024*1024];
		for(int i=0; i < bytes.length; ++i) {
			bytes[i] = (byte)(i*5-128);
		}
		//bytes = new byte[]{0,1,2,3,4,5,6,7,8,9,10,11};
		ArrayImg<UnsignedByteType, ByteArray> img = ArrayImgs.unsignedBytes(bytes, bytes.length);
		ZstandardCompression compressor = new ZstandardCompression();
		compressor.setSetCloseFrameOnFlush(true);
		N5Utils.save(img, zarr, dset, new int[]{1024}, compressor);
		
		byte[] compressedBytes = Files.readAllBytes(Paths.get(root, dset, "0"));
		System.out.println(Zstd.getFrameContentSize(compressedBytes));
	}
}

[mkitti]

Basically the problem is that at the time the Zstandard frame header is written it does not seem to know the size of the input. Thus it marks it as unknown. numcodecs does not know what to do with an unknown size.

Rather than using the stream API we may need to a buffer API.

To address the issue rather specifically, we may need to use setPledgedSrcSize.
https://www.javadoc.io/doc/com.github.luben/zstd-jni/latest/com/github/luben/zstd/ZstdCompressCtx.html

[bogovicj]

thanks for investigating @mkitti

[mkitti]

This PR to n5-zstandard fixes the issue for me.

JaneliaSciComp/n5-zstandard#3

[bogovicj]

N5Factory().openWriter( root )

comes from n5-universe.. The tests I was running involved adding Zstandard compression to the list of options in the imagej export plugin in https://github.com/saalfeldlab/n5-ij

[mkitti]

@bogovicj I updated n5-zstandard to version 1.0.2