Filters for object dtype

[alimanfoo]

Currently using object dtype can be unsafe, because without a filter to encode the objects (currently either Pickle or MsgPack) you can get segfaults.

One option would be to check for object dtype and no filters provided by user, and then insert either MsgPack or Pickle into the filters. Question is what to do if the user also provides some filters.

[jakirkham]

I guess raising an error could already be an improvement over a segfault.

[alimanfoo]

Yes, although what do you check for to raise the error? No filters provided by user?

[jakirkham]

I guess check they are using MsgPack or Pickle at least. Though IDK if there are other ways they could avoid the issue.

[alimanfoo]

Yep that would be the safe option. Difficulty is that in theory someone could implement their own codec to handle objects, e.g., if they wanted to explore better ways of encoding strings. I we raise an error if neither of the known object codecs are present, we prevent them from using their own codec. I don't know if anyone will do this but I wouldn't want to shut down the possibility.

…

On Tue, Nov 28, 2017 at 1:40 AM, jakirkham ***@***.***> wrote: I guess check they are using MsgPack or Pickle at least. Though IDK if there other ways they could avoid the issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/208#issuecomment-347385961>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QsAuR7XjXUVygmuMt86XwuFXYLv8ks5s62SBgaJpZM4QsjgK> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[jakirkham]

Maybe we could soften it to a warning? Guessing the segmentation fault occurs when storing data.

[shoyer]

I'd just like to stress that pickle should never be used without an explicit opt-in because could is a serious security vulnerability: unpickling can execute arbitrary code. Untrusted data is somewhat rare in scientific computing but I would still prefer not to build in these sorts of vulnerabilities.

Message pack is more palatable as a default, but it's not part of the zarr spec, so it would still be surprising to me to see data encoded that way.

I agree that segfaults are bad, but the preferred alternative is probably to raise an error in Python, not to assume any particular codec (all of which have tradeoffs).

[jakirkham]

Maybe we could get the best of both worlds. Namely create a special ZarrObjectWarning (or similar) that we issue if we detect object might by serialized. We can then use simplefilter to set default behavior of this warning to "error". Then users get a traceback by default if they don't add MsgPack or Pickle filters (as opposed to a seg fault). Though users are free to turn this back into a warning (not an error) if they want to try some custom filter. It just will be default behavior to error instead.

[mrocklin]

I'm in favor of msgpack by default. It's widely used and decently fast.

[mrocklin]

Or, if we're very conservative, would JSON be a sensible default?

[alimanfoo]

Found another way that arrays with object dtype are currently broken if no filters are provided:

In [1]: import zarr

In [2]: z = zarr.zeros(5, chunks=2, dtype=object)

In [3]: z[0] = 'foo'

In [5]: z[1] = 'bar'
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-98f6ee1706b9> in <module>()
----> 1 z[1] = 'bar'

~/src/github/alimanfoo/zarr/zarr/core.py in __setitem__(self, selection, value)
   1095 
   1096         fields, selection = pop_fields(selection)
-> 1097         self.set_basic_selection(selection, value, fields=fields)
   1098 
   1099     def set_basic_selection(self, selection, value, fields=None):

~/src/github/alimanfoo/zarr/zarr/core.py in set_basic_selection(self, selection, value, fields)
   1190             return self._set_basic_selection_zd(selection, value, fields=fields)
   1191         else:
-> 1192             return self._set_basic_selection_nd(selection, value, fields=fields)
   1193 
   1194     def set_orthogonal_selection(self, selection, value, fields=None):

~/src/github/alimanfoo/zarr/zarr/core.py in _set_basic_selection_nd(self, selection, value, fields)
   1481         indexer = BasicIndexer(selection, self)
   1482 
-> 1483         self._set_selection(indexer, value, fields=fields)
   1484 
   1485     def _set_selection(self, indexer, value, fields=None):

~/src/github/alimanfoo/zarr/zarr/core.py in _set_selection(self, indexer, value, fields)
   1523 
   1524             # put data
-> 1525             self._chunk_setitem(chunk_coords, chunk_selection, chunk_value, fields=fields)
   1526 
   1527     def _chunk_getitem(self, chunk_coords, chunk_selection, out, out_selection,

~/src/github/alimanfoo/zarr/zarr/core.py in _chunk_setitem(self, chunk_coords, chunk_selection, value, fields)
   1627         with lock:
   1628             self._chunk_setitem_nosync(chunk_coords, chunk_selection, value,
-> 1629                                        fields=fields)
   1630 
   1631     def _chunk_setitem_nosync(self, chunk_coords, chunk_selection, value, fields=None):

~/src/github/alimanfoo/zarr/zarr/core.py in _chunk_setitem_nosync(self, chunk_coords, chunk_selection, value, fields)
   1686 
   1687                 # decode chunk
-> 1688                 chunk = self._decode_chunk(cdata)
   1689                 if not chunk.flags.writeable:
   1690                     chunk = chunk.copy(order='K')

~/src/github/alimanfoo/zarr/zarr/core.py in _decode_chunk(self, cdata)
   1724             chunk = chunk.view(self._dtype)
   1725         else:
-> 1726             chunk = np.frombuffer(chunk, self._dtype)
   1727 
   1728         # reshape

ValueError: cannot create an OBJECT array from memory buffer

[alimanfoo]

More brokenness:

In [23]: z = zarr.zeros(5, chunks=2, dtype=object)

In [24]: z[0] = b'foo'

In [25]: z[:]
Out[25]: Segmentation fault (core dumped)

[alimanfoo]

Stumbled on some work in fastparquet, looks like a lot of good ideas for how to handle object encoding:

https://github.com/dask/fastparquet/blob/master/fastparquet/writer.py
https://github.com/dask/fastparquet/blob/master/fastparquet/speedups.pyx

[mrocklin]

@martindurant might be glad to hear that :)

…

On Thu, Nov 30, 2017 at 5:28 AM, Alistair Miles ***@***.***> wrote: Stumbled on some work in fastparquet, looks like a lot of good ideas for how to handle object encoding: https://github.com/dask/fastparquet/blob/master/fastparquet/writer.py https://github.com/dask/fastparquet/blob/master/fastparquet/speedups.pyx — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/issues/208#issuecomment-348147282>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszLyYiTWEzbeAe-7_pBJaZ7CQgQyqks5s7oNDgaJpZM4QsjgK> .