Add nested directory store

[alimanfoo]

This PR adds a new store class which stores chunk files in a nested directory structure rather than all chunk files for a given array in the same directory. Resolves #155.

[alimanfoo]

This is working, e.g., comparison between standard directory store and nested directory store:

In [23]: import zarr

In [24]: store = zarr.DirectoryStore('flat')

In [25]: root = zarr.group(store=store)

In [26]: foo = root.create_group('foo')

In [27]: z = foo.zeros('bar', shape=(10, 10), chunks=(5, 5))

In [28]: z[:] = 42

In [29]: !tree -a flat
flat
├── foo
│   ├── bar
│   │   ├── 0.0
│   │   ├── 0.1
│   │   ├── 1.0
│   │   ├── 1.1
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── .zattrs
│   └── .zgroup
├── .zattrs
└── .zgroup

2 directories, 10 files

In [30]: store = zarr.NestedDirectoryStore('nested')

In [31]: root = zarr.group(store=store)

In [32]: foo = root.create_group('foo')

In [33]: z = foo.zeros('bar', shape=(10, 10), chunks=(5, 5))

In [34]: z[:] = 42

In [35]: !tree -a nested
nested
├── foo
│   ├── bar
│   │   ├── 0
│   │   │   ├── 0
│   │   │   └── 1
│   │   ├── 1
│   │   │   ├── 0
│   │   │   └── 1
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── .zattrs
│   └── .zgroup
├── .zattrs
└── .zgroup

4 directories, 10 files

cc @axtimwalde

[axtimwalde]

Cool!

[jakirkham]

Will this become the new default?

[alimanfoo]

No, I was going to leave the default as DirectoryStore for compatibility. NestedDirectoryStore is intended as alternative if on a file system where too many files in a single dir causes problems, have not hit this problem myself but was reported for an analogous project.

On Wed, 1 Nov 2017 at 04:44, jakirkham ***@***.***> wrote: Will this become the new default? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/pull/177#issuecomment-340976964>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QgsM6QOf_flLA5vKFqihQt-P1_Asks5sx_dIgaJpZM4QNoIJ> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: [email protected] Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo