Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/core/terminology-hierarchy.excalidraw.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/core/terminology-read.excalidraw.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
86 changes: 54 additions & 32 deletions docs/core/v3.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,13 @@ Concepts and terminology
This section introduces and defines some key terms and explains the
conceptual model underpinning the Zarr format.

The following figure illustrates the first part of the terminology:

..
The following image was produced with https://excalidraw.com/
and can be loaded there, as the source is embedded in the png.
.. image:: terminology-hierarchy.excalidraw.png

.. _hierarchy:

*Hierarchy*
Expand Down Expand Up @@ -360,6 +367,44 @@ conceptual model underpinning the Zarr format.
ordering of elements_, but extensions may define other
memory layouts.

.. _metadata document:
.. _metadata documents:

*Metadata document*

Each array_ in a hierarchy_ is represented by a metadata document,
which is a machine-readable document containing essential
processing information about the node. For example, an array_
metadata document will specify the number of dimensions_, shape_,
`data type`_, grid_, `memory layout`_ and compressor_ for that
array_.

Groups can have a optional metadata document which provide extra
information about a group.

.. _store:
.. _stores:

*Store*

The `metadata documents`_ and encoded chunk_ data for all nodes in a
hierarchy_ are held in a store as a sequence of bytes. To enable a variety
of different store types to be used, this specification defines an
`Abstract store interface`_ which is a common set of operations that stores
may provide. For example, a directory in a file system can be a zarr store,
where keys are file names, values are file contents, and files can be read,
written, listed or deleted via the operating system. Equally, an S3 bucket
can provide this interface, where keys are resource names, values are
resource contents, and resources can be read, written or deleted via HTTP.

The following figure illustrates the codec, store and storage transformer
terminology on the example of reading from an array:

..
The following image was produced with https://excalidraw.com/
and can be loaded there, as the source is embedded in the png.
.. image:: terminology-read.excalidraw.png

.. _compressor:
.. _compressors:

Expand All @@ -384,43 +429,20 @@ conceptual model underpinning the Zarr format.
process, for example, to add support for one or more filter codecs
to be applied prior to compression.

.. _metadata document:
.. _metadata documents:

*Metadata document*

Each array_ in a hierarchy_ is represented by a metadata document,
which is a machine-readable document containing essential
processing information about the node. For example, an array_
metadata document will specify the number of dimensions_, shape_,
`data type`_, grid_, `memory layout`_ and compressor_ for that
array_.

Groups can have a optional metadata document which provide extra
information about a group.

.. _store:
.. _stores:

*Store*

The `metadata documents`_ and encoded chunk_ data for all nodes in
a hierarchy_ are held in a store. To enable a variety of different
store types to be used, this specification defines an `Abstract store
interface`_ which is a common set of operations that stores may
provide.

.. _storage transformer:
.. _storage transformers:

*Storage transformer*

To enhance the storage capabilities, storage transformers may
change the storage structure and behaviour of data coming from
an array_ in the underlying store_. Upon retrieval the original data
is restored within the transformer. Any number of `predefined storage
transformers`_ can be registered and stacked.
See the `storage transformers details`_ below.
To enhance the storage capabilities, storage transformers may change the
storage structure and behaviour of data coming from an array_ in the
underlying store_, being able to manipulate the stored bytes and the key
they are stored at. Upon retrieval the original data is restored within the
transformer. Any number of `predefined storage transformers`_ can be
registered and stacked. They allow to have common behavior that is not store
specific, but in the contrast to codecs they rather affect the storage
format of a complete array, rather than single chunks. See the
`storage transformers details`_ below.

.. _`storage transformers details`: #storage-transformers-1

Expand Down