Skip to content

Core protocol v3.0 - chunk memory layouts #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alimanfoo merged 2 commits into from
May 21, 2019
Merged

Core protocol v3.0 - chunk memory layouts #24

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0a7b70a
initial drafting on memory layouts section
alimanfoo May 8, 2019
34566d1
strike "Fortran"
alimanfoo May 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion docs/protocol/core/v3.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,48 @@ TODO define regular chunk grids, including how to form a key for each chunk in a
Memory layouts
--------------

TODO define "C" and "F" memory layouts
An array has a memory layout, which defines the way that the binary
values of the array elements are organised within each chunk to form a
contiguous sequence of bytes. This contiguous binary representation of
a chunk is then the input to the array's chunk encoding pipeline,
described in later sections. Typically, when reading data, an
implementation will load this binary representation into a contiguous
memory buffer to allow direct access to array elements without having
to copy data.

The core protocol defines two types of contiguous memory
layout. However, protocol extensions may define other memory
layouts. Note that there may be an interdependency between memory
layouts and data types, such that certain memory layouts may only be
applicable to arrays with certain data types.

C contiguous memory layout
~~~~~~~~~~~~~~~~~~~~~~~~~~

In this memory layout, the binary values of the array elements are
organised into a sequence such that the last dimension of the array is
the fastest changing dimension, also known as "row-major" order. This
layout is only applicable to arrays with fixed size data types.

For example, for a two-dimensional array with chunk shape (dx, dy),
the binary values for a given chunk are taken from chunk elements in
the order (0, 0), (0, 1), (0, 2), ..., (dx - 1, dy - 3), (dx - 1, dy -
2), (dx - 1, dy - 1).
Copy link
Copy Markdown
Member

@martindurant martindurant May 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe worth mentioning here, that the memory offset/location within the chunk is determined by this ordering and the size of the dtype. Will endianness be in the dtype section only (I know it's mentioned already). That could be be loosely considered memory layout, especially for sub-byte types.

Copy link
Copy Markdown
Member Author

@alimanfoo alimanfoo May 20, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Could you suggest an edit?


F contiguous memory layout
~~~~~~~~~~~~~~~~~~~~~~~~~~

In this memory layout, the binary values of the array elements are
organised into a sequence such that the first dimension of the array
is the fastest changing dimension, also known as "column-major"
order. This layout is only applicable to arrays with fixed size data
types.

For example, for a two-dimensional array with chunk shape (dx, dy),
the binary values for a given chunk are taken from chunk elements in
the order (0, 0), (1, 0), (2, 0), ..., (dx - 3, dy - 1), (dx - 2, dy -
1), (dx - 1, dy - 1).


Codec interface
---------------
Expand Down