From dc8b06a6be81f4f135b51297e7881e82e8d57273 Mon Sep 17 00:00:00 2001 From: Christian Haudum Date: Thu, 12 Dec 2024 09:51:20 +0100 Subject: [PATCH 1/2] chore: Update ASCII diagram of chunk binary encoding Signed-off-by: Christian Haudum --- pkg/chunkenc/README.md | 76 ++++++++++++++++++++++++++++-------------- 1 file changed, 51 insertions(+), 25 deletions(-) diff --git a/pkg/chunkenc/README.md b/pkg/chunkenc/README.md index f697bb107853c..aee781d67bd31 100644 --- a/pkg/chunkenc/README.md +++ b/pkg/chunkenc/README.md @@ -1,28 +1,54 @@ -# Chunk format +# Chunk v4 format ``` - | | | - | MagicNumber(4b) | version(1b) | - | | | - -------------------------------------------------- - | block-1 bytes | checksum (4b) | - -------------------------------------------------- - | block-2 bytes | checksum (4b) | - -------------------------------------------------- - | block-n bytes | checksum (4b) | - -------------------------------------------------- - | #blocks (uvarint) | - -------------------------------------------------- - | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) | - ------------------------------------------------------------------------------------------------ - | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) | - ------------------------------------------------------------------------------------------------ - | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) | - ------------------------------------------------------------------------------------------------ - | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) | - ------------------------------------------------------------------------------------------------ - | checksum(from #blocks) | - ------------------------------------------------------------------- - | metasOffset - offset to the point with #blocks | - -------------------------------------------------- +// Header ++-----------------------------------+ +| Magic Number (uint32, 4 bytes) | ++-----------------------------------+ +| Version (1 byte) | ++-----------------------------------+ +| Encoding (1 byte) | ++-----------------------------------+ + +// Blocks ++--------------------+----------------------------+ +| block 1 (n bytes) | checksum (uint32, 4 bytes) | ++--------------------+----------------------------+ +| block 1 (n bytes) | checksum (uint32, 4 bytes) | ++--------------------+----------------------------+ +| ... | ++--------------------+----------------------------+ +| block N (n bytes) | checksum (uint32, 4 bytes) | ++--------------------+----------------------------+ + +// Metas ++------------------------------------------------------------------------------------------------------------------------+ +| #blocks (uvarint) | ++--------------------+-----------------+-----------------+------------------+---------------+----------------------------+ +| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) | ++--------------------+-----------------+-----------------+------------------+---------------+----------------------------+ +| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) | ++--------------------+-----------------+-----------------+------------------+---------------+----------------------------+ +| ... | ++--------------------+-----------------+-----------------+------------------+---------------+----------------------------+ +| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) | ++--------------------+-----------------+-----------------+------------------+---------------+----------------------------+ +| checksum (uint32, 4 bytes) | ++------------------------------------------------------------------------------------------------------------------------+ + +// Structured Metadata ++---------------------------------+ +| #labels (uvarint) | ++---------------+-----------------+ +| len (uvarint) | value (n bytes) | ++---------------+-----------------+ +| checksum (uint32, 4 bytes) | ++---------------------------------+ + +// Footer ++-----------------------+--------------------------+ +| len (uint64, 8 bytes) | offset (uint64, 8 bytes) | // offset to Structured Metadata ++-----------------------+--------------------------+ +| len (uint64, 8 bytes) | offset (uint64, 8 bytes) | // offset to Metas ++-----------------------+--------------------------+ ``` From 92edc351dab64f3c059368e8aac79756bf71b5cb Mon Sep 17 00:00:00 2001 From: Christian Haudum Date: Sat, 14 Dec 2024 09:16:39 +0100 Subject: [PATCH 2/2] fixup! chore: Update ASCII diagram of chunk binary encoding Signed-off-by: Christian Haudum --- docs/sources/operations/storage/_index.md | 77 +++++++++++++++-------- pkg/chunkenc/README.md | 2 + 2 files changed, 54 insertions(+), 25 deletions(-) diff --git a/docs/sources/operations/storage/_index.md b/docs/sources/operations/storage/_index.md index 3b6bef94f9dce..f8918f87009a9 100644 --- a/docs/sources/operations/storage/_index.md +++ b/docs/sources/operations/storage/_index.md @@ -135,29 +135,56 @@ See the [IBM Cloud Object Storage section](https://grafana.com/docs/loki/