Skip to content

Core: Add KLL Datasketch as standard blob types to puffin file #8202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
simhadri-g wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Core: Add KLL Datasketch as standard blob types to puffin file #8202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
69df56f
Core: Add KLL Datasketch and Hive ColumnStatisticsObj as standard blo…
simhadri-g Aug 1, 2023
abf4509
Addressed review comments
simhadri-g Aug 2, 2023
e8f311f
Addressed review comments and moved the doc update from iceberg-doc t…
simhadri-g Aug 17, 2023
7f0c727
Update a typo in format/puffin-spec.md
simhadri-g Aug 17, 2023
a4fdf6c
Fix checkstyle error flagged by :iceberg-core:spotlessJavaCheck
simhadri-g Aug 17, 2023
c0bba5d
removed Hive ColumnStatsObject
deniskuzZ Feb 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@ private StandardBlobTypes() {}
*/
public static final String APACHE_DATASKETCHES_THETA_V1 = "apache-datasketches-theta-v1";

/**
* A serialized form of a KLL sketch, a very compact quantiles sketch, produced by the <a
* href="https://datasketches.apache.org/">Apache DataSketches</a> library
*/
public static final String APACHE_DATASKETCHES_KLL_SKETCH = "apache-datasketches-kll-v1";

/** A serialized deletion vector according to the Iceberg spec */
public static final String DV_V1 = "deletion-vector-v1";
}
9 changes: 9 additions & 0 deletions format/puffin-spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,15 @@ for Puffin v1.
[roaring-bitmap-portable-serialization]: https://github.com/RoaringBitmap/RoaringFormatSpec?tab=readme-ov-file#extension-for-64-bit-implementations
[roaring-bitmap-general-layout]: https://github.com/RoaringBitmap/RoaringFormatSpec?tab=readme-ov-file#general-layout

#### `apache-datasketches-kll-v1` blob type

A serialized form of a KLL sketch, a very compact quantiles sketch, produced by the
[Apache DataSketches](https://datasketches.apache.org/) library.
KLL quantiles sketch is a mergeable streaming algorithm to estimate
the distribution of values, and approximately answer queries about the rank of a value,
probability mass function of the distribution (PMF) or histogram,
cumulative distribution function (CDF), and quantiles (median, min, max, 95th percentile and such)

### Compression codecs

The data can also be uncompressed. If it is compressed the codec should be one of
Expand Down