feat(function): Implement theta sketch math functions

[nmahadevuni]

Required for the new Iceberg statistics introduced in Presto Java. Ported from https://github.com/apache/datasketches-cpp/tree/master/theta.

Sketches are data structures that can approximately answer particular questions about a dataset when full accuracy is not required. The benefit of approximate answers is that they are often faster and more efficient to compute than functions which result in full accuracy.

Theta sketches enable distinct value counting on datasets and also provide the ability to perform set operations. For more information on Theta sketches, please see the Apache Datasketches Theta sketch documentation

The Presto PR which introduced these changes is prestodb/presto#20993. A brief intro to these functions

New Sketch Functions

Iceberg's Puffin spec defines the format that NDVs must be written in. Currently, the only available format is a binary
blob representing an Apache Datasketches Theta Sketch, so we implemented three basic functions which expose the sketch so that Iceberg can eventually consume it when writing statistics.

1. sketch_theta(<column>) -> varbinary: An aggregation function which accepts a column and generates a binary representation of the org.apache.datasketches.theta.CompactSketch. Applications can easily consume this raw binary
   format to gain access to a CompactSketch instance and associated methods.

2. sketch_theta_estimate(<varbinary sketch>) -> double: A scalar function which consumes a raw binary sketch and produces the estimate. This is effectively the same as calling CompactSketch::getEstimate. I've exposed this as a convenience for checking the sketch's output

3. sketch_theta_summary(<varbinary sketch>) -> row(estimate double, theta double, upper_bound_std1 double, lower_bound_std1 double, retained_entries int): This is another scalar function, but returns a row type containing
   more human-readable information about the sketch such as the theta parameter as well as upper and lower bounds
   for 1 standard deviation from the estimate

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	b8ba93b
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/685cdb337deca20008e97288

[aditi-pandit]

@nmahadevuni : The theta code adapted from Data sketches is better fit under velox/external directory than velox/common. Can you try moving it there ?

[nmahadevuni]

Thanks @aditi-pandit. I have moved the code to external directory.

[yingsu00]

@nmahadevuni Where is the function registration stuff?

[nmahadevuni]

@nmahadevuni Where is the function registration stuff?

I'm working on it in a separate PR. Adding it here would make it even bigger.

[PingLiuPing]

The sketch document link is wrong.
Besides the unit tests, how did you verify this works as expected when integrate with Prestissimo?

[majetideepak]

@nmahadevuni Is this the Java PR that describes the sketches prestodb/presto#20993?
Can you update the PR description about the usage of these sketches? Does Presto optimizer currently use these sketches? How can we test these? Will Gluten also benefit from this?
It looks like we will need other functions from the library in the future as well. So does it make sense to add a dependency instead?

[majetideepak]

@nmahadevuni Can we use KLL sketches, which are already present in Velox?

[yingsu00]

@nmahadevuni Can we use KLL sketches, which are already present in Velox?

Unfortunately Iceberg doesn't use KLL sketch. In Apache Iceberg’s Puffin statistics format, only Theta sketches (apache‑datasketches‑theta‑v1) and deletion vectors (deletion‑vector‑v1) are defined as built‑in blob types. KLL was discussed but not planned. see: Request to add KLL Datasketch and hive ColumnStatisticsObj and as standard blob types to puffin file.

[nmahadevuni]

The sketch document link is wrong. Besides the unit tests, how did you verify this works as expected when integrate with Prestissimo?

@PingLiuPing Thank you. Sorry for the late reply. I have tested by updating my local presto-native-execution/velox and making sure the test output from Java functions match with Velox functions with a few known exceptions which were discussed and confirmed with datasketch project architect in apache/datasketches-cpp#460

Corrected the documentation link.

[nmahadevuni]

Once this is merged, I will write integration tests in presto-native-execution module

[nmahadevuni]

@nmahadevuni Is this the Java PR that describes the sketches prestodb/presto#20993? Can you update the PR description about the usage of these sketches? Does Presto optimizer currently use these sketches? How can we test these? Will Gluten also benefit from this? It looks like we will need other functions from the library in the future as well. So does it make sense to add a dependency instead?

Yes, prestodb/presto#20993 is the Java PR that added support for these functions. The write and read support for these statistics was added in the same PR, so optimizer would see the new statistics if they were written.

We can add a dependency too. Do you want to switch to adding a dependency?

[nmahadevuni]

btw, KLL sketches are already implemented in Velox. velox/functions/lib/KllSketch.cpp

[assignUser]

Please add a proper section to the NOTICE.txt, you will have to add the contents of the Apache DataSketches NOTICE.txt there too, as is required by the ASL 2.0.

[majetideepak]

@nmahadevuni It is better to add a dependency since the code size is huge. Tim, @aditi-pandit, and I believe we can start by adding this to Presto C++ and then moving to Velox once we have more Iceberg functionality implemented here.

[nmahadevuni]

@nmahadevuni It is better to add a dependency since the code size is huge. Tim, @aditi-pandit, and I believe we can start by adding this to Presto C++ and then moving to Velox once we have more Iceberg functionality implemented here.

How does it work adding a dependency to Presto C++? We have to add aggregate and scalar functions in Velox which depends on this.

[majetideepak]

We have to add aggregate and scalar functions in Velox which depends on this.

Can't we register these functions in Presto C++?

[nmahadevuni]

We have to add aggregate and scalar functions in Velox which depends on this.

Can't we register these functions in Presto C++?

We should implement the aggregate function also in PrestoC++, I don't see any prior example of this.

[aditi-pandit]

We have to add aggregate and scalar functions in Velox which depends on this.

Can't we register these functions in Presto C++?

We should implement the aggregate function also in PrestoC++, I don't see any prior example of this.

@nmahadevuni : Agree there aren't prior pure Presto aggregate functions. What is needed is for you to move your code :
i) External theta sketch library code to a folder in https://github.com/prestodb/presto/tree/master/presto-native-execution/presto_cpp/external
ii) Add function code in a new folder 'functions' in https://github.com/prestodb/presto/tree/master/presto-native-execution/presto_cpp/main
iii) Add function registration in https://github.com/prestodb/presto/blob/master/presto-native-execution/presto_cpp/main/PrestoServer.cpp#L1320
iv) Write e2e test SQL for the functions in some file in https://github.com/prestodb/presto/tree/master/presto-native-execution/presto_cpp/main

[majetideepak]

External theta sketch library code to a folder in

This needs to be a dependency instead.

[nmahadevuni]

Closing this for prestodb/presto#25685