[SPARK-6761][SQL] Approximate quantile for DataFrame

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-6761

Compute approximate quantile based on the paper Greenwald, Michael and Khanna, Sanjeev, "Space-efficient Online Computation of Quantile Summaries," SIGMOD '01.

[SparkQA]

Test build #32337 has finished for PR 6042 at commit 1086537.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks - I was thinking about having this as a UDAF after we have the new UDAF interface merged. Let's revisit this after 1.4 is stable enough.

[SparkQA]

Test build #37769 timed out for PR 6042 at commit 1086537 after a configured wait of 150m.

[rxin]

@viirya it would be great to have this as an aggregate function. Can you look into the feasibility of that?

[viirya]

@rxin ok. I will update this later.

[viirya]

@rxin I opened a new PR at #8459 for the aggregation function.

[viirya]

ping @thunterdb

[viirya]

retest this please.

[mengxr]

test this please

[thunterdb]

@viirya sorry I missed your email, I will look at your PR today.

[SparkQA]

Test build #51182 has finished for PR 6042 at commit 437aaea.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51185 has finished for PR 6042 at commit ac4bc97.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51187 has finished for PR 6042 at commit d320fd2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #51538 has finished for PR 6042 at commit daaa196.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

@thunterdb Your pull request looks good. Thanks. I've merged it. I will take another look later.

[SparkQA]

Test build #51624 has finished for PR 6042 at commit 9314f1e.

• This patch fails RAT tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class QuantileSummaries(
  • case class Stats(value: Double, g: Int, delta: Int)

[SparkQA]

Test build #51628 has finished for PR 6042 at commit 47cde05.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51632 has finished for PR 6042 at commit a36891b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@viirya I had an offline discussion with @thunterdb today. I'm going to add some comments inline and then merge this PR. @thunterdb will send another PR to address my comment because he already knew the context. After that, could you add approximate quantiles to describe()? Thanks!

[mengxr]

epsilon is not documented. It might be better to call it relerr or relativeError because epsilon doesn't carry any information.

[thunterdb]

Done

[viirya]

@mengxr ok. thank you.

[mengxr]

I wonder how much speedup we can get by merging the two lists manually compared to (thisSampled ++ otherSampled).sorted. Did you run some tests?

[thunterdb]

I agree that the current implementation is too complicated, and that probably just merging/sorting the two arrays directly is more efficient for the size considered.

When running some performance testing, the cost of the algorithm was dominated by the cost of accessing the content of Rows. Only 4% of the running time was spent on insertion+merging, so this cost was negligible at this point.

I am going to do as you suggest. If it happens to be a bottleneck when we use UDAFs later, directly manipulating ArrayBuffers would be more efficient than pattern-matching on lists anyway. Rerunning the synthetic benchmark with the suggested changes did not yield runtime changes.

[mengxr]

Merged into master. @thunterdb Please send another PR to address my comments. Thanks!

[thunterdb]

@mengxr thanks for the review, will do in another PR.