[SPARK-3250] Implement Gap Sampling optimization for random sampling

[erikerlandson]

More efficient sampling, based on Gap Sampling optimization:
http://erikerlandson.github.io/blog/2014/09/11/faster-random-samples-with-gap-sampling/

[mengxr]

Did you test whether rdd.randomSplit() will produce non-overlapping subsets with this change?

[mengxr]

add to whitelist

[mengxr]

this is ok to test

[SparkQA]

Can one of the admins verify this patch?

[erikerlandson]

@mengxr you're right, a data partitioning use case like rddSplit doesn't work with gap sampling, so I restored a "partitioning" RandomSampler subclass that works for those cases.

[mengxr]

test this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20790/

[erikerlandson]

Jenkins looks like it failed trying to fetch the repo.

[mengxr]

@erikerlandson Jenkins is not very stable. You are on the whitelist, feel free to ask Jenkins to retest this PR.

[mengxr]

test this please

[SparkQA]

QA tests have started for PR 2455 at commit e29a0ae.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2455 at commit e29a0ae.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class BernoulliPartitionSampler[T](lb: Double, ub: Double, complement: Boolean = false)
  • class BernoulliSampler[T: ClassTag](fraction: Double) extends RandomSampler[T, T]
  • class PoissonSampler[T: ClassTag](fraction: Double) extends RandomSampler[T, T]
  • class GapSamplingIterator[T: ClassTag](var data: Iterator[T], f: Double,
  • class GapSamplingReplacementIterator[T: ClassTag](var data: Iterator[T], f: Double,

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20811/

[erikerlandson]

test this please

[mengxr]

test this please

[SparkQA]

QA tests have started for PR 2455 at commit b89b591.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2455 at commit b89b591.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class BernoulliPartitionSampler[T](lb: Double, ub: Double, complement: Boolean = false)
  • class BernoulliSampler[T: ClassTag](fraction: Double) extends RandomSampler[T, T]
  • class PoissonSampler[T: ClassTag](fraction: Double) extends RandomSampler[T, T]
  • class GapSamplingIterator[T: ClassTag](var data: Iterator[T], f: Double,
  • class GapSamplingReplacementIterator[T: ClassTag](var data: Iterator[T], f: Double,

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20828/

[rxin]

can you change the comment to use javadoc style? e.g.

/**
 * Default gap sampling maximum.
 * ...
 */

[erikerlandson]

@srowen, regarding the testing for iterator types, inside of 'dd', that was the only way I found (so far) that scala would accept. The best solution (imo) would be if Scala defined a random-access-optimized iterator subclass that I could match on, but there is no such animal. I've been considering requesting one in a Scala PR.

[mengxr]

@erikerlandson The feature freeze deadline for v1.2 is this Sat. Just want to check with you and see whether you are going to update the PR this week.

[erikerlandson]

@mengxr, coincidentally I'm working through the PR comments today, I plan to have an update pushed this evening

[mengxr]

@erikerlandson Great! Thanks for the heads up.

[erikerlandson]

Was about to push, but looks like commit for SPARK-4022 broke my updates so I'm going to have to make more edits to rebase

[SparkQA]

Test build #22475 has started for PR 2455 at commit 46cb9fa.

• This patch merges cleanly.

[SparkQA]

Test build #22475 has finished for PR 2455 at commit 46cb9fa.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class BernoulliCellSampler[T](lb: Double, ub: Double, complement: Boolean = false)
  • class BernoulliSampler[T: ClassTag](fraction: Double) extends RandomSampler[T, T]
  • class PoissonSampler[T: ClassTag](fraction: Double) extends RandomSampler[T, T]
  • class GapSamplingIterator[T: ClassTag](var data: Iterator[T], f: Double,
  • class GapSamplingReplacementIterator[T: ClassTag](var data: Iterator[T], f: Double,
  • class DeferredObjectAdapter(oi: ObjectInspector) extends DeferredObject

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22475/
Test PASSed.

[erikerlandson]

@mengxr latest updates are rebased and passing Jenkins

[mengxr]

0.5 -> 0.4?

[erikerlandson]

0.5 is what I recommend as an initial guess if one is using a new RNG. (0.4 is what I got by experimenting with the current RNG)

[erikerlandson]

@mengxr, I changed fractionEpsilon to rngEpsilon, which is more suggestive of its purpose. I also updated its documentation, which I think is also now more clear about what rngEpsilon is for.

[SparkQA]

Test build #22576 has started for PR 2455 at commit 72496bc.

• This patch merges cleanly.

[SparkQA]

Test build #22576 has finished for PR 2455 at commit 72496bc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22576/
Test PASSed.

[mengxr]

LGTM. Merged into master. Thanks for implementing gap sampling!