SPARK-5270 [CORE] Elegantly check if RDD is empty

[srowen]

Pretty minor, but submitted for consideration -- this would at least help people make this check in the most efficient way I know.

[SparkQA]

Test build #25667 has started for PR 4074 at commit de6b95e.

• This patch merges cleanly.

[srowen]

(Oh of course, if this looks good I can add this to Java / Python too)

[SparkQA]

Test build #25667 has finished for PR 4074 at commit de6b95e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25667/
Test PASSed.

[ksakellis]

LTGM. What is the use case? is this part of a bigger pr?

[srowen]

This is all there is to it. It's just a convenience method that implements the check efficiently. Given several questions on the list, it seems that people do want to test for an empty RDD and there hasn't been an accepted way to do it that is faster than count() == 0:

http://apache-spark-user-list.1001560.n3.nabble.com/Testing-if-an-RDD-is-empty-td1678.html#a1679
... and of course
http://issues.apache.org/jira/browse/SPARK-5270

[pwendell]

Seems reasonable to have since it's non obvious how to do it - @srowen could you add this in Java and Python?

[SparkQA]

Test build #25682 has started for PR 4074 at commit d76f8e3.

• This patch merges cleanly.

[SparkQA]

Test build #25682 has finished for PR 4074 at commit d76f8e3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25682/
Test FAILed.

[srowen]

Jenkins, retest this please.

[SparkQA]

Test build #25701 has started for PR 4074 at commit d76f8e3.

• This patch merges cleanly.

[SparkQA]

Test build #25701 has finished for PR 4074 at commit d76f8e3.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25701/
Test FAILed.

[ash211]

I don't think this tests the case where there are multiple partitions but no data in any of the partitions. Maybe add something like

assert(sc.parallelize(Seq(1,2,3), 3).filter(_ < 0).isEmpty())

[srowen]

I think the sc.parallelize(Seq[Int]() case actually has multiple partitions but I'll add this too. Also, I'll check the case where the first partition is empty but others aren't.

[SparkQA]

Test build #25730 has started for PR 4074 at commit 191bb9f.

• This patch does not merge cleanly.

[SparkQA]

Test build #25730 has finished for PR 4074 at commit 191bb9f.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25730/
Test PASSed.

[SparkQA]

Test build #25731 has started for PR 4074 at commit 66885b8.

• This patch merges cleanly.

[SparkQA]

Test build #25731 has finished for PR 4074 at commit 66885b8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25731/
Test PASSed.

[pwendell]

LGTM @srowen - are you still working on it or is it good from your end? Will leave a bit of time for others to comment as well.

[srowen]

@pwendell No more changes from my side.

[pwendell]

@srowen Thanks Sean, I committed this with a minor re-word of the title.