Added validation check for parallelizing a seq

[bijaybisht]

This fixes a bug where in a Seq can be converted into a RDD with partitions more than the number of elements it has.

Also fixes the bug in the handling of the NumericRange.

[AmplabJenkins]

Can one of the admins verify this patch?

[pwendell]

Thanks for reporting this! Could you add unit tests for both fixes? Also, would you mind creating a JIRA for this and explaining what the exception was in the old case? It just makes it easier to track fixes for this.

[liancheng]

Hi @bijaybisht, thanks for fixing this :) I had once noticed this issue, but at last decided not to change this. And my reasons are:

1. Although not specified in the ScalaDoc, the numSlices parameter of SparkContext.parallelize specifies the exact partition number of the result RDD. This PR actually changes semantics of the numSlices parameter.
2. An RDD can have more partitions than its elements. For example, RDD.filter may result empty partitions.
3. For APIs like RDD.zipPartitions, partition number is significant, and this change may break some existing code. For example:

// Using coalesce to ensure we have exactly 4 partitions
val x = sc.textFile("input", 4).coalesce(4) 
val y = sc.parallelize(1 to 3, 4)
val z = x.zipPartitions(y) { (i, j) =>
  ...
}

(x.zipPartitions(y) requires x & y have exactly the same partition number):

[rxin]

I agree with @liancheng that it is best not to change the existing semantics. At least I've been using parallelize to just launch a bunch of tasks without passing it a Seq that is big enough. What we should make sure is Spark doesn't crash in this case (parallelize s iterable who size < num partitions).

[pwendell]

Ah I see - so the reported "bug" here was not that there were any failures/exceptions but just that you'd have extra empty partitions?

[mateiz]

We should allow the empty partitions, this was intentional behavior. It's used quite a bit in our unit tests for example. Users who want fewer partitions can always check the size of the Seq themselves.

[pwendell]

@bijaybisht mind closing this? I think this is a check we don't want to have in the API.

[bijaybisht]

Sure, ill close this. I presume that the change for the NumericRange which results in a more balanced partitions (which is part of the fix) is also something that is not required.