SPARK-1597: Add a version of reduceByKey that takes the Partitioner as a...

[techaddict]

... second argument

Most of our shuffle methods can take a Partitioner or a number of partitions as a second argument, but for some reason reduceByKey takes the Partitioner as a first argument: http://spark.apache.org/docs/0.9.1/api/core/#org.apache.spark.rdd.PairRDDFunctions.
Deprecated that version and added one where the Partitioner is the second argument.

[techaddict]

We'll need to specify the parameter types for function passed to reduceByKey
reduceByKey((x: Long, y: Long) => x + y, 10) instead of reduceByKey(_ + _, 10)
For detailed discussion on compiler issue causing this,
https://groups.google.com/forum/#!topic/scala-user/Qhd3vJ2rAWM

@mateiz IMHO we should leave the method as it is, as this will make the code ugly.

[AmplabJenkins]

Can one of the admins verify this patch?

[mateiz]

Ah, wow, I never knew that. So if one takes a Partitioner first and one takes a function, the types are inferred, but if both take a function first, they're not?

In that case we might want to change our other methods too, like cogroup and groupByKey, to take a Partitioner first. Wouldn't this problem also affect them?

[mateiz]

CC @rxin, @pwendell

[techaddict]

@mateiz I think this only applies with anon function's, thus isn't affecting either cogroup or groupByKey.

[rxin]

This line is over 100 chars wide

[techaddict]

@rxin will fix this as soon as, a decision is made over whether we want to do this or not.

[rxin]

I never even realized we had a version of reduceByKey where the first argument is not the closure ...

[rxin]

I have one solution to this, although it is technically an API change, so just throwing it out there for discussion. We can remove all the numPartitions: Int arguments, and add an implicit conversion from int to HashPartitioner.

[techaddict]

@rxin +1

[mateiz]

I'd rather not add the implicit conversion from int to partitioner, it will be very hard to discover on its own. Instead maybe we can just leave this API as is. It's strange but there's a good reason for it.

[SparkQA]

QA tests have started for PR 550. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17978/consoleFull

[SparkQA]

QA results for PR 550:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17978/consoleFull

[pwendell]

It sounds like the conclusion here is to close this issue then.