SPARK-2978. Transformation with MR shuffle semantics

[sryza]

I didn't add this to the transformations list in the docs because it's kind of obscure, but would be happy to do so if others think it would be helpful.

[SparkQA]

QA tests have started for PR 2274 at commit a75f277.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2274 at commit a75f277.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 2274 at commit a1ef807.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2274 at commit a1ef807.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class BlockManagerMaster(
  • class AttributeMap[A](baseMap: Map[ExprId, (Attribute, A)])

[davies]

How about re-arrange the parameters to follow the function name? such as:

repartitionAndSortWithinPartition(self, numPartitions=None, partitionFunc=portable_hash, 
                                          ascending=True, keyfunc=lambda x: x)

[sryza]

Updated patch removes Python version, adds Java version, and adds some additional doc.

[mateiz]

Just a nit, it should probably be called repartitionAndSortWithinPartition_s_.

Also, this name is pretty long. Another one I'd reconsider is repartitionWithSort, but not sure what other people think.

Finally I think it should be a policy to add all these APIs to Python, and implement them there too. Basically there are two options -- if you're doing this to support a slightly easier transition from MR jobs, but you don't want to do it in Python, you could just have it as a document, or an example, or maybe even a third-party package that takes a Hadoop JobConf and runs it on Spark. But if you want it in Spark, we need to put it in each language. The reason is to allow people to easily read code in one supported language and run it in others -- it's always disappointing when some operators turn out to be missing in yours.

[rxin]

@mateiz

The reason to add this is because this is a smaller API that we can support (both source and binary compatibility) in the long run before finalizing ShuffledRDD (since that one has been in flux and changing in multiple past releases). Perhaps we can mark this new API as DeveloperApi but commit to maintaining it. What do you think?

The naming is long, but I'm worried repartitionWithSort in a way implies the data are sorted globally.

[rxin]

u can put this on the previous line ...

[mateiz]

Ah, I see. Then we can add it, but in that case I'd also add it in Python.

[sryza]

Updated patch adds Python back in and adds the 's' at the end.

[rxin]

Thanks, Sandy. Can you add a unit test in Java to make sure the thing is callable from Java?

[davies]

These are removed by accident during merging?

[sryza]

Yup, my bad

[SparkQA]

QA tests have started for PR 2274 at commit c04b447.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2274 at commit c04b447.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 2274 at commit 4a5332a.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2274 at commit 4a5332a.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

LGTM, thanks.

[mateiz]

Thanks Sandy! I've merged this.