[SPARK-20333] HashPartitioner should be compatible with num of child RDD's partitions.

[jinxing64]

What changes were proposed in this pull request?

Fix test
"don't submit stage until its dependencies map outputs are registered (SPARK-5259)" ,
"run trivial shuffle with out-of-band executor failure and retry",
"reduce tasks should be placed locally with map output"
in DAGSchedulerSuite.

[jinxing64]

I found this when do #17533

[SparkQA]

Test build #75787 has finished for PR 17634 at commit e9d8297.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #75788 has finished for PR 17634 at commit cb80dbd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jinxing64]

@squito @srowen
Could you help comment on this :)

[jinxing64]

@kayousterhout @mridulm
Does this pr make sense?
Could you please take a look this when you have time :)

[jiangxb1987]

The test cases are not falling on master branch, are there some good reasons to change them?

[jinxing64]

@jiangxb1987
Thank you so much taking time looking into this.
Yes, it is not failing in existing code. But I think it's quite straightforward that partitioner should be compatible with num of child RDD's partitions, right?
In #17533, I have to query the MapStatus based on the correctness of partitioner(i.e. num of partitions). That's why I proposed this.
Thanks again taking time reviewing this :) 👍

[jiangxb1987]

Don't think we could fail on the original test case for any reason, otherwise the legacy user code could fail unexpectedly, isn't it?

[jinxing64]

Yes. it doesn't fail for sure.
I just think it's fairly straightforward that partitioner should be compatible with num of child RDD's partitions. I find no reason the num of partitions are not equal in those test.

[squito]

lgtm.
sorry for such a late review. Since its been a while I'll trigger tests again just to be safe before merging.

[SparkQA]

Test build #3767 has finished for PR 17634 at commit cb80dbd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[squito]

merged to master

[jinxing64]

@squito
Thanks a lot for merging :)