[SPARK-13071] [hadoop-2.7] Coalescing HadoopRDD overwrites existing input metrics

[andrewor14]

This issue is causing tests to fail consistently in master with Hadoop 2.6 / 2.7. This is because for Hadoop 2.5+ we overwrite existing values of InputMetrics#bytesRead in each call to HadoopRDD#compute. In the case of coalesce, e.g.

sc.textFile(..., 4).coalesce(2).count()

we will call compute multiple times in the same task, overwriting bytesRead values from previous calls to compute.

For a regression test, see InputOutputMetricsSuite.input metrics for old hadoop with coalesce. I did not add a new regression test because it's impossible without significant refactoring; there's a lot of existing duplicate code in this corner of Spark.

This was caused by #10835.

[andrewor14]

@yhuai @JoshRosen

[SparkQA]

Test build #50335 has finished for PR 10973 at commit c5a97fc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

test this please

[yhuai]

maybe also put hadoop2.7 in the title, so jenkins can test it?

[SparkQA]

Test build #50348 has finished for PR 10973 at commit c5a97fc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

like this? I didn't know Jenkins can do that

[andrewor14]

retest this please

[andrewor14]

let's try again. retest this please

[nongli]

LGTM

[SparkQA]

Test build #50394 has finished for PR 10973 at commit c5a97fc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50395 has finished for PR 10973 at commit c5a97fc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

retest this please

[SparkQA]

Test build #50408 has finished for PR 10973 at commit c5a97fc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2478 has finished for PR 10973 at commit c5a97fc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Merging into master.