[SPARK-6369] [SQL] Uses commit coordinator to help committing Hive and Parquet tables

[liancheng]

This PR leverages the output commit coordinator introduced in #4066 to help committing Hive and Parquet tables.

This PR extracts output commit code in SparkHadoopWriter.commit to SparkHadoopMapRedUtil.commitTask, and reuses it for committing Parquet and Hive tables on executor side.

TODO

• Add tests

[SparkQA]

Test build #29004 has started for PR 5139 at commit dfdf3ef.

• This patch merges cleanly.

[marmbrus]

@aarondav if you have time, I'd appreciate your input here.

[SparkQA]

Test build #29004 has finished for PR 5139 at commit dfdf3ef.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29004/
Test PASSed.

[liancheng]

@JoshRosen It would be great if you can also help reviewing this. Thanks in advance!

[aarondav]

Please add documentation to this guy mentioning what it means to commitTask (i.e., we may contact the driver to become authorized to commit to ensure speculative tasks do not override each other, and that this may cause us to abort the task by throwing a CommitDeniedException if we cannot become authorized as such), pointing to the JIRA that this fixes (the original one).

[aarondav]

LGTM from my side, but @JoshRosen should confirm the driver side should be happy with this. Only comment was that now that it's extracted and used in a common location, we need to make sure its API is well-documented.

[liancheng]

@aarondav Thanks! Added javadoc for this.

[SparkQA]

Test build #29405 has started for PR 5139 at commit 9a4b82b.

[SparkQA]

Test build #29405 has finished for PR 5139 at commit 9a4b82b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29405/
Test PASSed.

[JoshRosen]

"Commit output coordinator" -> "Output commit coordinator"

[JoshRosen]

LGTM from a Spark core point-of-view. One of the biggest risks here is passing the Long-valued parameters in the wrong order, but it looks like we've done it correctly here. I suppose that another risk might be calling the commit function with values of jobId, splitId, and attemptId that don't match / correspond to the ones used in the MapReduceTaskAttemptContext (that would undermine the whole scheme because the coordination wouldn't necessarily be guarding the right output paths), but our usage here looks fine as far as I can tell.

[SparkQA]

Test build #29434 has started for PR 5139 at commit 72eb628.

[liancheng]

Thanks @JoshRosen! Fixed the typo. I'm merging this to master and 1.3.

[SparkQA]

Test build #29434 has finished for PR 5139 at commit 72eb628.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29434/
Test PASSed.