[SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRelation job initialization for Hadoop 1.x #6669
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
cc @yhuai |
liancheng
force-pushed
the
spark-8121
branch
3 times, most recently
from
046a01e to
910b8aa
Compare
|
Test build #34264 has finished for PR 6669 at commit
|
|
Test build #34267 has finished for PR 6669 at commit
|
Contributor
There was a problem hiding this comment.
Seems we have protected def configuration = sparkContext.hadoopConfiguration in SQLTestUtils and we are not cloning it at here?
|
Test build #34344 has finished for PR 6669 at commit
|
|
Test build #34355 has finished for PR 6669 at commit
|
|
Test build #34390 has finished for PR 6669 at commit
|
Contributor
|
Should we log the output committer used by parquet? |
Contributor
There was a problem hiding this comment.
Maybe we can get the instance of outputCommitter first and then log its class?
val outputCommitter = ...
logInfo(...)
outputCommitter
|
Test build #34407 has finished for PR 6669 at commit
|
liancheng
added a commit
to liancheng/spark
that referenced
this pull request
Jun 8, 2015
|
Test build #34431 has finished for PR 6669 at commit
|
Contributor
|
LGTM. I am merging it to master. |
asfgit
pushed a commit
that referenced
this pull request
Jun 8, 2015
…n for Hadoop 1.x (branch 1.4 backport based on #6669)
nemccarthy
pushed a commit
to nemccarthy/spark
that referenced
this pull request
Jun 19, 2015
…n for Hadoop 1.x For Hadoop 1.x, `TaskAttemptContext` constructor clones the `Configuration` argument, thus configurations done in `HadoopFsRelation.prepareForWriteJob()` are not populated to *driver* side `TaskAttemptContext` (executor side configurations are properly populated). Currently this should only affect Parquet output committer class configuration. Author: Cheng Lian <[email protected]> Closes apache#6669 from liancheng/spark-8121 and squashes the following commits: 73819e8 [Cheng Lian] Minor logging fix fce089c [Cheng Lian] Adds more logging b6f78a6 [Cheng Lian] Fixes compilation error introduced while rebasing 963a1aa [Cheng Lian] Addresses @yhuai's comment c3a0b1a [Cheng Lian] Fixes InsertIntoHadoopFsRelation job initialization
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For Hadoop 1.x,
TaskAttemptContextconstructor clones theConfigurationargument, thus configurations done inHadoopFsRelation.prepareForWriteJob()are not populated to driver sideTaskAttemptContext(executor side configurations are properly populated). Currently this should only affect Parquet output committer class configuration.