-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-27194][SPARK-29302][SQL] Define a spark staging committer to resolve FileAlreadyExistingException #28989
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
gentle ping @cloud-fan Hi, we found a new solution to fix the issues when dynamic partition overwrite is enabled.
In this PR, we define a new type of OutputCommitter and leverage the OutputCommitCoordinator to coordinate the task commits. |
07066e7 to
8f202c9
Compare
|
aslo cc @Ngone51 |
cd42f42 to
b09a665
Compare
|
ok to test |
|
Test build #125132 has started for PR 28989 at commit |
|
test this please |
|
Test build #125143 has finished for PR 28989 at commit
|
|
will try to fix it. |
…solve file already exist exception
b09a665 to
0492977
Compare
|
Test build #125196 has finished for PR 28989 at commit
|
|
Test build #125205 has finished for PR 28989 at commit
|
|
Test build #125212 has finished for PR 28989 at commit
|
|
Test build #125275 has finished for PR 28989 at commit
|
|
close this, Prefer to #29000 |
What changes were proposed in this pull request?
For dynamic partition overwrite, its working dir is
.spark-staging-{jobId}.Task file name formatted
part-$taskId-$jobId$ext(regardless task attempt Id).Each task writes its output to:
.spark-staging-{jobId}/partitionPath1/taskFileName1.spark-staging-{jobId}/partitionPath2/taskFileName2.spark-staging-{jobId}/partitionPathN/taskFileNameNIf speculation is enabled, there may be several tasks, which have same taskId and different attemptId, write to the same files concurrently.
For distributedFileSystem, it only allow one task to hold the lease to write a file, if two tasks want to write the same file, an exception like
no lease on inodewould be thrown.Even speculation is not enabled, if a task aborted due to Executor OOM, its output would not be cleaned up.
Then a new task launched to write the same file, because parquet disallows overwriting, a
FileAlreadyExistsExceptionwould be thrown, like.It is a critical issue and would cause job failed.
In this Pr, we define a spark staging output committer to fix this issue:
Why are the changes needed?
Without this PR, dynamic partition overwrite operation might fail.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added UT.