Skip to content

[WIP][SPARK-29037][Core] Spark gives duplicate result when an application was killed #25795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
turboFei wants to merge 11 commits into from
Closed

[WIP][SPARK-29037][Core] Spark gives duplicate result when an application was killed #25795

Changes from 2 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8070497
[SPARK-29037] Make staging dir identified with applicationId
turboFei Sep 10, 2019
3e8b69b
[SPARK-29037] Spark gives duplicate result when an application was ki…
turboFei Sep 15, 2019
82d2173
unit test
turboFei Sep 15, 2019
c3aff4e
fix code style
turboFei Sep 15, 2019
4e44550
move ut to sql/core
turboFei Sep 15, 2019
656668e
fix ut
turboFei Sep 15, 2019
804ac8c
also skip setupJob and abortJob when dynamicPartitionOverwrite is true
turboFei Sep 16, 2019
3275685
just to triger a build
turboFei Sep 17, 2019
144ec63
just to triger a new building
turboFei Sep 17, 2019
979fbe4
add comment
turboFei Sep 17, 2019
f0b6c84
add insert datasource operations conflict check when dynamicPartition…
turboFei Sep 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,9 @@ class HadoopMapReduceCommitProtocol(
}

override def commitJob(jobContext: JobContext, taskCommits: Seq[TaskCommitMessage]): Unit = {
committer.commitJob(jobContext)
if (!dynamicPartitionOverwrite) {
committer.commitJob(jobContext)
}
Comment thread
turboFei marked this conversation as resolved.

if (hasValidPath) {
val (allAbsPathFiles, allPartitionPaths) =
Expand Down