Skip to content

[SPARK-23253][Core][Shuffle]Only write shuffle temporary index file when there is not an existing one #20422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 8 commits into from
Closed

[SPARK-23253][Core][Shuffle]Only write shuffle temporary index file when there is not an existing one #20422

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
98ea6a7
Only write shuffle temporary index file when there is not an existing…
yaooqinn Jan 29, 2018
6196770
add a ut
yaooqinn Jan 30, 2018
87e6dc0
modify ut
yaooqinn Jan 31, 2018
a96f6c4
merge uts together
yaooqinn Feb 1, 2018
246dbca
minor
yaooqinn Feb 1, 2018
f3f3627
comments
yaooqinn Feb 1, 2018
5677448
review
yaooqinn Feb 2, 2018
a3b85f1
blank line
yaooqinn Feb 2, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 13 additions & 14 deletions core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,19 +141,6 @@ private[spark] class IndexShuffleBlockResolver(
val indexFile = getIndexFile(shuffleId, mapId)
val indexTmp = Utils.tempFileWith(indexFile)
try {
val out = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(indexTmp)))
Utils.tryWithSafeFinally {
// We take in lengths of each block, need to convert it to offsets.
var offset = 0L
out.writeLong(offset)
for (length <- lengths) {
offset += length
out.writeLong(offset)
}
} {
out.close()
}

val dataFile = getDataFile(shuffleId, mapId)
// There is only one IndexShuffleBlockResolver per executor, this synchronization make sure
// the following check and rename are atomic.
Expand All @@ -166,8 +153,20 @@ private[spark] class IndexShuffleBlockResolver(
if (dataTmp != null && dataTmp.exists()) {
dataTmp.delete()
}
indexTmp.delete()
} else {
val out = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(indexTmp)))

@squito squito Jan 30, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move this below the comment "This is the first successul attempt".

I'd also include a comment about why we write to a temporary file, even though we're always going to rename (because in case the task dies somehow, we'd prefer to not leave a half-written index file in the final location).

Utils.tryWithSafeFinally {
// We take in lengths of each block, need to convert it to offsets.
var offset = 0L
out.writeLong(offset)
for (length <- lengths) {
offset += length
out.writeLong(offset)
}
} {
out.close()
}

// This is the first successful attempt in writing the map outputs for this task,
// so override any existing index and data files with the ones we wrote.
if (indexFile.exists()) {
Expand Down