Skip to content

Commit 0d4d199

Browse files
committed
Bump up shuffle.memoryFraction to make tests pass.
We'll want to revisit this before merging, since the large minimum memory usage means that minimum memory requirements for shuffle may be fairly high for local tests.
1 parent b3b1924 commit 0d4d199

File tree

2 files changed

+4
-1
lines changed

2 files changed

+4
-1
lines changed

core/src/main/java/org/apache/spark/shuffle/unsafe/UnsafeShuffleExternalSorter.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -337,7 +337,7 @@ private void allocateSpaceForRecord(int requiredSpace) throws IOException {
337337
final long memoryAcquiredAfterSpilling = shuffleMemoryManager.tryToAcquire(PAGE_SIZE);
338338
if (memoryAcquiredAfterSpilling != PAGE_SIZE) {
339339
shuffleMemoryManager.release(memoryAcquiredAfterSpilling);
340-
throw new IOException("Can't allocate memory!");
340+
throw new IOException("Unable to acquire " + PAGE_SIZE + " bytes of memory");
341341
}
342342
}
343343
}

core/src/test/scala/org/apache/spark/shuffle/unsafe/UnsafeShuffleSuite.scala

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,5 +26,8 @@ class UnsafeShuffleSuite extends ShuffleSuite with BeforeAndAfterAll {
2626

2727
override def beforeAll() {
2828
conf.set("spark.shuffle.manager", "unsafe")
29+
// UnsafeShuffleManager requires at least 128 MB of memory per task in order to be able to sort
30+
// shuffle records.
31+
conf.set("spark.shuffle.memoryFraction", "0.5")
2932
}
3033
}

0 commit comments

Comments
 (0)