Skip to content

Commit 52b6cd2

Browse files
committed
[SPARK-3889] Attempt to avoid SIGBUS by not mmapping files in ConnectionManager
In general, individual shuffle blocks are frequently small, so mmapping them often creates a lot of waste. It may not be bad to mmap the larger ones, but it is pretty inconvenient to get configuration into ManagedBuffer, and besides it is unlikely to help all that much. Note that user of ManagedBuffer#nioByteBuffer() seems generally bad practice, and would ideally never be used for data that may be large. Users of such data would ideally stream the data instead.
1 parent 4e9b551 commit 52b6cd2

File tree

1 file changed

+4
-1
lines changed

1 file changed

+4
-1
lines changed

core/src/main/scala/org/apache/spark/network/ManagedBuffer.scala

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,10 @@ final class FileSegmentManagedBuffer(val file: File, val offset: Long, val lengt
7272
var channel: FileChannel = null
7373
try {
7474
channel = new RandomAccessFile(file, "r").getChannel
75-
channel.map(MapMode.READ_ONLY, offset, length)
75+
val buf = ByteBuffer.allocate(length.toInt)
76+
channel.read(buf, offset)
77+
buf.flip()
78+
buf
7679
} catch {
7780
case e: IOException =>
7881
Try(channel.size).toOption match {

0 commit comments

Comments
 (0)