Skip to content

Commit f77270b

Browse files
10110346srowen
authored andcommitted
[SPARK-23358][CORE] When the number of partitions is greater than 2^28, it will result in an error result
## What changes were proposed in this pull request? In the `checkIndexAndDataFile`,the `blocks` is the ` Int` type, when it is greater than 2^28, `blocks*8` will overflow, and this will result in an error result. In fact, `blocks` is actually the number of partitions. ## How was this patch tested? Manual test Author: liuxian <[email protected]> Closes #20544 from 10110346/overflow.
1 parent 4b4ee26 commit f77270b

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ private[spark] class IndexShuffleBlockResolver(
8484
*/
8585
private def checkIndexAndDataFile(index: File, data: File, blocks: Int): Array[Long] = {
8686
// the index file should have `block + 1` longs as offset.
87-
if (index.length() != (blocks + 1) * 8) {
87+
if (index.length() != (blocks + 1) * 8L) {
8888
return null
8989
}
9090
val lengths = new Array[Long](blocks)

0 commit comments

Comments
 (0)