Skip to content
This repository was archived by the owner on May 9, 2024. It is now read-only.

Commit 39e4e7e

Browse files
eglp-slindemannliancheng
authored andcommitted
[SPARK-8841] [SQL] Fix partition pruning percentage log message
When pruning partitions for a query plan, a message is logged indicating what how many partitions were selected based on predicate criteria, and what percent were pruned. The current release erroneously uses `1 - total/selected` to compute this quantity, leading to nonsense messages like "pruned -1000% partitions". The fix is simple and obvious. Author: Steve Lindemann <[email protected]> Closes apache#7227 from srlindemann/master and squashes the following commits: c788061 [Steve Lindemann] fix percentPruned log message
1 parent 86768b7 commit 39e4e7e

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/sources/DataSourceStrategy.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ private[sql] object DataSourceStrategy extends Strategy with Logging {
6565
logInfo {
6666
val total = t.partitionSpec.partitions.length
6767
val selected = selectedPartitions.length
68-
val percentPruned = (1 - total.toDouble / selected.toDouble) * 100
68+
val percentPruned = (1 - selected.toDouble / total.toDouble) * 100
6969
s"Selected $selected partitions out of $total, pruned $percentPruned% partitions."
7070
}
7171

0 commit comments

Comments
 (0)