Skip to content

Commit b4d5441

Browse files
viiryaliancheng
authored andcommitted
[SPARK-8141] [SQL] Precompute datatypes for partition columns and reuse it
JIRA: https://issues.apache.org/jira/browse/SPARK-8141 Author: Liang-Chi Hsieh <[email protected]> Closes #6687 from viirya/reuse_partition_column_types and squashes the following commits: dab0688 [Liang-Chi Hsieh] Reuse partitionColumnTypes. (cherry picked from commit 26d07f1) Signed-off-by: Cheng Lian <[email protected]>
1 parent 9d1f4d6 commit b4d5441

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -435,8 +435,9 @@ abstract class HadoopFsRelation private[sql](maybePartitionSpec: Option[Partitio
435435
// partition values.
436436
userDefinedPartitionColumns.map { partitionSchema =>
437437
val spec = discoverPartitions()
438+
val partitionColumnTypes = spec.partitionColumns.map(_.dataType)
438439
val castedPartitions = spec.partitions.map { case p @ Partition(values, path) =>
439-
val literals = values.toSeq.zip(spec.partitionColumns.map(_.dataType)).map {
440+
val literals = values.toSeq.zip(partitionColumnTypes).map {
440441
case (value, dataType) => Literal.create(value, dataType)
441442
}
442443
val castedValues = partitionSchema.zip(literals).map { case (field, literal) =>

0 commit comments

Comments
 (0)