Skip to content

Commit a8f5a0f

Browse files
committed
Fix
1 parent 3ec5cae commit a8f5a0f

File tree

2 files changed

+10
-8
lines changed

2 files changed

+10
-8
lines changed

docs/sql-programming-guide.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1783,6 +1783,8 @@ options.
17831783

17841784
- Since Spark 2.3, when all inputs are binary, `functions.concat()` returns an output as binary. Otherwise, it returns as a string. Until Spark 2.3, it always returns as a string despite of input types. To keep the old behavior, set `spark.sql.function.concatBinaryAsString` to `true`.
17851785

1786+
- Since Spark 2.3, when all inputs are binary, SQL `elt()` returns an output as binary. Otherwise, it returns as a string. Until Spark 2.3, it always returns as a string despite of input types. To keep the old behavior, set `spark.sql.function.eltOutputAsString` to `true`.
1787+
17861788
## Upgrading From Spark SQL 2.1 to 2.2
17871789

17881790
- Spark 2.1.1 introduced a new configuration key: `spark.sql.hive.caseSensitiveInferenceMode`. It had a default setting of `NEVER_INFER`, which kept behavior identical to 2.1.0. However, Spark 2.2.0 changes this setting's default value to `INFER_AND_SAVE` to restore compatibility with reading Hive metastore tables whose underlying file schema have mixed-case column names. With the `INFER_AND_SAVE` configuration value, on first access Spark will perform schema inference on any Hive metastore table for which it has not already saved an inferred schema. Note that schema inference can be a very time consuming operation for tables with thousands of partitions. If compatibility with mixed-case column names is not a concern, you can safely set `spark.sql.hive.caseSensitiveInferenceMode` to `NEVER_INFER` to avoid the initial overhead of schema inference. Note that with the new default `INFER_AND_SAVE` setting, the results of the schema inference are saved as a metastore key for future use. Therefore, the initial schema inference occurs only at a table's first access.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -697,18 +697,18 @@ object TypeCoercion {
697697
p transformExpressionsUp {
698698
// Skip nodes if unresolved or not enough children
699699
case c @ Elt(children) if !c.childrenResolved || children.size < 2 => c
700-
case c @ Elt(children) if conf.eltOutputAsString ||
701-
!children.tail.map(_.dataType).forall(_ == BinaryType) =>
700+
case c @ Elt(children) =>
702701
val index = children.head
703702
val newIndex = ImplicitTypeCasts.implicitCast(index, IntegerType).getOrElse(index)
704-
val newInputs = children.tail.map { e =>
705-
ImplicitTypeCasts.implicitCast(e, StringType).getOrElse(e)
703+
val newInputs = if (conf.eltOutputAsString ||
704+
!children.tail.map(_.dataType).forall(_ == BinaryType)) {
705+
children.tail.map { e =>
706+
ImplicitTypeCasts.implicitCast(e, StringType).getOrElse(e)
707+
}
708+
} else {
709+
children.tail
706710
}
707711
c.copy(children = newIndex +: newInputs)
708-
case c @ Elt(children) =>
709-
val index = children.head
710-
val newIndex = ImplicitTypeCasts.implicitCast(index, IntegerType).getOrElse(index)
711-
c.copy(children = newIndex +: children.tail)
712712
}
713713
}
714714
}

0 commit comments

Comments
 (0)