Skip to content

[SPARK-27671][SQL] Fix error when casting from a nested null in a struct #24576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 3 commits into from
Closed

[SPARK-27671][SQL] Fix error when casting from a nested null in a struct #24576

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
397409b
Fix error when casting from a nested null in a struct.
viirya May 10, 2019
d38b371
Address comment.
viirya May 11, 2019
a5c2416
Add one more test.
viirya May 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -620,6 +620,12 @@ case class Cast(child: Expression, dataType: DataType, timeZoneId: Option[String
// We can return what the children return. Same thing should happen in the codegen path.
if (DataType.equalsStructurally(from, to)) {
identity
} else if (from == NullType) {
// According to `canCast`, NullType can be casted to any type.
// For primitive types, we don't reach here because the guard of `nullSafeEval`.
// But for nested types like struct, we might reach here for nested null type field.
// We won't call the returned function actually, but returns a placeholder.
_ => throw new SparkException(s"should not directly cast from NullType to $to.")
Copy link
Contributor

@cloud-fan cloud-fan May 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when will we hit this exception?

Copy link
Member Author

@viirya viirya May 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case of casting a nested field in a struct or array, if we find it is a null, we don't call the cast function but set the destination as null directly. If we call this cast function, we hit this exception.

Copy link
Member Author

@viirya viirya May 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do the thing as follow:

buildCast[InternalRow](_, row => {
  val newRow = new GenericInternalRow(from.fields.length)
  var i = 0
  while (i < row.numFields) {
    newRow.update(i,
      // We don't call cast function but directly set a null, if finding a null.
      if (row.isNullAt(i)) null else castFuncs(i)(row.get(i, from.apply(i).dataType)))
    i += 1
  }
  newRow
})

Copy link
Contributor

@cloud-fan cloud-fan May 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which means, it's a bug if the exception is thrown?

Copy link
Member Author

@viirya viirya May 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If reaching here, means we call the cast function for a null, unnecessarily.

} else {
to match {
case dt if dt == from => identity[Any]
Expand Down
15 changes: 15 additions & 0 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -990,4 +990,19 @@ class CastSuite extends SparkFunSuite with ExpressionEvalHelper {
}
}
}

test("SPARK-27671: cast from nested null type in struct") {
import DataTypeTestUtils._

atomicTypes.foreach { atomicType =>
val struct = Literal.create(
InternalRow(null),
StructType(Seq(StructField("a", NullType, nullable = true))))

val ret = cast(struct, StructType(Seq(
StructField("a", atomicType, nullable = true))))
assert(ret.resolved)
checkEvaluation(ret, InternalRow(null))
}
}
}
9 changes: 9 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2157,4 +2157,13 @@ class DataFrameSuite extends QueryTest with SharedSQLContext {
|*(1) Range (0, 10, step=1, splits=2)""".stripMargin))
}
}

test("SPARK-27671: Fix analysis exception when casting null in nested field in struct") {
val df = sql("SELECT * FROM VALUES (('a', (10, null))), (('b', (10, 50))), " +
"(('c', null)) AS tab(x, y)")
checkAnswer(df, Row("a", Row(10, null)) :: Row("b", Row(10, 50)) :: Row("c", null) :: Nil)

val cast = sql("SELECT cast(struct(1, null) AS struct<a:int,b:int>)")
checkAnswer(cast, Row(Row(1, null)) :: Nil)
}
}