Skip to content

[SPARK-24762][SQL][Followup] Enable Option of Product encoders #23143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 2 commits into from
Closed

[SPARK-24762][SQL][Followup] Enable Option of Product encoders #23143

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f6bfe3d
Inline isOptionType.
viirya Nov 26, 2018
f3f0507
Fix style.
viirya Nov 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,19 +251,16 @@ case class ExpressionEncoder[T](
*/
def isSerializedAsStruct: Boolean = objSerializer.dataType.isInstanceOf[StructType]

/**
* Returns true if the type `T` is an `Option` type.
*/
def isOptionType: Boolean = classOf[Option[_]].isAssignableFrom(clsTag.runtimeClass)

/**
* If the type `T` is serialized as a struct, when it is encoded to a Spark SQL row, fields in
* the struct are naturally mapped to top-level columns in a row. In other words, the serialized
* struct is flattened to row. But in case of the `T` is also an `Option` type, it can't be
* flattened to top-level row, because in Spark SQL top-level row can't be null. This method
* returns true if `T` is serialized as struct and is not `Option` type.
*/
def isSerializedAsStructForTopLevel: Boolean = isSerializedAsStruct && !isOptionType
def isSerializedAsStructForTopLevel: Boolean = {
isSerializedAsStruct && !classOf[Option[_]].isAssignableFrom(clsTag.runtimeClass)
}

// serializer expressions are used to encode an object to a row, while the object is usually an
// intermediate value produced inside an operator, not from the output of the child operator. This
Expand Down