Skip to content
Closed
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,8 @@ class Analyzer(
Batch("Nondeterministic", Once,
PullOutNondeterministic),
Batch("UDF", Once,
HandleNullInputsForUDF),
HandleNullInputsForUDF,
ResolveEncodersInUDF),
Batch("UpdateNullability", Once,
UpdateAttributeNullability),
Batch("Subquery", Once,
Expand Down Expand Up @@ -2847,6 +2848,45 @@ class Analyzer(
}
}

/**
* Resolve the encoders for the UDF by explicitly given the attributes. We give the
* attributes explicitly in order to handle the case where the data type of the input
* value is not the same with the internal schema of the encoder, which could cause
* data loss. For example, the encoder should not cast the input value to Decimal(38, 18)
* if the actual data type is Decimal(30, 0).
*
* The resolved encoders then will be used to deserialize the internal row to Scala value.
*/
object ResolveEncodersInUDF extends Rule[LogicalPlan] {
Comment thread
cloud-fan marked this conversation as resolved.
override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
case p if !p.resolved => p // Skip unresolved nodes.

case p => p transformExpressionsUp {

case udf @ ScalaUDF(_, _, inputs, encoders, _, _, _) if encoders.nonEmpty =>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we avoid argument matching? It's actually an anti-pattern - https://github.com/databricks/scala-style-guide#pattern-matching

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks!

val resolvedEncoders = encoders.zipWithIndex.map { case (encOpt, i) =>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe to call it boundEncoders.

val dataType = inputs(i).dataType
if (dataType.isInstanceOf[UserDefinedType[_]]) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about struct/array/map of UDT?

// for UDT, we use `CatalystTypeConverters`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

encoder does support UDT. We can figure it out later.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, but just doesn't support upcast from the subclass to the parent class. So, when the input data type from the child is the subclass of the input parameter data type of the udf, resolveAndBind can fail.

I think this may need a separate fix.

None
} else {
encOpt.map { enc =>
val attrs = if (enc.isSerializedAsStructForTopLevel) {
dataType.asInstanceOf[StructType].toAttributes
} else {
// the field name doesn't matter here, so we use
// a simple literal to avoid any overhead
new StructType().add("input", dataType).toAttributes
}
Comment thread
cloud-fan marked this conversation as resolved.
enc.resolveAndBind(attrs)
}
}
}
udf.copy(inputEncoders = resolvedEncoders)
}
}
}

/**
* Check and add proper window frames for all window functions.
*/
Expand Down
Loading