Skip to content

[SPARK-32159][SQL] Fix integration between Aggregator[Array[_], _, _] and UnresolvedMapObjects #28983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
erikerlandson wants to merge 19 commits into from
Closed

[SPARK-32159][SQL] Fix integration between Aggregator[Array[_], _, _] and UnresolvedMapObjects #28983

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
1a501d9
fix SPARK-32159 - intercept null function in MapObjects
erikerlandson Jul 2, 2020
73299e8
revert println
erikerlandson Jul 5, 2020
20012b3
add informative exception for null function
erikerlandson Jul 5, 2020
a8dd23d
move resolution of ScalaAggregator inputEncoder to new rule ResolveEn…
erikerlandson Jul 5, 2020
bc2d880
add ResolveEncodersInScalaAgg rule to TestHive
erikerlandson Jul 5, 2020
399cbab
add unit test for array input type
erikerlandson Jul 5, 2020
2139c14
add ResolveEncodersInScalaAgg rule to HiveSessionStateBuilder
erikerlandson Jul 6, 2020
2092b3a
revert session extension in TestHive
erikerlandson Jul 6, 2020
ac22ccf
revert spacing
erikerlandson Jul 6, 2020
1351b76
revert space
erikerlandson Jul 7, 2020
e923d2f
move jira into test name
erikerlandson Jul 7, 2020
a4858d5
add comment to new rule
erikerlandson Jul 7, 2020
d3c5d4d
make ResolveEncodersInScalaAgg an object
erikerlandson Jul 7, 2020
814956c
add bufferEncoder as a parameter to ScalaAggregator to make resolutio…
erikerlandson Jul 7, 2020
aca7b51
test Array type with input, buffer and output encoders
erikerlandson Jul 7, 2020
e679c01
more detail on array test name
erikerlandson Jul 8, 2020
c632437
add type link to scaladoc
erikerlandson Jul 8, 2020
ee96cc0
improved exception message and corresponding comments
erikerlandson Jul 8, 2020
622ac1c
too much 'an'
erikerlandson Jul 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3066,9 +3066,12 @@ class Analyzer(
inputAttributes
}

// println(s"\n\ndeserializer= $deserializer\n")
// println(s"\n\ninputs= $inputs\n")
Comment thread
erikerlandson marked this conversation as resolved.
Outdated
validateTopLevelTupleFields(deserializer, inputs)
val resolved = resolveExpressionBottomUp(
deserializer, LocalRelation(inputs), throws = true)
// println(s"\n\nresolved= $resolved\n")
val result = resolved transformDown {
case UnresolvedMapObjects(func, inputData, cls) if inputData.resolved =>
inputData.dataType match {
Expand Down
5 changes: 4 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -679,7 +679,10 @@ object MapObjects {
elementNullable: Boolean = true,
customCollectionCls: Option[Class[_]] = None): MapObjects = {
val loopVar = LambdaVariable("MapObject", elementType, elementNullable)
MapObjects(loopVar, function(loopVar), inputData, customCollectionCls)
// println(s"\n\n $function, $inputData, $elementType, $elementNullable, $customCollectionCls \n")
val fOfLV = if (function == null) loopVar else function(loopVar)
Comment thread
erikerlandson marked this conversation as resolved.
Outdated
// MapObjects(loopVar, function(loopVar), inputData, customCollectionCls)
MapObjects(loopVar, fOfLV, inputData, customCollectionCls)
Comment thread
erikerlandson marked this conversation as resolved.
Outdated
}
}

Expand Down