Skip to content

Commit 4c8403e

Browse files
rednaxelafxHyukjinKwon
authored andcommitted
[SPARK-34596][SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in NewInstance.doGenCode
### What changes were proposed in this pull request? Use `Utils.getSimpleName` to avoid hitting `Malformed class name` error in `NewInstance.doGenCode`. ### Why are the changes needed? On older JDK versions (e.g. JDK8u), nested Scala classes may trigger `java.lang.Class.getSimpleName` to throw an `java.lang.InternalError: Malformed class name` error. In this particular case, creating an `ExpressionEncoder` on such a nested Scala class would create a `NewInstance` expression under the hood, which will trigger the problem during codegen. Similar to #29050, we should use Spark's `Utils.getSimpleName` utility function in place of `Class.getSimpleName` to avoid hitting the issue. There are two other occurrences of `java.lang.Class.getSimpleName` in the same file, but they're safe because they're only guaranteed to be only used on Java classes, which don't have this problem, e.g.: ```scala // Make a copy of the data if it's unsafe-backed def makeCopyIfInstanceOf(clazz: Class[_ <: Any], value: String) = s"$value instanceof ${clazz.getSimpleName}? ${value}.copy() : $value" val genFunctionValue: String = lambdaFunction.dataType match { case StructType(_) => makeCopyIfInstanceOf(classOf[UnsafeRow], genFunction.value) case ArrayType(_, _) => makeCopyIfInstanceOf(classOf[UnsafeArrayData], genFunction.value) case MapType(_, _, _) => makeCopyIfInstanceOf(classOf[UnsafeMapData], genFunction.value) case _ => genFunction.value } ``` The Unsafe-* family of types are all Java types, so they're okay. ### Does this PR introduce _any_ user-facing change? Fixes a bug that throws an error when using `ExpressionEncoder` on some nested Scala types, otherwise no changes. ### How was this patch tested? Added a test case to `org.apache.spark.sql.catalyst.encoders.ExpressionEncoderSuite`. It'll fail on JDK8u before the fix, and pass after the fix. Closes #31709 from rednaxelafx/spark-34596-master. Authored-by: Kris Mok <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit ecf4811) Signed-off-by: HyukjinKwon <[email protected]>
1 parent 28f2d51 commit 4c8403e

File tree

2 files changed

+13
-1
lines changed

2 files changed

+13
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -489,7 +489,7 @@ case class NewInstance(
489489
// that might be defined on the companion object.
490490
case 0 => s"$className$$.MODULE$$.apply($argString)"
491491
case _ => outer.map { gen =>
492-
s"${gen.value}.new ${cls.getSimpleName}($argString)"
492+
s"${gen.value}.new ${Utils.getSimpleName(cls)}($argString)"
493493
}.getOrElse {
494494
s"new $className($argString)"
495495
}

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,18 @@ class ExpressionEncoderSuite extends CodegenInterpretedPlanTest with AnalysisTes
205205

206206
encodeDecodeTest(Array(Option(InnerClass(1))), "array of optional inner class")
207207

208+
// holder class to trigger Class.getSimpleName issue
209+
object MalformedClassObject extends Serializable {
210+
case class MalformedNameExample(x: Int)
211+
}
212+
213+
{
214+
OuterScopes.addOuterScope(MalformedClassObject)
215+
encodeDecodeTest(
216+
MalformedClassObject.MalformedNameExample(42),
217+
"nested Scala class should work")
218+
}
219+
208220
productTest(PrimitiveData(1, 1, 1, 1, 1, 1, true))
209221

210222
productTest(

0 commit comments

Comments
 (0)