Skip to content

Commit fe1d1e2

Browse files
HeartSaVioRdongjoon-hyun
authored andcommitted
[SPARK-31214][BUILD] Upgrade Janino to 3.1.2
### What changes were proposed in this pull request? This PR proposes to upgrade Janino to 3.1.2 which is released recently. Major changes were done for refactoring, as well as there're lots of "no commit message". Belows are the pairs of (commit title, commit) which seem to deal with some bugs or specific improvements (not purposed to refactor) after 3.0.15. * Issue #119: Guarantee executing popOperand() in popUninitializedVariableOperand() via moving popOperand() out of "assert" * Issue #116: replace operand to final target type if boxing conversion and widening reference conversion happen together * Merged pull request `#114` "Grow the code for relocatables, and do fixup, and relocate". * janino-compiler/janino@367c58e * issue `#107`: Janino requires "org.codehaus.commons.compiler.io", but commons-compiler does not export this package * janino-compiler/janino@f7d9959 * Throw an NYI CompileException when a static interface method is invoked. * janino-compiler/janino@efd3884 * Fixed the promotion of the array access index expression (see JLS7 15.13 Array Access Expressions) * janino-compiler/janino@32fdb5f * Issue `#104`: ClassLoaderIClassLoader 's ClassNotFoundException handle mechanism enhancement * janino-compiler/janino@6e8a97d You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html ### Why are the changes needed? We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details. Janino 3.1.1 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly. I've also fixed a couple of more bugs as 3.1.1 made Spark UTs fail - hence we need to upgrade to 3.1.2. Furthermore, from my testing, janino-compiler/janino#90 (which Josh Rosen filed before) seems to be also resolved in 3.1.2 as well. Looks like Janino is maintained by one person and there's no even version branches and releases/tags so we can't expect Janino maintainer to release a new bugfix version - hence have to try out new minor version. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing UTs. Closes #27860 from HeartSaVioR/SPARK-31101. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 91148f4 commit fe1d1e2

File tree

5 files changed

+14
-12
lines changed

5 files changed

+14
-12
lines changed

dev/deps/spark-deps-hadoop-2.7-hive-1.2

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar
3535
commons-cli/1.2//commons-cli-1.2.jar
3636
commons-codec/1.10//commons-codec-1.10.jar
3737
commons-collections/3.2.2//commons-collections-3.2.2.jar
38-
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
38+
commons-compiler/3.1.2//commons-compiler-3.1.2.jar
3939
commons-compress/1.8.1//commons-compress-1.8.1.jar
4040
commons-configuration/1.6//commons-configuration-1.6.jar
4141
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
@@ -106,7 +106,7 @@ jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
106106
jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar
107107
jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar
108108
jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
109-
janino/3.0.16//janino-3.0.16.jar
109+
janino/3.1.2//janino-3.1.2.jar
110110
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
111111
javax.inject/1//javax.inject-1.jar
112112
javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar

dev/deps/spark-deps-hadoop-2.7-hive-2.3

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar
3333
commons-cli/1.2//commons-cli-1.2.jar
3434
commons-codec/1.10//commons-codec-1.10.jar
3535
commons-collections/3.2.2//commons-collections-3.2.2.jar
36-
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
36+
commons-compiler/3.1.2//commons-compiler-3.1.2.jar
3737
commons-compress/1.8.1//commons-compress-1.8.1.jar
3838
commons-configuration/1.6//commons-configuration-1.6.jar
3939
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
@@ -119,7 +119,7 @@ jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
119119
jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar
120120
jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar
121121
jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
122-
janino/3.0.16//janino-3.0.16.jar
122+
janino/3.1.2//janino-3.1.2.jar
123123
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
124124
javax.inject/1//javax.inject-1.jar
125125
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar

dev/deps/spark-deps-hadoop-3.2-hive-2.3

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar
3030
commons-cli/1.2//commons-cli-1.2.jar
3131
commons-codec/1.10//commons-codec-1.10.jar
3232
commons-collections/3.2.2//commons-collections-3.2.2.jar
33-
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
33+
commons-compiler/3.1.2//commons-compiler-3.1.2.jar
3434
commons-compress/1.8.1//commons-compress-1.8.1.jar
3535
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
3636
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
@@ -118,7 +118,7 @@ jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
118118
jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar
119119
jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar
120120
jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
121-
janino/3.0.16//janino-3.0.16.jar
121+
janino/3.1.2//janino-3.1.2.jar
122122
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
123123
javax.inject/1//javax.inject-1.jar
124124
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar

pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@
179179
<commons-pool2.version>2.6.2</commons-pool2.version>
180180
<datanucleus-core.version>4.1.17</datanucleus-core.version>
181181
<guava.version>14.0.1</guava.version>
182-
<janino.version>3.0.16</janino.version>
182+
<janino.version>3.1.2</janino.version>
183183
<jersey.version>2.30</jersey.version>
184184
<joda.version>2.10.5</joda.version>
185185
<jodd.version>3.5.2</jodd.version>

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,9 @@ import scala.util.control.NonFatal
2727

2828
import com.google.common.cache.{CacheBuilder, CacheLoader}
2929
import com.google.common.util.concurrent.{ExecutionError, UncheckedExecutionException}
30-
import org.codehaus.commons.compiler.CompileException
31-
import org.codehaus.janino.{ByteArrayClassLoader, ClassBodyEvaluator, InternalCompilerException, SimpleCompiler}
30+
import org.codehaus.commons.compiler.{CompileException, InternalCompilerException}
31+
import org.codehaus.commons.compiler.util.reflect.ByteArrayClassLoader
32+
import org.codehaus.janino.{ClassBodyEvaluator, SimpleCompiler}
3233
import org.codehaus.janino.util.ClassFile
3334

3435
import org.apache.spark.{TaskContext, TaskKilledException}
@@ -1419,9 +1420,10 @@ object CodeGenerator extends Logging {
14191420
private def updateAndGetCompilationStats(evaluator: ClassBodyEvaluator): ByteCodeStats = {
14201421
// First retrieve the generated classes.
14211422
val classes = {
1422-
val resultField = classOf[SimpleCompiler].getDeclaredField("result")
1423-
resultField.setAccessible(true)
1424-
val loader = resultField.get(evaluator).asInstanceOf[ByteArrayClassLoader]
1423+
val scField = classOf[ClassBodyEvaluator].getDeclaredField("sc")
1424+
scField.setAccessible(true)
1425+
val compiler = scField.get(evaluator).asInstanceOf[SimpleCompiler]
1426+
val loader = compiler.getClassLoader.asInstanceOf[ByteArrayClassLoader]
14251427
val classesField = loader.getClass.getDeclaredField("classes")
14261428
classesField.setAccessible(true)
14271429
classesField.get(loader).asInstanceOf[JavaMap[String, Array[Byte]]].asScala

0 commit comments

Comments
 (0)