[SPARK-27176][SQL] Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4 #23788
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1414,6 +1414,23 @@ | |
| <groupId>commons-logging</groupId> | ||
| <artifactId>commons-logging</artifactId> | ||
| </exclusion> | ||
| <!-- Hive 2.3.4 --> | ||
| <exclusion> | ||
|
Member
Author
There was a problem hiding this comment. Exclude build/sbt clean package -Phadoop-3.2 -Phive
...
[error] /home/yumwang/opensource/spark/core/src/main/scala/org/apache/spark/SSLOptions.scala:78: value setTrustStorePath is not a member of org.eclipse.jetty.util.ssl.SslContextFactory
[error] trustStore.foreach(file => sslContextFactory.setTrustStorePath(file.getAbsolutePath))
[error] |
||
| <groupId>org.eclipse.jetty.aggregate</groupId> | ||
| <artifactId>jetty-all</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
|
Member
Author
There was a problem hiding this comment.
build/sbt clean package -Phadoop-3.2 -Phive
...
[error] /home/yumwang/opensource/spark/core/src/main/scala/org/apache/spark/internal/Logging.scala:236: value getLevel is not a member of org.apache.log4j.spi.LoggingEvent
[error] if (!loggingEvent.getLevel().eq(rootLevel)) {
[error] ^
[error] /home/yumwang/opensource/spark/core/src/main/scala/org/apache/spark/internal/Logging.scala:239: value getLogger is not a member of org.apache.log4j.spi.LoggingEvent
[error] var logger = loggingEvent.getLogger() |
||
| <groupId>org.eclipse.jetty.orbit</groupId> | ||
| <artifactId>javax.servlet</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>com.github.joshelser</groupId> | ||
| <artifactId>dropwizard-metrics-hadoop-metrics2-reporter</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.logging.log4j</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
|
wangyum marked this conversation as resolved.
|
||
| </exclusions> | ||
| </dependency> | ||
|
|
||
|
|
@@ -1532,6 +1549,31 @@ | |
| <groupId>org.json</groupId> | ||
| <artifactId>json</artifactId> | ||
|
wangyum marked this conversation as resolved.
|
||
| </exclusion> | ||
| <!-- Hive 2.3.4 --> | ||
| <exclusion> | ||
| <groupId>${hive.group}</groupId> | ||
| <artifactId>hive-vector-code-gen</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>${hive.group}</groupId> | ||
| <artifactId>hive-llap-tez</artifactId> | ||
|
|
||
| </exclusion> | ||
| <exclusion> | ||
|
Member
Author
There was a problem hiding this comment. Exclude |
||
| <groupId>org.apache.orc</groupId> | ||
| <artifactId>orc-tools</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.calcite.avatica</groupId> | ||
| <artifactId>avatica-metrics</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.calcite.avatica</groupId> | ||
| <artifactId>avatica</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.logging.log4j</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
| <dependency> | ||
|
|
@@ -1697,6 +1739,35 @@ | |
| <groupId>org.codehaus.groovy</groupId> | ||
| <artifactId>groovy-all</artifactId> | ||
| </exclusion> | ||
| <!-- Hive 2.3.4 --> | ||
| <exclusion> | ||
|
Member
Author
There was a problem hiding this comment. Exclude build/sbt clean package -Phadoop-3.2 -Phive
...
[error] /home/yumwang/opensource/spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:36: value JobSummaryLevel is not a member of object org.apache.parquet.hadoop.ParquetOutputFormat
[error] import org.apache.parquet.hadoop.ParquetOutputFormat.JobSummaryLevel
Member
There was a problem hiding this comment. These several exclusions would apply to both Hive 2 and Hive 1 in the build as it is now. That's probably OK; maybe they don't even exist in Hive 1. But some like this one I'm not as sure about?
Member
Author
There was a problem hiding this comment. Yes. |
||
| <groupId>org.apache.hbase</groupId> | ||
| <artifactId>hbase-client</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <artifactId>HikariCP</artifactId> | ||
| <groupId>com.zaxxer</groupId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>co.cask.tephra</groupId> | ||
| <artifactId>*</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.eclipse.jetty.aggregate</groupId> | ||
| <artifactId>jetty-all</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.parquet</groupId> | ||
| <artifactId>parquet-hadoop-bundle</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>tomcat</groupId> | ||
| <artifactId>jasper-compiler</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>tomcat</groupId> | ||
| <artifactId>jasper-runtime</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
|
|
@@ -1762,6 +1833,11 @@ | |
| <groupId>org.codehaus.groovy</groupId> | ||
| <artifactId>groovy-all</artifactId> | ||
| </exclusion> | ||
| <!-- Hive 2.3.4 --> | ||
| <exclusion> | ||
|
Member
Author
There was a problem hiding this comment. Exclude $ build/sbt clean package -Phadoop-3.2 -Phive
$ export SPARK_PREPEND_CLASSES=true
$ bin/spark-shell
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/AbstractLoggerAdapter
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:36)
at org.apache.spark.internal.Logging$.org$apache$spark$internal$Logging$$isLog4j12(Logging.scala:217)
at org.apache.spark.internal.Logging.initializeLogging(Logging.scala:122)
at org.apache.spark.internal.Logging.initializeLogIfNecessary(Logging.scala:111)
at org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:105)
at org.apache.spark.deploy.SparkSubmit.initializeLogIfNecessary(SparkSubmit.scala:73)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:939)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:948)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.spi.AbstractLoggerAdapter
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 22 more |
||
| <groupId>org.apache.logging.log4j</groupId> | ||
| <artifactId>log4j-slf4j-impl</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
| <dependency> | ||
|
|
@@ -2656,7 +2732,67 @@ | |
| <hadoop.version>3.2.0</hadoop.version> | ||
| <curator.version>2.13.0</curator.version> | ||
| <zookeeper.version>3.4.13</zookeeper.version> | ||
| <hive.group>org.apache.hive</hive.group> | ||
| <hive.classifier>core</hive.classifier> | ||
| <hive.version>2.3.4</hive.version> | ||
| <hive.version.short>${hive.version}</hive.version.short> | ||
| <hive.parquet.version>${parquet.version}</hive.parquet.version> | ||
| <orc.classifier></orc.classifier> | ||
| <hive.parquet.group>org.apache.parquet</hive.parquet.group> | ||
| <datanucleus-core.version>4.1.17</datanucleus-core.version> | ||
| </properties> | ||
| <dependencies> | ||
| <dependency> | ||
| <groupId>${hive.group}</groupId> | ||
| <artifactId>hive-common</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>${hive.group}</groupId> | ||
| <artifactId>hive-serde</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>${hive.group}</groupId> | ||
| <artifactId>hive-shims</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>${hive.group}</groupId> | ||
| <artifactId>hive-llap-client</artifactId> | ||
|
Member
Author
There was a problem hiding this comment. Need this dependency, otherwise: build/sbt "hive/testOnly *.StatisticsSuite" -Phadoop-3.1
sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/llap/security/LlapSigner$Signable
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:79)
at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:208)
at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:201)
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:500)
at org.apache.spark.sql.hive.test.TestHiveSparkSession.<init>(TestHive.scala:521)
at org.apache.spark.sql.hive.test.TestHiveSparkSession.<init>(TestHive.scala:181)
at org.apache.spark.sql.hive.test.TestHiveContext.<init>(TestHive.scala:129)
at org.apache.spark.sql.hive.test.TestHive$.<init>(TestHive.scala:53)
at org.apache.spark.sql.hive.test.TestHive$.<clinit>(TestHive.scala)
at org.apache.spark.sql.hive.test.TestHiveSingleton.$init$(TestHiveSingleton.scala:30)
at org.apache.spark.sql.hive.StatisticsSuite.<init>(StatisticsSuite.scala:45)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call(ForkMain.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException: org.apache.hadoop.hive.llap.security.LlapSigner$Signable
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:79)
at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:208)
at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:201)
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:500)
at org.apache.spark.sql.hive.test.TestHiveSparkSession.<init>(TestHive.scala:521)
at org.apache.spark.sql.hive.test.TestHiveSparkSession.<init>(TestHive.scala:181)
at org.apache.spark.sql.hive.test.TestHiveContext.<init>(TestHive.scala:129)
at org.apache.spark.sql.hive.test.TestHive$.<init>(TestHive.scala:53)
at org.apache.spark.sql.hive.test.TestHive$.<clinit>(TestHive.scala)
at org.apache.spark.sql.hive.test.TestHiveSingleton.$init$(TestHiveSingleton.scala:30)
at org.apache.spark.sql.hive.StatisticsSuite.<init>(StatisticsSuite.scala:45)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
at sbt.ForkMain$Run$2.call(ForkMain.java:296)
at sbt.ForkMain$Run$2.call(ForkMain.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) |
||
| <version>${hive.version}</version> | ||
| <scope>${hive.deps.scope}</scope> | ||
| <exclusions> | ||
| <exclusion> | ||
| <groupId>${hive.group}</groupId> | ||
| <artifactId>hive-common</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>${hive.group}</groupId> | ||
| <artifactId>hive-serde</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.zookeeper</groupId> | ||
| <artifactId>zookeeper</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.curator</groupId> | ||
| <artifactId>curator-framework</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
|
Member
Author
There was a problem hiding this comment. Refer to Lines 1478 to 1489 in 9c0af74 |
||
| <groupId>org.apache.curator</groupId> | ||
| <artifactId>apache-curator</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.slf4j</groupId> | ||
| <artifactId>slf4j-api</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>org.apache.hive</groupId> | ||
| <artifactId>hive-storage-api</artifactId> | ||
|
wangyum marked this conversation as resolved.
|
||
| <version>2.6.0</version> | ||
|
Member
There was a problem hiding this comment. This matches what 2.3.4 needs, and should it be provided or use hive.deps.scope?
Member
Author
There was a problem hiding this comment. No, Both
scala> spark.range(10).write.saveAsTable("test2")
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:85)
at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDF(Registry.java:177)
at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDF(Registry.java:170)
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:209)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:247)
at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:388)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:332)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:312)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
at org.apache.spark.sql.hive.client.HiveClientImpl.client(HiveClientImpl.scala:258)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:280)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:270)
at org.apache.spark.sql.hive.client.HiveClientImpl.databaseExists(HiveClientImpl.scala:361)
at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$databaseExists$1(HiveExternalCatalog.scala:217)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:217)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:139)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:129)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:40)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.$anonfun$catalog$1(HiveSessionStateBuilder.scala:55)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog$lzycompute(SessionCatalog.scala:90)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog(SessionCatalog.scala:90)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:420)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:446)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:441)
... 47 elided
Caused by: java.lang.reflect.InvocationTargetException: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/io/HiveDecimalWritable
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:83)
... 75 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/io/HiveDecimalWritable
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFFloorCeilBase.<init>(GenericUDFFloorCeilBase.java:48)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFFloor.<init>(GenericUDFFloor.java:41)
... 80 more
scala> spark.range(10).write.orc("test3")
19/04/01 21:47:40 WARN DAGScheduler: Broadcasting large task binary with size 172.4 KiB
[Stage 0:> (0 + 4) / 4]19/04/01 21:47:41 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/vector/ColumnVector
at org.apache.spark.sql.execution.datasources.orc.OrcSerializer.createOrcValue(OrcSerializer.scala:226)
at org.apache.spark.sql.execution.datasources.orc.OrcSerializer.<init>(OrcSerializer.scala:36)
at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:37)
at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:120)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:124)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:109)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:236)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$14(FileFormatWriter.scala:177)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:428)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1321)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:431)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.exec.vector.ColumnVector
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more |
||
| </dependency> | ||
| </dependencies> | ||
| </profile> | ||
|
|
||
| <profile> | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,6 +18,7 @@ | |
| package org.apache.spark.sql.hive | ||
|
|
||
| import java.io.{InputStream, OutputStream} | ||
| import java.lang.reflect.Method | ||
| import java.rmi.server.UID | ||
|
|
||
| import scala.collection.JavaConverters._ | ||
|
|
@@ -28,15 +29,13 @@ import com.google.common.base.Objects | |
| import org.apache.avro.Schema | ||
| import org.apache.hadoop.conf.Configuration | ||
| import org.apache.hadoop.fs.Path | ||
| import org.apache.hadoop.hive.ql.exec.UDF | ||
| import org.apache.hadoop.hive.ql.plan.{FileSinkDesc, TableDesc} | ||
| import org.apache.hadoop.hive.ql.udf.generic.GenericUDFMacro | ||
| import org.apache.hadoop.hive.serde2.ColumnProjectionUtils | ||
| import org.apache.hadoop.hive.serde2.avro.{AvroGenericRecordWritable, AvroSerdeUtils} | ||
| import org.apache.hadoop.hive.serde2.objectinspector.primitive.HiveDecimalObjectInspector | ||
| import org.apache.hadoop.io.Writable | ||
|
|
||
| import org.apache.spark.internal.Logging | ||
| import org.apache.spark.sql.types.Decimal | ||
|
|
@@ -146,34 +145,60 @@ private[hive] object HiveShim { | |
| case _ => false | ||
| } | ||
|
|
||
| private lazy val serUtilClass = | ||
| Utils.classForName("org.apache.hadoop.hive.ql.exec.SerializationUtilities") | ||
| private lazy val utilClass = Utils.classForName("org.apache.hadoop.hive.ql.exec.Utilities") | ||
| private val deserializeMethodName = "deserializeObjectByKryo" | ||
| private val serializeMethodName = "serializeObjectByKryo" | ||
|
|
||
| private def findMethod(klass: Class[_], name: String, args: Class[_]*): Method = { | ||
| val method = klass.getDeclaredMethod(name, args: _*) | ||
| method.setAccessible(true) | ||
| method | ||
| } | ||
|
|
||
| def deserializePlan[UDFType](is: java.io.InputStream, clazz: Class[_]): UDFType = { | ||
| if (HiveUtils.isHive2) { | ||
|
Member
Author
There was a problem hiding this comment. Hive 2.x(HIVE-12302): import org.apache.hadoop.hive.ql.exec.SerializationUtilities
val kryo = SerializationUtilities.borrowKryo()
try {
SerializationUtilities.deserializeObjectByKryo(kryo, is, clazz).asInstanceOf[UDFType]
} finally {
SerializationUtilities.releaseKryo(kryo)
}Hive 1.x: import org.apache.hadoop.hive.ql.exec.Utilities
Utilities.deserializeObjectByKryo(Utilities.runtimeSerializationKryo.get(), is, clazz)
.asInstanceOf[UDFType] |
||
| val borrowKryo = serUtilClass.getMethod("borrowKryo") | ||
| val kryo = borrowKryo.invoke(serUtilClass) | ||
| val deserializeObjectByKryo = findMethod(serUtilClass, deserializeMethodName, | ||
| kryo.getClass.getSuperclass, classOf[InputStream], classOf[Class[_]]) | ||
| try { | ||
| deserializeObjectByKryo.invoke(null, kryo, is, clazz).asInstanceOf[UDFType] | ||
| } finally { | ||
| serUtilClass.getMethod("releaseKryo", kryo.getClass.getSuperclass).invoke(null, kryo) | ||
| } | ||
| } else { | ||
| val runtimeSerializationKryo = utilClass.getField("runtimeSerializationKryo") | ||
| val threadLocalValue = runtimeSerializationKryo.get(utilClass) | ||
| val getMethod = threadLocalValue.getClass.getMethod("get") | ||
| val kryo = getMethod.invoke(threadLocalValue) | ||
| val deserializeObjectByKryo = findMethod(utilClass, deserializeMethodName, | ||
| kryo.getClass, classOf[InputStream], classOf[Class[_]]) | ||
| deserializeObjectByKryo.invoke(null, kryo, is, clazz).asInstanceOf[UDFType] | ||
| } | ||
| } | ||
|
|
||
| def serializePlan(function: AnyRef, out: java.io.OutputStream): Unit = { | ||
| if (HiveUtils.isHive2) { | ||
| val borrowKryo = serUtilClass.getMethod("borrowKryo") | ||
| val kryo = borrowKryo.invoke(serUtilClass) | ||
| val serializeObjectByKryo = findMethod(serUtilClass, serializeMethodName, | ||
| kryo.getClass.getSuperclass, classOf[Object], classOf[OutputStream]) | ||
| try { | ||
| serializeObjectByKryo.invoke(null, kryo, function, out) | ||
| } finally { | ||
| serUtilClass.getMethod("releaseKryo", kryo.getClass.getSuperclass).invoke(null, kryo) | ||
| } | ||
| } else { | ||
| val runtimeSerializationKryo = utilClass.getField("runtimeSerializationKryo") | ||
| val threadLocalValue = runtimeSerializationKryo.get(utilClass) | ||
| val getMethod = threadLocalValue.getClass.getMethod("get") | ||
| val kryo = getMethod.invoke(threadLocalValue) | ||
| val serializeObjectByKryo = findMethod(utilClass, serializeMethodName, | ||
| kryo.getClass, classOf[Object], classOf[OutputStream]) | ||
| serializeObjectByKryo.invoke(null, kryo, function, out) | ||
| } | ||
| } | ||
|
|
||
| def writeExternal(out: java.io.ObjectOutput) { | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.