[DO-NOT-MERGE] Try to update Hive to 2.3.2 #20659

wangyum · 2018-02-22T07:04:30Z

What changes were proposed in this pull request?

Check if there is any test failed.

* Update Hive to 2.3.2

SparkQA · 2018-02-22T07:10:30Z

Test build #87611 has finished for PR 20659 at commit 915e68f.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-02-22T10:12:18Z

Test build #87613 has finished for PR 20659 at commit a5bb731.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-02-22T14:03:33Z

Test build #87616 has finished for PR 20659 at commit 80fd8a8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

# Conflicts: # sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala # sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala # sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala # sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala # sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala

SparkQA · 2018-03-05T17:08:10Z

Test build #87960 has finished for PR 20659 at commit 1110ede.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-03-06T05:17:12Z

Nice try! Could you fix the remaining failure?

wangyum · 2018-03-06T07:33:10Z

Yes, I'm doing it

dongjoon-hyun · 2018-03-06T17:01:19Z

sql/core/pom.xml

+    <dependency>
+      <groupId>org.apache.hive</groupId>
+      <artifactId>hive-storage-api</artifactId>
+    </dependency>


BTW, you should not introduce hive into sql/core.
Apache ORC should be used as Apache ORC.

dongjoon-hyun · 2018-03-06T17:01:36Z

sql/core/pom.xml

    <dependency>
      <groupId>org.apache.orc</groupId>
      <artifactId>orc-core</artifactId>
-      <classifier>${orc.classifier}</classifier>


Please don't remove this.

dongjoon-hyun · 2018-03-06T17:02:30Z

...ore/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java

 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.ql.exec.vector.*;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;


Please revert this.

dongjoon-hyun · 2018-03-06T17:03:56Z

sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala

    withTable("jt2") {
      withView("testView") {
-        sql("CREATE VIEW testView AS SELECT id FROM jt")
+        sql("CREATE VIEW testView AS SELECT 1 as c1 FROM jt")


It is unrelated change and looks wrong.

dongjoon-hyun · 2018-03-06T17:06:20Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

+            sql("ALTER TABLE t1 ADD COLUMNS (C2 string)")
            assert(spark.table("t1").schema ==
-              new StructType().add("c1", IntegerType).add("C1", StringType))
+              new StructType().add("c1", IntegerType).add("C2", StringType))


You are intentionally removing case sensitive test.

dongjoon-hyun · 2018-03-06T17:08:23Z

In general, this should keep all existing test coverage. Otherwise, it will not considered as a feasible solution.

HyukjinKwon · 2018-03-09T13:22:18Z

just out of curiosity, what does DNM stands for?

dongjoon-hyun · 2018-03-09T17:37:54Z

I guess Do Not Merge. Actually, this is a try to see the result. :)

gatorsmile · 2018-03-09T17:48:26Z

This is a very interesting work. Thanks for your effort!

When we implement such a feature, please remember we have to ensure it is configurable.

jerryshao · 2018-03-12T08:09:48Z

@gatorsmile @cloud-fan what is our plan for such thing? Currently we're using our own forked version, which blocks the support of Hadoop 3 (SPARK-18673). I'm wondering if we're going to upgrade Hive like what is done in this PR, or we still uses forked branch, and patch that branch.

What is your thought and plan on it?

jerryshao · 2018-03-12T08:12:25Z

@wangyum can you please create a relative JIRA, so that we discuss on the JIRA. I think it is better to discuss and get a conclusion first.

SparkQA · 2018-03-13T16:30:15Z

Test build #88206 has finished for PR 20659 at commit b35daa0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-03-14T01:22:29Z

Test build #88217 has finished for PR 20659 at commit b418909.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-03-14T01:31:05Z

retest this please

SparkQA · 2018-03-15T12:07:31Z

Test build #88260 has finished for PR 20659 at commit 03138d8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-03-15T19:03:16Z

Test build #88269 has finished for PR 20659 at commit e041704.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-03-15T20:03:50Z

retest this please

SparkQA · 2018-03-16T00:20:34Z

Test build #88275 has finished for PR 20659 at commit e041704.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-03-16T11:13:11Z

Test build #88299 has finished for PR 20659 at commit b92918b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-03-16T12:50:37Z

Test build #88310 has finished for PR 20659 at commit 71bcb25.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-03-16T16:08:22Z

Test build #88311 has finished for PR 20659 at commit 75e4cc9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-03-16T16:20:50Z

retest this please

SparkQA · 2018-03-16T20:21:08Z

Test build #88317 has finished for PR 20659 at commit 75e4cc9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-03-16T23:39:11Z

Thanks everyone, we can move to SPARK-23710 to discuss.

dongjoon-hyun · 2018-03-17T01:24:53Z

@wangyum .
Please read again my previous comment.

Although I support your efforts, Spark must provide ORC data source without -Phive.

dongjoon-hyun · 2018-03-17T01:26:46Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcDeserializer.scala


 package org.apache.spark.sql.execution.datasources.orc

+import org.apache.hadoop.hive.serde2.io.{DateWritable, HiveDecimalWritable}


Please revert this.

dongjoon-hyun · 2018-03-17T01:26:55Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala

-import org.apache.orc.storage.serde2.io.HiveDecimalWritable
+import org.apache.hadoop.hive.ql.io.sarg.{PredicateLeaf, SearchArgument, SearchArgumentFactory}
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument.Builder
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable


Please revert this.

dongjoon-hyun · 2018-03-17T01:27:05Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcSerializer.scala

 import org.apache.orc.TypeDescription
 import org.apache.orc.mapred.{OrcList, OrcMap, OrcStruct, OrcTimestamp}
-import org.apache.orc.storage.common.`type`.HiveDecimal
-import org.apache.orc.storage.serde2.io.{DateWritable, HiveDecimalWritable}


Please revert this.

dongjoon-hyun · 2018-03-17T01:27:23Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala

 import scala.collection.JavaConverters._

-import org.apache.orc.storage.ql.io.sarg.{PredicateLeaf, SearchArgument}
+import org.apache.hadoop.hive.ql.io.sarg.{PredicateLeaf, SearchArgument}


Please revert this.

dongjoon-hyun · 2018-03-17T01:28:57Z

I added more specific comments about ORC parts, @wangyum .

SparkQA · 2018-03-17T07:05:02Z

Test build #88337 has finished for PR 20659 at commit e4c0624.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-03-17T07:07:40Z

retest this please

SparkQA · 2018-03-17T11:28:33Z

Test build #88338 has finished for PR 20659 at commit e4c0624.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Bidek56 · 2018-03-28T00:48:09Z

What will take to merge this PR in? It has passed all the tests.

srowen · 2018-04-23T13:59:02Z

@wangyum you can close this experiment?

Update Hive to 2.3.2

915e68f

* Update Hive to 2.3.2

replace manifest

a5bb731

Fix javaunidoc error

80fd8a8

wangyum closed this Feb 22, 2018

wangyum reopened this Mar 5, 2018

dongjoon-hyun reviewed Mar 6, 2018

View reviewed changes

wangyum changed the title ~~[DNM] Try to update Hive to 2.3.2~~ [DO-NOT-MERGE] Try to update Hive to 2.3.2 Mar 10, 2018

Fix dependency

566fa59

Add org.apache.derby.* to shared class

b35daa0

ignore backward compatibility

b418909

ignore SPARK-23598 test

e041704

wangyum added 2 commits March 16, 2018 15:00

Merge remote-tracking branch 'upstream/master' into hive-2.3.2-jenkins

133a628

Hive.getDelegationToken -> HiveMetaStoreClient.getDelegationToken

b92918b

Revert derby.version to 10.12.1.1

71bcb25

Revert derby.version to 10.12.1.1

75e4cc9

dongjoon-hyun reviewed Mar 17, 2018

View reviewed changes

Change Hive dependency to provided

e4c0624

wangyum closed this Apr 23, 2018

HyukjinKwon mentioned this pull request Oct 13, 2018

[SPARK-24590][BUILD] Make Jenkins tests passed with hadoop 3 profile #21588

Closed

wangyum deleted the hive-2.3.x branch September 10, 2019 05:31


		package org.apache.spark.sql.execution.datasources.orc

		import org.apache.hadoop.hive.serde2.io.{DateWritable, HiveDecimalWritable}

[DO-NOT-MERGE] Try to update Hive to 2.3.2 #20659

[DO-NOT-MERGE] Try to update Hive to 2.3.2 #20659

Uh oh!

Conversation

wangyum commented Feb 22, 2018

What changes were proposed in this pull request?

Uh oh!

SparkQA commented Feb 22, 2018

Uh oh!

SparkQA commented Feb 22, 2018

Uh oh!

SparkQA commented Feb 22, 2018

Uh oh!

SparkQA commented Mar 5, 2018

Uh oh!

dongjoon-hyun commented Mar 6, 2018

Uh oh!

wangyum commented Mar 6, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 6, 2018

Uh oh!

HyukjinKwon commented Mar 9, 2018

Uh oh!

dongjoon-hyun commented Mar 9, 2018

Uh oh!

gatorsmile commented Mar 9, 2018

Uh oh!

jerryshao commented Mar 12, 2018

Uh oh!

jerryshao commented Mar 12, 2018

Uh oh!

SparkQA commented Mar 13, 2018

Uh oh!

SparkQA commented Mar 14, 2018

Uh oh!

wangyum commented Mar 14, 2018

Uh oh!

SparkQA commented Mar 15, 2018

Uh oh!

SparkQA commented Mar 15, 2018

Uh oh!

wangyum commented Mar 15, 2018

Uh oh!

SparkQA commented Mar 16, 2018

Uh oh!

SparkQA commented Mar 16, 2018

Uh oh!

SparkQA commented Mar 16, 2018

Uh oh!

SparkQA commented Mar 16, 2018

Uh oh!

wangyum commented Mar 16, 2018

Uh oh!

SparkQA commented Mar 16, 2018

Uh oh!

wangyum commented Mar 16, 2018

Uh oh!

dongjoon-hyun commented Mar 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 17, 2018

Uh oh!

dongjoon-hyun commented Mar 17, 2018 •

edited

Loading