[SPARK-27749][SQL] hadoop-3.2 support hive-thriftserver #24628

wangyum · 2019-05-16T16:30:19Z

What changes were proposed in this pull request?

This PR mainly makes the following changes to make hadoop-3.2 support sql/hive-thriftserver:

Upgrade TCLIService.thrift and related code to Hive 2.3.5 because of HIVE-12442(Note that we only migrate code without adding features, such as HIVE-4924 and HIVE-15473).
Use slf4j as logging facade because of HIVE-12237.
Port HIVE-13169 to compatible with Hive 2.3.

How was this patch tested?

Exiting test

SparkQA · 2019-05-16T18:21:07Z

Test build #105462 has finished for PR 24628 at commit 89b6e00.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-18T13:04:38Z

Test build #105512 has finished for PR 24628 at commit ddb0d69.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-05-18T13:47:35Z

retest this please

SparkQA · 2019-05-18T17:43:09Z

Test build #105514 has finished for PR 24628 at commit ddb0d69.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-05-20T02:34:26Z

@wangyum . I reopened https://issues.apache.org/jira/browse/SPARK-27402 . Please make an independent follow-up PR using ID SPARK-27402. That will give us traceability.

SparkQA · 2019-05-20T03:19:17Z

Test build #105538 has finished for PR 24628 at commit a7bc4e8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-22T02:37:07Z

Test build #105653 has finished for PR 24628 at commit c9a003f.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala

...ftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala

dongjoon-hyun · 2019-05-22T04:36:56Z

So, the files under v2.3.5 are from Hive 2.3.5, not Hive 2.3.4, right?

SparkQA · 2019-05-22T04:44:30Z

Test build #105656 has finished for PR 24628 at commit 38ae34e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

Please replace 2.3.4 to 2.3.5 to pass the UTs. Thanks.

cc @gatorsmile since this is a massive change.

SparkQA · 2019-05-23T02:43:44Z

Test build #105707 has finished for PR 24628 at commit a0e52aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-05-24T02:50:20Z

@wangyum . Just to be clear, did you regenerate Thrift related code always for Spark?

(Note that we only migrate code without adding features, such as HIVE-4924 and HIVE-15473).

I assumed that this generated code came from Hive 3.5.3, but the generated code is not the same with Hive 3.5.3. Please describe how you get this code in the PR description.

cc @gatorsmile

wangyum · 2019-05-24T14:32:32Z

Yes @dongjoon-hyun I did some changes to workaround mvn install build failure with error: not found: type AsyncFrameBuffer....
Maybe @srowen knows the reason. He also changed these files in #23069.

dongjoon-hyun · 2019-05-24T17:49:34Z

Thank you for the confirmation and the pointers!

SparkQA · 2019-05-25T04:40:36Z

Test build #105777 has finished for PR 24628 at commit b38c906.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-05-25T04:55:19Z

retest this please

SparkQA · 2019-05-25T07:04:34Z

Test build #105778 has finished for PR 24628 at commit b38c906.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

juliuszsompolski · 2019-05-28T10:27:56Z

For readability, would it make sens to split this PR into 2: one to move change the v2.3.4 code to v2.3.5, and the other to start using it?

SparkQA · 2019-05-29T07:05:01Z

Test build #105893 has finished for PR 24628 at commit cee5636.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-05-29T07:10:37Z

retest this please

SparkQA · 2019-05-29T09:33:02Z

Test build #105895 has finished for PR 24628 at commit cee5636.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

juliuszsompolski · 2019-05-29T11:29:57Z

LGTM, though I would do the change of actually upgrading v2.3.4 to v2.3.5 in a separate PR
cc @gatorsmile

gatorsmile · 2019-06-05T00:36:53Z

retest this please

gatorsmile · 2019-06-05T00:46:55Z

@juliuszsompolski Actually, there does not exist code changes in this component from Hive 2.3.4 to 2.3.5. :-) See the change history. https://github.com/apache/hive/commits/branch-2.3

gatorsmile · 2019-06-05T00:47:16Z

LGTM

I will merge the code when the tests pass.

gatorsmile · 2019-06-05T00:48:19Z

After the tests pass, we also need to remove [test-hadoop3.2] tag and then retrigger the test again.

SparkQA · 2019-06-05T02:25:38Z

Test build #106176 has finished for PR 24628 at commit cee5636.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-06-05T02:29:27Z

retest this please

SparkQA · 2019-06-05T05:01:24Z

Test build #106182 has finished for PR 24628 at commit cee5636.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-06-05T05:07:34Z

retest this please

SparkQA · 2019-06-05T06:48:27Z

Test build #106187 has finished for PR 24628 at commit cee5636.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-06-05T06:53:51Z

retest this please

SparkQA · 2019-06-05T07:05:02Z

Test build #106191 has finished for PR 24628 at commit cee5636.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-06-05T07:16:26Z

retest this please

SparkQA · 2019-06-05T09:11:27Z

Test build #106192 has finished for PR 24628 at commit cee5636.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-06-05T09:21:46Z

Test build #106193 has finished for PR 24628 at commit cee5636.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2019-06-05T09:35:47Z

All tests passed:
hadoop-3.2: #24628 (comment)
hadoop-2.7: #24628 (comment)

gatorsmile · 2019-06-05T15:39:36Z

Thanks! Merged to master.

## What changes were proposed in this pull request? This PR mainly makes the following changes to make `hadoop-3.2` support `sql/hive-thriftserver`: 1. Upgrade [`TCLIService.thrift`](https://github.com/apache/hive/blob/rel/release-2.3.5/service-rpc/if/TCLIService.thrift) and related code to Hive 2.3.5 because of [HIVE-12442](https://issues.apache.org/jira/browse/HIVE-12442)(Note that we only migrate code without adding features, such as [HIVE-4924](https://issues.apache.org/jira/browse/HIVE-4924) and [HIVE-15473](https://issues.apache.org/jira/browse/HIVE-15473)). 2. Use slf4j as logging facade because of [HIVE-12237](https://issues.apache.org/jira/browse/HIVE-12237). 3. Port [HIVE-13169](https://issues.apache.org/jira/browse/HIVE-13169) to compatible with Hive 2.3. ## How was this patch tested? Exiting test Closes apache#24628 from wangyum/SPARK-27749. Authored-by: Yuming Wang <[email protected]> Signed-off-by: gatorsmile <[email protected]>

### What changes were proposed in this pull request? This PR change the Github Actions build command from `mvn package` to `mvn install` to build Scaladoc jars. ### Why are the changes needed? Sometimes `mvn install` build failure with error: `not found: type ClassName...`. More details: #24628 (comment) ### Does this PR introduce any user-facing change? No. ### How was this patch tested? N/A Closes #26414 from wangyum/github-action-install. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

This PR change the Github Actions build command from `mvn package` to `mvn install` to build Scaladoc jars. Sometimes `mvn install` build failure with error: `not found: type ClassName...`. More details: #24628 (comment) No. N/A Closes #26414 from wangyum/github-action-install. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit e5c176a) Signed-off-by: Dongjoon Hyun <[email protected]>

This PR change the Github Actions build command from `mvn package` to `mvn install` to build Scaladoc jars. Sometimes `mvn install` build failure with error: `not found: type ClassName...`. More details: apache#24628 (comment) No. N/A Closes apache#26414 from wangyum/github-action-install. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit e5c176a) Signed-off-by: Dongjoon Hyun <[email protected]>

wangyum changed the title ~~[WIP][SPARK-27749][SQL] Fix hive-thriftserver module in hadoop-3.2 test issue~~ [WIP][SPARK-27749][SQL][test-hadoop3.2] Fix hive-thriftserver module in hadoop-3.2 test issue May 16, 2019

wangyum changed the title ~~[WIP][SPARK-27749][SQL][test-hadoop3.2] Fix hive-thriftserver module in hadoop-3.2 test issue~~ [SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver May 18, 2019

wangyum mentioned this pull request May 20, 2019

[SPARK-27699][FOLLOW-UP][SQL][test-hadoop3.2][test-maven] Fix hadoop-3.2 test error #24639

Closed

wangyum changed the title ~~[SPARK-27749][SQL][test-hadoop3.2][test-maven] hadoop-3.2 support hive-thriftserver~~ [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver May 22, 2019

dongjoon-hyun reviewed May 22, 2019

View reviewed changes

sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed May 22, 2019

View reviewed changes

...ftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed May 22, 2019

View reviewed changes

...ftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed May 22, 2019

View reviewed changes

...ftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala Outdated Show resolved Hide resolved

dongjoon-hyun requested changes May 22, 2019

View reviewed changes

wangyum mentioned this pull request May 28, 2019

[SPARK-27737][SQL][FOLLOW-UP] Move sql/hive-thriftserver/v2.3.4 to sql/hive-thriftserver/v2.3.5 #24728

Closed

Merge master

cee5636

wangyum changed the title ~~[SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver~~ [SPARK-27749][SQL] hadoop-3.2 support hive-thriftserver Jun 5, 2019

gatorsmile closed this in 3f102a8 Jun 5, 2019

wangyum mentioned this pull request Nov 6, 2019

[MINOR][INFRA] Change the Github Actions build command to mvn install #26414

Closed

[SPARK-27749][SQL] hadoop-3.2 support hive-thriftserver #24628

[SPARK-27749][SQL] hadoop-3.2 support hive-thriftserver #24628

Uh oh!

Conversation

wangyum commented May 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 16, 2019

Uh oh!

SparkQA commented May 18, 2019

Uh oh!

wangyum commented May 18, 2019

Uh oh!

SparkQA commented May 18, 2019

Uh oh!

dongjoon-hyun commented May 20, 2019

Uh oh!

SparkQA commented May 20, 2019

Uh oh!

SparkQA commented May 22, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun commented May 22, 2019

Uh oh!

SparkQA commented May 22, 2019

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 23, 2019

Uh oh!

dongjoon-hyun commented May 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wangyum commented May 24, 2019

Uh oh!

dongjoon-hyun commented May 24, 2019

Uh oh!

SparkQA commented May 25, 2019

Uh oh!

wangyum commented May 25, 2019

Uh oh!

SparkQA commented May 25, 2019

Uh oh!

juliuszsompolski commented May 28, 2019

Uh oh!

SparkQA commented May 29, 2019

Uh oh!

wangyum commented May 29, 2019

Uh oh!

SparkQA commented May 29, 2019

Uh oh!

juliuszsompolski commented May 29, 2019

Uh oh!

gatorsmile commented Jun 5, 2019

Uh oh!

gatorsmile commented Jun 5, 2019

Uh oh!

gatorsmile commented Jun 5, 2019

Uh oh!

gatorsmile commented Jun 5, 2019

Uh oh!

SparkQA commented Jun 5, 2019

Uh oh!

wangyum commented Jun 5, 2019

Uh oh!

SparkQA commented Jun 5, 2019

Uh oh!

wangyum commented Jun 5, 2019

Uh oh!

SparkQA commented Jun 5, 2019

Uh oh!

wangyum commented Jun 5, 2019

Uh oh!

SparkQA commented Jun 5, 2019

wangyum commented May 16, 2019 •

edited

Loading

dongjoon-hyun commented May 24, 2019 •

edited

Loading