Upgrade hive-apache to new version 3.0.0-6 #16545

shangxinli · 2021-07-31T15:02:42Z

Test plan - (Please fill in how you tested your changes)

Please make sure your submission complies with our Development, Formatting, and Commit Message guidelines. Don't forget to follow our attribution guidelines for any code copied from other projects.

Fill in the release notes towards the bottom of the PR description.
See Release Notes Guidelines for details.

== RELEASE NOTES ==

General Changes
Upgrade hive-apache to new version 3.0.0-6

Hive Changes
Fix NoClassDefFoundError for JsonNode

shangxinli · 2021-08-06T04:16:58Z

To solve the test error below, we need to make several other changes in Hive and presto-hive-apache repro.

java.lang.NoClassDefFoundError: com/facebook/presto/hive/$internal/org/codehaus/jackson/JsonNode
presto-worker_1 | at org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(AvroSerDe.java:178)

We need the PR to fix the above failure.
The commit need to be ported to Hive 3.0+( presto-hive-apache uses 3.0.0-5). And then release a new version of Hive 3.0+, and then upgrade presto-hive-apache.

beinan · 2021-08-20T19:32:32Z

Thank you! Looks like this one skipped quite a few test, I might hesitate to merge it. Not sure the impact on FB side.

btw, could you please update the release note section in this PR? Thanks!

shangxinli · 2021-08-23T16:11:16Z

Thanks @beinan for having a look! Yeah, this PR disabled quite a few tests. I am not sure if that is acceptable. The root cause is the new version of Avro changed the signature of Avro Schema(not backward compatible), and Hive has a hard dependency on the old signature. A fix in Hive is needed and that is done in Hive 2.x.x but not Hive 3.x.x, but our Hive-apache repo is using Hive 3.x.x.

So there are 3 solutions now. 1) Push Hive community to port the fix from 2.x.x to 3.x.x and have a new release. 2) Revert #15805 to let Presto have its own Parquet upgrading path independent of the hive-apache repo. 3) Disable tests(not sure if real user scenarios also having issues).

#1 seems a better solution but it seems the Hive community has no interest, for now, to help on releasing Hive 3.x.x. I started an email thread for asking Hive community but not response.

beinan · 2021-08-23T17:11:55Z

Thanks @beinan for having a look! Yeah, this PR disabled quite a few tests. I am not sure if that is acceptable. The root cause is the new version of Avro changed the signature of Avro Schema(not backward compatible), and Hive has a hard dependency on the old signature. A fix in Hive is needed and that is done in Hive 2.x.x but not Hive 3.x.x, but our Hive-apache repo is using Hive 3.x.x.

So there are 3 solutions now. 1) Push Hive community to port the fix from 2.x.x to 3.x.x and have a new release. 2) Revert #15805 to let Presto have its own Parquet upgrading path independent of the hive-apache repo. 3) Disable tests(not sure if real user scenarios also having issues).

#1 seems a better solution but it seems the Hive community has no interest, for now, to help on releasing Hive 3.x.x. I started an email thread for asking Hive community but not response.

Thanks a lot for these solutions!
Any chance to override the change you made in hive in presto-hive-apache? we might be able to put AvroDeserializer.java into here https://github.com/prestodb/presto-hive-apache/tree/master/src/main/java/org/apache/hadoop/hive/serde2/avro
I'm just guessing, not sure if it works or not. Thanks!

shangxinli · 2021-08-25T01:39:32Z

@beinan The change I made in pesto-hive-apache helped JsonNode no class finding error. Now the failing tests are failing with the following stack.

at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.createAvroField(TypeInfoToSchema.java:76)
at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.convert(TypeInfoToSchema.java:61)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(AvroSerDe.java:179)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:117)
at com.facebook.presto.hive.HiveUtil.initializeDeserializer(HiveUtil.java:496)
at com.facebook.presto.hive.HiveUtil.getDeserializer(HiveUtil.java:456)
at com.facebook.presto.hive.GenericHiveRecordCursor.<init>(GenericHiveRecordCursor.java:144)
at com.facebook.presto.hive.GenericHiveRecordCursorProvider.lambda$createRecordCursor$1(GenericHiveRecordCursorProvider.java:79)
at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:81)
at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:75)
at com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:466)
at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:184)
at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:63)
at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:80)
at com.facebook.presto.operator.TableScanOperator.getOutput(TableScanOperator.java:249)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:599)
at com.facebook.presto.$gen.Presto_0_261_SNAPSHOT_280493b____20210824_183044_1.run(Unknown Source)

So I am not totally getting your point. Can you explain?

beinan · 2021-08-27T00:49:17Z

@beinan The change I made in pesto-hive-apache helped JsonNode no class finding error. Now the failing tests are failing with the following stack.

at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.createAvroField(TypeInfoToSchema.java:76)
at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.convert(TypeInfoToSchema.java:61)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(AvroSerDe.java:179)
at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:117)
at com.facebook.presto.hive.HiveUtil.initializeDeserializer(HiveUtil.java:496)
at com.facebook.presto.hive.HiveUtil.getDeserializer(HiveUtil.java:456)
at com.facebook.presto.hive.GenericHiveRecordCursor.<init>(GenericHiveRecordCursor.java:144)
at com.facebook.presto.hive.GenericHiveRecordCursorProvider.lambda$createRecordCursor$1(GenericHiveRecordCursorProvider.java:79)
at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:81)
at com.facebook.presto.hive.GenericHiveRecordCursorProvider.createRecordCursor(GenericHiveRecordCursorProvider.java:75)
at com.facebook.presto.hive.HivePageSourceProvider.createHivePageSource(HivePageSourceProvider.java:466)
at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:184)
at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:63)
at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:80)
at com.facebook.presto.operator.TableScanOperator.getOutput(TableScanOperator.java:249)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:599)
at com.facebook.presto.$gen.Presto_0_261_SNAPSHOT_280493b____20210824_183044_1.run(Unknown Source)

So I am not totally getting your point. Can you explain?

Sorry for my last misleading comment.
I mean we can shift the change you made in hive to presto-hive-apache. Because we can override hive's class in pesto-hive-apache

shangxinli · 2021-08-27T15:41:48Z

@beinan Thanks for the explanation! The change is inside the Hive repo for the file TypeInfoToSchema.java (you can refer above call stack). If we want to shift the change to presto-hive-apache, we would need to copy all the code from hive.serde or at least hive.serde2.avro, right? Or do you suggest we can shade Hive 2.x.x and in HiveUtil.java we only call the shaded one?

BTW, if you have some ideas to implement it, feel free to take them over. I saw you have another PR for the same upgrading too.

shangxinli · 2021-08-27T17:51:48Z

The way I can think of is to shade the Hive jar. 1) Create a new project which references hive-2.3.8 and shade it 2) Include this project in presto-hive-apache 3) Change presto to reference the shaded class

beinan · 2021-08-27T20:41:44Z

The way I can think of is to shade the Hive jar. 1) Create a new project which references hive-2.3.8 and shade it 2) Include this project in presto-hive-apache 3) Change presto to reference the shaded class
gotcha, let me try something in this weekend and get back to you:)

shangxinli · 2021-08-30T16:19:42Z

Did you find something @beinan?

shangxinli · 2021-09-07T15:27:35Z

The way I can think of is to shade the Hive jar. 1) Create a new project which references hive-2.3.8 and shade it 2) Include this project in presto-hive-apache 3) Change presto to reference the shaded class
gotcha, let me try something in this weekend and get back to you:)

Do you have updates @beinan?

beinan · 2021-09-10T00:17:06Z

The way I can think of is to shade the Hive jar. 1) Create a new project which references hive-2.3.8 and shade it 2) Include this project in presto-hive-apache 3) Change presto to reference the shaded class
gotcha, let me try something in this weekend and get back to you:)

Do you have updates @beinan?

Hi @shangxinli , sorry for my delay, I post a PR for this issue prestodb/presto-hive-apache#51
I copied some of classes from hive and made change on it. The pr fixed some of the unit tests, but there still are tests failing. Could you help take a look? I think you would be more familiar with avro stuff. Thanks a lot! Feel free take over the PR if you got some ideas.

aweisberg · 2021-09-23T23:52:20Z

I'll take a look at this soon and try deploying the whole thing.

aweisberg · 2021-09-27T15:11:44Z

I deployed this to a cluster and ran 20k test queries. There was one failure that I haven't resolved yet, but I am guessing it was transient and specific to that query and am retesting it now.

beinan · 2021-10-05T22:34:55Z

I deployed this to a cluster and ran 20k test queries. There was one failure that I haven't resolved yet, but I am guessing it was transient and specific to that query and am retesting it now.

Thank you @aweisberg ! any updates? looks like the ci is still failing.

aweisberg · 2021-10-06T16:09:08Z

There is something wrong with snapshots. I can upload them and even see them in https://oss.sonatype.org/#nexus-search;gav~com.facebook.presto.hive~hive-apache~~~~kw,versionexpand, but I can't get them to download.

I also get the error

Unable to identify component: org.sonatype.nexus.proxy.ItemNotFoundException: Path com/facebook/presto/hive/hive-apache/3.0.0-7-SNAPSHOT/hive-apache-3.0.0-7-20211006.154738-15.jar not found in local storage of repository "Snapshots" [id=snapshots]

When I try to look at component info in the Nexus web UI.

How snapshots are deployed is defined in airbase and it looks correct.

    <repositories>
        <repository>
            <id>sonatype-nexus-snapshots</id>
            <name>Sonatype Nexus Snapshots</name>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

The build tries to fetch from that location

[ERROR] Failed to execute goal on project presto-hive-metastore: Could not resolve dependencies for project com.facebook.presto:presto-hive-metastore:jar:0.264-SNAPSHOT: Could not find artifact com.facebook.presto.hive:hive-apache:jar:3.0.0-7-20211006.154738-15 in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project presto-hive-metastore: Could not resolve dependencies for project com.facebook.presto:presto-hive-metastore:jar:0.264-SNAPSHOT: Could not find artifact com.facebook.presto.hive:hive-apache:jar:3.0.0-7-20211006.154738-15 in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots)

aweisberg · 2021-10-06T16:30:36Z

https://stackoverflow.com/questions/43960518/the-timestamp-in-snapshot-jars-maven-metadata-xml-is-more-than-a-second-than-th

aweisberg · 2021-10-06T17:28:30Z

I think it's working now. I had to deploy a few times to get the timestamps to match.

beinan · 2021-10-06T19:48:15Z

Awesome!!! @aweisberg Thank you!

Hello @shangxinli, looks like all good except the iceberg connector one, I will take a look soon.

beinan · 2021-10-06T20:06:33Z

Awesome!!! @aweisberg Thank you!

Hello @shangxinli, looks like all good except the iceberg connector one, I will take a look soon.

@shangxili I'm not sure the iceberg test is just flaky or not, could you rebase the master and push again? Thanks!

shangxinli force-pushed the upstream branch from ddf7603 to 0329ae7 Compare August 2, 2021 16:40

shangxinli mentioned this pull request Aug 6, 2021

Fix NoClassDefFoundError for JsonNode prestodb/presto-hive-apache#49

Merged

shangxinli force-pushed the upstream branch from a7c92eb to 0783b45 Compare August 20, 2021 17:03

shangxinli force-pushed the upstream branch from d847e68 to 4345e2f Compare August 26, 2021 20:32

shangxinli mentioned this pull request Sep 16, 2021

Fix avro test failures in prestodb project prestodb/presto-hive-apache#51

Closed

shangxinli force-pushed the upstream branch from dd39dc6 to ea882f8 Compare September 16, 2021 16:00

Upgrade hive-apache to new version 3.0.0-6

d850c96

shangxinli force-pushed the upstream branch from fa9c7bc to d850c96 Compare September 16, 2021 16:09

shangxinli changed the title ~~Upgrade hive-apache to new version 3.0.0-5~~ Upgrade hive-apache to new version 3.0.0-6 Sep 17, 2021

yingsu00 requested review from aweisberg, yingsu00 and zhenxiao September 23, 2021 02:56

Upgrade the presto-hive-apache version to 3.0.0-7-SNAPSHOT

ab62739

beinan mentioned this pull request Oct 6, 2021

Presto fails to write to Iceberg tables #16242

Open

shangxinli closed this Dec 31, 2021

Upgrade hive-apache to new version 3.0.0-6 #16545

Upgrade hive-apache to new version 3.0.0-6 #16545

Uh oh!

Conversation

shangxinli commented Jul 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shangxinli commented Aug 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beinan commented Aug 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shangxinli commented Aug 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beinan commented Aug 23, 2021

Uh oh!

shangxinli commented Aug 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beinan commented Aug 27, 2021

Uh oh!

shangxinli commented Aug 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shangxinli commented Aug 27, 2021

Uh oh!

beinan commented Aug 27, 2021

Uh oh!

shangxinli commented Aug 30, 2021

Uh oh!

shangxinli commented Sep 7, 2021

Uh oh!

beinan commented Sep 10, 2021

Uh oh!

aweisberg commented Sep 23, 2021

Uh oh!

aweisberg commented Sep 27, 2021

Uh oh!

beinan commented Oct 5, 2021

Uh oh!

aweisberg commented Oct 6, 2021

Uh oh!

aweisberg commented Oct 6, 2021

Uh oh!

aweisberg commented Oct 6, 2021

Uh oh!

beinan commented Oct 6, 2021

Uh oh!

beinan commented Oct 6, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shangxinli commented Jul 31, 2021 •

edited

Loading

shangxinli commented Aug 6, 2021 •

edited

Loading

beinan commented Aug 20, 2021 •

edited

Loading

shangxinli commented Aug 23, 2021 •

edited

Loading

shangxinli commented Aug 25, 2021 •

edited

Loading

shangxinli commented Aug 27, 2021 •

edited

Loading