Skip to content

Update hive-apache to 3.0.0-10#21428

Merged
tdcmeehan merged 1 commit intoprestodb:masterfrom
wypb:update_hive_apache
Dec 5, 2023
Merged

Update hive-apache to 3.0.0-10#21428
tdcmeehan merged 1 commit intoprestodb:masterfrom
wypb:update_hive_apache

Conversation

@wypb
Copy link
Contributor

@wypb wypb commented Nov 21, 2023

Description

Update hive-apache to 3.0.0-10, because #21189 relies on it.

Motivation and Context

Impact

Test Plan

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

== NO RELEASE NOTE ==

CC: @tdcmeehan

@wypb wypb requested a review from presto-oss November 21, 2023 11:17
@wypb wypb force-pushed the update_hive_apache branch 4 times, most recently from 460b177 to 39af5ab Compare November 22, 2023 02:17
@wypb
Copy link
Contributor Author

wypb commented Nov 22, 2023

Hi @wanglinsong Thanks for releasing a new version of presto-hive-apache. However, several hive tests reported errors. The stack is as follows(https://github.com/prestodb/presto/actions/runs/6951744933/job/18914243328?pr=21428):

Caused by: java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getBufferFromBytes(AvroSerdeUtils.java:249)
	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getBufferFromDecimal(AvroSerdeUtils.java:259)
	at org.apache.hadoop.hive.serde2.avro.AvroSerializer.serializePrimitive(AvroSerializer.java:204)
	at org.apache.hadoop.hive.serde2.avro.AvroSerializer.serialize(AvroSerializer.java:124)
	at org.apache.hadoop.hive.serde2.avro.AvroSerializer.serialize(AvroSerializer.java:95)
	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.serialize(AvroSerDe.java:222)
	at com.facebook.presto.hive.RecordFileWriter.appendRow(RecordFileWriter.java:170)
	at com.facebook.presto.hive.RecordFileWriter.appendRows(RecordFileWriter.java:153)
	at com.facebook.presto.hive.HiveWriter.append(HiveWriter.java:95)
	at com.facebook.presto.hive.HivePageSink.writePage(HivePageSink.java:409)
	at com.facebook.presto.hive.HivePageSink.doAppend(HivePageSink.java:361)
	at com.facebook.presto.hive.HivePageSink.lambda$appendPage$4(HivePageSink.java:347)
	at com.facebook.presto.hive.authentication.HdfsAuthentication.lambda$doAs$0(HdfsAuthentication.java:24)
	at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
	at com.facebook.presto.hive.authentication.HdfsAuthentication.doAs(HdfsAuthentication.java:23)
	at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:86)
	at com.facebook.presto.hive.HivePageSink.appendPage(HivePageSink.java:347)
	at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSink.appendPage(ClassLoaderSafeConnectorPageSink.java:74)
	at com.facebook.presto.operator.TableWriterOperator.addInput(TableWriterOperator.java:343)
	at com.facebook.presto.operator.Driver.processInternal(Driver.java:438)
	at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:311)
	at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:732)
	at com.facebook.presto.operator.Driver.processFor(Driver.java:304)
	at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1079)
	at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:165)
	at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:621)
	at com.facebook.presto.$gen.Presto_0_286_SNAPSHOT_bb78c35__testversion____20231122_014848_4606.run(Unknown Source)
	... 3 more

Error:  com.facebook.presto.hive.TestHivePushdownIntegrationSmokeTest.testInsert  Time elapsed: 1.969 s  <<< FAILURE!

I am not sure whether it is a problem with version 3.0.0-9, I compiled the presto-hive-apache module locally and installed it into the local maven repository. This test can run successfully, but it does not work with 3.0.0-9. Can you check it for me?

@wypb
Copy link
Contributor Author

wypb commented Nov 28, 2023

Hi @tdcmeehan @wanglinsong Could you please help me review this, Thank you.

@tdcmeehan
Copy link
Contributor

I suspect this is because 3.0.0-9 was built with JDK9+. I created prestodb/presto-hive-apache#63, I will try to create a snapshot build to see if this fixes the issue.

@wypb
Copy link
Contributor Author

wypb commented Nov 30, 2023

HI @tdcmeehan thank you for your reply. I used JDK11 to compile presto-hive-apache locally, and installed it into the local maven repository, and then ran the com.facebook.presto.hive.TestHivePushdownIntegrationSmokeTest.testInsert test, I found the same exception as above. I also compiled it with mvn clean install -P java-8-api command in JDK11, and still got an error. And I found that there is no problem in compiling with jdk8.

@tdcmeehan
Copy link
Contributor

I found the same. I need some time to find out why this is the case, otherwise I'll try to publish a new artifact using JDK8 tomorrow or Monday.

@tdcmeehan
Copy link
Contributor

@wypb I have created 3.0.0-10--can you give it a try?

@wypb wypb force-pushed the update_hive_apache branch from 5472401 to fb5dc4f Compare December 5, 2023 01:11
@wypb wypb changed the title Update hive-apache to 3.0.0-9 Update hive-apache to 3.0.0-10 Dec 5, 2023
@tdcmeehan tdcmeehan merged commit c361c45 into prestodb:master Dec 5, 2023
@wypb wypb deleted the update_hive_apache branch December 5, 2023 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants