Skip to content

Upgrade Hive JDBC version to 3.1.3#5

Closed
ebyhr wants to merge 3 commits into
trinodb:masterfrom
ebyhr:ebi/3.1.3
Closed

Upgrade Hive JDBC version to 3.1.3#5
ebyhr wants to merge 3 commits into
trinodb:masterfrom
ebyhr:ebi/3.1.3

Conversation

@ebyhr
Copy link
Copy Markdown
Member

@ebyhr ebyhr commented May 29, 2023

The previous version 0.13.0 throws an exception
when executing a query on Spark 3.4.0.

Relates to trinodb/trino#17667

@cla-bot cla-bot Bot added the cla-signed label May 29, 2023
ebyhr added 3 commits May 29, 2023 12:49
Otherwise, it throws an IllegalArgumentException.
The previous version 0.13.0 throws an exception
when executing an query on Spark 3.4.0.
Copy link
Copy Markdown
Member

@electrum electrum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating this. We need to either add more exclusions or adjust the shading rules. If you look at the current release JAR, everything is under one of these:

io/trino/hive/jdbc/$internal
org/apache/hive/jdbc

The version from this PR has lots of extra stuff, which you can see by running the following:

jar tf target/hive-apache-jdbc-*-SNAPSHOT.jar | sort

I suspect that we can exclude most of these things. It might help to look at the exclusion rules from our trino-hive-apache project, as those will match the current Hive version.

Comment thread pom.xml
<dep.guava.version>11.0.2</dep.guava.version>
<dep.slf4j.version>1.7.10</dep.slf4j.version>
<dep.hadoop.version>1.2.1</dep.hadoop.version>
<dep.hive.version>3.1.3</dep.hive.version>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the project version to 3.1.3-1-SNAPSHOT so that it matches the Hive version.

Update the README to use the new version, or simply remove the Hive version from the README.

@ebyhr
Copy link
Copy Markdown
Member Author

ebyhr commented Jun 5, 2023

Hmm, it seems the new Hive JDBC driver can't connect to the old Hive server (e.g. hdp2.6-hive in product tests). Delta Lake 2.4.0 requires at least Hive 3.0.0 JDBC driver, but the version isn't compatible with the above HDP environment.

@ebyhr
Copy link
Copy Markdown
Member Author

ebyhr commented Jun 8, 2023

Installing standalone Hive JDBC driver at runtime like trinodb/trino#17667 looks better because we need to use two JDBC versions for connecting to old Hive server and new Spark server.

@ebyhr ebyhr closed this Jun 8, 2023
@ebyhr ebyhr deleted the ebi/3.1.3 branch June 8, 2023 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

3 participants