[SPARK-33696][BUILD][SQL] Upgrade built-in Hive to 2.3.8 #756
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rshkv
reviewed
Apr 19, 2021
rshkv
approved these changes
Apr 19, 2021
Hive 2.3.8 changes: HIVE-19662: Upgrade Avro to 1.8.2 HIVE-24324: Remove deprecated API usage from Avro HIVE-23980: Shade Guava from hive-exec in Hive 2.3 HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue HIVE-24512: Exclude calcite in packaging. HIVE-22708: Fix for HttpTransport to replace String.equals HIVE-24551: Hive should include transitive dependencies from calcite after shading it HIVE-24553: Exclude calcite from test-jar dependency of hive-exec Upgrade Avro and Parquet to latest version. No. Existing test add test try to upgrade Parquet to 1.11.1 and Avro to 1.10.1: apache#30517 Closes apache#30657 from wangyum/SPARK-33696. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
LorenzoMartini
force-pushed
the
lmartini/hive-final
branch
from
c6c3927 to
d2ecef0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Original pr message
Hive 2.3.8 changes:
HIVE-19662: Upgrade Avro to 1.8.2
HIVE-24324: Remove deprecated API usage from Avro
HIVE-23980: Shade Guava from hive-exec in Hive 2.3
HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue
HIVE-24512: Exclude calcite in packaging.
HIVE-22708: Fix for HttpTransport to replace String.equals
HIVE-24551: Hive should include transitive dependencies from calcite after shading it
HIVE-24553: Exclude calcite from test-jar dependency of hive-exec
Upgrade Avro and Parquet to latest version.
No.
Existing test add test try to upgrade Parquet to 1.11.1 and Avro to 1.10.1: apache#30517
Closes apache#30657 from wangyum/SPARK-33696.
Authored-by: Yuming Wang [email protected]
Signed-off-by: Dongjoon Hyun [email protected]
Upstream SPARK-XXXXX ticket and PR link (if not applicable, explain)
[SPARK-33696][BUILD][SQL] Upgrade built-in Hive to 2.3.8
apache#30657 or apache@c87b008
What changes were proposed in this pull request?
Bump Hive version to 2.3.8. From bump pr:
Hive 2.3.8 changes:
HIVE-19662: Upgrade Avro to 1.8.2
HIVE-24324: Remove deprecated API usage from Avro
HIVE-23980: Shade Guava from hive-exec in Hive 2.3
HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue
HIVE-24512: Exclude calcite in packaging.
HIVE-22708: Fix for HttpTransport to replace String.equals
HIVE-24551: Hive should include transitive dependencies from calcite after shading it
HIVE-24553: Exclude calcite from test-jar dependency of hive-exec
The cherry-pick is not 100% clean as there are some small differences in the upstream history.
One commit we didn't pick up is apache@10b6466 that instroduced a test tha we don't have and therefore you won't see the change here.
There was also some refactoring (apache@a127387) with removal of references to avro 1.2 that added a small if/else codepath for avro version. We don't have that so I kept the code as-is.
The last difference is in the
spark-deps-hadoop-hivefiles. We only had 1 reference to hive, while upstream had many. Upstream change includes changes in all those, while for us running locks doesn't change any of those.Why are the changes needed?
We need this bump to make the Avro bump work
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests