Skip to content

[WIP] Upgrade to Hadoop 3.x Hive 3.x (Needs Rebase) #5292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rahil-c wants to merge 64 commits into from
Closed

[WIP] Upgrade to Hadoop 3.x Hive 3.x (Needs Rebase) #5292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
47399cc
[HUDI-2783] Upgrade HBase
codope Nov 17, 2021
569a4da
Fix metablock buffered read in hfile reader
codope Dec 8, 2021
d13cb78
Replace hbase-shaded-server and shade some more deps
codope Jan 3, 2022
9c87820
Rebase master
yihua Feb 17, 2022
0c11f72
Fix HoodieHFileReader
yihua Feb 18, 2022
de2302f
Revert hadoop upgrade
yihua Feb 18, 2022
199f1a6
Fix hudi-hadoop-mr-bundle to include more dependencies and address CI…
yihua Feb 19, 2022
ee57910
Fix hudi-utilities-bundle
yihua Feb 25, 2022
b20e206
Fix hudi-common
yihua Feb 25, 2022
fddd577
Fix other bundles
yihua Feb 26, 2022
215ece2
Use hadoop 2.10.1
yihua Feb 28, 2022
e56cb20
Shade dependencies in hudi-hadoop-mr-bundle
yihua Feb 28, 2022
2b9f24c
Shade dependencies in other bundles
yihua Feb 28, 2022
d5eed1b
Fix bundle shading
yihua Feb 28, 2022
ffec02c
Fix hbase shading
yihua Mar 2, 2022
99c1cc5
Fix hudi-integ-test-bundle
yihua Mar 2, 2022
84d55b4
Fix rebase issue
yihua Mar 2, 2022
c6806f4
Add hbase-site to skip version check
yihua Mar 4, 2022
879fc30
Add hbase-site.xml to hudi-common
yihua Mar 9, 2022
c38fdf8
Update presto setup in docker demo
yihua Mar 9, 2022
70e3a44
Some cleanup in docker setup
yihua Mar 9, 2022
5b09e51
Simplify dependency changes
yihua Mar 10, 2022
88c715f
Trim bundle dependencies
yihua Mar 10, 2022
b218046
Make adjustment to bundles
yihua Mar 10, 2022
4bef7fb
Change kafka-connect-bundle and timeline-server-bundle
yihua Mar 10, 2022
70c1029
Adjust hudi-integ-test-bundle
yihua Mar 12, 2022
1c50a8c
Added dep on "hbase-hadoop2-compat" bearing implementation for interf…
Dec 10, 2021
0fd2a8d
Package "hbase-metrics-api" into the bundles
Dec 10, 2021
9cbad00
Package "hbase-metrics" into the bundles
Dec 10, 2021
7bc0a0f
Bumped dep versions in Docker images
Dec 10, 2021
43694ad
Fixed missing deps
Dec 11, 2021
80f7bb2
Adjust demo setup log4j properties
Dec 11, 2021
7e69208
Missing `scala-library` dep
Dec 7, 2021
dd90af3
Unified Java source/target configs
Dec 7, 2021
db3d5e0
Bumped Maven's Java/Scala compiler plugins;
Dec 7, 2021
4230467
Bumped Maven jar/dependency plugins
Dec 7, 2021
0297a41
XXX
Dec 7, 2021
437e48c
Reverting accidental change
Dec 11, 2021
97ebc92
Bumped Hive version to 3.1.2 (to be compatible w/ Hadoop 3.x)
Dec 11, 2021
69ee2b3
Upgraded
Dec 11, 2021
62793b1
Upgraded Hadoop/Hive in Docker images
Dec 11, 2021
c91d9c1
Excluded Jetty from being transitively pulled in from Hive, Hadoop, H…
Dec 11, 2021
9fabdb6
Added `NoOpMetastoreUriResolverHook` to bypass issue of Hive 3.x tryi…
Dec 13, 2021
129f8a0
Fixed HDFS default `namenode` port changed from 50070 to 9870
Dec 13, 2021
da03159
Fixed Hive Metastore's Postgresql image version to fetch correct vers…
Dec 13, 2021
d3aa942
Setup Tez (required for Hive 3.x, MR support is removed) w/in `hive_b…
Dec 13, 2021
59acc8c
Tidying up
Dec 13, 2021
dbe2311
XXX
Dec 13, 2021
c2738a5
Fixed compilation
Mar 16, 2022
34fd9d0
Get Docker Setup Working
Mar 25, 2022
be625ba
Change Maven Dep plugin to 3.3.0
Mar 24, 2022
fa4d523
Remove Tez and set hive vector execution to false
Mar 31, 2022
d3cc5d8
Setup to use Spark 3.2.1
Apr 1, 2022
d9ae5d8
Hive configurations to get past hive related issues, and parquet mr c…
Apr 5, 2022
1098578
try trino with hadoop 3 and hive 3
Apr 6, 2022
c0c6d9f
Fix Docker IT issue with Hive defaulting to Tez
Apr 8, 2022
17088d9
Fix Avro field not found issue introduced by Avro 1.10
Feb 24, 2022
9b6aa30
Remove unused import, increase sleep time
Apr 8, 2022
664c985
Change Azure CI setup to run Spark 3.2.1
Apr 9, 2022
4f78b3f
Remove Define Parquet version as 1.10.1 in Hudi Hadoop Mr
Apr 11, 2022
eb4b3f5
[HUDI-3817] shade parquet dependency for hudi-hadoop-mr-bundle (#5250)
RexXiong Apr 11, 2022
da3ccca
Remove making Spark3 default profile
Apr 11, 2022
24dab32
Add Spark3 profile in Azure CI setup
Apr 11, 2022
0e1a4ab
Exclude Jetty in timeline server
Apr 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ pool:
variables:
MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER) -Dcheckstyle.skip=true -Drat.skip=true -Djacoco.skip=true'
SPARK_VERSION: '2.4.4'
HADOOP_VERSION: '2.7'
SPARK_VERSION: '3.2.1'
HADOOP_VERSION: '3.2'
SPARK_ARCHIVE: spark-$(SPARK_VERSION)-bin-hadoop$(HADOOP_VERSION)

stages:
Expand All @@ -48,7 +48,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'install'
options: -T 2.5C -DskipTests
options: -T 2.5C -Dspark3 -DskipTests
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -57,7 +57,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Punit-tests -pl hudi-common,hudi-flink,hudi-client/hudi-spark-client
options: -Punit-tests,spark3 -pl hudi-common,hudi-flink,hudi-client/hudi-spark-client
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -66,7 +66,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pfunctional-tests -pl hudi-common,hudi-flink
options: -Pfunctional-tests,spark3 -pl hudi-common,hudi-flink
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -87,7 +87,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'install'
options: -T 2.5C -DskipTests
options: -T 2.5C -Dspark3 -DskipTests
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -96,7 +96,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pfunctional-tests -pl hudi-client/hudi-spark-client
options: -Pfunctional-tests,spark3 -pl hudi-client/hudi-spark-client
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -117,7 +117,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'install'
options: -T 2.5C -DskipTests
options: -T 2.5C -Dspark3 -DskipTests
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -126,7 +126,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Punit-tests -pl hudi-client/hudi-client-common,hudi-client/hudi-flink-client,hudi-client/hudi-java-client,hudi-cli,hudi-utilities,hudi-sync/hudi-hive-sync
options: -Punit-tests,spark3 -pl hudi-client/hudi-client-common,hudi-client/hudi-flink-client,hudi-client/hudi-java-client,hudi-cli,hudi-utilities,hudi-sync/hudi-hive-sync
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -135,7 +135,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pfunctional-tests -pl hudi-client/hudi-client-common,hudi-client/hudi-flink-client,hudi-client/hudi-java-client,hudi-cli,hudi-utilities,hudi-sync/hudi-hive-sync
options: -Pfunctional-tests,spark3 -pl hudi-client/hudi-client-common,hudi-client/hudi-flink-client,hudi-client/hudi-java-client,hudi-cli,hudi-utilities,hudi-sync/hudi-hive-sync
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -156,7 +156,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'install'
options: -T 2.5C -DskipTests
options: -T 2.5C -Dspark3 -DskipTests
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -165,7 +165,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Punit-tests -pl !hudi-common,!hudi-flink,!hudi-client/hudi-spark-client,!hudi-client/hudi-client-common,!hudi-client/hudi-flink-client,!hudi-client/hudi-java-client,!hudi-cli,!hudi-utilities,!hudi-sync/hudi-hive-sync
options: -Punit-tests,spark3 -pl !hudi-common,!hudi-flink,!hudi-client/hudi-spark-client,!hudi-client/hudi-client-common,!hudi-client/hudi-flink-client,!hudi-client/hudi-java-client,!hudi-cli,!hudi-utilities,!hudi-sync/hudi-hive-sync
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -174,7 +174,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pfunctional-tests -pl !hudi-common,!hudi-flink,!hudi-client/hudi-spark-client,!hudi-client/hudi-client-common,!hudi-client/hudi-flink-client,!hudi-client/hudi-java-client,!hudi-cli,!hudi-utilities,!hudi-sync/hudi-hive-sync
options: -Pfunctional-tests,spark3 -pl !hudi-common,!hudi-flink,!hudi-client/hudi-spark-client,!hudi-client/hudi-client-common,!hudi-client/hudi-flink-client,!hudi-client/hudi-java-client,!hudi-cli,!hudi-utilities,!hudi-sync/hudi-hive-sync
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx2g $(MAVEN_OPTS)'
Expand All @@ -194,5 +194,5 @@ stages:
tar -xvf $(Pipeline.Workspace)/$(SPARK_ARCHIVE).tgz -C $(Pipeline.Workspace)/
mkdir /tmp/spark-events/
- script: |
mvn $(MAVEN_OPTS) -Pintegration-tests verify
mvn $(MAVEN_OPTS) -Pintegration-tests,spark3 verify
displayName: IT
40 changes: 20 additions & 20 deletions docker/compose/docker-compose_hadoop284_hive233_spark244.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ services:
presto-coordinator-1:
container_name: presto-coordinator-1
hostname: presto-coordinator-1
image: apachehudi/hudi-hadoop_2.8.4-prestobase_0.268:latest
image: apachehudi/hudi-hadoop_2.8.4-prestobase_0.271:latest
ports:
- '8090:8090'
environment:
Expand All @@ -201,25 +201,25 @@ services:
command: coordinator

presto-worker-1:
container_name: presto-worker-1
hostname: presto-worker-1
image: apachehudi/hudi-hadoop_2.8.4-prestobase_0.268:latest
depends_on: ["presto-coordinator-1"]
environment:
- PRESTO_JVM_MAX_HEAP=512M
- PRESTO_QUERY_MAX_MEMORY=1GB
- PRESTO_QUERY_MAX_MEMORY_PER_NODE=256MB
- PRESTO_QUERY_MAX_TOTAL_MEMORY_PER_NODE=384MB
- PRESTO_MEMORY_HEAP_HEADROOM_PER_NODE=100MB
- TERM=xterm
links:
- "hivemetastore"
- "hiveserver"
- "hive-metastore-postgresql"
- "namenode"
volumes:
- ${HUDI_WS}:/var/hoodie/ws
command: worker
container_name: presto-worker-1
hostname: presto-worker-1
image: apachehudi/hudi-hadoop_2.8.4-prestobase_0.271:latest
depends_on: [ "presto-coordinator-1" ]
environment:
- PRESTO_JVM_MAX_HEAP=512M
- PRESTO_QUERY_MAX_MEMORY=1GB
- PRESTO_QUERY_MAX_MEMORY_PER_NODE=256MB
- PRESTO_QUERY_MAX_TOTAL_MEMORY_PER_NODE=384MB
- PRESTO_MEMORY_HEAP_HEADROOM_PER_NODE=100MB
- TERM=xterm
links:
- "hivemetastore"
- "hiveserver"
- "hive-metastore-postgresql"
- "namenode"
volumes:
- ${HUDI_WS}:/var/hoodie/ws
command: worker

trino-coordinator-1:
container_name: trino-coordinator-1
Expand Down
Loading