Skip to content

[WIP] Support Hadoop 3.x Hive 3.x and Spark 3.2.x default #5402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rahil-c wants to merge 85 commits into from
Closed

[WIP] Support Hadoop 3.x Hive 3.x and Spark 3.2.x default #5402

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
30b26b2
Resolve HBase upgrade changes
codope Nov 17, 2021
546c2c3
Package "hbase-metrics-api" into the bundles
Dec 10, 2021
cccf8eb
Package "hbase-metrics" into the bundles
Dec 10, 2021
eb7ec46
Bumped dep versions in Docker images
Dec 10, 2021
7498b35
Fixed missing deps
Dec 11, 2021
7400ba4
Adjust demo setup log4j properties
Dec 11, 2021
82b2d81
Missing `scala-library` dep
Dec 7, 2021
99d8db8
Unified Java source/target configs
Dec 7, 2021
785c329
Bumped Maven's Java/Scala compiler plugins;
Dec 7, 2021
65abac3
Bumped Maven jar/dependency plugins
Dec 7, 2021
f8e0ee9
XXX
Dec 7, 2021
b2d8ab7
Reverting accidental change
Dec 11, 2021
3de3a23
Bumped Hive version to 3.1.2 (to be compatible w/ Hadoop 3.x)
Dec 11, 2021
35f0357
Upgraded
Dec 11, 2021
b22ad99
Upgraded Hadoop/Hive in Docker images
Dec 11, 2021
98eec69
Excluded Jetty from being transitively pulled in from Hive, Hadoop, H…
Dec 11, 2021
2b81de3
Added `NoOpMetastoreUriResolverHook` to bypass issue of Hive 3.x tryi…
Dec 13, 2021
101441a
Fixed HDFS default `namenode` port changed from 50070 to 9870
Dec 13, 2021
206ddda
Fixed Hive Metastore's Postgresql image version to fetch correct vers…
Dec 13, 2021
7082585
Setup Tez (required for Hive 3.x, MR support is removed) w/in `hive_b…
Dec 13, 2021
f80b22b
Tidying up
Dec 13, 2021
0a97c8e
XXX
Dec 13, 2021
905b13e
Fixed compilation
Mar 16, 2022
201d5d0
Get Docker Setup Working
Mar 25, 2022
8f25653
Change Maven Dep plugin to 3.3.0
Mar 24, 2022
34143ce
Remove Tez and set hive vector execution to false
Mar 31, 2022
20fc4a0
Setup to use Spark 3.2.1
Apr 1, 2022
1e43f4b
Hive configurations to get past hive related issues, and parquet mr c…
Apr 5, 2022
84357ce
try trino with hadoop 3 and hive 3
Apr 6, 2022
56c2609
Fix Docker IT issue with Hive defaulting to Tez
Apr 8, 2022
106b2af
Fix Avro field not found issue introduced by Avro 1.10
Feb 24, 2022
aa9fe29
Remove unused import, increase sleep time
Apr 8, 2022
43adafe
Change Azure CI setup to run Spark 3.2.1
Apr 9, 2022
01ad6ae
Remove Define Parquet version as 1.10.1 in Hudi Hadoop Mr
Apr 11, 2022
bea14c8
Remove making Spark3 default profile
Apr 11, 2022
4f1a306
Add Spark3 profile in Azure CI setup
Apr 11, 2022
e39373e
Exclude Jetty in timeline server
Apr 13, 2022
62f2935
Resolve rebasing error
yihua Apr 20, 2022
deb4bda
Remove redundant changes after rebasing
yihua Apr 20, 2022
222606e
[HUDI-3088] Use Spark 3.2 as default Spark version
xushiyan Jan 28, 2022
34869ae
Fix java ci issues for all profiles
Apr 24, 2022
5e6f9d7
Fix unit test TestHoodieAvroUtils
Apr 25, 2022
cab9ff2
Fix test TestHoodieReaderWriterBase
Apr 25, 2022
5d91a70
Make Spark3.2 default spark bundle version
Apr 26, 2022
1d92c16
Get fieldValue from GenericRecord for both versions of avro
Apr 26, 2022
f7281d1
Disable flink quickstart test
Apr 26, 2022
2d8f455
Add jetty dependencies in timeline service in order to avoid dep conf…
Apr 26, 2022
c479980
Disable MiniCluster Tests (Revisit later due to Hadoop 9.3 jetty conf…
May 7, 2022
0092578
Disable TestSparkHoodieHBaseIndex for same jetty 9.3 conflict
May 8, 2022
6241966
Add spark3 profiles to integ tests
yihua May 9, 2022
92a7e39
Set sparkbundle.version to 3 by default
yihua May 9, 2022
bc5c5aa
Increase timeout for hiveTest setup, lower fasterXml version to hudi …
May 9, 2022
cf9b48b
Increase timeout for hiveTest setup, lower fasterXml version to hudi …
May 9, 2022
e948d87
Increase timeout for hiveTest setup, lower fasterXml version to hudi …
May 9, 2022
723f027
Change docker configs for CI
yihua May 10, 2022
dd78d37
Merge remote-tracking branch 'refs/remotes/rahil/hdp3-hive3-spark3-br…
yihua May 10, 2022
d7d0cc8
Fix recursive expression cycle in fasterxml.spark3.version
yihua May 10, 2022
4a88bdd
Fix docker yaml and setup configs
yihua May 10, 2022
604bf7b
Disable unit tests in hudi-integ-test module
yihua May 11, 2022
e081680
Disable testMergeOnReadSnapshotRelationWithDeltaLogsFallback
May 21, 2022
e5cd9c9
Azure CI: run only hive sync and IT
May 21, 2022
fbb4a7e
Disable flaky test ITTestHoodieDataSource#testWriteNonPartitionedTable
May 21, 2022
ba68f3b
Azure CI Run only IT module
May 21, 2022
ec5e2c7
Fix HiveSyncTool test by adding tez deps and fixing hive confs. Enabl…
May 23, 2022
8cb26c7
Fix run_hoodie_app.sh to find correct spark bundle
May 23, 2022
c561688
Fix HiveTestService Setup
May 23, 2022
451b498
Add fail-never flag to get complete run
May 23, 2022
2cada82
Resolve log4j dependency conflict in hudi-cli
May 26, 2022
9040aa4
Resolve netty dependency conflict in hudi-cli
May 29, 2022
5105d15
Disable delta streamer tests that use mini dfs in hudi utilities
May 29, 2022
e0805de
Resolve log4j dependency conflict in hudi-integ-test
May 30, 2022
3e42649
Resolve log4j dependency conflict in hudi-integ-test 2
May 30, 2022
d67e07e
Remove fail never and enabled all azure ci modules
May 31, 2022
af5fcb1
Add fail-never again to certain modules
Jun 1, 2022
6b3338b
Fix HiveIncrementalPuller tests by increasing sleep, adding tez, and …
Jun 1, 2022
c02afe0
Fix log4j dep conflict with ITTestHoodieSyncCommand.testValidateSync,…
Jun 2, 2022
3007879
Fix run_hoodie scripts to use correct spark bundle
Jun 3, 2022
3694b86
Increase wait time for services to be up in ITTestBase
Jun 3, 2022
53c5159
Fix bundles to include hudi-spark3_2.12
Jun 3, 2022
d3897c6
Increase wait time in HiveTestService
Jun 5, 2022
4be202b
Increase timeout for IT in azure yml
Jun 6, 2022
590e909
Exclude spark's hadoop in hudi-cli, disable IT FlinkTest, remove fail…
Jun 7, 2022
a29afee
Disable flaky hive sync tests
Jun 7, 2022
6a6cbd6
increase wait time in hivetestservice
Jun 8, 2022
34d7cc3
remove failnever in IT test section
Jun 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 17 additions & 25 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ pool:

variables:
MAVEN_OPTS: '-Dcheckstyle.skip=true -Drat.skip=true -Djacoco.skip=true'
SPARK_VERSION: '2.4.4'
HADOOP_VERSION: '2.7'
SPARK_VERSION: '3.2.1'
HADOOP_VERSION: '3.2'
SPARK_ARCHIVE: spark-$(SPARK_VERSION)-bin-hadoop$(HADOOP_VERSION)
SPARK_PROFILE: scala-2.12,spark3
EXCLUDE_TESTED_MODULES: '!hudi-examples/hudi-examples-common,!hudi-examples/hudi-examples-flink,!hudi-examples/hudi-examples-java,!hudi-examples/hudi-examples-spark,!hudi-common,!hudi-flink-datasource/hudi-flink,!hudi-client/hudi-spark-client,!hudi-client/hudi-client-common,!hudi-client/hudi-flink-client,!hudi-client/hudi-java-client,!hudi-cli,!hudi-utilities,!hudi-sync/hudi-hive-sync'

stages:
Expand All @@ -40,7 +41,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'clean install'
options: -T 2.5C -DskipTests
options: -T 2.5C -DskipTests -P $(SPARK_PROFILE)
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -49,7 +50,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Punit-tests -pl hudi-common,hudi-flink-datasource/hudi-flink,hudi-client/hudi-spark-client
options: -P $(SPARK_PROFILE),unit-tests -pl hudi-common,hudi-flink-datasource/hudi-flink,hudi-client/hudi-spark-client
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -58,7 +59,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pfunctional-tests -pl hudi-common,hudi-flink-datasource/hudi-flink
options: -P $(SPARK_PROFILE),functional-tests -pl hudi-common,hudi-flink-datasource/hudi-flink
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -71,7 +72,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'clean install'
options: -T 2.5C -DskipTests
options: -T 2.5C -DskipTests -P $(SPARK_PROFILE)
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -80,7 +81,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pfunctional-tests -pl hudi-client/hudi-spark-client
options: -P $(SPARK_PROFILE),functional-tests -pl hudi-client/hudi-spark-client
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -93,7 +94,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'clean install'
options: -T 2.5C -DskipTests
options: -T 2.5C -P $(SPARK_PROFILE) -DskipTests
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -102,7 +103,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Punit-tests -pl hudi-client/hudi-client-common,hudi-client/hudi-flink-client,hudi-client/hudi-java-client,hudi-cli,hudi-utilities,hudi-sync/hudi-hive-sync
options: -P $(SPARK_PROFILE),unit-tests -pl hudi-client/hudi-client-common,hudi-client/hudi-flink-client,hudi-client/hudi-java-client,hudi-cli,hudi-utilities,hudi-sync/hudi-hive-sync
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -111,7 +112,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pfunctional-tests -pl hudi-client/hudi-client-common,hudi-client/hudi-flink-client,hudi-client/hudi-java-client,hudi-cli,hudi-utilities,hudi-sync/hudi-hive-sync
options: -P $(SPARK_PROFILE),functional-tests -pl hudi-client/hudi-client-common,hudi-client/hudi-flink-client,hudi-client/hudi-java-client,hudi-cli,hudi-utilities,hudi-sync/hudi-hive-sync
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -124,7 +125,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'clean install'
options: -T 2.5C -DskipTests
options: -T 2.5C -DskipTests -P $(SPARK_PROFILE)
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -133,7 +134,7 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Punit-tests -pl $(EXCLUDE_TESTED_MODULES)
options: -P $(SPARK_PROFILE),unit-tests -pl $(EXCLUDE_TESTED_MODULES)
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -142,29 +143,20 @@ stages:
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pfunctional-tests -pl $(EXCLUDE_TESTED_MODULES)
options: -P $(SPARK_PROFILE),functional-tests -pl $(EXCLUDE_TESTED_MODULES)
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
- job: IT
displayName: IT modules
timeoutInMinutes: '120'
timeoutInMinutes: '180'
steps:
- task: Maven@3
displayName: maven install
inputs:
mavenPomFile: 'pom.xml'
goals: 'clean install'
options: -T 2.5C -Pintegration-tests -DskipTests
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
- task: Maven@3
displayName: UT integ-test
inputs:
mavenPomFile: 'pom.xml'
goals: 'test'
options: -Pintegration-tests -DskipUTs=false -DskipITs=true -pl hudi-integ-test test
options: -T 2.5C -P$(SPARK_PROFILE),integration-tests -DskipTests
publishJUnitResults: false
jdkVersionOption: '1.8'
mavenOptions: '-Xmx4g $(MAVEN_OPTS)'
Expand All @@ -180,5 +172,5 @@ stages:
tar -xvf $(Pipeline.Workspace)/$(SPARK_ARCHIVE).tgz -C $(Pipeline.Workspace)/
mkdir /tmp/spark-events/
- script: |
mvn $(MAVEN_OPTS) -Pintegration-tests verify
mvn $(MAVEN_OPTS) -P $(SPARK_PROFILE),integration-tests verify
displayName: IT
Loading