Skip to content

[SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4 #37281

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sunchao wants to merge 4 commits into from
Closed

[SPARK-39863][BUILD] Upgrade Hadoop to 3.3.4 #37281

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fb0b852
initial commit
sunchao May 6, 2022
f5adac2
empty commit
sunchao Aug 3, 2022
fe8c63b
add staging to SparkSubmit
sunchao Aug 4, 2022
467a7fd
remove staging settings
sunchao Aug 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 11 additions & 13 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ automaton/1.11-8//automaton-1.11-8.jar
avro-ipc/1.11.0//avro-ipc-1.11.0.jar
avro-mapred/1.11.0//avro-mapred-1.11.0.jar
avro/1.11.0//avro-1.11.0.jar
aws-java-sdk-bundle/1.11.1026//aws-java-sdk-bundle-1.11.1026.jar
aws-java-sdk-bundle/1.12.132//aws-java-sdk-bundle-1.12.132.jar
Copy link
Member

@dongjoon-hyun dongjoon-hyun Aug 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunchao Oh, master branch found that this is inconsistent from the official Apache Hadoop 3.3.4.
The staging artifacts in this PR seems to be outdated somehow. Let me make a followup.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Aug 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HADOOP-18344 changes AWS SDK at Apache Hadoop 3.3.4 RC1. So, this RC0-based dependency file is not updated. I found that the dependency test is not triggered due to the Python Linter flaky failure.

Copy link
Contributor

@steveloughran steveloughran Aug 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are at 1.12.262. there's a CVE out on the aws sdk transfer manager for releases < 1.12.261, which the s3a connector isn't exposed to (it's only for downloads through that class), but which other apps using the same sdk may be.

azure-data-lake-store-sdk/2.3.9//azure-data-lake-store-sdk-2.3.9.jar
azure-keyvault-core/1.0.0//azure-keyvault-core-1.0.0.jar
azure-storage/7.0.1//azure-storage-7.0.1.jar
Expand All @@ -52,7 +52,6 @@ commons-math3/3.6.1//commons-math3-3.6.1.jar
commons-pool/1.5.4//commons-pool-1.5.4.jar
commons-text/1.9//commons-text-1.9.jar
compress-lzf/1.1//compress-lzf-1.1.jar
cos_api-bundle/5.6.19//cos_api-bundle-5.6.19.jar
curator-client/2.13.0//curator-client-2.13.0.jar
curator-framework/2.13.0//curator-framework-2.13.0.jar
curator-recipes/2.13.0//curator-recipes-2.13.0.jar
Expand All @@ -66,18 +65,17 @@ generex/1.0.2//generex-1.0.2.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
hadoop-aliyun/3.3.3//hadoop-aliyun-3.3.3.jar
hadoop-annotations/3.3.3//hadoop-annotations-3.3.3.jar
hadoop-aws/3.3.3//hadoop-aws-3.3.3.jar
hadoop-azure-datalake/3.3.3//hadoop-azure-datalake-3.3.3.jar
hadoop-azure/3.3.3//hadoop-azure-3.3.3.jar
hadoop-client-api/3.3.3//hadoop-client-api-3.3.3.jar
hadoop-client-runtime/3.3.3//hadoop-client-runtime-3.3.3.jar
hadoop-cloud-storage/3.3.3//hadoop-cloud-storage-3.3.3.jar
hadoop-cos/3.3.3//hadoop-cos-3.3.3.jar
hadoop-openstack/3.3.3//hadoop-openstack-3.3.3.jar
hadoop-aliyun/3.3.4//hadoop-aliyun-3.3.4.jar
hadoop-annotations/3.3.4//hadoop-annotations-3.3.4.jar
hadoop-aws/3.3.4//hadoop-aws-3.3.4.jar
hadoop-azure-datalake/3.3.4//hadoop-azure-datalake-3.3.4.jar
hadoop-azure/3.3.4//hadoop-azure-3.3.4.jar
hadoop-client-api/3.3.4//hadoop-client-api-3.3.4.jar
hadoop-client-runtime/3.3.4//hadoop-client-runtime-3.3.4.jar
hadoop-cloud-storage/3.3.4//hadoop-cloud-storage-3.3.4.jar
hadoop-openstack/3.3.4//hadoop-openstack-3.3.4.jar
hadoop-shaded-guava/1.1.1//hadoop-shaded-guava-1.1.1.jar
hadoop-yarn-server-web-proxy/3.3.3//hadoop-yarn-server-web-proxy-3.3.3.jar
hadoop-yarn-server-web-proxy/3.3.4//hadoop-yarn-server-web-proxy-3.3.4.jar
hive-beeline/2.3.9//hive-beeline-2.3.9.jar
hive-cli/2.3.9//hive-cli-2.3.9.jar
hive-common/2.3.9//hive-common-2.3.9.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@
<slf4j.version>1.7.32</slf4j.version>
<log4j.version>2.18.0</log4j.version>
<!-- make sure to update IsolatedClientLoader whenever this version is changed -->
<hadoop.version>3.3.3</hadoop.version>
<hadoop.version>3.3.4</hadoop.version>
<protobuf.version>2.5.0</protobuf.version>
<yarn.version>${hadoop.version}</yarn.version>
<zookeeper.version>3.6.2</zookeeper.version>
Expand Down
2 changes: 1 addition & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ private[hive] object IsolatedClientLoader extends Logging {
// If the error message contains hadoop, it is probably because the hadoop
// version cannot be resolved.
val fallbackVersion = if (VersionUtils.isHadoop3) {
"3.3.3"
"3.3.4"
} else {
"2.7.4"
}
Expand Down