Skip to content

[SPARK-49774][BUILD] Upgrade guava to 33.3.1-jre #48233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
panbingkun wants to merge 4 commits into from
Closed

[SPARK-49774][BUILD] Upgrade guava to 33.3.1-jre #48233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ce68299
[SPARK-49774][BUILD] Upgrade guava to 33.3.1-jre
panbingkun Sep 24, 2024
a45ef60
Merge branch 'master' into SPARK-49774
panbingkun Sep 25, 2024
36eea82
update
panbingkun Oct 22, 2024
f74b788
update
panbingkun Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion connector/docker-integration-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>33.1.0-jre</version>
<version>${guava.version}</version>
<scope>test</scope>
</dependency>
<dependency>
Expand Down
6 changes: 3 additions & 3 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ breeze-macros_2.13/2.1.0//breeze-macros_2.13-2.1.0.jar
breeze_2.13/2.1.0//breeze_2.13-2.1.0.jar
bundle/2.24.6//bundle-2.24.6.jar
cats-kernel_2.13/2.8.0//cats-kernel_2.13-2.8.0.jar
checker-qual/3.42.0//checker-qual-3.42.0.jar
checker-qual/3.43.0//checker-qual-3.43.0.jar
chill-java/0.10.0//chill-java-0.10.0.jar
chill_2.13/0.10.0//chill_2.13-0.10.0.jar
commons-cli/1.9.0//commons-cli-1.9.0.jar
Expand Down Expand Up @@ -63,14 +63,14 @@ derby/10.16.1.1//derby-10.16.1.1.jar
derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar
derbytools/10.16.1.1//derbytools-10.16.1.1.jar
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
error_prone_annotations/2.26.1//error_prone_annotations-2.26.1.jar
error_prone_annotations/2.28.0//error_prone_annotations-2.28.0.jar
esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar
failureaccess/1.0.2//failureaccess-1.0.2.jar
flatbuffers-java/24.3.25//flatbuffers-java-24.3.25.jar
gcs-connector/hadoop3-2.2.25/shaded/gcs-connector-hadoop3-2.2.25-shaded.jar
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.11.0//gson-2.11.0.jar
guava/33.2.1-jre//guava-33.2.1-jre.jar
guava/33.3.1-jre//guava-33.3.1-jre.jar
hadoop-aliyun/3.4.1//hadoop-aliyun-3.4.1.jar
hadoop-annotations/3.4.1//hadoop-annotations-3.4.1.jar
hadoop-aws/3.4.1//hadoop-aws-3.4.1.jar
Expand Down
4 changes: 2 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@
<!-- org.apache.commons/commons-pool2/-->
<commons-pool2.version>2.12.0</commons-pool2.version>
<datanucleus-core.version>4.1.17</datanucleus-core.version>
<guava.version>33.2.1-jre</guava.version>
Copy link
Contributor Author

@panbingkun panbingkun Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After #42493, I think we should be able to unify it now ? @LuciferYang @pan3793 @dongjoon-hyun

Copy link
Member

@dongjoon-hyun dongjoon-hyun Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, IIRC, that was the proposal.

Copy link
Contributor

@LuciferYang LuciferYang Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I believe that we can continue to retain this configuration way, as it may be beneficial to allow the connect module to use an independent version of Guava.

This is just my personal opinion. We can also hear what others think or discuss it in the mail list

<guava.version>33.3.1-jre</guava.version>
<gson.version>2.11.0</gson.version>
<janino.version>3.1.9</janino.version>
<jersey.version>3.0.12</jersey.version>
Expand Down Expand Up @@ -292,7 +292,7 @@
<spark.test.docker.removePulledImage>true</spark.test.docker.removePulledImage>

<!-- Version used in Connect -->
<connect.guava.version>33.2.1-jre</connect.guava.version>
<connect.guava.version>${guava.version}</connect.guava.version>
<guava.failureaccess.version>1.0.2</guava.failureaccess.version>
<io.grpc.version>1.62.2</io.grpc.version>
<mima.version>1.1.4</mima.version>
Expand Down
9 changes: 7 additions & 2 deletions project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -954,7 +954,12 @@ object Unsafe {
object DockerIntegrationTests {
// This serves to override the override specified in DependencyOverrides:
lazy val settings = Seq(
dependencyOverrides += "com.google.guava" % "guava" % "33.1.0-jre"
dependencyOverrides += {
val guavaVersion =
SbtPomKeys.effectivePom.value.getProperties.get(
"guava.version").asInstanceOf[String]
"com.google.guava" % "guava" % guavaVersion
Copy link
Member

@pan3793 pan3793 Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, how will the Guava version be resolved without this override?

}
)
}

Expand Down Expand Up @@ -1057,7 +1062,7 @@ object KubernetesIntegrationTests {
* Overrides to work around sbt's dependency resolution being different from Maven's.
*/
object DependencyOverrides {
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("33.1.0-jre")
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("33.3.1-jre")
lazy val settings = Seq(
dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
dependencyOverrides += "jline" % "jline" % "2.14.6",
Expand Down