[SPARK-54049][BUILD] Shade com.google.thirdparty package to fix Guava class conflicts in spark 4.0

[vinodkc]

We upgraded Guava from 14.0.1 to 30+ in  spark 4.0 . Guava 33.4.0 used in Spark 4 consists of two main packages:

• com.google.common
• com.google.thirdparty

Prior to this PR, only the com.google.common package was shaded into the spark-network-common jar, while classes under com.google.thirdparty remained unshaded in the spark-network-common jar. This partial shading causes classloading conflicts and runtime errors when a downstream project depends on both Spark and its own version of Guava.

Eg: calls to guava class com.google.common.net.InternetDomainName fails with the following error:

Caused by: java.lang.NoSuchFieldError: EXACT
        at com.google.common.net.InternetDomainName.findSuffixOfType(InternetDomainName.java:226)
        at com.google.common.net.InternetDomainName.publicSuffixIndex(InternetDomainName.java:185)
        at com.google.common.net.InternetDomainName.hasPublicSuffix(InternetDomainName.java:400)
        at com.eadx.Domain$.printDomainInfo(Domain.scala:16)
        at com.eadx.TestApp$.main(TestApp.scala:16)

Root Cause:
com.google.common.net.InternetDomainName uses classes from com.google.thirdparty.publicsuffix.
The classloader resolves com.google.common.net.InternetDomainName from the downstream Guava jar, while com.google.thirdparty.publicsuffix.PublicSuffixPatterns is loaded from Spark 4.x Guava classes, leading to binary incompatibility.

Example diagnostic:

InternetDomainName → guava-32.0.0-jre.jar
(target/.../guava-32.0.0-jre.jar)

PublicSuffixPatterns → spark-network-common_2.13-4.0.0.jar
(target/.../spark-network-common_2.13-4.0.0.jar)

What changes were proposed in this pull request?

This PR ensures package com.google.thirdparty is also shaded and isolated under the sparkproject namespace in Spark, preventing downstream class conflicts and runtime errors.

Why are the changes needed?

These changes are necessary to prevent runtime errors and class conflicts for downstream projects that depend on both Spark and Guava by restoring proper isolation of shaded Guava classes in spark

Does this PR introduce any user-facing change?

No

How was this patch tested?

No new test cases added; used existing UT and IT.

Was this patch authored or co-authored using generative AI tooling?

No

[pan3793]

This is a good catch! I checked the jar, and com.google.thirdparty is the only missing package.

$ jar -tf spark-network-common_2.13-4.1.0-preview3.jar | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF'
org/
org/apache/
com/
com/google/
com/google/thirdparty/
com/google/thirdparty/publicsuffix/
com/google/thirdparty/publicsuffix/PublicSuffixPatterns.class
com/google/thirdparty/publicsuffix/PublicSuffixType.class
com/google/thirdparty/publicsuffix/TrieParser.class

[pan3793]

cc @LuciferYang

[cloud-fan]

Is it a regression in spark master branch or it's a long standing issue?

[pan3793]

@cloud-fan it's a regression since 4.0, because we upgraded Guava from 14.0.1 to 30+ in 4.0

[vinodkc]

@cloud-fan , @pan3793 , Can we backport this change to 4.0.0 as well?
If yes, should I create a separate follow-up PR?

[vrozov]

Any changes necessary to SBT build?

[pan3793]

Any changes necessary to SBT build?

@vrozov IIRC, the current sbt building script does not process shading and relocation properly.

[LuciferYang]

@vinodkc Please update the PR title to more clearly explain what this PR is intended to do. The current description is somewhat misleading to me, leading me to mistakenly assume that after this PR, the Spark-network-common module would no longer shade Guava. Thanks

[vrozov]

@vrozov IIRC, the current sbt building script does not process shading and relocation properly.

@pan3793 Is there a JIRA that explains what is wrong with shading in SBT build?

[pan3793]

@vrozov IIRC, the current sbt building script does not process shading and relocation properly.

@pan3793 Is there a JIRA that explains what is wrong with shading in SBT build?

@vrozov I think no, so far, sbt is dev only, the developer is likely to only do improvements/fixes when something goes wrong, e.g., wrong dependency version resolution causes CI failure.

[vinodkc]

@LuciferYang , I've updated the PR title and description.
Thanks

[HyukjinKwon]

Merged to master and branch-4.1.

[pan3793]

@HyukjinKwon this should go branch-4.0 too, @vinodkc could you please open a 4.0 backport PR?