[WIP] [SPARK-3996]: Shade Jetty in Spark deliverables

[pwendell]

This patch piggy-back's on @vanzin's work to simplify the Guava shading,
and adds Jetty as a shaded library in Spark. Other than adding Jetty,
it consilidates the 's into the root pom. I found it was
a bit easier to follow that way, since you don't need to look into
child pom's to find out specific artifact sets included in shading.

[pwendell]

Jenkins, test this please.

[SparkQA]

Test build #26242 has started for PR 4252 at commit 62429c9.

• This patch merges cleanly.

[pwendell]

@vanzin - would you be able to take a look?

[vanzin]

Hi @pwendell ,

Did you check that jetty classes only show up in spark-core's jar? I see a jetty dependency in streaming/pom.xml, so I'm afraid that would cause jetty to show up there too.

That would mean doing the guava thing: declare dependencies as provided in the root pom, and promote them to compile in core/pom.xml.

[pwendell]

Hey @vanzin - here the jetty classes show up in both core and streaming. Is that a problem?

[vanzin]

It's not a real problem other than being wasteful, but it should be easy to fix it.

[SparkQA]

Test build #26242 has finished for PR 4252 at commit 62429c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class SparkListenerExecutorAdded(time: Long, executorId: String, executorInfo: ExecutorInfo)
  • case class SparkListenerExecutorRemoved(time: Long, executorId: String, reason: String)
  • class BlockMatrix(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26242/
Test PASSed.

[pwendell]

Changed approach to only shade inside of core jar per @vanzin's suggestion.

[vanzin]

nit: guava is promoted in network-common, so maybe rephrase this a little.

[SparkQA]

Test build #26250 has started for PR 4252 at commit ac2e214.

• This patch merges cleanly.

[vanzin]

Latest patch LGTM. Thanks for the extra cleanup too!

[vanzin]

Hey @pwendell,

Just remembered that there's an extra bit that needs to be taken care of when shading. core/pom.xml has a copy-dependencies task to copy certain artifacts into the build directory. That is used when running with SPARK_PREPEND_CLASSES since the raw compiled classes do not go through the relocation process.

That needs to be extended to Jetty too. Also I should probably take a look if that is still working for Guava since that's not a direct dependency anymore in core/...

[SparkQA]

Test build #26250 has finished for PR 4252 at commit ac2e214.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26250/
Test PASSed.

[pwendell]

@vanzin I've addressed your additional comments. Mind taking a look?

[vanzin]

LGTM. Ship it!

[SparkQA]

Test build #26337 has started for PR 4252 at commit 19f0710.

• This patch merges cleanly.

[SparkQA]

Test build #26337 has finished for PR 4252 at commit 19f0710.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26337/
Test PASSed.

[vanzin]

BTW I think @mccheah was interested in this originally.

[mccheah]

@ash211

Thanks for doing this. My colleagues and I will test this appropriately when we find the bandwidth!