[pos] Support building presto-on-spark for spark2.4 and spark3.4

[wangkewen]

Description

This initial PR prepares the release of Presto on Spark 3.4.1 by incorporating updates from this pull request by @shrinidhijoshi.

Prior to the stable release of Presto on Spark 3 (e.g., Spark 3.4.1), code compatibility is maintained for both Spark 2.4 (the current version) and Spark 3.4. To support this, a Maven profile named spark3 has been created to facilitate building Presto on Spark 3.

All code changes are made within the presto-spark-classloader-interface module. To support different implementations for Spark versions 2.4 and 3.4, two separate modules have been created: presto-spark-classloader-spark2 and presto-spark-classloader-spark3. These modules contain the version-specific code, including a new PrestoSparkUtils.

                             presto-spark-classloader-spark2  (spark 2.4) 
                                         /                             
                                        /                              
presto-spark-classloader-interface ---                                
                                        \                               
                                         \                              
                             presto-spark-classloader-spark3 (spark 3.4)

In this PR, it adds support for shuffle deserialize in PrestoSparkShuffleSerializer.java

Motivation and Context

It is needed for upgrading Presto on Spark to support Spark3 (e.g., Spark 3.4.1) .

Impact

Test Plan

Internal pt verifier test: x1125914587947600

Contributor checklist

• Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== NO RELEASE NOTE ==

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: wangkewen / name: Kewen Wang (cf2f3fa)

[unidevel]

@wangkewen Thansk for your contribution, can you sign the EasyCLA and resolve the conflicts first? Also since this is a new feature, it is better to provide release notes and documentation about setup and configurations.

[shrinidhijoshi]

@wangkewen Thanks for making the changes. I have left some initial comments on amount of classes that are duplicated.

I see quite a few pom issues that CI is flagging, let's work through fixing all those.

[shrinidhijoshi]

@wangkewen Thanks for addressing the comments. I did another pass and left some more comments.

Also, please update the PR description in line with presto contributor guidlines.

[wangkewen]

rebase on @shrinidhijoshi's PR-25794

[wangkewen]

rebase on master to include @shrinidhijoshi 's PR-25797

[shrinidhijoshi]

Thanks for addressing the comments @wangkewen . Based on the latest changes I left a few more comments.

[shrinidhijoshi]

@wangkewen Created #25833 to track adding spark3.4 based CI tests as a follow up

[shrinidhijoshi]

LGTM! Thanks for working through all the comments @wangkewen

[rschlussel]

Change looks good. Can you clean up the commits? (squash them and make sure the commit title and message follows our guidelines)? https://github.com/prestodb/presto/wiki/Review-and-Commit-guidelines#commit-formatting-and-pull-requests

[wangkewen]

@rschlussel I have updated it.

[facebook-github-bot]

@wangkewen has imported this pull request. If you are a Meta employee, you can view this in D80638570.