Skip to content

feat: Support both Spark2.4 and Spark3.4 in one module#26729

Open
wangkewen wants to merge 1 commit intoprestodb:masterfrom
wangkewen:pt-build-spark23
Open

feat: Support both Spark2.4 and Spark3.4 in one module#26729
wangkewen wants to merge 1 commit intoprestodb:masterfrom
wangkewen:pt-build-spark23

Conversation

@wangkewen
Copy link
Copy Markdown

@wangkewen wangkewen commented Dec 2, 2025

Description

Previous approach (#25552) supporting both Spark2.4 and Spark3.4 utilizes two separate modules which incurred some release inconsistence issues. This change merges this change into existing module without generating separate modules.

Motivation and Context

It tries to resolve one issue mentioned here #25833.

Impact

Test Plan

Build for Spark2

$ mvn clean install -U -DskipTests -Dmaven.javadoc.skip=true -T1C -pl '!presto-docs'
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  20:28 min (Wall Clock)
[INFO] Finished at: 2026-02-11T14:16:44-08:00
[INFO] ------------------------------------------------------------------------

Build for Spark3

$ mvn clean install -U -DskipTests -Dmaven.javadoc.skip=true -Dspark-version=3 -T1C -pl '!presto-docs'
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  19:45 min (Wall Clock)
[INFO] Finished at: 2026-02-11T13:17:55-08:00
[INFO] ------------------------------------------------------------------------


Contributor checklist

  • Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.
  • If adding new dependencies, verified they have an OpenSSF Scorecard score of 5.0 or higher (or obtained explicit TSC approval for lower scores).

Release Notes

Please follow release notes guidelines and fill in the release notes below.

If release note is NOT required, use:

== NO RELEASE NOTE ==

@wangkewen wangkewen requested review from a team and shrinidhijoshi as code owners December 2, 2025 18:43
@prestodb-ci prestodb-ci added the from:Meta PR from Meta label Dec 2, 2025
@wangkewen wangkewen requested a review from ghelmling December 3, 2025 00:55
@wangkewen wangkewen changed the title Support both Spark2.4 and Spark3.4 in one module feat(pos) Support both Spark2.4 and Spark3.4 in one module Jan 23, 2026
@wangkewen wangkewen changed the title feat(pos) Support both Spark2.4 and Spark3.4 in one module feat(pos): Support both Spark2.4 and Spark3.4 in one module Jan 23, 2026
@wangkewen wangkewen changed the title feat(pos): Support both Spark2.4 and Spark3.4 in one module feat: Support both Spark2.4 and Spark3.4 in one module Jan 23, 2026
@shrinidhijoshi
Copy link
Copy Markdown
Collaborator

Thanks for getting started on this @wangkewen .

It looks like there we might have to fix the destination directory path from
java-spark2/com.facebook.presto..... -> java-spark2/com/facebook/presto/.... ?

Copy link
Copy Markdown
Collaborator

@shrinidhijoshi shrinidhijoshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making changes. Left some comments

@wangkewen
Copy link
Copy Markdown
Author

Thanks for getting started on this @wangkewen .

It looks like there we might have to fix the destination directory path from java-spark2/com.facebook.presto..... -> java-spark2/com/facebook/presto/.... ?

yeah, updated.

@wangkewen wangkewen force-pushed the pt-build-spark23 branch 2 times, most recently from 2af8fff to 301453d Compare February 13, 2026 01:06
@czentgr
Copy link
Copy Markdown
Contributor

czentgr commented Feb 17, 2026

@wangkewen Please rebase. The CI test issues should be resolved now.

…er-interface

Delete two modules presto-spark-classloader-spark2 and presto-spark-classloader-spark3 in the previous approach to support both Spark2 and Spark3. Move the previous implementations into subdirs into existing module presto-spark-classloader-interface. Clean up dependencies in pom files of root, presto-spark-base and presto-spark-launcher.
@tdcmeehan
Copy link
Copy Markdown
Contributor

@wangkewen can you please rebase again to fix up the CI. We need this before we can cut the release.

@steveburnett
Copy link
Copy Markdown
Contributor

@wangkewen when you can, please rebase to re-run and hopefully pass the CI checks. This PR is needed for the next release of Presto.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants