Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-668] Drop the support of Spark 3.0, 3.1, 3.2 #1653

Merged
merged 5 commits into from
Oct 27, 2024
Merged

Conversation

jiayuasu
Copy link
Member

@jiayuasu jiayuasu commented Oct 24, 2024

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

Apache Spark has officially announced the EOL of Spark 3.0, 3.1, and 3.2 long time ago. It has been a huge overhead for us to support Sedona on all these versions. We will drop the support of Spark 3.0, 3.1, and 3.2

Here is a summary of current Spark versions on major Spark hosting services

Spark 3.0 Spark 3.1 Spark 3.2 Spark 3.3
Databricks EOL Dec 19, 2024 Mar 18, 2025 Mar 1, 2026
AWS Glue EOL Unknown, Glue 3.0 N/A Unknown, Glue 4.0
AWS EMR July 24, 2025 July 24, 2025 January 25, 2026 January 25, 2026
Microsoft Fabric EOL EOL EOL Unknown
Microsoft Synapse EOL EOL EOL 3/31/2025
Google Dataproc EOL EOL N/A 2024/12/31

This PR did the following:

  • Removed all customized code for Spark 3.0, 3.1, 3.2
  • Updated the doc
  • Remove Spark 3.0, 3.1, 3.2 tests

The following functions will be affected on Spark 3.0, 3.1, 3.2

  • GeoParquet reader and writer
  • GeoParquet metadata reader and writer
  • Shapefile DataFrame reader
  • GeoPackage reader
  • CREATE TABLE with Geometry type

How was this patch tested?

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation.

@jiayuasu
Copy link
Member Author

@jbampton Isort does not throw error on my local machine. It turns out isort often gives inconsistent behavior. I think we should remove it from pre-commit. PyCQA/isort#1889

This reverts commit 12443f0.
@jiayuasu jiayuasu added this to the sedona-1.7.0 milestone Oct 27, 2024
@jiayuasu jiayuasu merged commit 3d0d54d into master Oct 27, 2024
38 checks passed
@jbampton
Copy link
Member

@jbampton Isort does not throw error on my local machine. It turns out isort often gives inconsistent behavior. I think we should remove it from pre-commit. PyCQA/isort#1889

Just saw this message @jiayuasu

I had a house mate move out so I was busy for about a week.

Thanks.

@jiayuasu jiayuasu deleted the SEDONA-668 branch November 4, 2024 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants