Skip to content

Conversation

@fpgmaas
Copy link

@fpgmaas fpgmaas commented Oct 19, 2023

This PR does the following:

  • Facilitates running unittests for multiple versions of Python and pyspark on each PR, see here
  • Drops support for Python 3.7 since it is EOL
  • Also drops support for Pyspark 2.x due to the dropped support for Python 3.7
  • Updates some of the dependencies in the docs dependency group.
  • Fixes a few linting issues.
  • Temporarily fixes a failing unittest

Let me know your thoughts :)

@SemyonSinchenko
Copy link
Collaborator

Python 3.7 may be EOL, but Spark 2 is still using in a lot of companies... Do we really need to drop the support of Spark 2? @MrPowers what do you think about it?

@fpgmaas
Copy link
Author

fpgmaas commented Oct 24, 2023

Python 3.7 may be EOL, but Spark 2 is still using in a lot of companies... Do we really need to drop the support of Spark 2? @MrPowers what do you think about it?

Valid point. I proposed to drop support for Spark 2 as a consequence of dropping support for Python 3.7, since that seems to be the latest version that supports Spark 2, see here.

I tried to run the tests with Spark 2 and Python 3.8, but that failed.

So I believe that if we want to keep supporting Spark 2 then we should also keep supporting Python 3.7. But this will prevent us from updating some dependencies and it requires us to keep using some workarounds such as this one.

If it's preferred to not drop support for Python 3.7 and Spark 2, let me know! I will try to update the PR accordingly.

@MrPowers
Copy link
Collaborator

Let's keep Python 3.7 & Spark 2 support for now.

It's super annoying, but I know lots of users are on legacy versions. But let's drop soon.

@MrPowers MrPowers changed the title Test against multiple versions of Python and drop support for Python 3.7 Bump to quinn 1.0 Mar 14, 2024
@fpgmaas fpgmaas closed this Jul 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants