[SPARK-45990][SPARK-45987][PYTHON][CONNECT] Upgrade protobuf to 4.25.1 to support Python 3.11

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to upgrade protobuf from 3.20.3 to 4.25.1 to fix PySpark failures on Python 3.11 environment for Apache Spark 4.0.0.

Why are the changes needed?

Currently, Python 3.11 Daily CI is failing.

• https://github.com/apache/spark/actions/workflows/build_python.yml

v4.25.0 is the first release where protobuf starts to support Python 3.11 officially.

• https://pypi.org/project/protobuf/4.25.0/
• https://pypi.org/project/protobuf/4.25.1/ (the latest)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

I also verified this manually.

$ python/run-tests --modules=pyspark-connect --parallelism=1 --python-executables=python3.11
...
Finished test(python3.11): pyspark.sql.connect.window (6s)
Tests passed in 780 seconds

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @grundprinzip and @HyukjinKwon

[HyukjinKwon]

Hmmm.. I remember we set this protobuf to 3.X because of some compat problem ... do you remember @grundprinzip ?

[HyukjinKwon]

From #38693

[dongjoon-hyun]

If then, we may need to drop Python 3.11 support officially from Spark Connect module.

[dongjoon-hyun]

We had better match the server and client via 4.25.1 from Apache Spark 4.0.0. Otherwise, it's difficult to support Python 3.11 and Python 3.12 and more.

[dongjoon-hyun]

WDYT about upgrade Java part too in this PR, @grundprinzip and @HyukjinKwon ?

[HyukjinKwon]

I'm good with this change. I just vaguely remember that there was a compat problem so would defer to @grundprinzip though.

[HyukjinKwon]

I'm fine with upgrading Java side too @hvanhovell but I don't have a very good insight about compat. I'll cc you Herman here too.

[HyukjinKwon]

Oh btw should probably regen the python protobuf code (https://github.com/apache/spark/blob/master/dev/connect-gen-protos.sh)

[dongjoon-hyun]

Thank you, @HyukjinKwon .

For now, apparently, PySpark protobuf upgrade seems to pass all PySpark connect CIs.

Let me tackle the issues step-by-step. I'll make a JAVA part very soon with the regenerated code.

I'm working in the context of the umbrella JIRA .

• SPARK-45981 Improve Python language test coverage

I hope to achieve all Python 3.8/3.9/3.10/3.11/3.12 for Apache Spark 4.0.0 for Spark Connect project in the community Daily CI level, @HyukjinKwon , @hvanhovell , @grundprinzip . If we need to abandon Python 3.11 and 3.12 for protobuf to be compatible with Spark 3.5.x or 3.4.0, we can officially revert this.

[dongjoon-hyun]

Merged to master.

[dongjoon-hyun]

Here is the Java part PR.

• [SPARK-45991][BUILD] Upgrade protobuf-java to 3.25.1 to match with protobuf 4.25.1 #43886

[grundprinzip]

Thanks for doing this.