[SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.core package

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to release a separate pyspark-connect package, see also SPIP: Pure Python Package in PyPI (Spark Connect).

Today's PySpark package is roughly as follows:

pyspark
├── *.py               # *Core / No Spark Connect support*
├── mllib              # MLlib / No Spark Connect support
├── resource           # Resource profile API / No Spark Connect support
├── streaming          # DStream (deprecated) / No Spark Connect support
├── ml                 # ML 
│   └── connect            # Spark Connect for ML
├── pandas             # API on Spark with/without Spark Connect support
└── sql                # SQL
    └── connect            # Spark Connect for SQL
        └── streaming      # Spark Connect for Structured Streaming

There will be two packages available, pyspark and pyspark-connect.

pyspark

Same as today’s PySpark. But Core module is factored out to pyspark.core.*. User-facing interface stays the same at pyspark.*.

pyspark
├── core               # *Core / No Spark Connect support*
├── mllib              # MLlib / No Spark Connect support
├── resource           # Resource profile API / No Spark Connect support
├── streaming          # DStream (deprecated) / No Spark Connect support
├── ml                 # ML 
│   └── connect            # Spark Connect for ML
├── pandas             # API on Spark with/without Spark Connect support
└── sql                # SQL
    └── connect            # Spark Connect for SQL
        └── streaming      # Spark Connect for Structured Streaming

pyspark-connect

Package after excluding modules that do not support Spark Connect, also excluding jars, that are, ml without jars:

pyspark
├── ml
│   └── connect
├── pandas
└── sql
    └── connect
        └── streaming

Why are the changes needed?

To provide a pure Python library that does not depend on JVM.

See also SPIP: Pure Python Package in PyPI (Spark Connect).

Does this PR introduce any user-facing change?

Yes, users can install pure Python library via pip install pyspark-connect.

How was this patch tested?

Manually tested the basic set of tests.

./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`

cd python
python packaging/connect/setup.py sdist
cd dist
conda create -y -n clean-py-3.11 python=3.11
conda activate clean-py-3.11
pip install pyspark-connect-4.0.0.dev0.tar.gz
python

>>> import pyspark
>>> from pyspark.sql import SparkSession
>>> spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
>>> spark.range(10).show()

+---+
| id|
+---+
|  0|
|  1|
|  2|
|  3|
|  4|
|  5|
|  6|
|  7|
|  8|
|  9|
+---+

They will be separated added, and set as a scheduled job in CI.

Was this patch authored or co-authored using generative AI tooling?

No.

[HyukjinKwon]

cc @zhengruifeng @grundprinzip @ueshin @hvanhovell @itholic @WeichenXu123 @mengxr @allisonwang-db @xinrong-meng @gatorsmile @cloud-fan This is ready for a look (before merging, should wait one more day for the SPIP to pass though)

[HyukjinKwon]

I restored the references for our internal API. Explicitly private attributes starting _ are not restored.

[HyukjinKwon]

Merged to master.