[SPARK-46668][DOCS] Parallelize Sphinx build of Python API docs

[nchammas]

What changes were proposed in this pull request?

Upgrade to Sphinx 4.5.0, which is the latest in the 4.x line and includes the fix for parallel builds on macOS.

Enable parallel Sphinx workers to build the Python API docs.

I experimented with a few different values, and auto seems to work best. Configuring 4 workers seems to yield the same improvement as auto, suggesting parallelization beyond that is ineffective due to some sort of resource contention. But I left it as auto since that's more dynamic and should work better across varied environments.

On my 16-core Intel workstation, the runtime of make html was cut by ~60%.

# `make html` @ master
real    43m51.167s
user    41m43.526s
sys     0m39.651s

# `make html` with parallel workers
real    17m8.424s
user    174m42.051s
sys     5m8.824s

Here on CI, the "Run documentation build" step seems also to have improved from the usual of ~50 minutes down to ~31 minutes.

Why are the changes needed?

This saves developer time and CI time.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I manually built and reviewed the docs using:

SKIP_SCALADOC=1 SKIP_SQLDOC=1 SKIP_RDOC=1 time bundle exec jekyll build

Was this patch authored or co-authored using generative AI tooling?

No.

[nchammas]

cc @itholic and @HyukjinKwon. This PR builds on #44012.

[itholic]

LGTM when CI passing.

Nice fix.

[HyukjinKwon]

Merged to master.

[dongjoon-hyun]

Unfortunately, this breaks Python API doc generation in many core machines because this means the number of parallel SparkSubmit invocation of PySpark. In addition, given that each PySpark currently is launched with local[*], this ends up N * N pyspark.daemon.

As of today, this setting seems to work on low-core machine like GitHub Action runners (4 cores). This breaks Python documentations build even on M3 Max environment and this is worse on large EC2 machines (c7i.24xlarge). You can see the failure locally like this.

$ build/sbt package -Phive-thriftserver
$ cd python/docs
$ make html
...
java.lang.OutOfMemoryError: Java heap space
...
24/09/16 14:09:55 WARN PythonRunner: Incomplete task 7.0 in stage 30 (TID 177) interrupted: Attempting to kill Python Worker
...
make: *** [html] Error 2

[nchammas]

As of today, this setting seems to work on low-core machine like GitHub Action runners (4 cores).

It seems something about the build has changed since this PR was merged in ~8 months ago. As stated in the PR description, I tested auto on a 16-core Intel and it worked fine. And I remember running the build many times to get a representative runtime to report.

this ends up N * N pyspark.daemon

Perhaps this was not the case back in January, for some reason. 🤷‍♂️

[dongjoon-hyun]

Here is a PR with new default setting while keeping the speed of GitHub Action CI. The users also override this via SPHINXOPT.

• [SPARK-49680][PYTHON] Limit Sphinx build parallelism to 4 by default #48129