Skip to content

[SPARK-46668][DOCS] Parallelize Sphinx build of Python API docs #44680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nchammas wants to merge 3 commits into from
Closed

[SPARK-46668][DOCS] Parallelize Sphinx build of Python API docs #44680

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5d30cc4
sphinx -> 4.5
nchammas Jan 11, 2024
23cb58b
enable parallel sphinx workers
nchammas Jan 11, 2024
9c232ee
trigger build
nchammas Jan 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -750,7 +750,7 @@ jobs:
Rscript -e "devtools::install_version('preferably', version='0.4', repos='https://cloud.r-project.org')"
- name: Install dependencies for documentation generation
run: |
python3.9 -m pip install 'sphinx==4.2.0' mkdocs 'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0'
python3.9 -m pip install 'sphinx==4.5.0' mkdocs 'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0'
python3.9 -m pip install ipython_genutils # See SPARK-38517
python3.9 -m pip install sphinx_plotly_directive 'numpy>=1.20.0' pyarrow pandas 'plotly>=4.8'
python3.9 -m pip install 'docutils<0.18.0' # See SPARK-39421
Expand Down
2 changes: 1 addition & 1 deletion dev/create-release/spark-rm/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ ENV DEBCONF_NONINTERACTIVE_SEEN true
# These arguments are just for reuse and not really meant to be customized.
ARG APT_INSTALL="apt-get install --no-install-recommends -y"

ARG PIP_PKGS="sphinx==4.2.0 mkdocs==1.1.2 numpy==1.20.3 pydata_sphinx_theme==0.13.3 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==3.1.2 twine==3.4.1 sphinx-plotly-directive==0.1.3 sphinx-copybutton==0.5.2 pandas==1.5.3 pyarrow==3.0.0 plotly==5.4.0 markupsafe==2.0.1 docutils<0.17 grpcio==1.59.3 protobuf==4.21.6 grpcio-status==1.59.3 googleapis-common-protos==1.56.4"
ARG PIP_PKGS="sphinx==4.5.0 mkdocs==1.1.2 numpy==1.20.3 pydata_sphinx_theme==0.13.3 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==3.1.2 twine==3.4.1 sphinx-plotly-directive==0.1.3 sphinx-copybutton==0.5.2 pandas==1.5.3 pyarrow==3.0.0 plotly==5.4.0 markupsafe==2.0.1 docutils<0.17 grpcio==1.59.3 protobuf==4.21.6 grpcio-status==1.59.3 googleapis-common-protos==1.56.4"
ARG GEM_PKGS="bundler:2.3.8"

# Install extra needed repos and refresh.
Expand Down
3 changes: 1 addition & 2 deletions dev/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ ipython
nbsphinx
numpydoc
jinja2
sphinx==4.2.0
sphinx==4.5.0
sphinx-plotly-directive
sphinx-copybutton
docutils<0.18.0
Expand Down Expand Up @@ -67,4 +67,3 @@ torcheval

# DeepspeedTorchDistributor dependencies
deepspeed; sys_platform != 'darwin'

2 changes: 1 addition & 1 deletion python/docs/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# Minimal makefile for Sphinx documentation

# You can set these variables from the command line.
SPHINXOPTS ?= "-W"
SPHINXOPTS ?= "-W" "-j" "auto"
Copy link
Member

@dongjoon-hyun dongjoon-hyun Sep 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this breaks Python API doc generation in many core machines because this means the number of parallel SparkSubmit invocation of PySpark. In addition, given that each PySpark currently is launched with local[*], this ends up N * N pyspark.daemon.

As of today, this setting seems to work on low-core machine like GitHub Action runners (4 cores). This breaks Python documentations build even on M3 Max environment and this is worse on large EC2 machines (c7i.24xlarge). You can see the failure locally like this.

$ build/sbt package -Phive-thriftserver
$ cd python/docs
$ make html
...
java.lang.OutOfMemoryError: Java heap space
...
24/09/16 14:09:55 WARN PythonRunner: Incomplete task 7.0 in stage 30 (TID 177) interrupted: Attempting to kill Python Worker
...
make: *** [html] Error 2

Copy link
Contributor Author

@nchammas nchammas Sep 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of today, this setting seems to work on low-core machine like GitHub Action runners (4 cores).

It seems something about the build has changed since this PR was merged in ~8 months ago. As stated in the PR description, I tested auto on a 16-core Intel and it worked fine. And I remember running the build many times to get a representative runtime to report.

this ends up N * N pyspark.daemon

Perhaps this was not the case back in January, for some reason. 🤷‍♂️

SPHINXBUILD ?= sphinx-build
SOURCEDIR ?= source
BUILDDIR ?= build
Expand Down