[SPARK-46715][INFRA] Pin sphinxcontrib-*

[zhengruifeng]

What changes were proposed in this pull request?

Pin

• sphinxcontrib-applehelp==1.0.4
• sphinxcontrib-devhelp==1.0.2
• sphinxcontrib-htmlhelp==2.0.1
• sphinxcontrib-qthelp==1.0.3
• sphinxcontrib-serializinghtml==1.1.5

previously,
Install Python linter dependencies install sphinxcontrib-applehelp-1.0.7, and then Install dependencies for documentation generation reinstall it with sphinxcontrib-applehelp-1.0.4;

now,
Install Python linter dependencies install sphinxcontrib-applehelp-1.0.8, and Install dependencies for documentation generation keep this intallation:
Requirement already satisfied: sphinxcontrib-applehelp in /usr/local/lib/python3.9/dist-packages (from sphinx==4.5.0) (1.0.8)

Why are the changes needed?

doc build is failing with:

Sphinx version error:
The sphinxcontrib.applehelp extension used by this project needs at least Sphinx v5.0; it therefore cannot be built with this version.
make: *** [Makefile:35: html] Error 2
                    ------------------------------------------------
      Jekyll 4.3.3   Please append `--trace` to the `build` command 
                     for any additional information or backtrace. 
                    ------------------------------------------------
/__w/spark/spark/docs/_plugins/copy_api_dirs.rb:131:in `<top (required)>': Python doc generation failed (RuntimeError)
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.3/lib/jekyll/external.rb:57:in `require'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.3.3/lib/jekyll/external.rb:57:in `block in require_with_graceful_fail'

Sphinx version error:
The sphinxcontrib.devhelp extension used by this project needs at least Sphinx v5.0; it therefore cannot be built with this version.
make: *** [Makefile:35: html] Error 2
                    ------------------------------------------------
      Jekyll 4.3.3   Please append `--trace` to the `build` command 
                     for any additional information or backtrace. 
                    ------------------------------------------------
/__w/spark/spark/docs/_plugins/copy_api_dirs.rb:131:in `<top (required)>': Python doc generation failed (RuntimeError)

Does this PR introduce any user-facing change?

no

How was this patch tested?

ci

Was this patch authored or co-authored using generative AI tooling?

no

[yaooqinn]

Mind creating a JIRA ticket @zhengruifeng?

[zhengruifeng]

Mind creating a JIRA ticket @zhengruifeng?

sure, it's still under test, maybe we need to pin more deps.

[LuciferYang]

How about upgrade sphinx to 5.0.0 directly? I am conducting tests, and it seems ok:

#44729

https://github.com/LuciferYang/spark/actions/runs/7523170642/job/20476201559

[zhengruifeng]

How about upgrade sphinx to 5.0.0 directly? I am conducting tests, and it seems ok:

#44729

https://github.com/LuciferYang/spark/actions/runs/7523170642/job/20476201559

I remember @itholic had compared multiple versions of sphinx, and chose 4.5.0.
Other versions cause some render issues without failing in CI.

[LuciferYang]

How about upgrade sphinx to 5.0.0 directly? I am conducting tests, and it seems ok:
#44729
https://github.com/LuciferYang/spark/actions/runs/7523170642/job/20476201559

I remember @itholic had compared multiple versions of sphinx, and chose 4.5.0. Other versions cause some render issues without failing in CI.

OK, let's continue using 4.5.0 for now

[zhengruifeng]

FYI, 4.2.0 is chosen in #44012 (comment) and then 4.5.0 for performance in 5eec99d

let's stick to 4.x before more investigation on the built docs with higher versions.

[zhengruifeng]

thanks all

merged to master to restore CI

[nchammas]

The proximate reason for this failure is that we install pydata_sphinx_theme without a version qualifier here:

spark/.github/workflows/build_and_test.yml

Line 705 in 9f061b0

python3.9 -m pip install 'flake8==3.9.0' pydata_sphinx_theme 'mypy==0.982' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' numpydoc jinja2 'black==23.9.1'

Note that this is separate from the constraint that sphinx==4.5.0, so pip installs pydata_sphinx_theme at version 0.15.1. This in turn requires sphinx>=5.0, which pulls in the incompatible Sphinx extensions mentioned in the error you are seeing.

Collecting pydata_sphinx_theme
  Downloading pydata_sphinx_theme-0.15.1-py3-none-any.whl.metadata (7.3 kB)
Collecting sphinx>=5.0 (from pydata_sphinx_theme)
  Downloading sphinx-7.2.6-py3-none-any.whl.metadata (5.9 kB)
Collecting sphinxcontrib-applehelp (from sphinx>=5.0->pydata_sphinx_theme)
  Downloading sphinxcontrib_applehelp-1.0.8-py3-none-any.whl.metadata (2.3 kB)

A longer term solution would be to compile all of our requirements per Python version and install them in one go. That's the only way pip is going to have the information it needs to prevent this kind of problem from occurring.

[wbo4958]

@zhengruifeng, seems this issue is till in branch-3.5, is it possible to back-port to branch 3.5?

[zhengruifeng]

@wbo4958 It seems the pydata_sphinx_theme installations is not changed:
it installs pydata_sphinx_theme-0.15.1 in Install Python linter dependencies, and then pydata_sphinx_theme-0.13.3 in Install dependencies for documentation generation, in both a successful run (https://github.com/apache/spark/actions/runs/7507799677/job/20442119109) and a failed one (https://github.com/apache/spark/actions/runs/7522706520/job/20475105206)

The reason I think is pydata_sphinx_theme itself (both 0.15.1 and 0.13.3) doesn't specify the versions of sphinxcontrib-*:

│   ├── sphinxcontrib-applehelp               sphinxcontrib-applehelp is a Sphinx extension which outputs Apple help books
│   ├── sphinxcontrib-devhelp                 sphinxcontrib-devhelp is a sphinx extension which outputs Devhelp documents
│   ├── sphinxcontrib-htmlhelp>=2.0.0         sphinxcontrib-htmlhelp is a sphinx extension which renders HTML help files
│   ├── sphinxcontrib-jsmath                  A sphinx extension which renders display math in HTML via JavaScript
│   ├── sphinxcontrib-qthelp                  sphinxcontrib-qthelp is a sphinx extension which outputs QtHelp documents
│   └── sphinxcontrib-serializinghtml>=1.1.9  sphinxcontrib-serializinghtml is a sphinx extension which outputs "serialized"

then latest sphinxcontrib-* is not compatabile with sphinx.

As to 3.5, I am not sure, since it uses sphinx<3.1.0, probably needs a separate fix

[wbo4958]

Thx for your fix.