[SPARK-46103][PYTHON][INFRA][BUILD][DOCS] Enhancing PySpark documentation #44012

itholic · 2023-11-25T20:53:38Z

What changes were proposed in this pull request?

This PR proposes to enhance the PySpark documentation by leveraging modern Sphinx features and functionalities. The primary objective is to improve the overall user experience and readability of the documentation. To achieve this, the PR includes an upgrade of Sphinx and Jinja2 to their newer/latest versions, enabling us to use the latest pydata_sphinx_theme features such as light/dark mode toggling.

Why are the changes needed?

Currently, the PySpark documentation is unable to utilize many of the advanced features available in recent Sphinx versions due to older package versions. This limitation hinders the documentation's visual appeal and usability, particularly when compared to other projects like Pandas which have already adopted these enhancements. For example:

Pandas API reference (better layout / switching light & dark mode available)

Dark mode

Light mode

PySpark API reference (less readable compare to pandas / no light & dark mode)

By updating the Sphinx and Jinja2 versions, we can significantly improve the documentation's layout, design, and interactive features, thereby enhancing the end-user experience.

Does this PR introduce any user-facing change?

No API changes, but users will notice a more modern and user-friendly interface in the PySpark documentation. New features like light/dark mode and improved page layouts will be available as below:

Before

After

Dark mode

Light mode

How was this patch tested?

Manually built docs from local environment, and also tested combinations between various Jinja2, Sphinx and pydata_sphinx_theme versions for best document rendering.

Was this patch authored or co-authored using generative AI tooling?

No.

itholic · 2023-11-25T21:03:09Z

python/docs/source/reference/pyspark.pandas/frame.rst


 .. autosummary::
   :toctree: api/
+   :template: autosummary/accessor_method.rst


With the new version of Sphinx, the package name creation rules for rst files that are automatically created when building documents have changed, so we must manually adjust the package path using these templates.

This behavior is used in the same way in Pandas, so I referred to it.

For example, previously the rst file was created as follows:

pyspark.sql.SparkSession.builder.appName ======================================== .. currentmodule:: pyspark.sql.SparkSession .. automethod:: builder.appName

However, in newer Sphinx versions it is generated like this:

pyspark.sql.SparkSession.builder.appName ======================================== .. currentmodule:: pyspark.sql .. automethod:: SparkSession.builder.appName

In the case of functions used through internal classes or accessors like this, the package paths created in a new way will cause Sphinx build to fail. That's why we should use the customized template to correct the module path.

See also sphinx-doc/sphinx#7551.

itholic · 2023-11-25T21:20:21Z

python/docs/source/conf.py

+    "navbar_end": ["version-switcher", "theme-switcher"],
+    "logo": {
+        "image_light": "_static/spark-logo-light.png",
+        "image_dark": "_static/spark-logo-dark.png",


FYI: The default mode for light/dark is auto, which will choose a theme based on the system settings from user, but we can specify one of dark or light as default manually if we want.

itholic · 2023-11-25T21:31:32Z

python/docs/source/_templates/autosummary/class_with_docs.rst

    .. autosummary::
    {% for item in attributes %}
-       ~{{ name }}.{{ item }}
+        {% if not (item == 'uid') %}


We should manually exclude uid from documentation because it is an internal property. We don't include them our current documentation as well, but for some reason newer Sphinx version trying to generate the internal property unexpectedly.

itholic · 2023-11-25T21:37:15Z

python/docs/source/reference/pyspark.pandas/frame.rst

   :toctree: api/
+   :template: autosummary/accessor_method.rst

-   DataFrame.plot


In newer versions of Sphinx, the build will fail because DataFrame.plot and Series.plot are determined to be duplicates of the list of functions described below such as DataFrame.plot.area, DataFrame.plot.barh, DataFrame.plot.bar, etc.

In fact, this behavior seems reasonable since .plot is simply an accessor keyword and not a function, so I believe we can just simply leave it out of the document.

itholic · 2023-11-25T21:45:21Z

dev/requirements.txt

-jinja2<3.0.0
-sphinx<3.1.0
+jinja2
+sphinx==4.2.0


I see that other Sphinx versions do not generate documentation properly for some reason. I have tested as many combinations as possible with Jinja2 and pydata_sphinx_theme, but I have confirmed that Sphinx version 4.2.0 currently renders documents in the most optimal form. Will investigate further in the future to support the latest Sphinx if necessary.

itholic · 2023-11-25T21:49:12Z

dev/requirements.txt


 # Documentation (Python)
-pydata_sphinx_theme
+pydata_sphinx_theme==0.13


I followed the version used in Pandas, and actually I believe this version render the document in the most optimal form after doing several version testing.

HyukjinKwon · 2023-11-26T01:45:33Z

This is a nice improvement!

itholic · 2023-11-26T04:26:09Z

Documentation build passed: https://github.com/itholic/spark/actions/runs/6991575840/job/19023836477

Other failures seems not related to this PR, but let me re-trigger the CI just to be sure.

…_sphinx

HyukjinKwon · 2023-11-26T23:49:08Z

Merged to master.

…nd `Jinja2` ### What changes were proposed in this pull request? Delete comments on `Sphinx` and `Jinja2` ### Why are the changes needed? they had been upgraded in #44012 ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no Closes #44046 from zhengruifeng/infra_Sphinx_nit. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

### What changes were proposed in this pull request? #44012 unpinned jinja2 in doc build, this PR unpin it in Python linter. this pr is only for master branch, and won't affect branch-3.x daily build ### Why are the changes needed? to be consistent with requirements. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no Closes #44051 from zhengruifeng/infra_linter_jinja. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

### What changes were proposed in this pull request? 1.After pr #44012, the output format of some 'ipynb' tables displayed in HTML format has been disrupted. The pr aims to fix table format error in ipynb docs. - Before: <img width="792" alt="image" src="https://github.com/apache/spark/assets/15246973/2095a2ac-f0b5-44bd-a3c2-ce742d041243"> - After: <img width="739" alt="image" src="https://github.com/apache/spark/assets/15246973/ec0be72d-4dc0-44f4-ab75-d9668e32fc51"> 2.Fix some minor errors. ### Why are the changes needed? Fix bug. ### Does this PR introduce _any_ user-facing change? Yes, only for docs. ### How was this patch tested? Manually test. Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44049 from panbingkun/SPARK-46135. Authored-by: panbingkun <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

### What changes were proposed in this pull request? 1.After pr apache#44012, the output format of some 'ipynb' tables displayed in HTML format has been disrupted. The pr aims to fix table format error in ipynb docs. - Before: <img width="792" alt="image" src="https://github.com/apache/spark/assets/15246973/2095a2ac-f0b5-44bd-a3c2-ce742d041243"> - After: <img width="739" alt="image" src="https://github.com/apache/spark/assets/15246973/ec0be72d-4dc0-44f4-ab75-d9668e32fc51"> 2.Fix some minor errors. ### Why are the changes needed? Fix bug. ### Does this PR introduce _any_ user-facing change? Yes, only for docs. ### How was this patch tested? Manually test. Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#44049 from panbingkun/SPARK-46135. Authored-by: panbingkun <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

github-actions bot added BUILD DOCS PYTHON labels Nov 25, 2023

[SPARK-46103][PYTHON][BUILD][DOCS] Enhancing PySpark documentation

8dd74da

itholic force-pushed the upgrade_sphinx branch from be8a300 to 8dd74da Compare November 25, 2023 20:57

itholic commented Nov 25, 2023

View reviewed changes

Update build_and_test.yml

fcc368d

github-actions bot added the INFRA label Nov 25, 2023

itholic changed the title ~~[SPARK-46103][PYTHON][BUILD][DOCS] Enhancing PySpark documentation~~ [SPARK-46103][PYTHON][INFRA][BUILD][DOCS] Enhancing PySpark documentation Nov 25, 2023

itholic added 2 commits November 26, 2023 13:26

Merge branch 'master' of https://github.com/apache/spark into upgrade…

bc1bfc6

…_sphinx

Retrigger CI

d85133e

HyukjinKwon approved these changes Nov 26, 2023

View reviewed changes

HyukjinKwon closed this in 400db88 Nov 26, 2023

zhengruifeng mentioned this pull request Nov 28, 2023

[SPARK-32407][SPARK-35375][INFRA][DOCS] Delete comments on Sphinx and Jinja2 #44046

Closed

panbingkun mentioned this pull request Nov 28, 2023

[SPARK-46135][PYTHON][DOCS] Fix table format error in ipynb docs #44049

Closed

zhengruifeng mentioned this pull request Nov 28, 2023

[SPARK-46103][INFRA][FOLLOWUP] Unpin jinja2 in Python linter #44051

Closed

itholic deleted the upgrade_sphinx branch January 1, 2024 23:41

nchammas mentioned this pull request Jan 11, 2024

[SPARK-46668][DOCS] Parallelize Sphinx build of Python API docs #44680

Closed

zhengruifeng mentioned this pull request Jan 15, 2024

[SPARK-46715][INFRA] Pin sphinxcontrib-* #44727

Closed

ueshin mentioned this pull request Mar 4, 2024

[SPARK-47078][DOCS][PYTHON] Documentation for SparkSession-based Profilers #45269

Closed

panbingkun mentioned this pull request Mar 19, 2024

[WIP] Fix scheduled jobs for branch-3.4 & branch-3.5 #45551

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-46103][PYTHON][INFRA][BUILD][DOCS] Enhancing PySpark documentation #44012

[SPARK-46103][PYTHON][INFRA][BUILD][DOCS] Enhancing PySpark documentation #44012

Uh oh!

itholic commented Nov 25, 2023 •

edited

Loading

Uh oh!

itholic Nov 25, 2023

Uh oh!

itholic Nov 25, 2023

Uh oh!

itholic Nov 25, 2023 •

edited

Loading

Uh oh!

itholic Nov 25, 2023 •

edited

Loading

Uh oh!

itholic Nov 25, 2023 •

edited

Loading

Uh oh!

itholic Nov 25, 2023 •

edited

Loading

Uh oh!

itholic Nov 25, 2023 •

edited

Loading

Uh oh!

HyukjinKwon commented Nov 26, 2023

Uh oh!

itholic commented Nov 26, 2023 •

edited

Loading

Uh oh!

HyukjinKwon commented Nov 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[SPARK-46103][PYTHON][INFRA][BUILD][DOCS] Enhancing PySpark documentation #44012

[SPARK-46103][PYTHON][INFRA][BUILD][DOCS] Enhancing PySpark documentation #44012

Uh oh!

Conversation

itholic commented Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Pandas API reference (better layout / switching light & dark mode available)

Dark mode

Light mode

PySpark API reference (less readable compare to pandas / no light & dark mode)

Does this PR introduce any user-facing change?

Before

After

Dark mode

Light mode

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

itholic Nov 25, 2023

Choose a reason for hiding this comment

Uh oh!

itholic Nov 25, 2023

Choose a reason for hiding this comment

Uh oh!

itholic Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itholic Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itholic Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itholic Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itholic Nov 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Nov 26, 2023

Uh oh!

itholic commented Nov 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Nov 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

itholic commented Nov 25, 2023 •

edited

Loading

itholic Nov 25, 2023 •

edited

Loading

itholic Nov 25, 2023 •

edited

Loading

itholic Nov 25, 2023 •

edited

Loading

itholic Nov 25, 2023 •

edited

Loading

itholic Nov 25, 2023 •

edited

Loading

itholic commented Nov 26, 2023 •

edited

Loading