[SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base #29188

HyukjinKwon · 2020-07-22T10:09:41Z

What changes were proposed in this pull request?

This PR proposes to redesign the PySpark documentation.

I made a demo site to make it easier to review: https://hyukjin-spark.readthedocs.io/en/stable/reference/index.html.

Here is the initial draft for the final PySpark docs shape: https://hyukjin-spark.readthedocs.io/en/latest/index.html.

In more details, this PR proposes:

Use pydata_sphinx_theme theme - pandas and Koalas use this theme. The CSS overwrite is ported from Koalas. The colours in the CSS were actually chosen by designers to use in Spark.
Use the Sphinx option to separate source and build directories as the documentation pages will likely grow.
Port current API documentation into the new style. It mimics Koalas and pandas to use the theme most effectively.

One disadvantage of this approach is that you should list up APIs or classes; however, I think this isn't a big issue in PySpark since we're being conservative on adding APIs. I also intentionally listed classes only instead of functions in ML and MLlib to make it relatively easier to manage.

Why are the changes needed?

Often I hear the complaints, from the users, that current PySpark documentation is pretty messy to read - https://spark.apache.org/docs/latest/api/python/index.html compared other projects such as pandas and Koalas.

It would be nicer if we can make it more organised instead of just listing all classes, methods and attributes to make it easier to navigate.

Also, the documentation has been there from almost the very first version of PySpark. Maybe it's time to update it.

Does this PR introduce any user-facing change?

Yes, PySpark API documentation will be redesigned.

How was this patch tested?

Manually tested, and the demo site was made to show.

HyukjinKwon · 2020-07-22T10:12:48Z

.gitignore

 python/.eggs/
 python/deps
 python/docs/_site/
+python/docs/source/reference/api/


This is generated by autosummary plugin in Sphinx when autosummary_generate in conf.py is enabled. Each page of API or class under autosummary directive, for example, DataFrame.alias will be generated via that plugin as RST files.

HyukjinKwon

docs/img/spark-logo-reverse.png image is from "white logo" at http://spark.apache.org/faq.html.

HyukjinKwon · 2020-07-22T10:17:24Z

python/docs/source/_templates/class_with_docs.rst

+
+    {% endif %}
+    {% endblock %}
+


This is needed to let autosummary plugin document the methods in a class. For example, when we use this template, it describes methods documentation on the bottom. See pyspark.ml.Transformer as an example.

Without this template, it only lists the methods and attributes without showing the documentation in details. See pyspark.sql.DataFrameNaFunctions as an example.

HyukjinKwon · 2020-07-22T10:18:12Z

python/docs/source/conf.py

+# Remove previously generated rst files. Ignore errors just in case it stops
+# generating whole docs.
+shutil.rmtree(
+    "%s/reference/api" % os.path.dirname(os.path.abspath(__file__)), ignore_errors=True)


autosummary generates RST files but don't remove it back. Here we always remove the generated RST files so the leftover doesn't cause any side effect.

HyukjinKwon · 2020-07-22T10:21:42Z

@BryanCutler, @huaxingao, @ueshin, @viirya, @srowen, @dongjoon-hyun, @WeichenXu123, @zhengruifeng, @holdenk, @zero323, can you guys take a look when you are available?

srowen

Looks nice! yes will take a little more work to maintain the modules / class lists, so whatever we can do to keep it simple is welcome.

holdenk · 2020-07-22T12:58:21Z

Excited to see the site improve, I’ll take some time to review it this week.

viirya

It looks great! Besides visual effects like colors, it looks more structured.

zero323 · 2020-07-22T20:02:15Z

Looks nice. I miss a bit direct access to docstrings, but I guess that's a reasonable trade-off.

I wonder if there is some non-hacky way to organize functions into logical groups, similarly to what ScalaDoc does.

huaxingao · 2020-07-22T22:04:32Z

Looks really nice! It's more organized this way.

dev/requirements.txt

dongjoon-hyun · 2020-07-23T00:42:27Z

The demo website looks nice although I didn't generate manually this PR~

HyukjinKwon · 2020-07-23T02:27:24Z

I wonder if there is some non-hacky way to organize functions into logical groups, similarly to what ScalaDoc does.

I tried hard but looked difficult to do. I will take a look one more time.

HyukjinKwon · 2020-07-23T06:42:41Z

python/pyspark/sql/functions.py

    By default, it follows casting rules to :class:`pyspark.sql.types.DateType` if the format
    is omitted. Equivalent to ``col.cast("date")``.

+    .. _datetime pattern: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html


This is needed because we're now creating each page for each API. For example, see https://hyukjin-spark.readthedocs.io/en/stable/reference/api/pyspark.sql.functions.to_date.html

HyukjinKwon · 2020-07-23T12:00:58Z

I believe this is ready for a look or possibly ready to go.

SparkQA · 2020-07-24T04:29:36Z

Test build #126449 has finished for PR 29188 at commit d6d0117.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-27T07:47:12Z

retest this please

HyukjinKwon · 2020-07-27T07:47:56Z

I will merge and go ahead given that there are multiple positive feedback here.

HyukjinKwon · 2020-07-27T08:48:32Z

Merged to master.

HyukjinKwon · 2020-07-27T08:48:56Z

Let me know if you guys have any concern on this. I will be working on this to complete the other pages for a while.

SparkQA · 2020-07-27T11:08:23Z

Test build #126627 has finished for PR 29188 at commit d6d0117.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zhengruifeng · 2020-07-28T03:38:03Z

The demo website looks great!

### What changes were proposed in this pull request? This PR proposes to write the main page of PySpark documentation. The base work is finished at #29188. ### Why are the changes needed? For better usability and readability in PySpark documentation. ### Does this PR introduce _any_ user-facing change? Yes, it creates a new main page as below: ![Screen Shot 2020-07-31 at 10 02 44 PM](https://user-images.githubusercontent.com/6477701/89037618-d2d68880-d379-11ea-9a44-562f2aa0e3fd.png) ### How was this patch tested? Manually built the PySpark documentation. ```bash cd python make clean html ``` Closes #29320 from HyukjinKwon/SPARK-32507. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

probot-autolabeler bot added BUILD DOCS INFRA PYTHON labels Jul 22, 2020

HyukjinKwon commented Jul 22, 2020

View reviewed changes