[SPARK-34041][PYTHON][DOCS] Miscellaneous cleanup for new PySpark documentation #31082

HyukjinKwon · 2021-01-07T08:54:14Z

What changes were proposed in this pull request?

This PR proposes to:

Add a link of quick start in PySpark docs into "Programming Guides" in Spark main docs
ML / MLlib -> MLlib (DataFrame-based) / MLlib (RDD-based) in API reference page
Mention other user guides as well because the guide such as ML and SQL.
Mention other migration guides as well because PySpark can get affected by it.

Why are the changes needed?

For better documentation.

Does this PR introduce any user-facing change?

It fixes user-facing docs. However, it's not released out yet.

How was this patch tested?

Manually tested by running:

cd docs
SKIP_SCALADOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 jekyll serve --watch

SparkQA · 2021-01-07T09:43:00Z

Test build #133785 has finished for PR 31082 at commit 003b61f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-01-07T09:45:20Z

docs/_layouts/global.html

                                <a class="dropdown-item" href="ml-guide.html">MLlib (Machine Learning)</a>
                                <a class="dropdown-item" href="graphx-programming-guide.html">GraphX (Graph Processing)</a>
                                <a class="dropdown-item" href="sparkr.html">SparkR (R on Spark)</a>
+                                <a class="dropdown-item" href="api/python/getting_started/index.html">PySpark (Python on Spark)</a>


HyukjinKwon · 2021-01-07T09:45:40Z

docs/index.md

 * [MLlib](ml-guide.html): applying machine learning algorithms
 * [GraphX](graphx-programming-guide.html): processing graphs 
+* [SparkR](sparkr.html): processing data with Spark in R
+* [PySpark](api/python/getting_started/index.html): processing data with Spark in Python


HyukjinKwon · 2021-01-07T09:46:15Z

python/docs/source/getting_started/index.rst

 This page summarizes the basic steps required to setup and get started with PySpark.
+There are more guides shared with other languages such as
+`Quick Start <http://spark.apache.org/docs/latest/quick-start.html>`_ in Programming Guides
+at `the Spark documentation <http://spark.apache.org/docs/latest/index.html#where-to-go-from-here>`_.


HyukjinKwon · 2021-01-07T09:46:50Z

python/docs/source/user_guide/index.rst

+- `Structured Streaming Programming Guide <http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html>`_
+- `Spark Streaming Programming Guide <http://spark.apache.org/docs/latest/streaming-programming-guide.html>`_
+- `Machine Learning Library (MLlib) Guide <http://spark.apache.org/docs/latest/ml-guide.html>`_
+


HyukjinKwon · 2021-01-07T09:47:29Z

python/docs/source/migration_guide/index.rst

+- `Migration Guide: SQL, Datasets and DataFrame <http://spark.apache.org/docs/latest/sql-migration-guide.html>`_
+- `Migration Guide: Structured Streaming <http://spark.apache.org/docs/latest/ss-migration-guide.html>`_
+- `Migration Guide: MLlib (Machine Learning) <http://spark.apache.org/docs/latest/ml-migration-guide.html>`_
+


HyukjinKwon · 2021-01-07T09:48:26Z

Thanks for post-reviewing @mengxr. Can you take a look please? @viirya, @srowen and @zero323 too FYI.

SparkQA · 2021-01-07T09:55:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38373/

SparkQA · 2021-01-07T10:23:10Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38373/

SparkQA · 2021-01-07T10:43:21Z

Test build #133789 has finished for PR 31082 at commit 0cacfd0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-01-07T10:59:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38378/

SparkQA · 2021-01-07T11:04:34Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38378/

zero323 · 2021-01-07T19:47:29Z

python/docs/source/reference/pyspark.ml.rst


-ML
-==
+MLlib (DataFrame-based)


I have mixed about it. I am aware that MLlib is the official name applied to both, but majority of users, I interacted with, prefers to use ML when speaking about DataFrame-based API.

This one is actually the feedback who has a lot of major contributions in ML, @mengxr :-). I think that this name was picked based on what we documented here https://spark.apache.org/docs/latest/ml-guide.html which I think makes sense:

What is “Spark ML”?

“Spark ML” is not an official name but occasionally used to refer to the MLlib DataFrame-based API. This is majorly due to the org.apache.spark.ml Scala package name used by the DataFrame-based API, and the “Spark ML Pipelines” term we used initially to emphasize the pipeline concept.

viirya · 2021-01-07T21:20:51Z

python/docs/source/getting_started/index.rst


 This page summarizes the basic steps required to setup and get started with PySpark.
+There are more guides shared with other languages such as
+`Quick Start <http://spark.apache.org/docs/latest/quick-start.html>`_ in Programming Guides


Is this needed to be absolute path? Cannot we use relative path?

Yeah ... I tried so hard to find a good way but failed. This is because the link here is outer side of the PySpark documentation build. So it can't resolve the link when PySpark documentation builds.

viirya

Looks good. Just one question about absolute path.

HyukjinKwon · 2021-01-08T00:28:19Z

Thanks guys. Let me merge this in.

…umentation ### What changes were proposed in this pull request? This PR proposes to: - Add a link of quick start in PySpark docs into "Programming Guides" in Spark main docs - `ML` / `MLlib` -> `MLlib (DataFrame-based)` / `MLlib (RDD-based)` in API reference page - Mention other user guides as well because the guide such as [ML](http://spark.apache.org/docs/latest/ml-guide.html) and [SQL](http://spark.apache.org/docs/latest/sql-programming-guide.html). - Mention other migration guides as well because PySpark can get affected by it. ### Why are the changes needed? For better documentation. ### Does this PR introduce _any_ user-facing change? It fixes user-facing docs. However, it's not released out yet. ### How was this patch tested? Manually tested by running: ```bash cd docs SKIP_SCALADOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 jekyll serve --watch ``` Closes #31082 from HyukjinKwon/SPARK-34041. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit aa388cf) Signed-off-by: HyukjinKwon <[email protected]>

HyukjinKwon marked this pull request as draft January 7, 2021 09:02

Miscellaneous cleanup for new PySpark documentation

0cacfd0

HyukjinKwon force-pushed the SPARK-34041 branch from 003b61f to 0cacfd0 Compare January 7, 2021 09:40

HyukjinKwon commented Jan 7, 2021

View reviewed changes

HyukjinKwon marked this pull request as ready for review January 7, 2021 09:47

github-actions bot added DOCS PYTHON labels Jan 7, 2021

srowen approved these changes Jan 7, 2021

View reviewed changes

zero323 reviewed Jan 7, 2021

View reviewed changes

zero323 approved these changes Jan 7, 2021

View reviewed changes

viirya reviewed Jan 7, 2021

View reviewed changes

viirya approved these changes Jan 7, 2021

View reviewed changes

HyukjinKwon closed this in aa388cf Jan 8, 2021

HyukjinKwon deleted the SPARK-34041 branch January 4, 2022 00:55

[SPARK-34041][PYTHON][DOCS] Miscellaneous cleanup for new PySpark documentation #31082

[SPARK-34041][PYTHON][DOCS] Miscellaneous cleanup for new PySpark documentation #31082

Uh oh!

Conversation

HyukjinKwon commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Jan 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Jan 7, 2021

Uh oh!

SparkQA commented Jan 7, 2021

Uh oh!

SparkQA commented Jan 7, 2021

Uh oh!

SparkQA commented Jan 7, 2021

Uh oh!

SparkQA commented Jan 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jan 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HyukjinKwon commented Jan 7, 2021 •

edited

Loading

HyukjinKwon Jan 7, 2021 •

edited

Loading

HyukjinKwon commented Jan 7, 2021 •

edited

Loading