Skip to content

[MINOR][PySpark][DOCS] Fix examples in PySpark documentation #15242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
HyukjinKwon wants to merge 5 commits into from
Closed

[MINOR][PySpark][DOCS] Fix examples in PySpark documentation #15242

HyukjinKwon wants to merge 5 commits into from

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Sep 26, 2016 •
edited
Loading

What changes were proposed in this pull request?

This PR proposes to fix wrongly indented examples in PySpark documentation

-        >>> json_sdf = spark.readStream.format("json")\
-                                       .schema(sdf_schema)\
-                                       .load(tempfile.mkdtemp())
+        >>> json_sdf = spark.readStream.format("json") \\
+        ...     .schema(sdf_schema) \\
+        ...     .load(tempfile.mkdtemp())

-        people.filter(people.age > 30).join(department, people.deptId == department.id)\
+        people.filter(people.age > 30).join(department, people.deptId == department.id) \\

-        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, 1.23), (2, 4.56)])), \
-                        LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]
+        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, 1.23), (2, 4.56)])),
+        ...             LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]

-        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, -1.23), (2, 4.56e-7)])), \
-                        LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]
+        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, -1.23), (2, 4.56e-7)])),
+        ...             LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]

-        ...      for x in iterator:
-        ...           print(x)
+        ...     for x in iterator:
+        ...          print(x)

How was this patch tested?

Manually tested.

Before

2016-09-26 8 36 02

2016-09-26 9 22 16

2016-09-27 2 29 27

2016-09-27 2 29 58

2016-09-27 2 30 05

After

2016-09-26 9 29 47

2016-09-26 9 30 24

2016-09-27 2 28 19

2016-09-27 3 50 59

2016-09-27 3 51 03

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Sep 26, 2016

Could you take a look @srowen ? I made this change before across PySpark documentation but it seems I missed one before and then another one is introduced somewhere after my previous PR.

srowen
srowen reviewed Sep 26, 2016
View reviewed changes
python/pyspark/sql/streaming.py
>>> json_sdf = spark.readStream.format("json")\
.schema(sdf_schema)\
.load(tempfile.mkdtemp())
>>> json_sdf = spark.readStream.format("json") \\
Copy link
Member

@srowen srowen Sep 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, does the \ vs \ make the difference here? are there possibly other instances of this?

Copy link
Member Author

@HyukjinKwon HyukjinKwon Sep 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it seems it does. I took a look and a sweep before - #14063

I will take another look across the documentation again tomorrow.

@HyukjinKwon HyukjinKwon changed the title [MINOR][PySpark] Fix examples in PySpark documentation [MINOR][PySpark][DOC] Fix examples in PySpark documentation Sep 26, 2016
@HyukjinKwon HyukjinKwon changed the title [MINOR][PySpark][DOC] Fix examples in PySpark documentation [MINOR][PySpark][DOCS] Fix examples in PySpark documentation Sep 26, 2016
@SparkQA
Copy link

SparkQA commented Sep 26, 2016

Test build #65911 has finished for PR 15242 at commit 9ac3088.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Sep 26, 2016

retest this please

@SparkQA
Copy link

SparkQA commented Sep 26, 2016

Test build #65915 has finished for PR 15242 at commit 9ac3088.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Sep 27, 2016

@srowen I just took a scan twice and I think they should be all.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Sep 27, 2016

Let me update the PR description just in case.

@SparkQA
Copy link

SparkQA commented Sep 27, 2016

Test build #65947 has finished for PR 15242 at commit 88c1069.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 27, 2016

Test build #65949 has finished for PR 15242 at commit 85cda01.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 27, 2016

Test build #65951 has finished for PR 15242 at commit 6de11ec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Sep 27, 2016

(PR description is updated again)

@srowen
Copy link
Member

srowen commented Sep 28, 2016

Merged to master/2.0

asfgit pushed a commit that referenced this pull request Sep 28, 2016
@HyukjinKwon @srowen
[MINOR][PYSPARK][DOCS] Fix examples in PySpark documentation
4d73d5c 
## What changes were proposed in this pull request?

This PR proposes to fix wrongly indented examples in PySpark documentation

```
-        >>> json_sdf = spark.readStream.format("json")\
-                                       .schema(sdf_schema)\
-                                       .load(tempfile.mkdtemp())
+        >>> json_sdf = spark.readStream.format("json") \\
+        ...     .schema(sdf_schema) \\
+        ...     .load(tempfile.mkdtemp())
```

```
-        people.filter(people.age > 30).join(department, people.deptId == department.id)\
+        people.filter(people.age > 30).join(department, people.deptId == department.id) \\
```

```
-        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, 1.23), (2, 4.56)])), \
-                        LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]
+        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, 1.23), (2, 4.56)])),
+        ...             LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]
```

```
-        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, -1.23), (2, 4.56e-7)])), \
-                        LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]
+        >>> examples = [LabeledPoint(1.1, Vectors.sparse(3, [(0, -1.23), (2, 4.56e-7)])),
+        ...             LabeledPoint(0.0, Vectors.dense([1.01, 2.02, 3.03]))]
```

```
-        ...      for x in iterator:
-        ...           print(x)
+        ...     for x in iterator:
+        ...          print(x)
```

## How was this patch tested?

Manually tested.

**Before**

![2016-09-26 8 36 02](https://cloud.githubusercontent.com/assets/6477701/18834471/05c7a478-8431-11e6-94bb-09aa37b12ddb.png)

![2016-09-26 9 22 16](https://cloud.githubusercontent.com/assets/6477701/18834472/06c8735c-8431-11e6-8775-78631eab0411.png)

<img width="601" alt="2016-09-27 2 29 27" src="https://cloud.githubusercontent.com/assets/6477701/18861294/29c0d5b4-84bf-11e6-99c5-3c9d913c125d.png">

<img width="1056" alt="2016-09-27 2 29 58" src="https://cloud.githubusercontent.com/assets/6477701/18861298/31694cd8-84bf-11e6-9e61-9888cb8c2089.png">

<img width="1079" alt="2016-09-27 2 30 05" src="https://cloud.githubusercontent.com/assets/6477701/18861301/359722da-84bf-11e6-97f9-5f5365582d14.png">

**After**

![2016-09-26 9 29 47](https://cloud.githubusercontent.com/assets/6477701/18834467/0367f9da-8431-11e6-86d9-a490d3297339.png)

![2016-09-26 9 30 24](https://cloud.githubusercontent.com/assets/6477701/18834463/f870fae0-8430-11e6-9482-01fc47898492.png)

<img width="515" alt="2016-09-27 2 28 19" src="https://cloud.githubusercontent.com/assets/6477701/18861305/3ff88b88-84bf-11e6-902c-9f725e8a8b10.png">

<img width="652" alt="2016-09-27 3 50 59" src="https://cloud.githubusercontent.com/assets/6477701/18863053/592fbc74-84ca-11e6-8dbf-99cf57947de8.png">

<img width="709" alt="2016-09-27 3 51 03" src="https://cloud.githubusercontent.com/assets/6477701/18863060/601607be-84ca-11e6-80aa-a401df41c321.png">

Author: hyukjinkwon <[email protected]>

Closes #15242 from HyukjinKwon/minor-example-pyspark.

(cherry picked from commit 2190037)
Signed-off-by: Sean Owen <[email protected]>
@asfgit asfgit closed this in 2190037 Sep 28, 2016
@HyukjinKwon HyukjinKwon deleted the minor-example-pyspark branch January 2, 2018 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowen srowen srowen left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@HyukjinKwon @SparkQA @srowen