Skip to content

Conversation

@HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Nov 15, 2016

What changes were proposed in this pull request?

It seems in Scala/Java,

  • Note:
  • NOTE:
  • Note that
  • '''Note:'''
  • @note

This PR proposes to fix those to @note to be consistent.

Before

  • Scala
    2016-11-17 6 16 39

  • Java
    2016-11-17 6 14 41

After

  • Scala
    2016-11-17 6 16 44

  • Java
    2016-11-17 6 13 39

How was this patch tested?

The notes were found via

grep -r "NOTE: " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// NOTE: " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \ # note that this is a regular expression. So actual matches were mostly `org/apache/spark/api/java/functions ...`
-e 'org.apache.spark.api.r' \
...
grep -r "Note that " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// Note that " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...
grep -r "Note: " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// Note: " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...
grep -r "'''Note:'''" . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// '''Note:''' " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...

And then fixed one by one comparing with API documentation/access modifiers.

After that, manually tested via jekyll build.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Nov 15, 2016

I am going to, double check any missed one first, and then sweep them again and again by leaving comments with images from the built API doc each by each change to double-check.

@SparkQA
Copy link

SparkQA commented Nov 15, 2016

Test build #68660 has finished for PR 15889 at commit 2acb56f.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

Hi @sandeepmreddy, thanks for approving but actually it is WIP. It would be great if it is reviewed when it's complete.

@SparkQA
Copy link

SparkQA commented Nov 16, 2016

Test build #68721 has finished for PR 15889 at commit 30794b4.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon HyukjinKwon changed the title [WIP][SPARK-18445][DOCS] Fix the markdown for Note:/NOTE:/Note that across Scala/Java API documentation [WIP][SPARK-18445][DOCS] Fix the markdown for Note:/NOTE:/Note that/'''Note:''' across Scala/Java API documentation Nov 16, 2016
@SparkQA
Copy link

SparkQA commented Nov 16, 2016

Test build #68723 has finished for PR 15889 at commit b5fd61e.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 16, 2016

Test build #68727 has finished for PR 15889 at commit 436fabf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Nov 16, 2016

@HyukjinKwon this is ready to go right?

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Nov 16, 2016

Ah, I couldn't check all of the java documentation yet because I have some problems with javadoc8 because of several errors (although they look already existing ones). So, I am thinking of trying to use different version. (Maybe I am doing something wrong assuming it is not mentioned in docs/README.md).

Will check several times more the built documentation and also update PR description within today.I am also fine if you are worried of conflicts from merging others. I can resolve them.

@SparkQA
Copy link

SparkQA commented Nov 17, 2016

Test build #68756 has started for PR 15889 at commit f2b201c.

Copy link
Member Author

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems @note plays fine with existing formats.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before

  • Scala
    2016-11-17 6 30 55
  • Java
    2016-11-17 6 24 01

After

  • Scala
    2016-11-17 6 30 49
  • Java
    2016-11-17 6 24 23

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before

  • Scala
    2016-11-17 6 33 27
  • Java
    2016-11-17 6 33 51

After

  • Scala
    2016-11-17 6 34 31
  • Java
    2016-11-17 6 37 16

@HyukjinKwon HyukjinKwon changed the title [WIP][SPARK-18445][DOCS] Fix the markdown for Note:/NOTE:/Note that/'''Note:''' across Scala/Java API documentation [SPARK-18445][DOCS] Fix the markdown for Note:/NOTE:/Note that/'''Note:''' across Scala/Java API documentation Nov 17, 2016
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srowen, this was added to deal with @note annotation in Java documentation. It seems @note for scaladoc not for javadoc. So, I had to manually deal with this.

pom.xml Outdated
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the equivalent one with SparkBuild.scala.

@HyukjinKwon HyukjinKwon changed the title [SPARK-18445][DOCS] Fix the markdown for Note:/NOTE:/Note that/'''Note:''' across Scala/Java API documentation [SPARK-18445][BUILD][DOCS] Fix the markdown for Note:/NOTE:/Note that/'''Note:''' across Scala/Java API documentation Nov 17, 2016
@SparkQA
Copy link

SparkQA commented Nov 17, 2016

Test build #68761 has finished for PR 15889 at commit d22136e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

HyukjinKwon commented Nov 17, 2016

(it was simply rebased and added (in 39873dc) 5 instances introduced in another PR)

*
* @note This is NOT guaranteed to provide exactly the fraction of the count
* of the given [[Dataset]].
*
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Scala
    2016-11-17 8 56 08
  • Java
    2016-11-17 8 55 46

@HyukjinKwon
Copy link
Member Author

Thank you for approving Sean!

@SparkQA
Copy link

SparkQA commented Nov 17, 2016

Test build #68775 has finished for PR 15889 at commit 8cd3cdd.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member Author

retest this please

@SparkQA
Copy link

SparkQA commented Nov 17, 2016

Test build #68783 has finished for PR 15889 at commit 8cd3cdd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 17, 2016

Test build #68785 has finished for PR 15889 at commit 8cd3cdd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@srowen srowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few last questions now that I skimmed it again, actually

* "combined type" C. Note that V and C can be different -- for example, one might group an
* "combined type" C.
*
* @note V and C can be different -- for example, one might group an
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was skimming the changes again and noticed this. This causes this note to render pretty separately from this text, but it seems to belong with the sentence above.

Does it cause the bullet points below to be in the note? because they also seem unrelated to this 'note'.

Could be OK but wasn't sure whether this was really a stand-alone note as much as a usage of the word "note"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yes. It seems so. Will check the same instances again. Thank you.

* supporting the key and value types K and V in this RDD.
*
* Note that, we should make sure our tasks are idempotent when speculation is enabled, i.e. do
* @note We should make sure our tasks are idempotent when speculation is enabled, i.e. do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise I wonder if these following sentences all go into the note? they go together.

Copy link
Member Author

@HyukjinKwon HyukjinKwon Nov 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I left and removed an useless comment. Yes, it seems it goes together.

2016-11-18 1 41 26

2016-11-18 1 42 02

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It almost looks like it should be a code comment, not scaladoc. Well, I don't feel strongly about changing it if it was already in the scaladoc.

* it will process either one or all of the RDDs returned by the queue.
*
* NOTE:
* @note
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise here just want to make sure that this actually causes the following 3 lines are in the note?

Copy link
Member Author

@HyukjinKwon HyukjinKwon Nov 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, it seems keeping the original format at least.

Before

  • Scala
    2016-11-18 1 45 48
  • Java
    2016-11-18 1 46 07

After

  • Scala
    2016-11-18 1 45 26
  • Java
    2016-11-18 1 44 58

@SparkQA
Copy link

SparkQA commented Nov 18, 2016

Test build #68828 has started for PR 15889 at commit 1ef229a.

* items with the same key).
*
* @note V and C can be different -- for example, one might group a RDD of type (Int, Int) into
* an RDD of type (Int, List[Int]).
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems fine now.

2016-11-18 4 47 33
2016-11-18 4 47 17

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it works alright as a standalone note. It could have stayed an inline sentence too. OK either way.

*
* @note Vectors to be transformed must be the same length
* as the source vectors given to [[PCA.fit()]].
* @note Vectors to be transformed must be the same lengtt as the source vectors given
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh sorry @HyukjinKwon but there's now a typo here: lengtt

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doh! I will check the same instances. I am so sorry.

@SparkQA
Copy link

SparkQA commented Nov 18, 2016

Test build #68839 has finished for PR 15889 at commit b61ab72.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 18, 2016

Test build #68850 has finished for PR 15889 at commit 826c950.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

* In addition, users can control the partitioning of the output RDD, and whether to perform
* map-side aggregation (if a mapper can produce multiple items with the same key).
*
* @note V and C can be different -- for example, one might group a RDD of type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"an RDD" is correct actually. Could you make this one last fix by reverting just that change? I bet there are occurrences of "a RDD" in the docs elsewhere and you're welcome to fix them too if it makes it easier to search and replace it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah... I see. Sure. I thought "a RDD" is correct but just realised it after googling it.. Will fix and replace. I see. It should be an

@SparkQA
Copy link

SparkQA commented Nov 18, 2016

Test build #68859 has finished for PR 15889 at commit 4576f7d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 18, 2016

Test build #68861 has finished for PR 15889 at commit ee5b035.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Nov 19, 2016
… that`/`'''Note:'''` across Scala/Java API documentation

It seems in Scala/Java,

- `Note:`
- `NOTE:`
- `Note that`
- `'''Note:'''`
- `note`

This PR proposes to fix those to `note` to be consistent.

**Before**

- Scala
  ![2016-11-17 6 16 39](https://cloud.githubusercontent.com/assets/6477701/20383180/1a7aed8c-acf2-11e6-9611-5eaf6d52c2e0.png)

- Java
  ![2016-11-17 6 14 41](https://cloud.githubusercontent.com/assets/6477701/20383096/c8ffc680-acf1-11e6-914a-33460bf1401d.png)

**After**

- Scala
  ![2016-11-17 6 16 44](https://cloud.githubusercontent.com/assets/6477701/20383167/09940490-acf2-11e6-937a-0d5e1dc2cadf.png)

- Java
  ![2016-11-17 6 13 39](https://cloud.githubusercontent.com/assets/6477701/20383132/e7c2a57e-acf1-11e6-9c47-b849674d4d88.png)

The notes were found via

```bash
grep -r "NOTE: " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// NOTE: " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \ # note that this is a regular expression. So actual matches were mostly `org/apache/spark/api/java/functions ...`
-e 'org.apache.spark.api.r' \
...
```

```bash
grep -r "Note that " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// Note that " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...
```

```bash
grep -r "Note: " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// Note: " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...
```

```bash
grep -r "'''Note:'''" . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// '''Note:''' " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...
```

And then fixed one by one comparing with API documentation/access modifiers.

After that, manually tested via `jekyll build`.

Author: hyukjinkwon <[email protected]>

Closes #15889 from HyukjinKwon/SPARK-18437.

(cherry picked from commit d5b1d5f)
Signed-off-by: Sean Owen <[email protected]>
@srowen
Copy link
Member

srowen commented Nov 19, 2016

Merged to master/2.1

@asfgit asfgit closed this in d5b1d5f Nov 19, 2016
@HyukjinKwon
Copy link
Member Author

Thank you!!

}

/** Write a RDD partition out in a single Spark task. */
/** Write an RDD partition out in a single Spark task. */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you changing this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because a RDD was incorrect in English.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I missing something? I'm not familiar with English grammer, but shouldn't we use a when the RDD start with R?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I searched the code base and there are both a RDD and an RDD.

Copy link
Member Author

@HyukjinKwon HyukjinKwon Nov 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should replace them to an RDD.

I think on my Mac I found and replaced them by

find . -type f -name "*.scala"  | xargs sed -i '' 's/ A RDD / An RDD /g'
find . -type f -name "*.scala"  | xargs sed -i '' 's/ A RDD\./ An RDD\./g'
find . -type f -name "*.scala"  | xargs sed -i '' 's/ a RDD / an RDD /g'
find . -type f -name "*.scala"  | xargs sed -i '' 's/ a RDD\./ an RDD\./g'
find . -type f -name "*.java"  | xargs sed -i '' 's/ A RDD / An RDD /g'
find . -type f -name "*.java"  | xargs sed -i '' 's/ A RDD\./ An RDD\./g'
find . -type f -name "*.java"  | xargs sed -i '' 's/ a RDD / an RDD /g'
find . -type f -name "*.java"  | xargs sed -i '' 's/ a RDD\./ an RDD\./g'
find . -type f -name "*.r"  | xargs sed -i '' 's/ A RDD / An RDD /g'
find . -type f -name "*.r"  | xargs sed -i '' 's/ A RDD\./ An RDD\./g'
find . -type f -name "*.r"  | xargs sed -i '' 's/ a RDD / an RDD /g'
find . -type f -name "*.r"  | xargs sed -i '' 's/ a RDD\./ an RDD\./g'
find . -type f -name "*.py"  | xargs sed -i '' 's/ A RDD / An RDD /g'
find . -type f -name "*.py"  | xargs sed -i '' 's/ A RDD\./ An RDD\./g'
find . -type f -name "*.py"  | xargs sed -i '' 's/ a RDD / an RDD /g'
find . -type f -name "*.py"  | xargs sed -i '' 's/ a RDD\./ an RDD\./g'
find . -type f -name "*.md"  | xargs sed -i '' 's/ A RDD / An RDD /g'
find . -type f -name "*.md"  | xargs sed -i '' 's/ A RDD\./ An RDD\./g'
find . -type f -name "*.md"  | xargs sed -i '' 's/ a RDD / an RDD /g'
find . -type f -name "*.md"  | xargs sed -i '' 's/ a RDD\./ an RDD\./g'

If there are more, we should fix. Actually, i was preparing a minor PR to fix all of them including the same instances such as a X.. to an X...

Copy link
Member Author

@HyukjinKwon HyukjinKwon Nov 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I also made this mistake and found actually an RDD is correct after googling :).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you point where a RDD is? I will try to include them in a similar way above.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, "an RDD" is correct because "a" precedes consonant sounds and "an" precedes vowel sounds. So it's "an arr-dee-dee".

We don't need to fix up every last instance of this. It was a good idea to fix it all while fixing some that overlapped with this PR though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I will include

./R/pkg/inst/tests/testthat/test_sparkSQL.R:test_that("jsonRDD() on a RDD with json string", {
./R/pkg/R/RDD.R:#' @param x A RDD.

It seems I should have used -name "*.R" instead.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Sure, let me just keep it and will leave some comments in the related PRs in the future if possible.

uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
… that`/`'''Note:'''` across Scala/Java API documentation

## What changes were proposed in this pull request?

It seems in Scala/Java,

- `Note:`
- `NOTE:`
- `Note that`
- `'''Note:'''`
- `note`

This PR proposes to fix those to `note` to be consistent.

**Before**

- Scala
  ![2016-11-17 6 16 39](https://cloud.githubusercontent.com/assets/6477701/20383180/1a7aed8c-acf2-11e6-9611-5eaf6d52c2e0.png)

- Java
  ![2016-11-17 6 14 41](https://cloud.githubusercontent.com/assets/6477701/20383096/c8ffc680-acf1-11e6-914a-33460bf1401d.png)

**After**

- Scala
  ![2016-11-17 6 16 44](https://cloud.githubusercontent.com/assets/6477701/20383167/09940490-acf2-11e6-937a-0d5e1dc2cadf.png)

- Java
  ![2016-11-17 6 13 39](https://cloud.githubusercontent.com/assets/6477701/20383132/e7c2a57e-acf1-11e6-9c47-b849674d4d88.png)

## How was this patch tested?

The notes were found via

```bash
grep -r "NOTE: " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// NOTE: " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \ # note that this is a regular expression. So actual matches were mostly `org/apache/spark/api/java/functions ...`
-e 'org.apache.spark.api.r' \
...
```

```bash
grep -r "Note that " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// Note that " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...
```

```bash
grep -r "Note: " . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// Note: " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...
```

```bash
grep -r "'''Note:'''" . | \ # Note:|NOTE:|Note that|'''Note:'''
grep -v "// '''Note:''' " | \  # starting with // does not appear in API documentation.
grep -E '.scala|.java' | \ # java/scala files
grep -v Suite | \ # exclude tests
grep -v Test | \ # exclude tests
grep -e 'org.apache.spark.api.java' \ # packages appear in API documenation
-e 'org.apache.spark.api.java.function' \
-e 'org.apache.spark.api.r' \
...
```

And then fixed one by one comparing with API documentation/access modifiers.

After that, manually tested via `jekyll build`.

Author: hyukjinkwon <[email protected]>

Closes apache#15889 from HyukjinKwon/SPARK-18437.
@HyukjinKwon HyukjinKwon deleted the SPARK-18437 branch January 2, 2018 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants