[SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs #29056

GuoPhilipse · 2020-07-09T13:57:14Z

What changes were proposed in this pull request?

update sql-ref docs, the following key words will be added in this PR.

CASE/ELSE
WHEN/THEN
MAP KEYS TERMINATED BY
NULL DEFINED AS
LINES TERMINATED BY
ESCAPED BY
COLLECTION ITEMS TERMINATED BY
PIVOT
LATERAL VIEW OUTER?
ROW FORMAT SERDE
ROW FORMAT DELIMITED
FIELDS TERMINATED BY
IGNORE NULLS
FIRST
LAST

Why are the changes needed?

let more users know the sql key words usage

Does this PR introduce any user-facing change?

How was this patch tested?

No

update masterbranch

sync

docs/sql-ref-syntax-qry-select-case.md

maropu · 2020-07-10T01:28:54Z

cc: @huaxingao

maropu · 2020-07-10T01:31:43Z

Thanks for the work, @GuoPhilipse ! Please list up the keywords in the PR description that this PR will add.

GuoPhilipse · 2020-07-10T01:36:27Z

Thanks for the work, @GuoPhilipse ! Please list up the keywords in the PR description that this PR will add.
sure, will add it later.

docs/sql-ref-syntax-qry-select-case.md

docs/sql-ref-syntax-qry-select-pivot.md

maropu · 2020-07-17T14:07:52Z

btw, could you update the PR description, too? e.g., remove FROM

GuoPhilipse · 2020-07-17T14:08:47Z

btw, could you update the PR description, too? e.g., remove FROM

sure ,wil update soon.

docs/sql-ref-syntax-ddl-create-table-hiveformat.md

docs/sql-ref-syntax-qry-select-case.md

docs/sql-ref-syntax-qry-select-groupby.md

huaxingao · 2020-07-17T23:13:05Z

also edit the menu-sql.yaml to update the sidebar menu?

docs/sql-ref-syntax-qry-select-lateral-view.md

docs/sql-ref-syntax-qry-select-pivot.md

docs/sql-ref-syntax-ddl-create-table-hiveformat.md

docs/sql-ref-syntax-qry-select-case.md

docs/sql-ref-syntax-qry-select-pivot.md

docs/sql-ref-syntax-qry-select.md

huaxingao · 2020-07-19T03:04:02Z

@GuoPhilipse Sorry I have more nitpicks on grammar. I know this is very tedious. I went through all these when I did the doc PRs. Thanks a lot for doing this!
I have no more comments :)

GuoPhilipse · 2020-07-19T13:48:18Z

Thanks @maropu @huaxingao, really appreciated by your earnest and carefull review. I have leant a lot from this PR. Thanks for your time.
BTW @maropu do you have any new comments?

docs/sql-ref-syntax-qry-select.md

docs/sql-ref-syntax-qry-select-pivot.md

maropu · 2020-07-20T01:09:53Z

docs/sql-ref-syntax-qry-select-pivot.md

+
+```sql
+PIVOT ( { aggregate_expression [ AS aggregate_expression_alias ] } [ , ... ]
+    FOR column_list IN ( expression_list ) )


cc: @maryannxue @gatorsmile

docs/sql-ref-syntax-qry-select-pivot.md

docs/sql-ref-syntax-qry-select-lateral-view.md

docs/sql-ref-syntax-qry-select-case.md

docs/sql-ref-syntax-ddl-create-table-hiveformat.md

docs/sql-ref-syntax-qry-select-groupby.md

maropu · 2020-07-21T07:19:49Z

docs/sql-ref-syntax-qry-select-groupby.md

    Specifies an aggregate function name (MIN, MAX, COUNT, SUM, AVG, etc.).
+    Note that `FIRST` and `LAST` have an optional `IGNORE NULLS` clause; when the option specified,
+    it will returns the first or last value that is not null (or null if all values are null). 
+    **Syntax:** `[ FIRST | LAST ] ( [ distinct ] expression [ IGNORE NULLS ] ) [ FILTER ( WHERE boolean_expression ) ]`


We still need **Syntax:** [ FIRST | LAST ] ( [ distinct ] expression [ IGNORE NULLS ] ) [ FILTER ( WHERE boolean_expression ) ]? It looks verbose to me though.

now have have aggregate functions above and examples in the end, looks enough,let me move it

maropu · 2020-07-21T07:26:19Z

Thanks for the work, @GuoPhilipse ! Looks ok except for one comment. Could you check for the final sign-off? @srowen @dongjoon-hyun @huaxingao @dilipbiswal Note: since all the SQL syntaxes in this update has been implemented in 3.0.0, we can merge this into branch-3.0.

maropu · 2020-07-21T07:34:10Z

@GuoPhilipse Could you generate HTML docs and check if all things are okay for this udpate?
In most PRs for doc updates, the screenshots of HTML docs are often attached in the PR description, e.g., #28672

maropu · 2020-07-21T12:02:33Z

@GuoPhilipse Btw, we still have more document improvement issues;

Improve the structure of auto-generated built-in function pages in SQL references, https://issues.apache.org/jira/browse/SPARK-31513
Add group tags (ExpressionDescription) to all the built-in functions: (Related JIRA: https://issues.apache.org/jira/browse/SPARK-31429)

As you know, we currently have the two duplicated document pages for built-in functions;

If we finish adding the tags, we might be able to remove the former page. If you're interested in more contributions, feel free to take them over. Its very helpful. cc: @HyukjinKwon @huaxingao

GuoPhilipse · 2020-07-21T12:17:07Z

@GuoPhilipse Btw, we still have more document improvement issues;

Improve the structure of auto-generated built-in function pages in SQL references, https://issues.apache.org/jira/browse/SPARK-31513

Add group tags (ExpressionDescription) to all the built-in functions: (Related JIRA: https://issues.apache.org/jira/browse/SPARK-31429)

As you know, we currently have the two duplicated document pages for built-in functions;

https://spark.apache.org/docs/latest/api/sql/index.html

https://spark.apache.org/docs/latest/sql-ref-functions.html#built-in-functions

If we finish adding the tags, we might be able to remove the former page. If you're interested in more contributions, feel free to take them over. Its very helpful. cc: @HyukjinKwon @huaxingao

Thanks @maropu , will glad to take more look on this.

GuoPhilipse · 2020-07-21T13:31:48Z

@GuoPhilipse Could you generate HTML docs and check if all things are okay for this udpate?
In most PRs for doc updates, the screenshots of HTML docs are often attached in the PR description, e.g., #28672
will append later

maropu · 2020-07-23T00:51:07Z

ok to test

maropu · 2020-07-23T00:53:32Z

Note: If nobody has comments, this is just a doc improvement, so I will merge in a few days.

SparkQA · 2020-07-23T01:07:30Z

Test build #126368 has finished for PR 29056 at commit ffa0603.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? update sql-ref docs, the following key words will be added in this PR. CASE/ELSE WHEN/THEN MAP KEYS TERMINATED BY NULL DEFINED AS LINES TERMINATED BY ESCAPED BY COLLECTION ITEMS TERMINATED BY PIVOT LATERAL VIEW OUTER? ROW FORMAT SERDE ROW FORMAT DELIMITED FIELDS TERMINATED BY IGNORE NULLS FIRST LAST ### Why are the changes needed? let more users know the sql key words usage ### Does this PR introduce _any_ user-facing change? ![image](https://user-images.githubusercontent.com/46367746/88148830-c6dc1f80-cc31-11ea-81ea-13bc9dc34550.png) ![image](https://user-images.githubusercontent.com/46367746/88148968-fb4fdb80-cc31-11ea-8649-e8297cf5813e.png) ![image](https://user-images.githubusercontent.com/46367746/88149000-073b9d80-cc32-11ea-9aa4-f914ecd72663.png) ![image](https://user-images.githubusercontent.com/46367746/88149021-0f93d880-cc32-11ea-86ed-7db8672b5aac.png) ### How was this patch tested? No Closes #29056 from GuoPhilipse/add-missing-keywords. Lead-authored-by: GuoPhilipse <[email protected]> Co-authored-by: GuoPhilipse <[email protected]> Signed-off-by: Takeshi Yamamuro <[email protected]> (cherry picked from commit 8de4333) Signed-off-by: Takeshi Yamamuro <[email protected]>

maropu · 2020-07-28T00:42:48Z

Merged to master/branch-3.0. Thanks, @GuoPhilipse !

gatorsmile · 2020-09-25T07:45:56Z

docs/sql-ref-syntax-ddl-create-table-hiveformat.md

        | ( col_name1, col_name2, ... ) ]
    [ ROW FORMAT row_format ]
    [ STORED AS file_format ]
    [ LOCATION path ]


The bucketSpec is still missing in CREATE HIVE FORMAT table, right?

[ CLUSTERED BY ( col_name3, col_name4, ... ) [ SORTED BY ( col_name [ ASC | DESC ], ... ) ] INTO num_buckets BUCKETS ]

Any reason we did not add it? @huaxingao @GuoPhilipse

Sorry, we missed that.
@GuoPhilipse Could you please have a follow-up to add bucketSpec? Thanks!

Yes, we need it, will add it soon.

GuoPhilipse and others added 23 commits May 21, 2020 07:36

Merge pull request #1 from apache/master

bb1efa2

update masterbranch

Merge pull request #2 from apache/master

1459d5b

sync

Merge pull request #3 from apache/master

88c40fe

sync

Merge pull request #4 from apache/master

df22083

sync

Merge pull request #5 from apache/master

0436611

sync

Merge pull request #6 from apache/master

ed80c84

sync

Merge pull request #7 from apache/master

39ca87c

sync

Merge pull request #8 from apache/master

c3b3c89

sync

Merge pull request #9 from apache/master

c3546eb

sync

Merge pull request #10 from apache/master

77a339a

sync

Merge pull request #11 from apache/master

a6b4f74

sync

Merge pull request #12 from apache/master

664277e

sync

Merge pull request #13 from apache/master

fd677c9

sync

Merge pull request #14 from apache/master

93b1f63

sync

Merge pull request #15 from apache/master

a5b5474

sync

Merge pull request #16 from apache/master

f4556a4

sync

Merge pull request #17 from apache/master

6071006

sync

Merge pull request #18 from apache/master

596b842

sync

Merge pull request #19 from apache/master

dc24541

sync

Merge pull request #20 from apache/master

c6f541b

sync

update case when docs

a80fa85

update case when docs

e4c2a17

update case when docs

24b64a8

probot-autolabeler bot added the DOCS label Jul 9, 2020

GuoPhilipse changed the title ~~[SPARK-31753][SQL][DOCS]Add missing keywords~~ [SPARK-31753][SQL][DOCS][WIP]Add missing keywords Jul 9, 2020

dongjoon-hyun reviewed Jul 9, 2020

View reviewed changes

docs/sql-ref-syntax-qry-select-case.md Outdated Show resolved Hide resolved

huaxingao reviewed Jul 10, 2020

View reviewed changes

docs/sql-ref-syntax-qry-select-case.md Outdated Show resolved Hide resolved

maropu reviewed Jul 17, 2020

View reviewed changes

docs/sql-ref-syntax-qry-select-pivot.md Outdated Show resolved Hide resolved

update docs

b281494

huaxingao reviewed Jul 17, 2020

View reviewed changes

huaxingao reviewed Jul 18, 2020

View reviewed changes

update docs\

bed618b

maropu reviewed Jul 18, 2020

View reviewed changes

docs/sql-ref-syntax-ddl-create-table-hiveformat.md Outdated Show resolved Hide resolved

huaxingao reviewed Jul 19, 2020

View reviewed changes

update docs

caa7130

maropu reviewed Jul 20, 2020

View reviewed changes

update docs

2813735

maropu reviewed Jul 21, 2020

View reviewed changes

update docs

ffa0603

maropu closed this in 8de4333 Jul 28, 2020

gatorsmile reviewed Sep 25, 2020

View reviewed changes

GuoPhilipse deleted the add-missing-keywords branch September 27, 2020 11:46

[SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs #29056

[SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs #29056

Uh oh!

Conversation

GuoPhilipse commented Jul 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

maropu commented Jul 10, 2020

Uh oh!

maropu commented Jul 10, 2020

Uh oh!

GuoPhilipse commented Jul 10, 2020

Uh oh!

Uh oh!

Uh oh!

maropu commented Jul 17, 2020

Uh oh!

GuoPhilipse commented Jul 17, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huaxingao commented Jul 17, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

huaxingao commented Jul 19, 2020

Uh oh!

GuoPhilipse commented Jul 19, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maropu Jul 20, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maropu Jul 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GuoPhilipse Jul 21, 2020

Choose a reason for hiding this comment

Uh oh!

maropu commented Jul 21, 2020

Uh oh!

maropu commented Jul 21, 2020

Uh oh!

maropu commented Jul 21, 2020

Uh oh!

GuoPhilipse commented Jul 21, 2020

Uh oh!

GuoPhilipse commented Jul 21, 2020

GuoPhilipse commented Jul 9, 2020 •

edited

Loading

maropu Jul 21, 2020 •

edited

Loading