[SPARK-32193][SQL][DOCS] Update regexp usage in SQL docs #29009

GuoPhilipse · 2020-07-06T10:03:41Z

What changes were proposed in this pull request?

update REGEXP usage and examples in sql-ref-syntx-qry-select-like.cmd

Why are the changes needed?

make the usage of REGEXP known to more users

Does this PR introduce any user-facing change?

No

How was this patch tested?

No tests

update masterbranch

sync

maropu · 2020-07-06T10:33:55Z

I think you don't need to file jira for this kind of minor doc fixes. Btw, any other systems supporting REGEXP for regular expressions other than Hive? I think we might be able to assign an alias name to it if it is a general name.

maropu · 2020-07-06T10:34:48Z

docs/sql-migration-guide.md

 * `ACOS(n)` If n < -1 or n > 1, Hive returns null, Spark SQL returns NaN.
 * `ASIN(n)` If n < -1 or n > 1, Hive returns null, Spark SQL returns NaN.
 * `CAST(n AS TIMESTAMP)` If n is integral numbers, Hive treats n as milliseconds, Spark SQL treats n as seconds.
+* `REGEXP(str, patten)` Hive support this function, Spark SQL use RLIKE instead.


nit: support -> supports and use -> uses

maropu · 2020-07-06T10:35:04Z

cc: @HyukjinKwon

GuoPhilipse · 2020-07-06T10:58:56Z

I think you don't need to file jira for this kind of minor doc fixes. Btw, any other systems supporting REGEXP for regular expressions other than Hive? I think we might be able to assign an alias name to it if it is a general name.

So far as i know ,mysql and hive support this key word .But i think a alias name seems a good idea.
mysql examples:
SELECT 'abc' REGEXP '([a-z]+)';
Ressult:
1

HyukjinKwon · 2020-07-06T11:05:00Z

Thanks @maropu. @GuoPhilipse, does MySQL also supports REGEXP as a function?

From reading the doc in Hive at https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF, it's same as Rlike which Spark supports. Maybe we should alias it.

FWIW, there are many unsupported expressions explicitly:

spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala

Lines 189 to 192 in 5d870ef

    
           // List of functions we are explicitly not supporting are: 
        
           // compute_stats, context_ngrams, create_union, 
        
           // current_user, ewah_bitmap, ewah_bitmap_and, ewah_bitmap_empty, ewah_bitmap_or, field, 
        
           // in_file, index, matchpath, ngrams, noop, noopstreaming, noopwithmap,

HyukjinKwon · 2020-07-06T11:05:52Z

docs/sql-migration-guide.md

 * `ACOS(n)` If n < -1 or n > 1, Hive returns null, Spark SQL returns NaN.
 * `ASIN(n)` If n < -1 or n > 1, Hive returns null, Spark SQL returns NaN.
 * `CAST(n AS TIMESTAMP)` If n is integral numbers, Hive treats n as milliseconds, Spark SQL treats n as seconds.
+* `REGEXP(str, patten)` Hive support this function, Spark SQL use RLIKE instead.


Also, technically this isn's in "the scenarios in which Hive and Spark generate different results".

GuoPhilipse · 2020-07-06T11:08:03Z

Thanks @maropu. @GuoPhilipse, does MySQL also supports REGEXP as a function?

From reading the doc in Hive at https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF, it's same as Rlike which Spark supports. Maybe we should alias it.

FWIW, there are many unsupported expressions explicitly:

spark/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala

Lines 189 to 192 in 5d870ef

// List of functions we are explicitly not supporting are:

// compute_stats, context_ngrams, create_union,

// current_user, ewah_bitmap, ewah_bitmap_and, ewah_bitmap_empty, ewah_bitmap_or, field,

// in_file, index, matchpath, ngrams, noop, noopstreaming, noopwithmap,

yes, i posted the REGEXP usage in mysql last comment.

HyukjinKwon · 2020-07-06T11:11:00Z

No, @GuoPhilipse, I meant if it's supported in SELECT REGEXP('abc', '([a-z]+)'); way.

maropu · 2020-07-06T12:36:39Z

I get a bit confiused and Hive really supports REGEX as a function?

// REGEXP case in hive (3.1.1)
hive> SELECT 'abc' REGEXP '([a-z]+)';
OK
true

hive> select REGEXP('abc','([a-z]+)');
NoViableAltException(251@[])
	at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:962)
    ...
FAILED: ParseException line 1:7 cannot recognize input near 'REGEXP' '(' ''abc'' in select clause

// RLIKE in hive
hive> SELECT 'abc' RLIKE '([a-z]+)';
OK
true

hive> select RLIKE('abc','([a-z]+)');
NoViableAltException(265@[])
	at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:962)
	...
FAILED: ParseException line 1:7 cannot recognize input near 'RLIKE' '(' ''abc'' in select clause

// REGEXP in mysql
mysql> SELECT 'abc' REGEXP '([a-z]+)';
+-------------------------+
| 'abc' REGEXP '([a-z]+)' |
+-------------------------+
|                       1 |
+-------------------------+

mysql> select REGEXP('abc','([a-z]+)');
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'REGEXP('abc','([a-z]+)')' at line 1

// RLIKE in mysql
mysql> SELECT 'abc' RLIKE '([a-z]+)';
+------------------------+
| 'abc' RLIKE '([a-z]+)' |
+------------------------+
|                      1 |
+------------------------+

mysql> select RLIKE('abc','([a-z]+)');
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'RLIKE('abc','([a-z]+)')' at line 1

GuoPhilipse · 2020-07-06T12:50:30Z

I tried in hive 1.1, and it works. seems hive is incompatible for old version

//RLIKE
hive> select RLIKE('abc','([a-z]+)');
true

hive> select 'abc' RLIKE'([a-z]+)';
true

//REGEXP
hive> select REGEXP('abc','([a-z]+)');
true

hive> select 'abc' REGEXP'([a-z]+)';
true

maropu · 2020-07-06T12:57:29Z

Yea, I checked the doc @HyukjinKwon put above, and it seems the current hive only supports REGEXP only in a SQL syntax.

sync

maropu · 2020-07-09T01:01:47Z

docs/sql-ref-syntax-qry-select-like.md


 ```sql
-[ NOT ] { LIKE search_pattern [ ESCAPE esc_char ] | RLIKE regex_pattern }
+[ NOT ] { LIKE search_pattern [ ESCAPE esc_char ] | RLIKE regex_pattern | REGEXP regex_pattern}


[ RLIKE | REGEXP ] regex_pattern cc: @huaxingao

I think its okay for this PR to just update this doc.

have updated and added examples for this usage.

Btw, this reminded me that there are still missing keywords in the SQL docs: https://issues.apache.org/jira/browse/SPARK-31753
Its very helpful if you take this over.

I am glad to take this over,will work on it later :)

Thanks! That's very helpful.

maropu · 2020-07-09T01:02:12Z

docs/sql-migration-guide.md


  - In Spark 3.1, `from_unixtime`, `unix_timestamp`,`to_unix_timestamp`, `to_timestamp` and `to_date` will fail if the specified datetime pattern is invalid. In Spark 3.0 or earlier, they result `NULL`.

+  - In Spark 3.1, we can use regexp function, it is the alias of rlike funtion, which functions the same with rike funtion.


We don't need this.

maropu · 2020-07-09T01:03:13Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

    expression[StringReplace]("replace"),
    expression[Overlay]("overlay"),
    expression[RLike]("rlike"),
+    expression[RLike]("regexp", true),


Need this? It seems hive and mysql doesn't support this though.

yes, let me will remove the function usage

HyukjinKwon · 2020-07-09T02:37:28Z

ok to test

SparkQA · 2020-07-09T02:56:39Z

Test build #125427 has finished for PR 29009 at commit 06cfd5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

Looks okay but let me leave it to @maropu

### What changes were proposed in this pull request? update REGEXP usage and examples in sql-ref-syntx-qry-select-like.cmd ### Why are the changes needed? make the usage of REGEXP known to more users ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No tests Closes #29009 from GuoPhilipse/update-migrate-guide. Lead-authored-by: GuoPhilipse <[email protected]> Co-authored-by: GuoPhilipse <[email protected]> Signed-off-by: Takeshi Yamamuro <[email protected]> (cherry picked from commit 09cc6c5) Signed-off-by: Takeshi Yamamuro <[email protected]>

maropu · 2020-07-09T07:15:32Z

Thanks! Merged to master/3.0.

GuoPhilipse and others added 20 commits May 21, 2020 07:36

Merge pull request #1 from apache/master

bb1efa2

update masterbranch

Merge pull request #2 from apache/master

1459d5b

sync

Merge pull request #3 from apache/master

88c40fe

sync

Merge pull request #4 from apache/master

df22083

sync

Merge pull request #5 from apache/master

0436611

sync

Merge pull request #6 from apache/master

ed80c84

sync

Merge pull request #7 from apache/master

39ca87c

sync

Merge pull request #8 from apache/master

c3b3c89

sync

Merge pull request #9 from apache/master

c3546eb

sync

Merge pull request #10 from apache/master

77a339a

sync

Merge pull request #11 from apache/master

a6b4f74

sync

Merge pull request #12 from apache/master

664277e

sync

Merge pull request #13 from apache/master

fd677c9

sync

Merge pull request #14 from apache/master

93b1f63

sync

Merge pull request #15 from apache/master

a5b5474

sync

Merge pull request #16 from apache/master

f4556a4

sync

Merge pull request #17 from apache/master

6071006

sync

Merge pull request #18 from apache/master

596b842

sync

Merge pull request #19 from apache/master

dc24541

sync

update sql migrate guide

d99c869

probot-autolabeler bot added the DOCS label Jul 6, 2020

maropu reviewed Jul 6, 2020

View reviewed changes

HyukjinKwon changed the title ~~[SPARK-32193][SQL]update migrate guide docs on regexp function~~ [SPARK-32193][SQL] Update migrate guide docs on regexp function Jul 6, 2020

HyukjinKwon reviewed Jul 6, 2020

View reviewed changes

add alias for function rlike

805c8c3

probot-autolabeler bot added the SQL label Jul 9, 2020

GuoPhilipse and others added 3 commits July 9, 2020 08:21

Merge pull request #20 from apache/master

c6f541b

sync

resolve confict

386888a

add migrate dos

62d2fb2

GuoPhilipse changed the title ~~[SPARK-32193][SQL] Update migrate guide docs on regexp function~~ [SPARK-32193][SQL] Add alias for function rlike Jul 9, 2020

maropu reviewed Jul 9, 2020

View reviewed changes

update docs

06cfd5d

GuoPhilipse changed the title ~~[SPARK-32193][SQL] Add alias for function rlike~~ [SPARK-32193][DOCS] Update regexp usage in docs Jul 9, 2020

HyukjinKwon approved these changes Jul 9, 2020

View reviewed changes

maropu changed the title ~~[SPARK-32193][DOCS] Update regexp usage in docs~~ [SPARK-32193][SQL][DOCS] Update regexp usage in SQL docs Jul 9, 2020

maropu approved these changes Jul 9, 2020

View reviewed changes

maropu closed this in 09cc6c5 Jul 9, 2020


		- In Spark 3.1, `from_unixtime`, `unix_timestamp`,`to_unix_timestamp`, `to_timestamp` and `to_date` will fail if the specified datetime pattern is invalid. In Spark 3.0 or earlier, they result `NULL`.

		- In Spark 3.1, we can use regexp function, it is the alias of rlike funtion, which functions the same with rike funtion.

[SPARK-32193][SQL][DOCS] Update regexp usage in SQL docs #29009

[SPARK-32193][SQL][DOCS] Update regexp usage in SQL docs #29009

Uh oh!

Conversation

GuoPhilipse commented Jul 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

maropu commented Jul 6, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maropu commented Jul 6, 2020

Uh oh!

GuoPhilipse commented Jul 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Jul 6, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GuoPhilipse commented Jul 6, 2020

Uh oh!

HyukjinKwon commented Jul 6, 2020

Uh oh!

maropu commented Jul 6, 2020

Uh oh!

GuoPhilipse commented Jul 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maropu commented Jul 6, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

maropu commented Jul 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

GuoPhilipse commented Jul 6, 2020 •

edited

Loading

GuoPhilipse commented Jul 6, 2020 •

edited

Loading

GuoPhilipse commented Jul 6, 2020 •

edited

Loading