Skip to content
Closed
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
bb1efa2
Merge pull request #1 from apache/master
GuoPhilipse May 20, 2020
1459d5b
Merge pull request #2 from apache/master
GuoPhilipse May 22, 2020
88c40fe
Merge pull request #3 from apache/master
GuoPhilipse May 22, 2020
df22083
Merge pull request #4 from apache/master
GuoPhilipse May 25, 2020
0436611
Merge pull request #5 from apache/master
GuoPhilipse May 27, 2020
ed80c84
Merge pull request #6 from apache/master
GuoPhilipse May 27, 2020
39ca87c
Merge pull request #7 from apache/master
GuoPhilipse May 28, 2020
c3b3c89
Merge pull request #8 from apache/master
GuoPhilipse May 28, 2020
c3546eb
Merge pull request #9 from apache/master
GuoPhilipse Jun 2, 2020
77a339a
Merge pull request #10 from apache/master
GuoPhilipse Jun 3, 2020
a6b4f74
Merge pull request #11 from apache/master
GuoPhilipse Jun 5, 2020
664277e
Merge pull request #12 from apache/master
GuoPhilipse Jun 7, 2020
fd677c9
Merge pull request #13 from apache/master
GuoPhilipse Jun 8, 2020
93b1f63
Merge pull request #14 from apache/master
GuoPhilipse Jun 10, 2020
a5b5474
Merge pull request #15 from apache/master
GuoPhilipse Jun 11, 2020
f4556a4
Merge pull request #16 from apache/master
GuoPhilipse Jun 11, 2020
6071006
Merge pull request #17 from apache/master
GuoPhilipse Jun 26, 2020
596b842
Merge pull request #18 from apache/master
GuoPhilipse Jun 29, 2020
dc24541
Merge pull request #19 from apache/master
GuoPhilipse Jul 5, 2020
d99c869
update sql migrate guide
GuoPhilipse Jul 6, 2020
805c8c3
add alias for function rlike
GuoPhilipse Jul 9, 2020
c6f541b
Merge pull request #20 from apache/master
GuoPhilipse Jul 9, 2020
386888a
resolve confict
GuoPhilipse Jul 9, 2020
62d2fb2
add migrate dos
GuoPhilipse Jul 9, 2020
06cfd5d
update docs
GuoPhilipse Jul 9, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/sql-migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ license: |

- In Spark 3.1, `from_unixtime`, `unix_timestamp`,`to_unix_timestamp`, `to_timestamp` and `to_date` will fail if the specified datetime pattern is invalid. In Spark 3.0 or earlier, they result `NULL`.

- In Spark 3.1, we can use regexp function, it is the alias of rlike funtion, which functions the same with rike funtion.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this.


## Upgrading from Spark SQL 3.0 to 3.0.1

- In Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option `inferTimestamp` to `true` to enable such type inference.
Expand Down
2 changes: 1 addition & 1 deletion docs/sql-ref-syntax-qry-select-like.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ A LIKE predicate is used to search for a specific pattern.
### Syntax

```sql
[ NOT ] { LIKE search_pattern [ ESCAPE esc_char ] | RLIKE regex_pattern }
[ NOT ] { LIKE search_pattern [ ESCAPE esc_char ] | RLIKE regex_pattern | REGEXP regex_pattern}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[ RLIKE | REGEXP ] regex_pattern cc: @huaxingao

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its okay for this PR to just update this doc.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have updated and added examples for this usage.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, this reminded me that there are still missing keywords in the SQL docs: https://issues.apache.org/jira/browse/SPARK-31753
Its very helpful if you take this over.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am glad to take this over,will work on it later :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! That's very helpful.

```

### Parameters
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,7 @@ object FunctionRegistry {
expression[StringReplace]("replace"),
expression[Overlay]("overlay"),
expression[RLike]("rlike"),
expression[RLike]("regexp", true),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need this? It seems hive and mysql doesn't support this though.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, let me will remove the function usage

expression[StringRPad]("rpad"),
expression[StringTrimRight]("rtrim"),
expression[Sentences]("sentences"),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ package object dsl {
def like(other: Expression, escapeChar: Char = '\\'): Expression =
Like(expr, other, escapeChar)
def rlike(other: Expression): Expression = RLike(expr, other)
def regexp(other: Expression): Expression = RLike(expr, other)
def contains(other: Expression): Expression = Contains(expr, other)
def startsWith(other: Expression): Expression = StartsWith(expr, other)
def endsWith(other: Expression): Expression = EndsWith(expr, other)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,43 @@ class RegexpExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
}
}

test("SPARK-32193: add alias for rlike function") {
checkLiteralRow(Literal.create(null, StringType) regexp _, "abdef", null)
checkEvaluation("abdef" regexp Literal.create(null, StringType), null)
checkEvaluation(Literal.create(null, StringType) regexp Literal.create(null, StringType), null)
checkEvaluation("abdef" regexp NonFoldableLiteral.create("abdef", StringType), true)
checkEvaluation("abdef" regexp NonFoldableLiteral.create(null, StringType), null)
checkEvaluation(
Literal.create(null, StringType) regexp NonFoldableLiteral.create("abdef", StringType), null)
checkEvaluation(
Literal.create(null, StringType) regexp NonFoldableLiteral.create(null, StringType), null)

checkLiteralRow("abdef" regexp _, "abdef", true)
checkLiteralRow("abbbbc" regexp _, "a.*c", true)

checkLiteralRow("fofo" regexp _, "^fo", true)
checkLiteralRow("fo\no" regexp _, "^fo\no$", true)
checkLiteralRow("Bn" regexp _, "^Ba*n", true)
checkLiteralRow("afofo" regexp _, "fo", true)
checkLiteralRow("afofo" regexp _, "^fo", false)
checkLiteralRow("Baan" regexp _, "^Ba?n", false)
checkLiteralRow("axe" regexp _, "pi|apa", false)
checkLiteralRow("pip" regexp _, "^(pi)*$", false)

checkLiteralRow("abc" regexp _, "^ab", true)
checkLiteralRow("abc" regexp _, "^bc", false)
checkLiteralRow("abc" regexp _, "^ab", true)
checkLiteralRow("abc" regexp _, "^bc", false)

intercept[java.util.regex.PatternSyntaxException] {
evaluateWithoutCodegen("abbbbc" regexp "**")
}
intercept[java.util.regex.PatternSyntaxException] {
val regex = 'a.string.at(0)
evaluateWithoutCodegen("abbbbc" regexp regex, create_row("**"))
}
}

test("RegexReplace") {
val row1 = create_row("100-200", "(\\d+)", "num")
val row2 = create_row("100-200", "(\\d+)", "###")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,7 @@
| org.apache.spark.sql.catalyst.expressions.Pow | power | SELECT power(2, 3) | struct<POWER(CAST(2 AS DOUBLE), CAST(3 AS DOUBLE)):double> |
| org.apache.spark.sql.catalyst.expressions.Quarter | quarter | SELECT quarter('2016-08-31') | struct<quarter(CAST(2016-08-31 AS DATE)):int> |
| org.apache.spark.sql.catalyst.expressions.RLike | rlike | SELECT '%SystemDrive%\Users\John' rlike '%SystemDrive%\\Users.*' | struct<%SystemDrive%UsersJohn RLIKE %SystemDrive%\Users.*:boolean> |
| org.apache.spark.sql.catalyst.expressions.RLike | regexp | SELECT '%SystemDrive%\Users\John' regexp '%SystemDrive%\\Users.*' | struct<%SystemDrive%UsersJohn REGEXP %SystemDrive%\Users.*:boolean> |
| org.apache.spark.sql.catalyst.expressions.Rand | random | SELECT random() | struct<rand():double> |
| org.apache.spark.sql.catalyst.expressions.Rand | rand | SELECT rand() | struct<rand():double> |
| org.apache.spark.sql.catalyst.expressions.Randn | randn | SELECT randn() | struct<randn():double> |
Expand Down