[SPARK-20909][SQL] Add build-int SQL function - DAYOFWEEK by wangyum · Pull Request #18134 · apache/spark

wangyum · 2017-05-28T04:45:59Z

What changes were proposed in this pull request?

Add build-int SQL function - DAYOFWEEK

How was this patch tested?

unit tests

SparkQA · 2017-05-28T07:02:09Z

Test build #77478 has finished for PR 18134 at commit 02b62b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class DayOfWeek(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

ueshin

LGTM except for a minor comment.

ueshin · 2017-05-29T02:26:02Z


 select to_timestamp(null), to_timestamp('2016-12-31 00:12:00'), to_timestamp('2016-12-31', 'yyyy-MM-dd');
+
+select dayofweek('2007-02-03'), dayofweek('2009-07-30'), dayofweek('2017-05-27'), dayofweek(null), dayofweek('1582-10-15 13:10:15');


nit: Can you add a line break at the end of file?

viirya · 2017-05-29T03:56:03Z

+
+  override protected def nullSafeEval(date: Any): Any = {
+    c.setTimeInMillis(date.asInstanceOf[Int] * 1000L * 3600L * 24L)
+    c.get(Calendar.DAY_OF_WEEK)


In WeekOfYear, we set Calendar.MONDAY as the first day of a week. Here seems we assume it's Calendar.SUNDAY. Is there any conflict we will encounter?

Keep pace with Hive's DayOfWeek
.

viirya · 2017-05-30T02:23:44Z

  }
 }

+// scalastyle:off line.size.limit


As Sunday, Saturday are included, it is not only weekday. Returns the day of the week ....

viirya · 2017-05-30T02:28:07Z

LGTM except for one comment about the function description.

SparkQA · 2017-05-30T04:25:29Z

Test build #77514 has finished for PR 18134 at commit 87defdc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedRelation(
case class StringReplace(srcExpr: Expression, searchExpr: Expression, replaceExpr: Expression)

SparkQA · 2017-05-30T04:51:48Z

Test build #77513 has finished for PR 18134 at commit dcae776.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2017-05-30T05:06:25Z

Test build #77515 has finished for PR 18134 at commit 69d4227.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2017-05-30T06:40:08Z

Thanks! Merging to master.

sergiobilello-eb · 2017-11-10T23:56:34Z

@ueshin This function is not reported in the API documentation: https://spark.apache.org/docs/2.2.0/api/java/index.html?org/apache/spark/sql/functions.html or https://spark.apache.org/docs/2.1.0/api/java/index.html?org/apache/spark/sql/functions.html ?
How is it possible? I am trying to use it from spark-sql on spark 2.1.0 cluster
Thanks

gatorsmile · 2017-11-11T00:02:47Z

@sergiobilello-eb This is just a SQL function. You can call it in SQL interface or using the df.select(expr("dayofweek('2009-07-30')")). It is not part of the DataFrame functions. You can submit a PR or report it as an issue to add such an API.

sergiobilello-eb · 2017-11-11T00:05:27Z

thanks @gatorsmile :) how can I call it from spark-sql?

sergiobilello-eb · 2017-11-11T00:08:36Z

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/

Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_131
Branch
Compiled by user jenkins on 2017-06-30T22:58:04Z
Revision
Url
Type --help for more information.
➜  ~


spark-sql> select dayofweek('2007-02-03'), dayofweek('2009-07-30'), dayofweek('2017-05-27'), dayofweek(null), dayofweek('1582-10-15 13:10:15');
17/11/10 16:08:23 INFO SparkSqlParser: Parsing command: select dayofweek('2007-02-03'), dayofweek('2009-07-30'), dayofweek('2017-05-27'), dayofweek(null), dayofweek('1582-10-15 13:10:15')
17/11/10 16:08:23 INFO HiveMetaStore: 0: get_database: default
17/11/10 16:08:23 INFO audit: ugi=sergio.bilello	ip=unknown-ip-addr	cmd=get_database: default
17/11/10 16:08:23 INFO HiveMetaStore: 0: get_database: default
17/11/10 16:08:23 INFO audit: ugi=sergio.bilello	ip=unknown-ip-addr	cmd=get_database: default
17/11/10 16:08:23 INFO HiveMetaStore: 0: get_function: default.dayofweek
17/11/10 16:08:23 INFO audit: ugi=sergio.bilello	ip=unknown-ip-addr	cmd=get_function: default.dayofweek
Error in query: Undefined function: 'dayofweek'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7```

gatorsmile · 2017-11-11T00:12:44Z

https://issues.apache.org/jira/browse/SPARK-20909

This is not part of 2.2. Based on JIRA, it will be included in 2.3

sergiobilello-eb · 2017-11-11T00:16:30Z

thanks @gatorsmile! Do you suggest any workaround until then? I mean not rebuilding spark with that patch.... Can I register my UDF that contains that logic?

gatorsmile · 2017-11-11T00:18:48Z

Yeah, you always can implement such a UDF.

Add build-int SQL function - DAYOFWEEK

02b62b4

ueshin reviewed May 29, 2017

View reviewed changes

viirya reviewed May 29, 2017

View reviewed changes

wangyum added 2 commits May 30, 2017 09:52

Add a line break at the end of datetime.sql

dcae776

Merge branch 'master' into SPARK-20909

87defdc

viirya reviewed May 30, 2017

View reviewed changes

Update function desc.

69d4227

asfgit closed this in d797ed0 May 30, 2017


		select to_timestamp(null), to_timestamp('2016-12-31 00:12:00'), to_timestamp('2016-12-31', 'yyyy-MM-dd');

		select dayofweek('2007-02-03'), dayofweek('2009-07-30'), dayofweek('2017-05-27'), dayofweek(null), dayofweek('1582-10-15 13:10:15'); No newline at end of file

Conversation

wangyum commented May 28, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 28, 2017

Uh oh!

ueshin left a comment

Choose a reason for hiding this comment

Uh oh!

ueshin May 29, 2017

Choose a reason for hiding this comment

Uh oh!

viirya May 29, 2017

Choose a reason for hiding this comment

Uh oh!

wangyum May 30, 2017

Choose a reason for hiding this comment

Uh oh!

viirya May 30, 2017

Choose a reason for hiding this comment

Uh oh!

viirya commented May 30, 2017

Uh oh!

SparkQA commented May 30, 2017

Uh oh!

SparkQA commented May 30, 2017

Uh oh!

SparkQA commented May 30, 2017

Uh oh!

ueshin commented May 30, 2017

Uh oh!

sergiobilello-eb commented Nov 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Nov 11, 2017

Uh oh!

sergiobilello-eb commented Nov 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergiobilello-eb commented Nov 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Nov 11, 2017

Uh oh!

sergiobilello-eb commented Nov 11, 2017

Uh oh!

gatorsmile commented Nov 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sergiobilello-eb commented Nov 10, 2017 •

edited

Loading

sergiobilello-eb commented Nov 11, 2017 •

edited

Loading

sergiobilello-eb commented Nov 11, 2017 •

edited

Loading