[SPARK-23927][SQL] Add "sequence" expression #21155

wajda · 2018-04-25T13:28:16Z

What changes were proposed in this pull request?

The PR adds the SQL function sequence.
https://issues.apache.org/jira/browse/SPARK-23927

The behavior of the function is based on Presto's one.
Ref: https://prestodb.io/docs/current/functions/array.html

sequence(start, stop) → array<bigint>
Generate a sequence of integers from start to stop, incrementing by 1 if start is less than or equal to stop, otherwise -1.
sequence(start, stop, step) → array<bigint>
Generate a sequence of integers from start to stop, incrementing by step.
sequence(start_date, stop_date) → array<date>
Generate a sequence of dates from start_date to stop_date, incrementing by interval 1 day if start_date is less than or equal to stop_date, otherwise - interval 1 day.
sequence(start_date, stop_date, step_interval) → array<date>
Generate a sequence of dates from start_date to stop_date, incrementing by step_interval. The type of step_interval is CalendarInterval.
sequence(start_timestemp, stop_timestemp) → array<timestamp>
Generate a sequence of timestamps from start_timestamps to stop_timestamps, incrementing by interval 1 day if start_date is less than or equal to stop_date, otherwise - interval 1 day.
sequence(start_timestamp, stop_timestamp, step_interval) → array<timestamp>
Generate a sequence of timestamps from start_timestamps to stop_timestamps, incrementing by step_interval. The type of step_interval is CalendarInterval.

How was this patch tested?

Added unit tests.

gatorsmile · 2018-04-25T16:17:14Z

ok to test

gatorsmile · 2018-04-25T16:17:25Z

cc @ueshin

SparkQA · 2018-04-25T16:19:45Z

Test build #89851 has finished for PR 21155 at commit 8fb427c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedLiteral(child: Expression, literalResolver: DataType => Literal)
trait SequenceImpl
case class Sequence(left: Expression,
class IntegralSequenceImpl[T: ClassTag](implicit num: Integral[T]) extends SequenceImpl
abstract class TemporalSequence[T: ClassTag](dt: IntegralType, scale: Long, fromLong: Long => T)

kiszk · 2018-04-25T16:28:24Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

Is there any reason to use lazy here?

No reason. It's a left over, sorry.
Fixed.

SparkQA · 2018-04-25T19:36:54Z

Test build #89854 has finished for PR 21155 at commit 88fde1c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-25T19:42:27Z

Test build #89852 has finished for PR 21155 at commit 0b4d868.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-26T00:58:04Z

Test build #89860 has finished for PR 21155 at commit a7c0ccd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin

Could you annotate the pr title as [SPARK-23927][SQL]?
I left some comments, but since this is kind of large, let me have some more time to review thoroughly.
Thanks!

ueshin · 2018-05-04T09:16:50Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala

What if step is interval?

Maybe we should use ExpectsInputTypes or ImplicitCastInputTypes instead of TypeCoercion.

Good catch, thank you. I've added a unit test and fixed coercion when the step is interval.
Regarding ImplicitCastInputTypes I tried it, but it isn't flexible enough for my case. To support actual type coercion for both integral and temporal sequences I had to resort to TypeCoercion. Please check the fixed version. Is there any better way how to do it?

ueshin · 2018-05-04T09:18:27Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

nit: style

case class Sequence( left: Expression, middle: Expression, right: Expression, timeZoneId: Option[String] = None) extends TernaryExpression with TimeZoneAwareExpression { ...

And should we use start, stop, step instead of left, middle, right respectively for readability?

How about making step: Option[Expression] so we don't need UnresolvedLiteral, I guess. Similar usage is in ArrayJoin expression.

Changed to Option[Expression], removed UnresolvedLiteral. Thanks for suggestion.

ueshin · 2018-05-04T09:21:44Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

start, stop?

ueshin · 2018-05-04T09:21:56Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

start, stop, step?

ueshin · 2018-05-04T09:27:17Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

Maybe we need to fix indent in the usage, arguments, examples.

ueshin · 2018-05-04T09:51:30Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

start, stop, step?

ueshin · 2018-05-04T09:55:39Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

nit: style

def genCode (start: String, stop: String, step: String) (arr: String, elemType: String) (implicit ctx: CodegenContext): String

ueshin · 2018-05-04T09:56:55Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

ueshin · 2018-05-04T09:57:48Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

nit: indent

ueshin · 2018-05-04T09:58:06Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

nit: indent

wajda · 2018-05-04T14:19:20Z

Thanks you for the comments. I'll review and make changes shortly.

wajda · 2018-05-04T14:28:42Z

Regarding step, stop, step vs. arg1, arg2, arg3, left, middle, right and so on. Originally I used those to avoid hiding variables from the outer scope, but after the final refactoring it doesn't seem to be an issue anymore. I'll rename those, but I would keep ones as is for the def eval(input1: Any, input2: Any, input3: Any) method implementations. The reason is to distinguish the argument of type Any from their cast references as below:

 override def eval(input1: Any, input2: Any, input3: Any): Array[T] = {
      val start = input1.asInstanceOf[T]
      val stop = input2.asInstanceOf[T]
      val step = input3.asInstanceOf[T]

SparkQA · 2018-05-04T22:18:32Z

Test build #90207 has finished for PR 21155 at commit d15dad8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2018-05-07T18:05:39Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

How about val StartType = start.dataType? Then, replace stopType with stop.dataType at line 1540.

SparkQA · 2018-05-07T19:44:31Z

Test build #90330 has finished for PR 21155 at commit 383f180.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-05-07T20:20:10Z

Test build #90329 has finished for PR 21155 at commit 8786192.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2018-05-10T00:39:58Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

We might need to override checkInputDataTypes here to check if elemType is one of DateType, TimestampType or IntegralType. And also the validity of type of step expression.

Hm, isn't it exactly what I had before? Just yesterday I followed @ueshin suggestion and replaced checkInputDataTypes with ExpectsInputTypes trait + inputTypes

Both implementations basically do the same thing - check the validity of the element and step types. Unless I am missing something. There are unit tests for mixed types cases.

Default checkInputDataTypes just checks if data types of children matches inputTypes. Looks like for now here we don't have any limit. So even an invalid data type is given, checkInputDataTypes won't complain because inputTypes simply picks up children's data types.

Seems there are no test cases with invalid data types.

It would fail with match error, but I see your point.
Reverted to checkInputDataTypes, improved type check conditions, added tests for invalid data types.

I'm wondering what was wrong with checkInputDataTypes, but I can't see the difference because the commit was just overwritten.

SparkQA · 2018-05-10T01:59:31Z

Test build #90428 has finished for PR 21155 at commit 22bde31.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-05-10T15:49:39Z

Test build #90456 has finished for PR 21155 at commit 0a0a1e6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-05-21T12:40:04Z

Test build #90892 has finished for PR 21155 at commit 1b74d6f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-05-21T17:13:07Z

Test build #90899 has finished for PR 21155 at commit 4b8af57.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Sequence(

SparkQA · 2018-05-21T17:16:54Z

Test build #90893 has finished for PR 21155 at commit 3ef9566.

This patch fails SparkR unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
case class Sequence(

SparkQA · 2018-05-21T20:43:41Z

Test build #90909 has finished for PR 21155 at commit 2ccb9bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Sequence(

wajda · 2018-05-23T11:44:29Z

Any other comments on this one?

ueshin

I'm sorry for the super delay.
I left some comments. Thanks.

ueshin · 2018-06-07T21:50:46Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

We can use withNewChildren instead?

what is the purpose? Copy constructor works the best in this case as it's type safe and does exactly what I need to do.
I tried to use withNewChildren like this:

def castChildrenTo(widerType: DataType): Expression = withNewChildren(Seq( Cast(start, widerType), Cast(stop, widerType)) ++ stepOpt.map(step => if (step.dataType != CalendarIntervalType) Cast(step, widerType) else step))

but it doesn't work as expected for some reason. I didn't grasp all the magic it does yet, but do we really need to complicate things? Why is a copy constructor a bad choice in this case?

I just wanted to confirm if we can use withNewChildren or not, and it's okay to use the copy constructor if it's hard to use the method.

ueshin · 2018-06-08T00:38:18Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

We should use ctx.addReferenctObj("timeZone", timeZone) instead to follow the implementations of datetime expressions? Doing TimeZone.getTimeZone() for every row should be slow.

Good catch. Done.

ueshin · 2018-06-08T00:42:09Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH instead of Int.MaxValue?

ueshin · 2018-06-08T00:43:15Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

Could you add more parenthesis to be clear the boundaries between && and ||? The combinations of && and || without parenthesis may cause an unexpected behavior.

( ... && ... ) || ( ... && ... ) || ...

ueshin · 2018-06-08T00:47:53Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

ueshin · 2018-06-08T00:48:46Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

Maybe we don't need this?

ueshin · 2018-06-08T00:55:55Z

...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala

Maybe we need additional words in the test title?

ueshin · 2018-06-08T00:59:45Z

...st/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala

Could you use a long name? 3-letter name might not work in some environment.

HyukjinKwon · 2018-06-09T09:08:40Z

ok to test

SparkQA · 2018-06-09T09:19:58Z

Test build #91617 has finished for PR 21155 at commit 2ccb9bb.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Sequence(

SparkQA · 2018-06-22T20:05:18Z

Test build #92217 has finished for PR 21155 at commit c54857a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin

Could you split the current updates into some PRs to add sequence function and refactor other functions?
We usually do one thing in one PR unless we have a special reason. Thanks!

ueshin · 2018-06-25T04:01:41Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

We can try to cast to CalendarIntervalType if widerType is DateType or TimestampType and step.dataType is StringType? We might want to use StringType for the interval.

How can it be useful? Do you have a use case example?
When using SQL notation an interval step is automatically parsed as a correct type, so I never get it as String in this method. Unless I am mistaken the only way for step to be of String type is if Sequence expression is created programmatically with a string type expression passed as 3rd argument. But in this case having strong typing is rather a good thing, isn't it?

Makes sense. Thanks!

ueshin · 2018-06-25T04:19:54Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

MAX_ROUNDED_ARRAY_LENGTH instead of Integer.MAX_VALUE?

SparkQA · 2018-06-25T11:52:03Z

Test build #92295 has finished for PR 21155 at commit def678b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

… Int.MaxValue

SparkQA · 2018-06-26T07:05:01Z

Test build #92326 has finished for PR 21155 at commit 140225d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wajda · 2018-06-26T07:27:01Z

please retest

ueshin · 2018-06-26T08:00:33Z

Jenkins, retest this please.

ueshin

LGTM except for one comment.

ueshin · 2018-06-26T08:16:08Z

...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala

+    require(
+      (step > num.zero && start <= stop)
+        || (step < num.zero && start >= stop)
+        || (step == 0 && start == stop),


num.zero instead of 0?

Fixed. Although I wonder what difference it makes besides the readability? Doesn't equals(0) work consistently on every numeric types in Scala? Or is it related to autoboxing somehow? It's not very clear to me in this context. Thanks.

SparkQA · 2018-06-26T11:44:03Z

Test build #92331 has finished for PR 21155 at commit 140225d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-06-26T13:47:27Z

Test build #92334 has finished for PR 21155 at commit d6de8cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2018-06-27T02:51:33Z

Thanks! merging to master.

## What changes were proposed in this pull request? This pr is a follow-up pr of #21155. The #21155 removed unnecessary import at that time, but the import became necessary in another pr. ## How was this patch tested? Existing tests. Author: Takuya UESHIN <[email protected]> Closes #21646 from ueshin/issues/SPARK-23927/fup1.

wajda force-pushed the feature/array-api-sequence branch from 5cc7d87 to 8fb427c Compare April 25, 2018 13:45

kiszk reviewed Apr 25, 2018

View reviewed changes

ueshin reviewed May 4, 2018

View reviewed changes

wajda changed the title ~~SPARK-23927: Add "sequence" expression~~ [SPARK-23927][SQL] Add "sequence" expression May 4, 2018

wajda force-pushed the feature/array-api-sequence branch from a7c0ccd to d15dad8 Compare May 4, 2018 18:20

wajda force-pushed the feature/array-api-sequence branch from d15dad8 to 8786192 Compare May 7, 2018 16:34

kiszk reviewed May 7, 2018

View reviewed changes

viirya reviewed May 10, 2018

View reviewed changes

wajda force-pushed the feature/array-api-sequence branch from 0a0a1e6 to 3ef9566 Compare May 21, 2018 12:25

wajda force-pushed the feature/array-api-sequence branch 2 times, most recently from 4b8af57 to 2ccb9bb Compare May 21, 2018 16:59

ueshin reviewed Jun 8, 2018

View reviewed changes

wajda force-pushed the feature/array-api-sequence branch from 2ccb9bb to c54857a Compare June 22, 2018 16:21

ueshin reviewed Jun 25, 2018

View reviewed changes

wajda force-pushed the feature/array-api-sequence branch from c54857a to def678b Compare June 25, 2018 08:11

wajda added 5 commits June 26, 2018 08:37

SPARK-23927: Add "sequence" expression

61068e3

SPARK-23927: use ctx.addReferenceObj instead of TimeZone.getTimeZone

b1a5333

SPARK-23927: use ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH instead of…

de044ea

… Int.MaxValue

SPARK-23927: minor code style fixes

163a748

SPARK-23927: remove redundant "|boolean ${ev.isNull} = false;"

140225d

wajda force-pushed the feature/array-api-sequence branch from def678b to 140225d Compare June 26, 2018 06:38

ueshin reviewed Jun 26, 2018

View reviewed changes

SPARK-23927: num.zero instead of 0

d6de8cb

asfgit closed this in 2669b4d Jun 27, 2018

ueshin mentioned this pull request Jun 27, 2018

[SPARK-23927][SQL][FOLLOW-UP] Fix a build failure. #21646

Closed

MaxGekk mentioned this pull request Jul 15, 2019

[SPARK-28389][SQL] Use Java 8 API in add_months #25153

Closed

HyukjinKwon mentioned this pull request Jun 14, 2020

[SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates #28819

Closed

[SPARK-23927][SQL] Add "sequence" expression #21155

[SPARK-23927][SQL] Add "sequence" expression #21155

Uh oh!

Conversation

wajda commented Apr 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

gatorsmile commented Apr 25, 2018

Uh oh!

gatorsmile commented Apr 25, 2018

Uh oh!

SparkQA commented Apr 25, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 25, 2018

Uh oh!

SparkQA commented Apr 25, 2018

Uh oh!

SparkQA commented Apr 26, 2018

Uh oh!

ueshin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wajda commented May 4, 2018

Uh oh!

wajda commented May 4, 2018

Uh oh!

SparkQA commented May 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 7, 2018

Uh oh!

SparkQA commented May 7, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wajda May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wajda commented Apr 25, 2018 •

edited

Loading

wajda May 10, 2018 •

edited

Loading

viirya May 10, 2018 •

edited

Loading

wajda Jun 22, 2018 •

edited

Loading