Skip to content

[SPARK-32272][SQL] Add SQL standard command SET TIME ZONE #29064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 12 commits into from
Closed

[SPARK-32272][SQL] Add SQL standard command SET TIME ZONE #29064

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
7910c9a
[SPARK-32272][SQL] Add and extend SQL standard command SET TIME ZONE
yaooqinn Jul 10, 2020
13056b5
add more tests
yaooqinn Jul 10, 2020
225695f
fix parser
yaooqinn Jul 10, 2020
3a62ecc
support interval
yaooqinn Jul 13, 2020
ef1f3ed
add tests
yaooqinn Jul 13, 2020
95a756a
add doc
yaooqinn Jul 13, 2020
c95fa7d
rm list all api
yaooqinn Jul 13, 2020
5501213
check second precision
yaooqinn Jul 13, 2020
b3ac6f4
doc
yaooqinn Jul 15, 2020
e5aa3b3
address comments and fix tests
yaooqinn Jul 16, 2020
11f0fed
Merge remote-tracking branch 'ssh/master' into SPARK-32272
yaooqinn Jul 16, 2020
a4c60f3
doc
yaooqinn Jul 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/sql-ref-ansi-compliance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,7 @@ Below is a list of all the keywords in Spark SQL.
|TEMPORARY|non-reserved|non-reserved|non-reserved|
|TERMINATED|non-reserved|non-reserved|non-reserved|
|THEN|reserved|non-reserved|reserved|
|TIME|reserved|non-reserved|reserved|
|TO|reserved|non-reserved|reserved|
|TOUCH|non-reserved|non-reserved|non-reserved|
|TRAILING|reserved|non-reserved|reserved|
Expand Down Expand Up @@ -380,3 +381,4 @@ Below is a list of all the keywords in Spark SQL.
|WINDOW|non-reserved|non-reserved|reserved|
|WITH|reserved|non-reserved|reserved|
|YEAR|reserved|non-reserved|reserved|
|ZONE|non-reserved|non-reserved|non-reserved|
67 changes: 67 additions & 0 deletions docs/sql-ref-syntax-aux-conf-mgmt-set-timezone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
layout: global
title: SET TIME ZONE
displayTitle: SET TIME ZONE
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---

### Description

The `SET TIME ZONE` command sets the current default time zone(`spark.sql.session.timeZone`) displacement for the `SparkSession`.

@cloud-fan cloud-fan Jul 15, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SET TIME ZONE command sets the time zone of the current session.


### Syntax

```sql
SET TIME ZONE LOCAL
SET TIME ZONE 'timezone_value'
SET TIME ZONE INTERVAL interval_literal
```

### Parameters

* **LOCAL**

Respectively set the time zone the one specified in environment variable `TZ`, or to the operating system time zone if `TZ` is undefined.

@cloud-fan cloud-fan Jul 15, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set the time zone "to" the one specified

@cloud-fan cloud-fan Jul 15, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

operating system -> system? We actually use the JVM timezone which can be different from the operating system,

@yaooqinn yaooqinn Jul 16, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, it prefers user.timezone

@huaxingao huaxingao Jul 15, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Respectively set -> Sets?


* **timezone_value**

The ID of session local timezone in the format of either region-based zone IDs or zone offsets. Region IDs must have the form 'area/city', such as 'America/Los_Angeles'. Zone offsets must be in the format '`(+|-)HH`', '`(+|-)HH:mm`' or '`(+|-)HH:mm:ss`', e.g '-08', '+01:00' or '-13:33:33'. Also, 'UTC' and 'Z' are supported as aliases of '+00:00'. Other short names are not recommended to use because they can be ambiguous.

* **interval_literal**

The [interval literal](sql-ref-literals.html#interval-literal) represents the displacement of time zone to the 'UTC'. It must be in the range of [-18, 18] hours and max to second precision, e.g. `INTERVAL 2 HOURS 30 MINITUES` or `INTERVAL '15:40:32' HOUR TO SECOND`.

@huaxingao huaxingao Jul 15, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess something like this?
The interval literal represents the difference between the (system?) time zone and UTC.


### Examples

```sql
-- Set time zone to the system default.
SET TIME ZONE LOCAL;

-- Set time zone to the region-based zone ID.
SET TIME ZONE 'America/Los_Angeles'

@huaxingao huaxingao Jul 15, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ; in the end?


-- Set time zone to the Zone offset.
SET TIME ZONE '+08:00';

-- Set time zone with intervals.
SET TIME ZONE INTERVAL 1 HOUR 30 MINUTES;
SET TIME ZONE INTERVAL '08:30:00' HOUR TO SECOND;
```

Comment thread
cloud-fan marked this conversation as resolved.
### Related Statements

* [SET](sql-ref-syntax-aux-conf-mgmt-set.html)
1 change: 1 addition & 0 deletions docs/sql-ref-syntax-aux-conf-mgmt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ license: |

* [SET](sql-ref-syntax-aux-conf-mgmt-set.html)
* [RESET](sql-ref-syntax-aux-conf-mgmt-reset.html)
* [SET TIME ZONE](sql-ref-syntax-aux-conf-mgmt-set-timezone.html)
8 changes: 8 additions & 0 deletions sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,9 @@ statement
| MSCK REPAIR TABLE multipartIdentifier #repairTable
| op=(ADD | LIST) identifier (STRING | .*?) #manageResource
| SET ROLE .*? #failNativeCommand
| SET TIME ZONE interval #setTimeZone
| SET TIME ZONE timezone=(STRING | LOCAL) #setTimeZone
| SET TIME ZONE .*? #setTimeZone

@cloud-fan cloud-fan Jul 13, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we add this only for better parser message?

@yaooqinn yaooqinn Jul 13, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is used to fail invalid set time zone syntax explicitly, cuz' now we support

spark-sql (default)> set time zone abcd;
key	value
time zone abcd	<undefined>

@gatorsmile gatorsmile Aug 16, 2020 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It sounds like none of the systems are following ANSI SQL syntax completely.

@cloud-fan cloud-fan Aug 17, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are close to the DB2 syntax, except that we support interval and LOCAL, and we don't allow the optional SESSION keyword.

| SET .*? #setConfiguration
| RESET #resetConfiguration
| unsupportedHiveNativeCommands .*? #failNativeCommand
Expand Down Expand Up @@ -1190,6 +1193,7 @@ ansiNonReserved
| VIEW
| VIEWS
| WINDOW
| ZONE
//--ANSI-NON-RESERVED-END
;

Expand Down Expand Up @@ -1431,6 +1435,7 @@ nonReserved
| TEMPORARY
| TERMINATED
| THEN
| TIME
| TO
| TOUCH
| TRAILING
Expand Down Expand Up @@ -1459,6 +1464,7 @@ nonReserved
| WINDOW
| WITH
| YEAR
| ZONE
;

// NOTE: If you add a new token in the list below, you should update the list of keywords
Expand Down Expand Up @@ -1690,6 +1696,7 @@ TBLPROPERTIES: 'TBLPROPERTIES';
TEMPORARY: 'TEMPORARY' | 'TEMP';
TERMINATED: 'TERMINATED';
THEN: 'THEN';
TIME: 'TIME';
TO: 'TO';
TOUCH: 'TOUCH';
TRAILING: 'TRAILING';
Expand Down Expand Up @@ -1720,6 +1727,7 @@ WHERE: 'WHERE';
WINDOW: 'WINDOW';
WITH: 'WITH';
YEAR: 'YEAR';
ZONE: 'ZONE';
//--SPARK-KEYWORD-LIST-END
//============================
// End of the keywords list
Expand Down
11 changes: 9 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2089,14 +2089,21 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
* - from-to unit, for instance: interval '1-2' year to month.
*/
override def visitInterval(ctx: IntervalContext): Literal = withOrigin(ctx) {
Literal(parseIntervalLiteral(ctx), CalendarIntervalType)
}

/**
* Create a [[CalendarInterval]] object
*/
protected def parseIntervalLiteral(ctx: IntervalContext): CalendarInterval = withOrigin(ctx) {
if (ctx.errorCapturingMultiUnitsInterval != null) {
val innerCtx = ctx.errorCapturingMultiUnitsInterval
if (innerCtx.unitToUnitInterval != null) {
throw new ParseException(
"Can only have a single from-to unit in the interval literal syntax",
innerCtx.unitToUnitInterval)
}
Literal(visitMultiUnitsInterval(innerCtx.multiUnitsInterval), CalendarIntervalType)
visitMultiUnitsInterval(innerCtx.multiUnitsInterval)
} else if (ctx.errorCapturingUnitToUnitInterval != null) {
val innerCtx = ctx.errorCapturingUnitToUnitInterval
if (innerCtx.error1 != null || innerCtx.error2 != null) {
Expand All @@ -2105,7 +2112,7 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
"Can only have a single from-to unit in the interval literal syntax",
errorCtx)
}
Literal(visitUnitToUnitInterval(innerCtx.body), CalendarIntervalType)
visitUnitToUnitInterval(innerCtx.body)
} else {
throw new ParseException("at least one time unit should be given for interval literal", ctx)
}
Expand Down
6 changes: 3 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1723,9 +1723,9 @@ object SQLConf {
val SESSION_LOCAL_TIMEZONE = buildConf("spark.sql.session.timeZone")
.doc("The ID of session local timezone in the format of either region-based zone IDs or " +
"zone offsets. Region IDs must have the form 'area/city', such as 'America/Los_Angeles'. " +
"Zone offsets must be in the format '(+|-)HH:mm', for example '-08:00' or '+01:00'. " +
"Also 'UTC' and 'Z' are supported as aliases of '+00:00'. Other short names are not " +
"recommended to use because they can be ambiguous.")
"Zone offsets must be in the format '(+|-)HH', '(+|-)HH:mm' or '(+|-)HH:mm:ss', e.g '-08', " +
"'+01:00' or '-13:33:33'. Also 'UTC' and 'Z' are supported as aliases of '+00:00'. Other " +
"short names are not recommended to use because they can be ambiguous.")
.version("2.2.0")
.stringConf
.checkValue(isValidTimezone, s"Cannot resolve the given timezone with" +
Expand Down
39 changes: 38 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@

package org.apache.spark.sql.execution

import java.util.Locale
import java.time.ZoneOffset
import java.util.{Locale, TimeZone}
import javax.ws.rs.core.UriBuilder

import scala.collection.JavaConverters._
Expand All @@ -32,6 +33,7 @@ import org.apache.spark.sql.catalyst.expressions.Expression
import org.apache.spark.sql.catalyst.parser._
import org.apache.spark.sql.catalyst.parser.SqlBaseParser._
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.util.DateTimeConstants
import org.apache.spark.sql.execution.command._
import org.apache.spark.sql.execution.datasources._
import org.apache.spark.sql.internal.{HiveSerDe, SQLConf, VariableSubstitution}
Expand Down Expand Up @@ -90,6 +92,41 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf) {
ResetCommand
}

/**
* Create a [[SetCommand]] logical plan to set [[SQLConf.SESSION_LOCAL_TIMEZONE]]
* Example SQL :
* {{{
* SET TIME ZONE LOCAL;
* SET TIME ZONE 'Asia/Shanghai';
* SET TIME ZONE INTERVAL 10 HOURS;
* }}}
*/
override def visitSetTimeZone(ctx: SetTimeZoneContext): LogicalPlan = withOrigin(ctx) {
val key = SQLConf.SESSION_LOCAL_TIMEZONE.key
if (ctx.interval != null) {
val interval = parseIntervalLiteral(ctx.interval)
if (interval.months != 0 || interval.days != 0 ||
math.abs(interval.microseconds) > 18 * DateTimeConstants.MICROS_PER_HOUR ||
interval.microseconds % DateTimeConstants.MICROS_PER_SECOND != 0) {
throw new ParseException("The interval value must be in the range of [-18, +18] hours" +
" with second precision",
ctx.interval())
} else {
val seconds = (interval.microseconds / DateTimeConstants.MICROS_PER_SECOND).toInt
SetCommand(Some(key -> Some(ZoneOffset.ofTotalSeconds(seconds).toString)))
}
} else if (ctx.timezone != null) {
ctx.timezone.getType match {
case SqlBaseParser.LOCAL =>
SetCommand(Some(key -> Some(TimeZone.getDefault.getID)))
Comment thread
cloud-fan marked this conversation as resolved.
Comment thread
cloud-fan marked this conversation as resolved.
case _ =>
SetCommand(Some(key -> Some(string(ctx.STRING))))
}
} else {
throw new ParseException("Invalid time zone displacement value", ctx)
}
}

/**
* Create a [[RefreshResource]] logical plan.
*/
Expand Down
15 changes: 15 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/timezone.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
-- valid time zones
SET TIME ZONE 'Asia/Hong_Kong';
SET TIME ZONE 'GMT+1';
SET TIME ZONE INTERVAL 10 HOURS;
SET TIME ZONE INTERVAL '15:40:32' HOUR TO SECOND;
SET TIME ZONE LOCAL;

-- invalid time zone
SET TIME ZONE;
SET TIME ZONE 'invalid/zone';
SET TIME ZONE INTERVAL 3 DAYS;
SET TIME ZONE INTERVAL 24 HOURS;
SET TIME ZONE INTERVAL '19:40:32' HOUR TO SECOND;
SET TIME ZONE INTERVAL 10 HOURS 'GMT+1';
SET TIME ZONE INTERVAL 10 HOURS 1 MILLISECOND;
135 changes: 135 additions & 0 deletions sql/core/src/test/resources/sql-tests/results/timezone.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 12


-- !query
SET TIME ZONE 'Asia/Hong_Kong'
-- !query schema
struct<key:string,value:string>
-- !query output
spark.sql.session.timeZone Asia/Hong_Kong

@cloud-fan cloud-fan Jul 13, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a weird output for SET TIME ZONE 'Asia/Hong_Kong'. Shall we add a Project node on the SetCommand to give better output?

@yaooqinn yaooqinn Jul 13, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do we expect as output? only keep the value Asia/Hong_Kong here?

@cloud-fan cloud-fan Jul 13, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any reference?

@yaooqinn yaooqinn Jul 13, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

postgres=# SET TIME ZONE 'Asia/Hong_kong';
SET



-- !query
SET TIME ZONE 'GMT+1'
-- !query schema
struct<key:string,value:string>
-- !query output
spark.sql.session.timeZone GMT+1


-- !query
SET TIME ZONE INTERVAL 10 HOURS
-- !query schema
struct<key:string,value:string>
-- !query output
spark.sql.session.timeZone +10:00


-- !query
SET TIME ZONE INTERVAL '15:40:32' HOUR TO SECOND
-- !query schema
struct<key:string,value:string>
-- !query output
spark.sql.session.timeZone +15:40:32


-- !query
SET TIME ZONE LOCAL
-- !query schema
struct<key:string,value:string>
-- !query output
spark.sql.session.timeZone America/Los_Angeles


-- !query
SET TIME ZONE
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

Invalid time zone displacement value(line 1, pos 0)

== SQL ==
SET TIME ZONE
^^^


-- !query
SET TIME ZONE 'invalid/zone'
-- !query schema
struct<>
-- !query output
java.lang.IllegalArgumentException
Cannot resolve the given timezone with ZoneId.of(_, ZoneId.SHORT_IDS)


-- !query
SET TIME ZONE INTERVAL 3 DAYS
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

The interval value must be in the range of [-18, +18] hours with second precision(line 1, pos 14)

== SQL ==
SET TIME ZONE INTERVAL 3 DAYS
--------------^^^


-- !query
SET TIME ZONE INTERVAL 24 HOURS
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

The interval value must be in the range of [-18, +18] hours with second precision(line 1, pos 14)

== SQL ==
SET TIME ZONE INTERVAL 24 HOURS
--------------^^^


-- !query
SET TIME ZONE INTERVAL '19:40:32' HOUR TO SECOND
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

The interval value must be in the range of [-18, +18] hours with second precision(line 1, pos 14)

== SQL ==
SET TIME ZONE INTERVAL '19:40:32' HOUR TO SECOND
--------------^^^


-- !query
SET TIME ZONE INTERVAL 10 HOURS 'GMT+1'
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

Invalid time zone displacement value(line 1, pos 0)

== SQL ==
SET TIME ZONE INTERVAL 10 HOURS 'GMT+1'
^^^


-- !query
SET TIME ZONE INTERVAL 10 HOURS 1 MILLISECOND
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

The interval value must be in the range of [-18, +18] hours with second precision(line 1, pos 14)

== SQL ==
SET TIME ZONE INTERVAL 10 HOURS 1 MILLISECOND
--------------^^^
Loading