Skip to content

[SPARK-31879][SQL] Using GB as default Locale for datetime formatters #28692

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 3 commits into from
Closed

[SPARK-31879][SQL] Using GB as default Locale for datetime formatters #28692

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ab0517d
[SPARK-31879][SQL] Using GB as default Locale for datetime formatters
yaooqinn Jun 1, 2020
3339274
comment
yaooqinn Jun 1, 2020
f94c72b
use formatter
yaooqinn Jun 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,13 @@ class LegacySimpleDateFormatter(pattern: String, locale: Locale) extends LegacyD
object DateFormatter {
import LegacyDateFormats._

val defaultLocale: Locale = Locale.US
/**
* Before Spark 3.0, the first day-of-week is always Monday. Since Spark 3.0, it depends on the
* locale.
* We pick GB as the default locale instead of US, to be compatible with Spark 2.x, as US locale
* uses Sunday as the first day-of-week. See SPARK-31879.
*/
val defaultLocale: Locale = new Locale("en", "GB")

val defaultPattern: String = "yyyy-MM-dd"

Expand Down
8 changes: 7 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,13 @@ object LegacyDateFormats extends Enumeration {
object TimestampFormatter {
import LegacyDateFormats._

val defaultLocale: Locale = Locale.US
/**
* Before Spark 3.0, the first day-of-week is always Monday. Since Spark 3.0, it depends on the
* locale.
* We pick GB as the default locale instead of US, to be compatible with Spark 2.x, as US locale
* uses Sunday as the first day-of-week. See SPARK-31879.
*/
val defaultLocale: Locale = new Locale("en", "GB")

def defaultPattern(): String = s"${DateFormatter.defaultPattern} HH:mm:ss"

Expand Down
3 changes: 3 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/datetime.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,3 +164,6 @@ select from_csv('26/October/2015', 'date Date', map('dateFormat', 'dd/MMMMM/yyyy
select from_unixtime(1, 'yyyyyyyyyyy-MM-dd');
select date_format(timestamp '2018-11-17 13:33:33', 'yyyyyyyyyy-MM-dd HH:mm:ss');
select date_format(date '2018-11-17', 'yyyyyyyyyyy-MM-dd');

-- SPARK-31879: the first day of week
select to_timestamp('2020-01-01', 'YYYY-ww-uu');

@cloud-fan cloud-fan Jun 2, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use formatting in the test? so this doesn't conflict with #28706

10 changes: 9 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 119
-- Number of queries: 120


-- !query
Expand Down Expand Up @@ -1025,3 +1025,11 @@ struct<>
-- !query output
org.apache.spark.SparkUpgradeException
You may get a different result due to the upgrading of Spark 3.0: Fail to recognize 'yyyyyyyyyyy-MM-dd' pattern in the DateTimeFormatter. 1) You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0. 2) You can form a valid datetime pattern with the guide from https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html


-- !query
select to_timestamp('2020-01-01', 'YYYY-ww-uu')
-- !query schema
struct<to_timestamp(2020-01-01, YYYY-ww-uu):timestamp>
-- !query output
2019-12-30 00:00:00
10 changes: 9 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/datetime-legacy.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 119
-- Number of queries: 120


-- !query
Expand Down Expand Up @@ -980,3 +980,11 @@ select date_format(date '2018-11-17', 'yyyyyyyyyyy-MM-dd')
struct<date_format(CAST(DATE '2018-11-17' AS TIMESTAMP), yyyyyyyyyyy-MM-dd):string>
-- !query output
00000002018-11-17


-- !query
select to_timestamp('2020-01-01', 'YYYY-ww-uu')
-- !query schema
struct<to_timestamp(2020-01-01, YYYY-ww-uu):timestamp>
-- !query output
2019-12-30 00:00:00
10 changes: 9 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/datetime.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 119
-- Number of queries: 120


-- !query
Expand Down Expand Up @@ -997,3 +997,11 @@ struct<>
-- !query output
org.apache.spark.SparkUpgradeException
You may get a different result due to the upgrading of Spark 3.0: Fail to recognize 'yyyyyyyyyyy-MM-dd' pattern in the DateTimeFormatter. 1) You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0. 2) You can form a valid datetime pattern with the guide from https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html


-- !query
select to_timestamp('2020-01-01', 'YYYY-ww-uu')
-- !query schema
struct<to_timestamp(2020-01-01, YYYY-ww-uu):timestamp>
-- !query output
2019-12-30 00:00:00