Skip to content

Conversation

@MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Apr 10, 2021

What changes were proposed in this pull request?

  1. Extend IntervalUtils methods: toYearMonthIntervalString and toDayTimeIntervalString to support formatting of year-month/day-time intervals in Hive style. The methods get new parameter style which can have to values; HIVE_STYLE and ANSI_STYLE.
  2. Invoke toYearMonthIntervalString and toDayTimeIntervalString from the Cast expression with the style parameter is set to ANSI_STYLE.
  3. Invoke toYearMonthIntervalString and toDayTimeIntervalString from HiveResult with style is set to HIVE_STYLE.

Why are the changes needed?

The spark-sql shell formats its output in Hive style by using HiveResult.hiveResultString(). The changes are needed to match Hive behavior. For instance,

Hive:

0: jdbc:hive2://localhost:10000/default> select timestamp'2021-01-01 01:02:03.000001' - date'2020-12-31';
+-----------------------+
|          _c0          |
+-----------------------+
| 1 01:02:03.000001000  |
+-----------------------+

Spark before the changes:

spark-sql> select timestamp'2021-01-01 01:02:03.000001' - date'2020-12-31';
INTERVAL '1 01:02:03.000001' DAY TO SECOND

Also this should unblock #32099 which enables *.sql tests in SQLQueryTestSuite.

Does this PR introduce any user-facing change?

Yes. After the changes:

spark-sql> select timestamp'2021-01-01 01:02:03.000001' - date'2020-12-31';
1 01:02:03.000001000

How was this patch tested?

  1. Added new tests to IntervalUtilsSuite:
$  build/sbt "test:testOnly *IntervalUtilsSuite"
  1. Modified existing tests in HiveResultSuite:
$  build/sbt -Phive-2.3 -Phive-thriftserver "testOnly *HiveResultSuite"
  1. By running cast tests:
$ build/sbt "testOnly *CastSuite*"

@SparkQA
Copy link

SparkQA commented Apr 10, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41744/

@SparkQA
Copy link

SparkQA commented Apr 10, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41744/

@SparkQA
Copy link

SparkQA commented Apr 10, 2021

Test build #137165 has finished for PR 32120 at commit 3daaf3d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks fine.

@MaxGekk
Copy link
Member Author

MaxGekk commented Apr 11, 2021

Merging to master. Thank you @yaooqinn and @HyukjinKwon for your review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants