[SPARK-34357][SQL] Map JDBC SQL TIME type to TimestampType with time portion fixed regardless of timezone

[saikocat]

What changes were proposed in this pull request?

Due to user-experience (confusing to Spark users - java.sql.Time using milliseconds vs Spark using microseconds; and user losing useful functions like hour(), minute(), etc on the column), we have decided to revert back to use TimestampType but this time we will enforce the hour to be consistently across system timezone (via offset manipulation) and date part fixed to zero epoch.

Full Discussion with Wenchen Fan Wenchen Fan regarding this ticket is here #30902 (comment)

Why are the changes needed?

Revert and improvement to sql.Time handling

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit tests and integration tests

[saikocat]

CC: @cloud-fan @sarutak @skestle - regarding the updated changes to respect vendor db integrations

[sarutak]

ok to test.

[sarutak]

@saikocat Have you confirmed all the integration tests like PostgresIntegrationSuite pass?
Jenkins and GA don't run them so we need to confirm by ourselves.

[saikocat]

@saikocat Have you confirmed all the integration tests like PostgresIntegrationSuite pass?
Jenkins and GA don't run them so we need to confirm by ourselves.

Yup, I learned from my previous mistake. All IT all passed.

Edit: manual loaded the docker image for mssql works

[sarutak]

Yup, I learned from my previous mistake. All IT all passed.

Edit: manual loaded the docker image for mssql works

Thanks. Anyway, I'll run all the integration tests before this change merged.

[cloud-fan]

In most cases, session timezone should be the same as JVM default timezone, but ideally we should use session timezone for datetime operations: SQLConf.get.sessionLocalTimeZone

[cloud-fan]

Seems we can just do DateTimeUtils.toUTCTime(localTimeMicro, SQLConf.get.sessionLocalTimeZone)

[saikocat]

Superb! This is really more elegant. Thanks for this suggestion!

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39458/

[sarutak]

Confirmed all the JDBC integration tests pass for the latest commit 0b4a344.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39458/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39461/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39461/

[SparkQA]

Test build #134872 has finished for PR 31473 at commit 15e980f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• assert(types(1).equals(\"class java.sql.Timestamp\"))
• assert(types(5).equals(\"class java.sql.Timestamp\"))
• assert(types(1).equals(\"class java.sql.Timestamp\"))
• assert(types(2).equals(\"class java.sql.Timestamp\"))

[cloud-fan]

thanks, merging to master!

[SparkQA]

Test build #134878 has finished for PR 31473 at commit 0b4a344.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.