[SPARK-27199][SQL] Replace TimeZone by ZoneId in TimestampFormatter API #24141
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #103667 has finished for PR 24141 at commit
|
kiszk
reviewed
Mar 19, 2019
| pattern: String, | ||
| timeZone: TimeZone, | ||
| locale: Locale) extends TimestampFormatter with DateTimeFormatterHelper { | ||
| pattern: String, |
Member
Author
There was a problem hiding this comment.
thanx. Just wondering why scalastyle didn't catch that.
Member
|
Looks reasonable to me. Is there any behavior change? I don't think so, just checking. |
Member
Author
Potentially we could observe a behavoir change in parsing time zone id strings but because we did it twice in |
|
Test build #103675 has started for PR 24141 at commit |
Member
Author
|
jenkins, retest this, please |
|
Test build #103693 has finished for PR 24141 at commit
|
HyukjinKwon
approved these changes
Mar 20, 2019
Member
|
Merged to master. |
cloud-fan
pushed a commit
that referenced
this pull request
Apr 12, 2019
… and FromUnixTime ## What changes were proposed in this pull request? SPARK-27199 introduced the use of `ZoneId` instead of `TimeZone` in a few date/time expressions. There were 3 occurrences of `ctx.addReferenceObj("zoneId", zoneId)` in that PR, which had a bug because while the `java.time.ZoneId` base type is public, the actual concrete implementation classes are not public, so using the 2-arg version of `CodegenContext.addReferenceObj` would incorrectly generate code that reference non-public types (`java.time.ZoneRegion`, to be specific). The 3-arg version should be used, with the class name of the referenced object explicitly specified to the public base type. One of such occurrences was caught in testing in the main PR of SPARK-27199 (#24141), for `DateFormatClass`. But the other 2 occurrences slipped through because there were no test cases that covered them. Example of this bug in the current Apache Spark master, in a Spark Shell: ``` scala> Seq(("2016-04-08", "yyyy-MM-dd")).toDF("s", "f").repartition(1).selectExpr("to_unix_timestamp(s, f)").show ... java.lang.IllegalAccessError: tried to access class java.time.ZoneRegion from class org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1 ``` This PR fixes the codegen issues and adds the corresponding unit tests. ## How was this patch tested? Enhanced tests in `DateExpressionsSuite` for `to_unix_timestamp` and `from_unixtime`. Closes #24352 from rednaxelafx/fix-spark-27199. Authored-by: Kris Mok <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In the PR, I propose to use
ZoneIdinstead ofTimeZonein:applyandgetFractionFormattermethods of theTimestampFormatterobject,TimestampFormattertrait likeFractionTimestampFormatter.The reason of the changes is to avoid unnecessary conversion from
TimeZonetoZoneIdbecauseZoneIdis used inTimestampFormatterimplementations internally, and the conversion is performed viaStringwhich is not for free. Also taking into account thatTimeZoneinstances are converted fromStringin some cases, the worse case looks likeString->TimeZone->String->ZoneId. The PR eliminates the unneeded conversions.How was this patch tested?
It was tested by
DateExpressionsSuite,DateTimeUtilsSuiteandTimestampFormatterSuite.