[SPARK-25496][SQL][followup] avoid using to_utc_timestamp #24319
Conversation
| val inputData = MemoryStream[Long] | ||
| val aggregated = | ||
| inputData.toDF() | ||
| .select(($"value" * DateTimeUtils.SECONDS_PER_DAY).cast("timestamp").as("value")) |
There was a problem hiding this comment.
This is basically the only change right? can you remove tz too?
Seems OK, especially if pulling timezones out of the equation here.
There was a problem hiding this comment.
there is one more: I replaced current_date with current_timestamp().cast("date").
current_date returns date in UTC. We can't compare it with the value of casting timestamp column to date. There is a timezone shift when casting between date and timestamp.
|
Test build #104391 has finished for PR 24319 at commit
|
|
Test build #104389 has finished for PR 24319 at commit
|
attilapiros
left a comment
attilapiros
left a comment
There was a problem hiding this comment.
LGTM
Tip for the next reviewer: after switching on "Hide whitespace changes" the diff is quite small and easier to review.
|
thanks, merging to master! |
6 participants
What changes were proposed in this pull request?
in #24195 , we deprecate
from/to_utc_timestamp.This PR removes unnecessary use of
to_utc_timestampin the test.How was this patch tested?
test only PR