[SPARK-19758][SQL] Resolving timezone aware expressions with time zone when resolving inline table #17114
Conversation
|
cc @ueshin |
|
Can we put the test case in a sql file? |
|
Ok. I will put it in inline-table.sql. |
|
Test build #73671 has finished for PR 17114 at commit
|
|
Test build #73675 has finished for PR 17114 at commit
|
| Cast(e, targetType) | ||
| } | ||
| castedExpr match { | ||
| case te: TimeZoneAwareExpression => te.withTimeZone(conf.sessionLocalTimeZone).eval() |
There was a problem hiding this comment.
Shouldn't we traverse the entire tree expression to check for time zone aware expressions?
There was a problem hiding this comment.
Or did ResolveTimeZone already process the input expression? In that case just add the timezone to the cast.
There was a problem hiding this comment.
We should. ResolveTimeZone is performed after all resolution rules finish.
| select * from values ("one", 2.0), ("two", 3.0D) as data(a, b); | ||
|
|
||
| -- string to timestamp | ||
| select * from values timestamp('1991-12-06 00:00:00.0') as data(a); |
There was a problem hiding this comment.
Could you put this at the end of the file? That reduces the size of the diff.
hvanhovell
left a comment
hvanhovell
left a comment
There was a problem hiding this comment.
LGTM - pending jenkins
|
Thanks @hvanhovell @rxin |
|
Test build #73844 has finished for PR 17114 at commit
|
|
LGTM - merging to master. Thanks! |
| ResolveAggregateFunctions :: | ||
| TimeWindowing :: | ||
| ResolveInlineTables :: | ||
| ResolveInlineTables(conf) :: |
There was a problem hiding this comment.
is it possible to fix this bug by re-order analyzer rules?
There was a problem hiding this comment.
I tried before. But the resolution rules will add new timezone-aware expressions, so it still needs the rule to resolve timezone-aware expressions after resolution rules.
5 participants
What changes were proposed in this pull request?
When we resolve inline tables in analyzer, we will evaluate the expressions of inline tables.
When it evaluates a
TimeZoneAwareExpressionexpression, an error will happen because theTimeZoneAwareExpressionis not associated with timezone yet.So we need to resolve these
TimeZoneAwareExpressions with time zone when resolving inline tables.How was this patch tested?
Jenkins tests.
Please review http://spark.apache.org/contributing.html before opening a pull request.