[SPARK-31818][SQL] Fix pushing down filters with java.time.Instant values in ORC
#28636
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@cloud-fan @HyukjinKwon @dongjoon-hyun Please, review this PR. It is similar to #28261 but for timestamps. |
cloud-fan
reviewed
May 25, 2020
| implicit def bigDecimalToLiteral(d: java.math.BigDecimal): Literal = Literal(d) | ||
| implicit def decimalToLiteral(d: Decimal): Literal = Literal(d) | ||
| implicit def timestampToLiteral(t: Timestamp): Literal = Literal(t) | ||
| implicit def timestampToLiteral(t: Instant): Literal = Literal(t) |
Contributor
There was a problem hiding this comment.
shall we call it instantToLiteral, to be consistent with localDateToLiteral?
cloud-fan
approved these changes
May 25, 2020
|
Test build #123084 has finished for PR 28636 at commit
|
MaxGekk
changed the title
java.time.Instant values in ORCjava.time.Instant values in ORC
Member
Author
|
jenkins, retest this, please |
dongjoon-hyun
approved these changes
May 25, 2020
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you for testing both hive profile, @MaxGekk .
|
Test build #123086 has finished for PR 28636 at commit
|
| implicit def bigDecimalToLiteral(d: java.math.BigDecimal): Literal = Literal(d) | ||
| implicit def decimalToLiteral(d: Decimal): Literal = Literal(d) | ||
| implicit def timestampToLiteral(t: Timestamp): Literal = Literal(t) | ||
| implicit def instantToLiteral(t: Instant): Literal = Literal(t) |
There was a problem hiding this comment.
variable name: i instead of t if want follow convention around
|
Test build #123093 has finished for PR 28636 at commit
|
|
Test build #123098 has finished for PR 28636 at commit
|
dongjoon-hyun
pushed a commit
that referenced
this pull request
May 26, 2020
…values in ORC ### What changes were proposed in this pull request? Convert `java.time.Instant` to `java.sql.Timestamp` in pushed down filters to ORC datasource when Java 8 time API enabled. ### Why are the changes needed? The changes fix the exception raised while pushing date filters when `spark.sql.datetime.java8API.enabled` is set to `true`: ``` java.lang.IllegalArgumentException: Wrong value class java.time.Instant for TIMESTAMP.EQUALS leaf at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.checkLiteralType(SearchArgumentImpl.java:192) at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.<init>(SearchArgumentImpl.java:75) ``` ### Does this PR introduce any user-facing change? Yes ### How was this patch tested? Added tests to `OrcFilterSuite`. Closes #28636 from MaxGekk/orc-timestamp-filter-pushdown. Authored-by: Max Gekk <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 6c80ebb) Signed-off-by: Dongjoon Hyun <[email protected]>
Member
|
Merged to master/3.0. Thank you, @MaxGekk , @cloud-fan , @seanli-rallyhealth . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Convert
java.time.Instanttojava.sql.Timestampin pushed down filters to ORC datasource when Java 8 time API enabled.Why are the changes needed?
The changes fix the exception raised while pushing date filters when
spark.sql.datetime.java8API.enabledis set totrue:Does this PR introduce any user-facing change?
Yes
How was this patch tested?
Added tests to
OrcFilterSuite.