Spark: IN clause on system function is not pushed down#9192
Spark: IN clause on system function is not pushed down#9192tmnd1991 wants to merge 1 commit intoapache:mainfrom
Conversation
tmnd1991
force-pushed
the
feature/9191
branch
from
fe35fa3 to
43cc632
Compare
|
Can I have a feedback on this one @ConeyLiu, too? Thanks a lot |
...-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala
Outdated
Show resolved
Hide resolved
|
@tmnd1991 thanks for your contribution. I think this is a valid fix. |
|
cc @nastra @dramaticlly @advancedxy for review |
advancedxy
left a comment
advancedxy
left a comment
There was a problem hiding this comment.
Left two minor comments. Overall, it's in good shape.
...-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala
Outdated
Show resolved
Hide resolved
...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java
Show resolved
Hide resolved
|
Thanks for the review, I addressed your concerns. If I get green light I proceed to copy-paste over 3.4 |
wypoon
left a comment
wypoon
left a comment
There was a problem hiding this comment.
The fix seems to make sense.
Just one comment about a test and some nits (also, please note that the Iceberg project doesn't use wildcard imports, so you should put back all the explicit imports plus additional ones needed).
...-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceStaticInvoke.scala
Outdated
Show resolved
Hide resolved
| if canReplace(systemFunction) && values.forall(_.foldable) => | ||
| in.copy(value = replaceStaticInvoke(systemFunction)) | ||
|
|
||
| case in@InSet(systemFunction: StaticInvoke, _) if canReplace(systemFunction) => |
There was a problem hiding this comment.
Nit: in @ InSet to be consistent with the style.
...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java
Outdated
Show resolved
Hide resolved
tmnd1991
force-pushed
the
feature/9191
branch
from
2a99af7 to
e0075c3
Compare
|
Thanks @wypoon for the review, I addressed your concerns |
| filter.copy(condition = newCondition) | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Unnecessary (and undesirable) whitespace change.
| } | ||
|
|
||
| private void testBucketLongFunctionIsNotReplacedWhenArgumentsAreNotLiterals() { | ||
| List<Integer> range = IntStream.range(0, 3).boxed().collect(Collectors.toList()); |
There was a problem hiding this comment.
range is not used. (I pointed this out before.)
| filter.copy(condition = newCondition) | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Unnecessary whitespace change.
| } | ||
|
|
||
| private void testBucketLongFunctionIsNotReplacedWhenArgumentsAreNotLiterals() { | ||
| List<Integer> range = IntStream.range(0, 3).boxed().collect(Collectors.toList()); |
|
@rdblue @szehon-ho @RussellSpitzer can you please review this? Also, can you please enable the CI for this PR? |
|
any chance to have this reviewed? |
|
@aokolnychyi could you take a look at this please? |
|
just FYI, looks like there's a related PR that @aokolnychyi opened for this: #9873 |
thanks, watching that 🙏 |
reformat Add impl and fix test Work also in the IN_SET case Add missing `canReplace` check in `In` case Refactor tests Apply suggestions Revert Add tests when replace should not happen Fix code style issues Fix code style issues Copy over 3.4 Fixed unused variables and whitespace changes
|
@aokolnychyi I see you fixed part of this in #9873 but Spark 3.4 looks stil bugged on main. Do you have any time to give me a feedback on this? |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
6 participants
PR to verify bug reported in issue #9191.
With some guidance I'm open to work on the fix too.