Use legacySizeOfNull argument to determine the behavior of Spark size function#10100
Use legacySizeOfNull argument to determine the behavior of Spark size function#10100philo-he wants to merge 8 commits intofacebookincubator:mainfrom
Conversation
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox canceled.
|
philo-he
changed the title
philo-he
changed the title
|
@rui-mo, do you have any comment? |
|
@rui-mo, do you have any other comment? |
rui-mo
left a comment
rui-mo
left a comment
There was a problem hiding this comment.
Thanks. Can we make the title more concise maybe like Use legacySizeOfNull parameter to determine the behavior of Spark size function?
philo-he
changed the title
|
Thanks guys. @rui-mo I forgot to mention, but please add the "ready-to-merge" tag to PRs which are reviewed and ready to be merged. This will alert our oncall. |
pedroerp
added
the
ready-to-merge
|
@pedroerp Thank you for the pointer. I understand. |
|
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Remove use of spark specific query config which is going to be removed in facebookincubator/velox#10100
|
Created a PR in presto repo to remove the use of the query config that is removed in this PR. prestodb/presto#23046 |
@bikramSingh91, thanks for your fix! |
Remove use of spark specific query config which is going to be removed in facebookincubator/velox#10100
|
@bikramSingh91, can this pr be merged? The CI failure is related to network error. |
|
@bikramSingh91, I just rebased this pr. All checks passed. |
|
Friendly ping @bikramSingh91. Thanks! |
|
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@bikramSingh91 merged this pull request in 258db51. |
|
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
… function (facebookincubator#10100) Summary: 1) Spark `size` function's legacySizeOfNull is specified either by other functions (e.g., `array_size` function) or by configuration. However, in the existing implementation, it only depends on the configuration property. This pr removes the configuration property, then just uses the legacySizeOfNull arg passed from framework like Gluten. 2) At Spark's analysis time, `array_size` is replaced with `size` function. And their implementations are same actually. This pr removes duplicate code. Pull Request resolved: facebookincubator#10100 Reviewed By: xiaoxmeng Differential Revision: D58825758 Pulled By: bikramSingh91 fbshipit-source-id: 5e1c9679832dfb6ac7dd15e3c0c1d979d3219b93
5 participants
sizefunction's legacySizeOfNull is specified either by other functions(e.g.,
array_sizefunction) or by configuration. However, in the existingimplementation, it only depends on the configuration property. This pr removes
the configuration property, then just uses the legacySizeOfNull arg passed from
framework like Gluten.
array_sizeis replaced withsizefunction. And theirimplementations are same actually. This pr removes duplicate code.