Spark 3.3 - Support bucket in FunctionCatalog#5513

Merged

rdblue merged 19 commits intoapache:masterfrom

kbendick:kb-add-spark-bucket-function-squash

Aug 18, 2022

Commits on Aug 15, 2022

Spark - Add Bucket Function for FunctionCatalog
kbendick
committed
Rename BucketHashUtil to BucketUtil and name methods hash instead
kbendick
committed
Make order of functions consistent for all subclasses
kbendick
committed
Give a concrete example with correct returned value for BucketFunction javadoc
kbendick
committed
Add note about missing UUID type
kbendick
committed
Remove redundant comments
kbendick
committed
Update BucketUtil to name all functions hash and use primitive types wherever possible
kbendick
committed
Remove Spark BucketFloat given that bucketing by float is disallowed by the spec
kbendick
committed
Fix hashBytes for slice with non-zero offset and add test
kbendick
committed
Remove comment regarding UUID type as that's not supported by spark
kbendick
committed
Update error message for invalid bucket column type
kbendick
committed
Remove test for magic function being invoked when using tinyint or shortint for number of buckets - already covered by test that checks that those can be used and implicitly by the check that magic…
kbendick
committed
Update canonical name of decimal bucket function as precision and scale only matter for the input type binding
kbendick
committed
Add static apply function in base class for handling post-hash logic
kbendick
committed
Use static base class apply method for the post-hash bucketing logic in all subclasses
kbendick
committed
Pass in sqlType for BucketDecimal to be consistent with the others, as precision and scale are not important to the hash function itself
kbendick
committed
Pass in type directly to all Bucket function constructors except BucketInteger, where we explicitly want Spark to cast short and byte to integer type
kbendick
committed
Add all examples from the spec to the tests and check just the hashed value (not the bucketed value)
kbendick
committed
Checkstyle
kbendick
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Spark 3.3 - Support bucket in FunctionCatalog#5513

Spark 3.3 - Support bucket in FunctionCatalog#5513
rdblue merged 19 commits intoapache:masterfrom
kbendick:kb-add-spark-bucket-function-squash

Commits on Aug 15, 2022

Spark - Add Bucket Function for FunctionCatalog

Rename BucketHashUtil to BucketUtil and name methods hash instead

Make order of functions consistent for all subclasses

Give a concrete example with correct returned value for BucketFunction javadoc

Add note about missing UUID type

Remove redundant comments

Update BucketUtil to name all functions hash and use primitive types wherever possible

Remove Spark BucketFloat given that bucketing by float is disallowed by the spec

Fix hashBytes for slice with non-zero offset and add test

Remove comment regarding UUID type as that's not supported by spark

Update error message for invalid bucket column type

Remove test for magic function being invoked when using tinyint or shortint for number of buckets - already covered by test that checks that those can be used and implicitly by the check that magic…

Update canonical name of decimal bucket function as precision and scale only matter for the input type binding

Add static apply function in base class for handling post-hash logic

Use static base class apply method for the post-hash bucketing logic in all subclasses

Pass in sqlType for BucketDecimal to be consistent with the others, as precision and scale are not important to the hash function itself

Pass in type directly to all Bucket function constructors except BucketInteger, where we explicitly want Spark to cast short and byte to integer type

Add all examples from the spec to the tests and check just the hashed value (not the bucketed value)

Checkstyle