Spark: Support truncate in FunctionCatalog#5431

Merged

aokolnychyi merged 27 commits intoapache:masterfrom

kbendick:kb-add-spark-truncate-function

Aug 12, 2022

Commits on Aug 11, 2022

Spark - Implement Truncate Partition Transform Function as a Spark V2 Function for Function Catalog
kbendick
committed
Use built-in code-point aware UTF8String::substring method
kbendick
committed
Remove varchar type and char type as Spark will convert them to Strings anyway
kbendick
committed
Remove width from the canonical names - leaving out the int width field entirely from the string as it's in the documentation anyhow
kbendick
committed
Add test case for truncating empty string
kbendick
committed
Make comment cleaner and rename width function
kbendick
committed
Change order of arguments in truncateBinaryUnsafe and phrasing in JavaDoc
kbendick
committed
Remove changes to UnicodeUtil - consider addressing efficiency in Truncate in another PR
kbendick
committed
Use sql types for invalid truncation column type error message and add test
kbendick
committed
Update phrasing for invalid truncation width type to use SQL types
kbendick
committed
Remove width function so that produceResult and magic functions behave the same with respect to null
kbendick
committed
Update javadoc
kbendick
committed
Skip isNullAt check for non-primitive types that already check for null input values in invoke
kbendick
committed
Add isNull check in all produceResults calls to avoid calling getXXXX on null on the special accesssors of certain row types that might not tolerate it
kbendick
committed
Add missing type tests for truncation
kbendick
committed
Use WIDTH_ORDINAL and VALUE_ORDINAL constants
kbendick
committed
Remove odd ternary expression
kbendick
committed
Add line between pulling types out of schema and validating the width type
kbendick
committed
Ensure string truncation has two byte, three byte, and four byte unicode codepoints in tests
kbendick
committed
Use constant set for width types as none have variable length and remove separate function for validation
kbendick
committed
Use inputType.size
kbendick
committed
Use assertNull where possible
kbendick
committed
Simplify exception message for wrong number of inputs and add test for fewer than 2 and more than 2 arguments
kbendick
committed
Use value consistently for the truncation value field
kbendick
committed
Ensure truncation width is non-null - need to validate this will be accepted in storage partitioned joins impl
kbendick
committed
Add check that dataframe with nullable integer will be rejected for use with truncate
kbendick
committed
Remove assertion that width field is non-null and associated tests per PR discussion
kbendick
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Spark: Support truncate in FunctionCatalog#5431

Spark: Support truncate in FunctionCatalog#5431
aokolnychyi merged 27 commits intoapache:masterfrom
kbendick:kb-add-spark-truncate-function

Commits on Aug 11, 2022

Spark - Implement Truncate Partition Transform Function as a Spark V2 Function for Function Catalog

Use built-in code-point aware UTF8String::substring method

Remove varchar type and char type as Spark will convert them to Strings anyway

Remove width from the canonical names - leaving out the int width field entirely from the string as it's in the documentation anyhow

Add test case for truncating empty string

Make comment cleaner and rename width function

Change order of arguments in truncateBinaryUnsafe and phrasing in JavaDoc

Remove changes to UnicodeUtil - consider addressing efficiency in Truncate in another PR

Use sql types for invalid truncation column type error message and add test

Update phrasing for invalid truncation width type to use SQL types

Remove width function so that produceResult and magic functions behave the same with respect to null

Update javadoc

Skip isNullAt check for non-primitive types that already check for null input values in invoke

Add isNull check in all produceResults calls to avoid calling getXXXX on null on the special accesssors of certain row types that might not tolerate it

Add missing type tests for truncation

Use WIDTH_ORDINAL and VALUE_ORDINAL constants

Remove odd ternary expression

Add line between pulling types out of schema and validating the width type

Ensure string truncation has two byte, three byte, and four byte unicode codepoints in tests

Use constant set for width types as none have variable length and remove separate function for validation

Use inputType.size

Use assertNull where possible

Simplify exception message for wrong number of inputs and add test for fewer than 2 and more than 2 arguments

Use value consistently for the truncation value field

Ensure truncation width is non-null - need to validate this will be accepted in storage partitioned joins impl

Add check that dataframe with nullable integer will be rejected for use with truncate

Remove assertion that width field is non-null and associated tests per PR discussion