Reading FIXED_LEN_BYTE_ARRAY columns with nulls is inefficient #6296
Labels
enhancement
Any new improvement worthy of a entry in the changelog
parquet
Changes to the parquet crate
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
When reading a Parquet file with FIXED_LEN_BYTE_ARRAY columns with nulls present one necessary operation is moving the fixed-length data into the correct location within the output buffer to take into account null slots. This is handled by the
pad_nulls
function in theValuesBuffer
trait. The inner loop of this functionworks well when the fixed width is low (
<= 4
), but for larger widths this loop is quite inefficient.Describe the solution you'd like
Rewriting the inner loop for longer fixed-size arrays can speed this operation up considerably. In particular, by copying slices of the buffer to another location in the buffer, the compiler can vectorize the move, e.g.
Describe alternatives you've considered
I tried
Vec::copy_within
but it was slower than the vectorized copy.Additional context
The text was updated successfully, but these errors were encountered: