perf: improve calculating length performance for GenericByteArray in row conversion
#9078
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n row conversion
Member
Author
|
run benchmark row_format |
|
🤖 |
This was referenced Dec 31, 2025
|
🤖: Benchmark completed Details
|
Contributor
|
run benchmark row_format |
|
🤖 |
|
🤖: Benchmark completed Details
|
alamb
added a commit
that referenced
this pull request
Jan 13, 2026
…ow conversion (#9080) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? 1. Instead of iterating over the bytes and getting the length from the byte slice, we use the offsets directly, this is faster as it saves us going to the buffer 2. Added new API for `GenericByteViewArray` (explained below) # Are these changes tested? Yes # Are there any user-facing changes? Yes, added `lengths` function to `GenericByteViewArray` to get an iterator over the lengths of the items in the array ----- Related to: - #9078 - #9079 --------- Co-authored-by: Andrew Lamb <[email protected]>
# Conflicts: # arrow-row/src/lib.rs
Member
Author
|
@alamb can you please review and hopefully merge if approved with no comments |
alamb
approved these changes
Jan 13, 2026
| .lengths() | ||
| .zip(nulls.iter()) | ||
| .map(|(length, is_valid)| if is_valid { Some(length) } else { None }) | ||
| .map(variable::padded_length), |
Contributor
There was a problem hiding this comment.
I verified that variable::encoded_len is calling padded_len
arrow-row/src/lib.rs
Outdated
| array | ||
| .offsets() | ||
| .lengths() | ||
| .map(Some) |
Contributor
There was a problem hiding this comment.
You could potentially avoid another branch by making a version of padded_length that takes usize instead of Option<usize>
Member
Author
There was a problem hiding this comment.
I hoped that it would be optimized away, but I see it is not.
fixed
Dandandan
approved these changes
Jan 14, 2026
Contributor
|
Very nice! |
Dandandan
pushed a commit
that referenced
this pull request
Jan 14, 2026
… conversion (#9079) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? 1. Instead of iterating over the rows and getting the length from the byte slice, we use the offsets directly, this 2. Added 3 new APIs for `Rows` (explained below) # Are these changes tested? Yes # Are there any user-facing changes? Yes, added 3 functions to `Rows`: - `row_len` - get the row length at index - `row_len_unchecked` - get the row length at index without bound checks - `lengths` - get iterator over the lengths of the rows ----- Related to: - #9078 - #9080 --------- Co-authored-by: Andrew Lamb <[email protected]>
alamb
reviewed
Jan 14, 2026
| pub fn padded_length(a: Option<usize>) -> usize { | ||
| match a { | ||
| Some(a) if a <= BLOCK_SIZE => 1 + ceil(a, MINI_BLOCK_SIZE) * (MINI_BLOCK_SIZE + 1), | ||
| Some(a) => non_null_padded_length(a), |
Dandandan
pushed a commit
to Dandandan/arrow-rs
that referenced
this pull request
Jan 15, 2026
…ow conversion (apache#9080) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? 1. Instead of iterating over the bytes and getting the length from the byte slice, we use the offsets directly, this is faster as it saves us going to the buffer 2. Added new API for `GenericByteViewArray` (explained below) # Are these changes tested? Yes # Are there any user-facing changes? Yes, added `lengths` function to `GenericByteViewArray` to get an iterator over the lengths of the items in the array ----- Related to: - apache#9078 - apache#9079 --------- Co-authored-by: Andrew Lamb <[email protected]>
Dandandan
pushed a commit
to Dandandan/arrow-rs
that referenced
this pull request
Jan 15, 2026
…n row conversion (apache#9078) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? Instead of iterating over the items in the array and getting the length from the byte slice, we use the offsets directly and zip with nulls if necessary # Are these changes tested? Existing tests # Are there any user-facing changes? Faster encoding ------ Split to 2 more PRs as the other 2 add a change to the public API Related to: - apache#9079 - apache#9080 --------- Co-authored-by: Andrew Lamb <[email protected]>
Dandandan
pushed a commit
to Dandandan/arrow-rs
that referenced
this pull request
Jan 15, 2026
… conversion (apache#9079) # Which issue does this PR close? N/A # Rationale for this change Making the row length calculation faster which result in faster row conversion # What changes are included in this PR? 1. Instead of iterating over the rows and getting the length from the byte slice, we use the offsets directly, this 2. Added 3 new APIs for `Rows` (explained below) # Are these changes tested? Yes # Are there any user-facing changes? Yes, added 3 functions to `Rows`: - `row_len` - get the row length at index - `row_len_unchecked` - get the row length at index without bound checks - `lengths` - get iterator over the lengths of the rows ----- Related to: - apache#9078 - apache#9080 --------- Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
Making the row length calculation faster which result in faster row conversion
What changes are included in this PR?
Instead of iterating over the items in the array and getting the length from the byte slice, we use the offsets directly and zip with nulls if necessary
Are these changes tested?
Existing tests
Are there any user-facing changes?
Faster encoding
Split to 2 more PRs as the other 2 add a change to the public API
Related to: