-
Notifications
You must be signed in to change notification settings - Fork 4k
Optimize column reader by bitmap counting definition levels #48732
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
| int64_t offset = 0; | ||
| for (; offset + 64 <= num_levels; offset += 64) { | ||
| const uint64_t bitmap = | ||
| internal::GreaterThanBitmap(levels + offset, 64, rhs); | ||
| count += static_cast<int64_t>(bit_util::PopCount(bitmap)); | ||
| } | ||
| if (offset < num_levels) { | ||
| const uint64_t bitmap = | ||
| internal::GreaterThanBitmap(levels + offset, num_levels - offset, rhs); | ||
| count += static_cast<int64_t>(bit_util::PopCount(bitmap)); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| int64_t offset = 0; | |
| for (; offset + 64 <= num_levels; offset += 64) { | |
| const uint64_t bitmap = | |
| internal::GreaterThanBitmap(levels + offset, 64, rhs); | |
| count += static_cast<int64_t>(bit_util::PopCount(bitmap)); | |
| } | |
| if (offset < num_levels) { | |
| const uint64_t bitmap = | |
| internal::GreaterThanBitmap(levels + offset, num_levels - offset, rhs); | |
| count += static_cast<int64_t>(bit_util::PopCount(bitmap)); | |
| } | |
| for (int64_t offset = 0; offset < num_levels; offset += 64) { | |
| const int64_t chunk_size = std::min<int64_t>(64, num_levels - offset); | |
| const uint64_t bitmap = | |
| internal::GreaterThanBitmap(levels + offset, static_cast<int>(chunk_size), rhs); | |
| count += static_cast<int64_t>(bit_util::PopCount(bitmap)); | |
| } |
I suggest simplifying the for loop and if statement into one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
okay got it
Rationale for this change
Fuse definition level counting into a bitmap-based pass to reduce extra memory traversal and improve cache/SIMD efficiency, addressing the TODO in column_reader.cc.
What changes are included in this PR?
Replace std::count with a bitmap/PopCount helper for counting max definition levels in ReadLevels.
Are these changes tested?
yes
cpp/build-asan/debug/parquet-encoding-test --gtest_filter=*
cpp/build-asan/debug/parquet-arrow-internals-test --gtest_filter=*
Are there any user-facing changes?
No.