ARROW-6131: [C++] Optimize the Arrow Non-Ascii-string-validation #5038
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guyuqi
force-pushed
the
utf8-validation-ARROW-6131
branch
from
23e090a to
0b36f0d
Compare
Member
|
What's the point of including this in Arrow if it's not used in the codebase? |
Member
|
I had been thinking that this would be exposed in some way to be invoked easily on If this patch is accepted we need to add something to LICENSE.txt |
Member
|
This also doesn't have any unit tests |
Member
Author
I'd like to add the unit test case for |
guyuqi
force-pushed
the
utf8-validation-ARROW-6131
branch
2 times, most recently
from
3fb2fa2 to
01fff3a
Compare
Member
Author
|
Add |
Member
Author
Member
|
I'm working on adding a more capable Aarch64 machine to the Ursabot cluster (48-core ThunderX) so that could help with providing more timely CI |
Member
|
Cool. But I still don't think we should hamper the full-ASCII case (which will be very common in e.g. CSV or JSON files). |
Member
|
I agree |
Member
Author
I'd like to propose a full-ASCII optimization for Arm64 later. |
guyuqi
force-pushed
the
utf8-validation-ARROW-6131
branch
from
01fff3a to
ec8b241
Compare
Member
Author
|
Add Ascii validation: After: Get ~ 1.9x speedup for Large Ascii |
Algorithm and code come from: https://github.com/cyb70289/utf8 (MIT LICENSE) From discussion in ARROW-6131: The patch would introduce a fast non-ASCII validation method with this new algorithm into Apache Arrow and give give a option to non-ascii case validation Change-Id: Idf7e13b71a91d5aa85fe47dd271b2159d19fde6a Signed-off-by: Yuqi Gu <[email protected]>
Change-Id: I306c15b60ed657a7de4167191425b0d2a6046d04
Before: ValidateTinyAscii 6.04 ns 6.03 ns 112754472 bytes_per_second=1.54331G/s ValidateSmallAscii 22.8 ns 22.7 ns 29583112 bytes_per_second=5.60858G/s ValidateLargeAscii 15166 ns 15164 ns 46164 bytes_per_second=6.14226G/s After: ValidateTinyAscii 8.06 ns 8.06 ns 87046764 bytes_per_second=1.15497G/s ValidateSmallAscii 14.5 ns 14.5 ns 48115118 bytes_per_second=8.77193G/s ValidateLargeAscii 7840 ns 7839 ns 89218 bytes_per_second=11.8821G/s Get ~ 1.9x speedup for Large Ascii Change-Id: Ia98e3f836dad259e9c4711f82b6af8de90bb9d77 Signed-off-by: Yuqi Gu <[email protected]>
guyuqi
force-pushed
the
utf8-validation-ARROW-6131
branch
from
ec8b241 to
34ed1e7
Compare
Member
Author
|
Rebase the code for file confliction and also have added the full-ASCII validation optimization here. |
Change-Id: Id2bcfde0d15bda004223fa476bb85fd119c417b8
Contributor
|
@guyuqi this fails to build on windows due to type truncation issues: https://github.com/apache/arrow/pull/5038/checks?check_run_id=380918799#step:4:510 |
Member
|
It would be useful to have an ASCII validation function (versus UTF-8 / non-UTF-8), would you be able to break out the ASCII validation changes into a new PR? |
Member
Author
Yes, of course. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Algorithm and code come from: https://github.com/cyb70289/utf8 (MIT LICENSE)
The patch would introduce a fast non-ASCII validation method with this new algorithm into Apache Arrow and give give a option to non-ascii case validation.
Arm64 platform
Benchmark
Origin:
The new algorithm:
x86 platform
Benchmark
Origin:
The new algorithm: