-
Notifications
You must be signed in to change notification settings - Fork 30
perf: Faster st_geometrytype() function
#90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
365e67d
faster st_geometrytype
2010YOUY01 5ed94af
fix test fail on null col
2010YOUY01 f7aee24
clean up
2010YOUY01 71f5bcd
move `execute_wkb_bytes_void()` to another generic instanciation
2010YOUY01 5b034e6
add todo to move the util to Wkb crate
2010YOUY01 df88be5
review
2010YOUY01 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -23,11 +23,9 @@ use datafusion_common::error::Result; | |||||
| use datafusion_expr::{ | ||||||
| scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility, | ||||||
| }; | ||||||
| use geo_traits::GeometryTrait; | ||||||
| use sedona_common::sedona_internal_err; | ||||||
| use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF}; | ||||||
| use sedona_schema::{datatypes::SedonaType, matchers::ArgMatcher}; | ||||||
| use wkb::reader::Wkb; | ||||||
|
|
||||||
| pub fn st_geometry_type_udf() -> SedonaScalarUDF { | ||||||
| SedonaScalarUDF::new( | ||||||
|
|
@@ -71,12 +69,12 @@ impl SedonaScalarKernel for STGeometryType { | |||||
| let min_output_size = "POINT".len() * executor.num_iterations(); | ||||||
| let mut builder = StringBuilder::with_capacity(executor.num_iterations(), min_output_size); | ||||||
|
|
||||||
| // We can do quite a lot better than this with some vectorized WKB processing, | ||||||
| // but for now we just do a slow iteration | ||||||
| executor.execute_wkb_void(|maybe_item| { | ||||||
| match maybe_item { | ||||||
| Some(item) => { | ||||||
| builder.append_option(invoke_scalar(&item)?); | ||||||
| // Iterate over raw WKB bytes for faster type inference | ||||||
| executor.execute_wkb_bytes_void(|maybe_bytes| { | ||||||
|
||||||
| executor.execute_wkb_bytes_void(|maybe_bytes| { | |
| executor.execute_wkb_void(|maybe_bytes| { |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would probably be better to use the
GeometryFactory/generic executor to do this. I believe something like this works (and is what we do for iterating over things that aren'twkb::Wkbfrom other libraries).