-
Notifications
You must be signed in to change notification settings - Fork 4k
ARROW-4466: [Rust] [DataFusion] Add support for Parquet data source #3851
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 28 commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
10710a2
Parquet datasource
andygrove ff3e5b7
test
andygrove 3a412b1
first parquet test passes
andygrove 322fc87
add test for reading strings from parquet
andygrove eaddafb
save
andygrove f46e6f7
save
andygrove aea9f8a
convert to use row iter
andygrove c3f71d7
add integration test
andygrove 5ce3086
revert to columnar reads
andygrove b4981ed
implement more parquet column types and tests
andygrove 6c3b7e2
add support for all primitive parquet types
andygrove debb2fb
code cleanup
andygrove 157512e
Remove invalid TODO comment
andygrove dddb7d7
update to use partition-aware changes from master
andygrove 7e1a98f
remove println and unwrap
andygrove c56510e
projection takes slice instead of vec
andygrove 6457c36
use parquet::reader::schema::parquet_to_arrow_schema
andygrove e8aa784
revert temp debug change to error messages
andygrove 607a29f
return result if there are null values
andygrove e6cbbaa
replace read_column! macro with generic
nevi-me 3c711a5
immediately allocate vec
nevi-me 306d07a
fmt
nevi-me 5a3368c
Remove unnecessary slice, fix null handling
andygrove 80cf303
add date support
andygrove 1503855
handle nulls for binary data
andygrove 639e13e
null handling for int96
andygrove 9d3047a
code cleanup
andygrove 2aeea24
remove println from tests
andygrove 02b2ed3
fix int96 conversion to read timestamps correctly
nevi-me 023dc25
Merge pull request #2 from nevi-me/ARROW-4466
andygrove 1ec815b
Clean up imports
andygrove 9b1308f
clean up handling of INT96 and DATE/TIME/TIMESTAMP types in schema co…
andygrove 25d34ac
Make INT32/64/96 handling consistent with C++ implementation
andygrove 73aa934
Remove println from test
andygrove 204db83
fix timestamp nano issue
andygrove 8d2df06
move schema projection function from arrow into datafusion
andygrove 549c829
Remove hard-coded batch size, fix nits
andygrove 3158529
add test for reading small batches
andygrove File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this belong in
datafusionas a free function? It doesn't seem like this will be used within thearrowsub-crate?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm 50/50 on this but I moved it to datafusion for now.