-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Support converting large dates (i.e. +10999-12-31) from string to Date32 #7074
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
alamb
merged 21 commits into
apache:main
from
spiceai:phillip/250205-handle-large-dates
Feb 12, 2025
Merged
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
93934bf
Support converting large dates (i.e. +10999-12-31) from string to Date32
phillipleblanc 4041947
Fix lint
phillipleblanc a584828
Update arrow-cast/src/parse.rs
phillipleblanc 458773c
fix: issue introduced in #6833 - less than equal check for scale in …
himadripal 3a8a001
minor: re-export `OffsetBufferBuilder` in `arrow` crate (#7077)
alamb cb54440
Add another decimal cast edge test case (#7078)
findepi 7e91c46
Support both 0x01 and 0x02 as type for list of booleans in thrift met…
jhorstmann bb5f3ae
Fix LocalFileSystem with range request that ends beyond end of file (…
kylebarron e199ccc
Introduce `UnsafeFlag` to manage disabling `ArrayData` validation (#7…
alamb 6ec6cd9
Refactor arrow-ipc: Rename `ArrayReader` to `RecodeBatchDecoder` (#7028)
alamb 02ee7d2
Minor: Update release schedule (#7086)
alamb ec1d17a
Refactor some decimal-related code and tests (#7062)
CurtHagenlocher 706a523
Refactor arrow-ipc: Move `create_*_array` methods into `RecordBatchDe…
alamb 0d943b9
Print Parquet BasicTypeInfo id when present (#7094)
devinrsmith b339382
Add a custom implementation `LocalFileSystem::list_with_offset` (#7019)
corwinjoy 1738b57
fix: first none/empty list in `ListArray` panics in `cast_with_option…
irenjj d74be2c
Benchmarks for Arrow IPC writer (#7090)
alamb 5f69b6e
Minor: Clarify documentation on `NullBufferBuilder::allocated_size` (…
alamb 6dcdde6
Add more tests for edge cases
phillipleblanc 05d500f
Add negative test case for incorrectly formatted large dates
phillipleblanc f0bcaf1
Merge remote-tracking branch 'origin/main' into phillip/250205-handle…
phillipleblanc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4229,6 +4229,48 @@ mod tests { | |
| } | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_cast_string_with_large_date_to_date32() { | ||
| let array = Arc::new(StringArray::from(vec![ | ||
| Some("+10999-12-31"), | ||
| Some("-0010-02-28"), | ||
| Some("0010-02-28"), | ||
| Some("0000-01-01"), | ||
| Some("-0000-01-01"), | ||
| Some("-0001-01-01"), | ||
| ])) as ArrayRef; | ||
| let to_type = DataType::Date32; | ||
| let options = CastOptions { | ||
phillipleblanc marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| safe: false, | ||
| format_options: FormatOptions::default(), | ||
| }; | ||
| let b = cast_with_options(&array, &to_type, &options).unwrap(); | ||
| let c = b.as_primitive::<Date32Type>(); | ||
| assert_eq!(3298139, c.value(0)); // 10999-12-31 | ||
| assert_eq!(-723122, c.value(1)); // -0010-02-28 | ||
| assert_eq!(-715817, c.value(2)); // 0010-02-28 | ||
| assert_eq!(c.value(3), c.value(4)); // Expect 0000-01-01 and -0000-01-01 to be parsed the same | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
| assert_eq!(-719528, c.value(3)); // 0000-01-01 | ||
| assert_eq!(-719528, c.value(4)); // -0000-01-01 | ||
| assert_eq!(-719893, c.value(5)); // -0001-01-01 | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_cast_invalid_string_with_large_date_to_date32() { | ||
| // Large dates need to be prefixed with a + or - sign, otherwise they are not parsed correctly | ||
| let array = Arc::new(StringArray::from(vec![Some("10999-12-31")])) as ArrayRef; | ||
| let to_type = DataType::Date32; | ||
| let options = CastOptions { | ||
| safe: false, | ||
| format_options: FormatOptions::default(), | ||
| }; | ||
| let err = cast_with_options(&array, &to_type, &options).unwrap_err(); | ||
| assert_eq!( | ||
| err.to_string(), | ||
| "Cast error: Cannot cast string '10999-12-31' to value of Date32 type" | ||
| ); | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_cast_string_format_yyyymmdd_to_date32() { | ||
| let a0 = Arc::new(StringViewArray::from(vec![ | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.