Skip to content

Variant Support for Arrow and Parquet [DRAFT] #7404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft

Variant Support for Arrow and Parquet [DRAFT] #7404

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
1657aa4
schema: add initial Variant type as a Canonical Extension Type
Feb 26, 2025
7b098f1
parquet: initial support for LogicalType and ConvertedType for Variant
Feb 26, 2025
2b659c6
schema: enforce required value field in Variant
Feb 26, 2025
7796a1e
parquet: add support for variant extension type conversion
Mar 10, 2025
b47df75
turn variant into primitive type, add roundtrip test
PinkCrow007 Mar 20, 2025
0220e97
test variant roundtrip with multiple columns RecordBatch; refine variant
PinkCrow007 Mar 25, 2025
36a96ab
update variant roundtrip test
PinkCrow007 Mar 25, 2025
a8ba629
add VariantArray and VariantBuilder draft
PinkCrow007 Mar 27, 2025
0eaa7f0
implement VariantArrayReader and encode_variant_array for variant rou…
PinkCrow007 Mar 27, 2025
01a2e70
update comment
PinkCrow007 Mar 27, 2025
f245655
implement get_metadata_length
PinkCrow007 Apr 1, 2025
d81959f
modify comments
PinkCrow007 Apr 3, 2025
816d189
create arrow-variant; implement variant metadata encoding
PinkCrow007 Apr 3, 2025
83d8048
implement sorted_string to metadata; encode value draft
PinkCrow007 Apr 7, 2025
d8d6dae
initial variant encoder and decoder
PinkCrow007 Apr 7, 2025
8de7de5
add json_variant_parquet_roundtrip test; refine variant <-> json
PinkCrow007 Apr 10, 2025
1313697
fix bug
PinkCrow007 Apr 11, 2025
2096395
Merge remote-tracking branch 'upstream/main' into struct_group_type
PinkCrow007 Apr 16, 2025
c1b6bf2
Make Variant an ExtensionType over Struct in Arrow;
PinkCrow007 Apr 17, 2025
626765e
Variant ExtensionType over Struct, GroupType in Parquet
PinkCrow007 Apr 21, 2025
f93c238
minor fix
PinkCrow007 Apr 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ members = [
"arrow-schema",
"arrow-select",
"arrow-string",
"arrow-variant",
"parquet",
"parquet_derive",
"parquet_derive_test",
Expand Down
1 change: 1 addition & 0 deletions arrow-array/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ all-features = true
[features]
ffi = ["arrow-schema/ffi", "arrow-data/ffi"]
force_validate = []
canonical_extension_types = ["arrow-schema/canonical_extension_types"]

[dev-dependencies]
rand = { version = "0.9", default-features = false, features = ["std", "std_rng", "thread_rng"] }
Expand Down
3 changes: 3 additions & 0 deletions arrow-array/src/array/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,9 @@ pub use list_view_array::*;

use crate::iterator::ArrayIter;

mod variant_array;
pub use variant_array::*;

/// An array in the [arrow columnar format](https://arrow.apache.org/docs/format/Columnar.html)
pub trait Array: std::fmt::Debug + Send + Sync {
/// Returns the array as [`Any`] so that it can be
Expand Down
Loading
Loading