[WIP] Support Shredded Lists/Array in `variant_get` #8354

sdf-jkl · 2025-09-16T03:27:40Z

Which issue does this PR close?

Pathinding PR for [Variant] Support Shredded Lists/Array in variant_get #8082.

Rationale for this change

We should be able to read lists using variant_get

What changes are included in this PR?

Are these changes tested?

I'm trying to start with some basic tests to do some TDD.

Are there any user-facing changes?

scovich

Left a couple comments that are hopefully helpful.

Also, we should (eventually) support nesting -- arrays and structs inside arrays.
Let's get simple lists of primitives working first, tho!

parquet-variant-compute/src/variant_get.rs

scovich · 2025-09-16T04:35:46Z

parquet-variant-compute/src/variant_get.rs

+        let main_struct = crate::variant_array::StructArrayBuilder::new()
+            .with_field("metadata", Arc::new(metadata_array))
+            .with_field("value", Arc::new(value_array))
+            .with_field("typed_value", Arc::new(typed_value_array))


Check the variant shredding spec for arrays -- the typed_value for a shredded variant array is a non-nullable group called element, with child fields typed_value and value for shredded and unshredded list elements, respectively.

And then we'll need to build an appropriate GenericListArray out of this string array you built, which gives the offsets for each sub-list.

Thanks for this too, I was under the wrong impression that the metadata encoding stores the offsets for the actual values. Reading your #8359 and rereading the Variant Encoding spec I see that the values offsets are within the value encoding itself.

So the outermost typed_value should be an GenericListArray of element - VariantObjects with {value and typed_value fields}?

yes, exactly! And element is non-nullable (**), while the two children are nullable.

(**) As always, in arrow, it can still have null entries, but only if its parent is already NULL for the same row (so nobody can ever observe a non-null element)

parquet-variant-compute/src/variant_get.rs

…ed_list_support

scovich

I'm not sure I understand how these unit tests will translate to variant_get?

sdf-jkl · 2025-09-19T15:45:45Z

I'm not sure I understand how these unit tests will translate to variant_get?

Could you elaborate please?

I am currently trying to build just the Shredded List VariantArray test case, and while doing so learning how we could build them in shred_variant later. Once have a good way of building simple Shredded List VariantArray it will be easy to work on the rest of the unit tests for variant_get

scovich · 2025-09-19T21:18:02Z

I'm not sure I understand how these unit tests will translate to variant_get?

Could you elaborate please?

I am currently trying to build just the Shredded List VariantArray test case, and while doing so learning how we could build them in shred_variant later. Once have a good way of building simple Shredded List VariantArray it will be easy to work on the rest of the unit tests for variant_get

No worries -- the current iteration does look it produces a correct shredded variant containing a list, so I should probably just be patient and let you finish!

sdf-jkl · 2025-09-23T21:53:42Z

Hey @scovich I see that your current implementation of follow_shredded_path_element for VariantPathElement::Field when following the shredded path is successful, it returns a ShreddedPathStep::Success(field.shredding_state()) that holds a ShreddingState::Typed that holds a reference to the typed_value array. (That we later use for the next steps)

My question is: does ShreddedPathStep::Success() necessarily have to require the input ShreddingState to be a reference?

The reason I am asking is that since we use the output of follow_shredded_path_element to get the values from the shredded VariantArray, shouldn't we be free to drop the outer array once we extract the relevant typed_value?

The only way to work with list arrays I came up with so far, is to build new arrays with arrow_select::take, combining the path index and GenericListArray offsets.
But by using this method we create new arrays within the scope of the function and can't use a reference to the array in the ShreddedPathStep::Success.
(I just pushed a commit with a non-working implementation of the idea)

~~Should we instead look for another way to represent a resulting array consisting of slices instead?~~

I just saw the #8392

…ed_list_support

sdf-jkl · 2025-09-23T22:03:45Z

parquet-variant-compute/src/variant_get.rs

+            // Build the list of indices to take
+            let mut take_indices = Vec::with_capacity(list_len);
+            for i in 0..list_len {
+                let start = offsets[i] as usize;
+                let end = offsets[i + 1] as usize;
+                let len = end - start;
+
+                if *index < len {
+                    take_indices.push(Some((start + index) as u32));
+                } else {
+                    take_indices.push(None);
+                }
+            }
+
+            let index_array = UInt32Array::from(take_indices);
+
+            // Use Arrow compute kernel to gather elements
+            let taken = take(field_array, &index_array, None)?;


You can see the basic idea here

…ed_list_support

sdf-jkl · 2025-09-25T21:38:58Z

Hey @scovich I made it work for a one of the simple tests and it doesn't go through with the second one because Variant to Arrow does not support utf8 yet.

Do we have an issue tracking variant_to_arrow types support? If not, I can make one.

scovich · 2025-09-26T13:27:02Z

I made it work for a one of the simple tests and it doesn't go through with the second one because Variant to Arrow does not support utf8 yet.

Do we have an issue tracking variant_to_arrow types support? If not, I can make one.

I'm not sure we have a tracking issue for utf8 support in variant_to_arrow, but I've also noticed that it's an annoying gap for unit testing (we all seem to reach for string values...)

Co-authored-by: Congxian Qiu <[email protected]>

Co-authored-by: Ryan Johnson <[email protected]>

…row-rs into variant_to_arrow_utf8

…ed_list_support

sdf-jkl

Building on top of the utf8 variant_to_arrow support PR.
Changes in generic_bytes_builder.rs, generic_bytes_view_builder.rs and variant_to_arrow.rs are irrelevant.
Some changes in variant_get.rs and variant_array.rs are also from the utf8 pr, so they can be safely skipped.
Main changes are:

Adding ShreddingStateCow enum
Adding VariantPathElement::Index support for unnested List VariantArray

sdf-jkl · 2025-10-24T16:57:21Z

parquet-variant-compute/src/variant_get.rs

+            let Some(list_array) = typed_value.as_any().downcast_ref::<GenericListArray<i32>>()
+            else {
+                // Downcast failure - if strict cast options are enabled, this should be an error
+                if !cast_options.safe {
+                    return Err(ArrowError::CastError(format!(
+                        "Cannot access index '{}' on non-list type: {}",
+                        index,
+                        typed_value.data_type()
+                    )));
+                }
+                // With safe cast options, return NULL (missing_path_step)
+                return Ok(missing_path_step());
+            };
+
+            let offsets = list_array.offsets();
+            let values = list_array.values(); // This is a StructArray
+
+            let Some(struct_array) = values.as_any().downcast_ref::<StructArray>() else {
+                return Ok(missing_path_step());
+            };
+
+            let Some(typed_array) = struct_array.column_by_name("typed_value") else {
+                return Ok(missing_path_step());
+            };
+
+            // Build the list of indices to take
+            let mut take_indices = Vec::with_capacity(list_array.len());
+            for i in 0..list_array.len() {
+                let start = offsets[i] as usize;
+                let end = offsets[i + 1] as usize;
+                let len = end - start;
+
+                if *index < len {
+                    take_indices.push(Some((start + index) as u32));
+                } else {
+                    take_indices.push(None);
+                }
+            }
+
+            let index_array = UInt32Array::from(take_indices);
+
+            // Use Arrow compute kernel to gather elements
+            let taken = take(typed_array, &index_array, None)?;
+
+            let metadata_array = BinaryViewArray::from_iter_values(std::iter::repeat_n(
+                EMPTY_VARIANT_METADATA_BYTES,
+                taken.len(),
+            ));
+
+            let struct_array = &StructArrayBuilder::new()
+                .with_field("metadata", Arc::new(metadata_array), false)
+                .with_field("typed_value", taken, true)
+                .build();
+
+            let state = ShreddingState::try_from(struct_array)?;
+            Ok(ShreddedPathStep::Success(state.into()))
        }
    }
 }


When we use variant_get on Struct Variant Array's it's relatively easy to extract the typed_value. For example, if we extract a.b because on the inside it's just:

VariantArray{ StructArray{ "typed_value": StructArray{ "typed_value": PrimiteArray, <- We can directly borrow the value into ShreddingState::Success() because the needed values in the array are contiguous "value": VariantArray

But if we try to extract "typed_value" from a List VariantArray it gets more complicated. For example, extracting 0.0:

VariantArray{ StructArray{ "typed_value": ListArray{ Offsets StructArray{ "typed_value": PrimiteArray, <- but the values are now not contiguous, and the output array can only be extracted using offsets, no borrow available "value": VariantArray

Because of this issue the output of follow_shredded_path_element -> ShreddedPathStep::Success can end up receiving BorrowedShreddingState or owned ShreddingState.

To make this work I added a ShreddingStateCow enum and made it the ShreddedPathStep::Success input.

parquet-variant-compute/src/variant_array.rs

sdf-jkl · 2025-10-24T17:00:53Z

parquet-variant-compute/src/variant_get.rs

                    );
                }
-                shredding_state = state;
+                shredding_state = ShreddingStateCow::Owned(state.into_owned());


Here I could not come up with a way to make the shredding_state for the next path_element be ither borrowed or owned depending on the follow_shredded_path_element output.

Made it into_owned() just to pass the borrow checker.

sdf-jkl · 2025-10-24T17:09:27Z

Hey @scovich, I'm ready for another go when you are available, thanks.

sdf-jkl · 2025-12-03T21:16:04Z

@alamb @klion26 can you take a look?

klion26 · 2025-12-04T07:19:00Z

@alamb @klion26 can you take a look?

Thanks for the remainder, I'll check it later today or tomorrow.

klion26

@sdf-jkl I've made a first round review for this, and left some comments. Please take a look.

parquet-variant-compute/src/variant_get.rs

klion26 · 2025-12-05T06:05:54Z

parquet-variant-compute/src/variant_get.rs

+        assert_eq!(result_variant.len(), 2);

+        // Row 0: expect 0 index = "comedy"
+        assert_eq!(result_variant.value(0), Variant::from("comedy"));


Do we need to cover the case that list[index] located in the value field of the input variant(such as VariantPath::from(1) here for ["horror", 123])

Maybe we can add some more test cases

the request index value located in the typed_value column

the request index value located in the value column

some nest struct(list in struct, or struct in list)

...

klion26 · 2025-12-05T06:48:56Z

parquet-variant-compute/src/variant_get.rs

+            let values = list_array.values(); // This is a StructArray
+
+            let Some(struct_array) = values.as_any().downcast_ref::<StructArray>() else {
+                return Ok(missing_path_step());


Double-checking here,
From the variant shredding spec, the element is a required group list. Does this return Ok(missing_path_step()); mean the input is not a valid variant?

optional group tags (VARIANT) { required binary metadata; optional binary value; optional group typed_value (LIST) { # must be optional to allow a null list repeated group list { required group element { # shredded element optional binary value; optional binary typed_value (STRING); } } } }

klion26 · 2025-12-05T06:58:40Z

parquet-variant-compute/src/variant_get.rs

-            Err(ArrowError::NotYetImplemented(
-                "Pathing into shredded variant array index".into(),
-            ))
+            let Some(list_array) = typed_value.as_any().downcast_ref::<GenericListArray<i32>>()


Is there any chance the list length exceeds i32?

Yes, I wanted to add two cases for i32 and i64 lists.

klion26 · 2025-12-05T07:11:55Z

parquet-variant-compute/src/variant_get.rs

+                let end = offsets[i + 1] as usize;
+                let len = end - start;
+
+                if *index < len {


Please correct me if I'm wrong. Here we assert that all the values will be typed_value column, and use the indices collected here to retrieve the final value.

What if the value is located in the value column instead of the typed_value column? (change the test test_shredded_list_as_string from VariantPath::from(0) to VariantPath::from(1) can see this)

This test is running variant_get with the as_type parameter, specifying which type we are looking for.

The example VariantArray is shredded by String type, therefore a String value cannot be outside the typed_value column and will return a null in this test case.

Given the ["comedy", "drama"], ["horror" 123] ListArray if we try take the value from index 1 instead 0, we will get:

thread 'variant_get::test::test_shredded_list_as_string' (51616) panicked at parquet-variant-compute\src\variant_get.rs:1735:9: assertion `left == right` failed left: StringArray [ "drama", null, ] right: StringArray [ "comedy", "horror", ]

Sorry for not describing it clearly.

The data["comedy", "drama"], ["horro", 123] translated into variant will be that

comedy, drama and horro in the typed_value column,

and 123 in the value column(it has an incompatible type).

Here, we retrieve all the results from the typed_value column(take in line 148), but ["hooro", 123](1)(the second item in the list) here will return null(if CastOptions::safe = true) and Err (if CastOptions::safe = false) -- currently, we return null for both of the cases.

Seems there may be something more tricky here(maybe we need to have a design note for this as this comment), such as

if the target_type here we request is not list/struct then, we can use the logic like here, and respect the CastOptions::safe

If we need to handle variant nesting here or somewhere else?

Here, I don't have any answer yet. I'll try to find some time next week for this.

NOTE: Other than (partially shredded) object fields, the shredding spec doesn't actually require any other type to shred merely because a compatible typed_value column exists. We have to assume that e.g. value could contain i8, i32, and i64 values even if typed_value is a 64-bit int. And AFAIK, we also have to assume that value could contain a variant array even if typed_value is a list. Super annoying.

Maybe this code already handles that case, but I wanted to make sure to flag it.

klion26 · 2025-12-05T07:28:51Z

parquet-variant-compute/src/variant_get.rs

+            let index_array = UInt32Array::from(take_indices);
+
+            // Use Arrow compute kernel to gather elements
+            let taken = take(typed_array, &index_array, None)?;


Seems this will create a new array, not sure if we can use some "view" here to avoid creating the new array here.

Yes, I believe this is the inconvenient part of working with non-contiguous data.

klion26 · 2025-12-05T07:56:34Z

parquet-variant-compute/src/variant_get.rs

+                if *index < len {
+                    take_indices.push(Some((start + index) as u32));
+                } else {
+                    take_indices.push(None);


Does this mean OutOfBound in the current list? Currently, we'll return null, not sure if we need to return an error in this case. Return an error or not, maybe we can log the behavior somewhere.

Lists can be variable length within a ListArray, so I don't think it should be an error if the list if not long enough for an index.

(GenericListArray docs) For example, the ListArray shown in the following diagram stores lists of strings. Note that [] represents an empty (length 0), but non NULL list. ┌─────────────┐ │ [A,B,C] │ ├─────────────┤ │ [] │ ├─────────────┤ │ NULL │ ├─────────────┤ │ [D] │ ├─────────────┤ │ [NULL, F] │ └─────────────┘

Yes, the list could have a different length, but do we need to return Err if there is an OutOfBound occurs (like the CastOptions:safe=false), so that the caller knows this.

sdf-jkl · 2025-12-30T21:25:35Z

@liamzwbao taking over with #8082

sdf-jkl and others added 3 commits September 15, 2025 13:49

Add test shredded variant list array

9c25cc4

Add basic tests

ed961a4

Merge branch 'apache:main' into shredded_list_support

03ecb95

github-actions bot added the parquet-variant parquet-variant* crates label Sep 16, 2025

sdf-jkl mentioned this pull request Sep 16, 2025

[Variant] Support Shredded Lists/Array in variant_get #8082

Open

scovich reviewed Sep 16, 2025

View reviewed changes

sdf-jkl and others added 2 commits September 16, 2025 16:21

Merge branch 'apache:main' into shredded_list_support

158d6d7

Redo test shredded array

d53c831

sdf-jkl commented Sep 17, 2025

View reviewed changes

parquet-variant-compute/src/variant_get.rs Show resolved Hide resolved

sdf-jkl commented Sep 17, 2025

View reviewed changes

parquet-variant-compute/src/variant_get.rs Outdated Show resolved Hide resolved

sdf-jkl commented Sep 17, 2025

View reviewed changes

parquet-variant-compute/src/variant_get.rs Outdated Show resolved Hide resolved

sdf-jkl commented Sep 17, 2025

View reviewed changes

parquet-variant-compute/src/variant_get.rs Outdated Show resolved Hide resolved

sdf-jkl commented Sep 17, 2025

View reviewed changes

parquet-variant-compute/src/variant_get.rs Outdated Show resolved Hide resolved

scovich mentioned this pull request Sep 18, 2025

[Variant] Define new shred_variant function #8366

Merged

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

174e429

…ed_list_support

scovich reviewed Sep 19, 2025

View reviewed changes

Rebuild the shredded list array

69de7d7

sdf-jkl added 2 commits September 23, 2025 17:54

Use select::take to build the output array

cc6d787

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

8f6ad1b

…ed_list_support

sdf-jkl commented Sep 23, 2025

View reviewed changes

sdf-jkl added 3 commits September 25, 2025 11:46

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

bc8abd9

…ed_list_support

Pass one test

c0d2065

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

85aaa3f

…ed_list_support

Get typed values directly

40b6311

Added support for utf8, largeUtf8, utf8view

f6e88ef

sdf-jkl and others added 12 commits October 14, 2025 09:59

Update parquet-variant-compute/src/variant_to_arrow.rs

2b6d280

Co-authored-by: Congxian Qiu <[email protected]>

Merge branch 'main' into variant_to_arrow_utf8

398b52d

Update parquet-variant-compute/src/variant_to_arrow.rs

defa07b

Co-authored-by: Ryan Johnson <[email protected]>

Support LargeUtf8, Utf8-View

5022acd

Merge branch 'main' into variant_to_arrow_utf8

ed66007

Fix Merge errors

196b5d4

Update arrow-array/src/builder/generic_bytes_builder.rs

642d192

Co-authored-by: Ryan Johnson <[email protected]>

Add docs for AVERAGE_STRING_LENGTH const

76b3c80

Merge branch 'variant_to_arrow_utf8' of https://github.com/sdf-jkl/ar…

35785d6

…row-rs into variant_to_arrow_utf8

cargo fmt

5914218

cargo fmt

216d401

Merge branch 'variant_to_arrow_utf8' into shredded_list_support

3aa6cf3

github-actions bot added the arrow Changes to the arrow crate label Oct 22, 2025

sdf-jkl and others added 3 commits October 22, 2025 13:54

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

15fc8be

…ed_list_support

Quick fix variant_get

04b9941

Merge branch 'main' into shredded_list_support

857f0e2

sdf-jkl commented Oct 24, 2025

View reviewed changes

Merge branch 'main' into shredded_list_support

1edfeca

github-actions bot removed the arrow Changes to the arrow crate label Nov 11, 2025

sdf-jkl requested a review from scovich November 11, 2025 03:24

sdf-jkl and others added 2 commits November 11, 2025 00:29

fix merge errors

6d6793d

Merge branch 'main' into shredded_list_support

71412b8

sdf-jkl marked this pull request as ready for review December 3, 2025 19:37

klion26 reviewed Dec 5, 2025

View reviewed changes

sdf-jkl closed this Dec 30, 2025

sdf-jkl deleted the shredded_list_support branch January 3, 2026 00:14

[WIP] Support Shredded Lists/Array in variant_get #8354

[WIP] Support Shredded Lists/Array in variant_get #8354

Uh oh!

Conversation

sdf-jkl commented Sep 16, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

scovich left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scovich commented Sep 19, 2025

Uh oh!

sdf-jkl commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Sep 25, 2025

Uh oh!

scovich commented Sep 26, 2025

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Oct 24, 2025

Uh oh!

sdf-jkl commented Dec 3, 2025

Uh oh!

klion26 commented Dec 4, 2025

Uh oh!

klion26 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

klion26 Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

[WIP] Support Shredded Lists/Array in `variant_get` #8354

[WIP] Support Shredded Lists/Array in `variant_get` #8354

scovich left a comment •

edited

Loading

sdf-jkl commented Sep 19, 2025 •

edited

Loading

sdf-jkl commented Sep 23, 2025 •

edited

Loading

klion26 Dec 11, 2025 •

edited

Loading

sdf-jkl Dec 10, 2025 •

edited

Loading