ARROW-8289: [Rust] Parquet Arrow writer with nested support

[nevi-me]

Note: I started making changes to #6785, and ended up deviating a lot, so I opted for making a new draft PR in case my approach is not suitable.

---

This is a draft to implement an arrow writer for parquet. It supports the following (no complete test coverage yet):

• writing primitives except for booleans and binary
• nested structs
• null values (via definition levels)

It does not yet support:

• Boolean arrays (have to be handled differently from numeric values)
• Binary arrays
• Dictionary arrays
• Union arrays (are they even possible?)

I have only added a test by creating a nested schema, which I tested on pyarrow.

# schema of test_complex.parquet

a: int32 not null
b: int32
c: struct<d: double, e: struct<f: float>> not null
  child 0, d: double
  child 1, e: struct<f: float>
      child 0, f: float

This PR potentially addresses:

• https://issues.apache.org/jira/browse/ARROW-8289
• https://issues.apache.org/jira/browse/ARROW-8423
• https://issues.apache.org/jira/browse/ARROW-8424
• https://issues.apache.org/jira/browse/ARROW-8425

And I would like to propose either opening new JIRAs for the above incomplete items, or renaming the last 3 above.

---

Help Needed

I'm implementing the definition and repetition levels on first principle from an old Parquet blog post from the Twitter engineering blog. It's likely that I'm not getting some concepts correct, so I would appreciate help with:

• Checking if my logic is correct
• Guidance or suggestions on how to more efficiently extract levels from arrays
• Adding tests - I suspect we might need a lot of tests, so far we only test writing 1 batch, so I don't know how paging would work when writing a large enough file

I also don't know if the various encoding levels (dictionary, RLE, etc.) and compression levels are applied automagically, or if that'd be something we need to explicitly enable.

CC @sunchao @sadikovi @andygrove @paddyhoran

Might be of interest to @mcassels @maxburke

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[github-actions]

https://issues.apache.org/jira/browse/ARROW-8289

[sunchao]

Thanks @nevi-me , left some comments. I think it we should think about how this code can evolve to handle other nested types (especially lists), and whether it will require significant changes and refactoring to achieve that goal.

IMO we can solve this in two steps: 1) have a top-level method to compute def/rep levels for all leaf arrays of a given an arrow array, and then 2) use column writer to write Parquet value/def/rep using the result from the previous step and the input leaf array.

[sunchao]

May add comments on what this is for

[nevi-me]

Documented, and removed the rows field. @andygrove I think I copied it as is from your initial draft. Did you want to be able to check how many rows have been written? I think it ends up being the same as the batch's length

[sunchao]

Hmm is this correct? if I have a deeply nested struct and if its parent is null, then its def level is 0?

[nevi-me]

I've rewritten this and fixed it

[sunchao]

Not sure if this is the best way to handle this. For one, we can skip the for-loop if all array elements are not-null. Also perhaps we should use array builder for this? Eventually we can have a write_arrow_batch method in column/writer.rs.

[nevi-me]

Yes, I agree that it's better to avoid this function if there are no nulls, I'll look into this.

[emkornfield]

IMO we can solve this in two steps: 1) have a top-level method to compute def/rep levels for all leaf arrays of a given an arrow array, and then 2) use column writer to write Parquet value/def/rep using the result from the previous step and the input leaf array.

FWIW, this is the approach the C++ code takes (supports arbitrary nessting). One thing to note which is currently a bug in C++ is once rep/def levels are computed for any anything
with deep nesting (any leaf column one or more direct struct/group ancestor), nullness should be determined rep/def-levels and not leaf-arrays (this is currently a bug in C++).

[maxburke]

Small comment about the API, something I found necessary was being able to pass in props to the writer interface: urbanlogiq@5e08cf6

The idea here is so that we can set the created_by attributes and compression, as an example.

[maxburke]

Oh! One more :) This one uses the ParquetWriter trait instead of std::fs::File: urbanlogiq@e15851e

(our primary usecase is we write parquet files to memory because they immediately get persisted to object storage like AWS S3 so we never deal with local files)

[nevi-me]

@maxburke there's been some interest from other people on this PR. I haven't been able to continue working on it because where I have a bit of free time I've been looking at the IPC/integration issues (Rust doesn't work with 0.15+ files).

Please feel free to push changes against this PR, or to open a PR against my fork with upstream changes. There's also someone who reached out to me on Twitter asking how they can continue with this.

[nevi-me]

... One thing to note which is currently a bug in C++ is once rep/def levels are computed for any anything
with deep nesting (any leaf column one or more direct struct/group ancestor), nullness should be determined rep/def-levels and not leaf-arrays (this is currently a bug in C++).

@emkornfield not sure if I understand this part, I'll try create a nested batch with a few levels, and have one record have the top level be nested. Would this cover the case above? I might also be limited by https://issues.apache.org/jira/browse/ARROW-5408 for now

[nevi-me]

@sunchao @andygrove (CC @wesm @kszucs @emkornfield) in the past few months we haven't had enough review bandwidth on Rust's Parquet implementation (mostly relying on Chao for non-trivial reviews), and given the amount of work needed for an Arrow writer + the interest so far (I think few people already using this fork), I'd like to propose:

• We create a temporary branch in the apache/arrow repo, where the arrow writer can temporarily live
• We can merge changes into the branch, esp if there aren't enough reviewers at the time
• When we're close to a release, we merge what's on the temp branch into the branch that's currently called master but will be renamed soon 😉

ITO this PR, I think I've gotten arbitrary nesting covered, but there's a lot more work that we can now divide more easily so others can contribute better. I'm also unsure of how to test deeply nested arrays directly in the code (I had to use Spark because Arrow reader doesn't yet support that).

I'll also bring this up in the mailing list for wider visibility

[andygrove]

@nevi-me Sounds good to me. Thanks.

[emkornfield]

@emkornfield not sure if I understand this part, I'll try create a nested batch with a few levels, and have one record have the top level be nested.

There are two bugs in C++ (one with an open PR).

The first bug is if you have a schema like nullable struct<list<nullable struct<nullable struct<int>>> you need to include all null values from the leaf to the list. The bug we had in C++ is we would only include the first level of nulls and drop the other ones (leading to inconsistent list size).

The second bug we have with no PR. if you have a schema nullable struct<nullable int> then the null validity buffer could look like [null, null, null] but the underlying int vector could have valid values [1, 2, 3]. For the purposes of writing to parquet the values should all be considered null. The only way to determine this is to re-walk the tree or use the already generated levels to generate a new bitmap for the leaf..

[emkornfield]

this isn't true. you can have many "empty" list which can have no items. each empty list adds a def and rep level.

[nevi-me]

Correct, this comment is incorrect. I wrote it before I tried to write an empty list. I'll remove it.
The arrow_writer_list test covers this scenario

[emkornfield]

I'm also unsure of how to test deeply nested arrays directly in the code (I had to use Spark because Arrow reader doesn't yet support that).

In C++ I covered at least partial testing by testing rep and def level generation directly

[sunchao]

+1 on the approach as well. Thanks @nevi-me for all the efforts!

[nevi-me]

Merged as 80a9c02, not sure why the merge tool didn't close the issue