Parquet: Add readers and writers for the internal object model

[ajantha-bhat]

• Refactor BaseParquetWriter and BaseParquetReaders to reuse for internal writers and readers.
• Added InternalWriter and InternalReader class for parquet that consumes and produces the Iceberg in-memory data model.
• Fixed some bugs in genric readers like UUID, timestamp millis, fixed length validatation etc.

[rdblue]

This looks fine to me.

[rdblue]

Here also, there's not much value in using Parquet in the class name. Since this will produce GenericRecord instances, how about RecordReader?

[rdblue]

When checking that name (RecordReader) for consistency, I noticed that there's already a RecordReader in GenericParquetReaders. You can reuse that class.

[ajantha-bhat]

Cannot reuse the class from GenericParquetReaders as it is based on Record interface, we need a class based on StructLike interface.

I will rename to StructLikeReader, just like the StructLikeWriter from InternalWriter class.

[rdblue]

This isn't correct. The unboxed reader will return a Binary for int96 columns. Instead, this needs to use the same logic as the Spark reader (which also uses the internal representation):

  private static class TimestampInt96Reader extends UnboxedReader<Long> {
    TimestampInt96Reader(ColumnDescriptor desc) {
      super(desc);
    }

    @Override
    public Long read(Long ignored) {
      return readLong();
    }
    @Override
    public long readLong() {
      final ByteBuffer byteBuffer =
          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
      return ParquetUtil.extractTimestampInt96(byteBuffer);
    }
  }

You can move that class into the parquet package to share it.

[rdblue]

I agree with this change, but please point these kinds of changes out for reviewers.

The old version worked because all of the supported logical type annotations had an equivalent ConvertedType (which is what OriginalType is called in Parquet format and the logical type docs).

[rdblue]

I don't think it makes sense to have the subclasses provide this visitor.

[rdblue]

I agree with moving the date/time reader classes here.

[rdblue]

This isn't correct. The unit of the incoming timestamp value still needs to be handled, even if the in-memory representation of the value is the same (a long).

[rdblue]

Looks like the Spark implementations for this should work well, just like the int96 cases.

[rdblue]

As with the Avro tests, I think this should extend DataTest. It is probably easier to do the Avro work first and then reuse it here.

[rdblue]

This PR should not introduce revapi failures. Instead, the new methods should have default implementations that match the previous behavior (returning the generic representations).

[ajantha-bhat]

New methods are abstract and abstract method cannot have default implementation. So, I think we have to handle revapi failures.

[ajantha-bhat]

Oh, I think what you mean is don't add it as abstract. Add it as methods with default implementation. I got it. I will update it today.

[rdblue]

This returns Record so I don't think it needed to be modified. It doesn't return any other subclass of StructLike.

[rdblue]

I think I commented on this in the last round of reviews. This isn't correct. Incoming values are of type UUID so this needs a writer that can convert UUID into a byte array. This should return ParquetValueWriters.uuids(desc).

There's also no need to add a method for this because it is the same between the generic and internal object models.

[ajantha-bhat]

Sorry, the existing testcases was the reason for confusion as I mentioned in #11904 (comment)

I will update the existing testcases of Arrow too in this PR.

[rdblue]

Should be ParquetValueReader<Record>

[ajantha-bhat]

@rdblue: Thanks for giving additional context for unresolved comments. I think I understood all the comments this time. PR is ready. It also fixed base code issues and testcases.

[rdblue]

This is valid for time but not for timestamp. I may have mixed up the timestamp reader and time reader in an earlier comment. This needs to be nextLong.

[rdblue]

I think the confusion was from this comment: #11904 (comment)

I was talking about the time type, but the code I pasted had the wrong class name, TimestampMillisReader should have been TimeMillisReader. Timestamps (millis) should use nextLong and time (millis) should use nextInteger.

[rdblue]

Fixed in ajantha-bhat#74

[ajantha-bhat]

Lack of test coverage in the base code for these milliseconds time, timestamp and int96 timestamps is the reason for back and forth. Tests would have caught this. Will try to add in a follow up.

[ajantha-bhat]

rebasing the PR as Flink hit a flaky test #11833 (comment)

[rdblue]

Thanks, @ajantha-bhat! Good to get this in.