feat(iceberg): Add support for writing iceberg tables (to be closed)

[imjalpreet]

This PR implements support for Iceberg table insertions in Velox, enabling write operations to Iceberg tables through the Hive connector.

Changes

• Implemented IcebergDataSink class that extends HiveDataSink to handle Iceberg-specific write operations.
• Added IcebergInsertTableHandle to manage Iceberg table insertion metadata.
• Added IcebergPartitionField, IcebergPartitionSpec to support Iceberg's partition transform specification, this PR only supports identity transform for now.
• Implemented JSON serialization for partition values to support Iceberg metadata requirements.

Design Doc

Implementing_Iceberg_Insertion_Design.md

Implementation Details

The implementation follows Iceberg's table format specification, particularly for handling partitioning and metadata. Key components include:

1. IcebergPartitionSpec: Manages partition specifications with support for different transform types.
2. IcebergDataSink: Extends HiveDataSink to handle Iceberg-specific write operations.
3. Add extra parameters to HiveDataSink for dataChannels.
4. Adjust HiveDataSink data members and methods visibility for IcebergDataSink to use.
   The PR also includes test infrastructure for validating Iceberg insertions with various partition strategies.

Testing

Added unit tests that verify:

• Basic table writes to non-partitioned tables
• Partitioned table writes with single column partition and multiple column partition

Limitation

This PR only support iceberg identity partition transform. And it only support primitive column type as partition column, nested column type such as struct is not supported.

All tests pass on the current codebase.

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	5882b63
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/68b5e3bb53455e00091052d4

[zhouyuan]

FWIW, there's a new project to add C++ support for iceberg:
apache/iceberg-cpp#2

[prestodb-ci]

@majetideepak imported this issue into IBM GitHub Enterprise

[yingsu00]

@imjalpreet Did you remove the tests?

[yingsu00]

@imjalpreet Can you please make the test file format agnostic? You can use ReaderFactory::createReader() to create the readers instead of directly calling into the Parquet reader constructor.

[PingLiuPing]

When I create iceberg table based on your code, I hit following error, time data type is not supported?

presto:iceberg> insert into partition_t2 values (TIMESTAMP '2025-02-28 14:00:02', 11, DATE '2024-02-28', TIME '14:00:33', 128);
Query 20250303_133535_00004_gh5dp failed: inferredType Failed to parse type [time]. Type not registered.

My create table DDL is:

create table partition_t2 (c_timestamp timestamp, c_int int, c_date date, c_time time, c_bigint bigint) with (format='PARQUET', partitioning=ARRAY['year(c_date)']);

[imjalpreet]

When I create iceberg table based on your code, I hit following error, time data type is not supported?

@PingLiuPing Velox does not yet support the Time Datatype, so we must first add support for it.

Types supported in Velox: https://facebookincubator.github.io/velox/develop/types.html

[jinchengchenghh]

It should be done in downstream, if you have the restriction, you need to verify it and describe what input is valid input in API comments.

[PingLiuPing]

It should be done in downstream, if you have the restriction, you need to verify it and describe what input is valid input in API comments.

Let's clarify is it insertion failed or reading failed? If it is insertion failed, can you please check if the data and column name been passed into velox correctly, e.g, in IcebergDataSink::appendData . If it is reading failed, are you able to read the same data using Spark?

[jinchengchenghh]

The read is failed, it is a unit test in Gluten, before native write, it can pass

[PingLiuPing]

The read is failed, it is a unit test in Gluten, before native write, it can pass

Can you query this data that is written by Gluten from Spark?
I cannot reproduce this error from Pretissimo.

[jinchengchenghh]

Yes, Gluten without native write can query.
I have fallback this case, let us focus on statistic collect first.

[PingLiuPing]

Yes, Gluten without native write can query. I have fallback this case, let us focus on statistic collect first.

Ok, what I mean is using C++ insert data, and this will generate a data file. Then are you able to query this data file from Java (Spark).
Just want to figure out which component causes this error.

[PingLiuPing]

@Yuhta can you help review again? Thank you very much in advance.

[PingLiuPing]

The code change in Prestissimo will not be able get merged until velox PR is merged. To prevent the build break in other CI pipelines revert the cmake target name change in hive/iceberg/CMakeLists.txt first.

[jinchengchenghh]

Hi, @Yuhta @mbasmanova Could you help review this PR? This PR has been integrated with Gluten and Presto, and pass several unit tests back port from Apache Iceberg, apache/incubator-gluten#9397, I fallback the partition table because this PR only supports identity partitioning and lacks for metadata which causes some tests failed. The following prs will support all partition transforms, functions and metadata, after that, I will enable all the test.

Willing to see your reply, much thanks!

[jinchengchenghh]

Could you split to unpartitioned table support only?

[PingLiuPing]

Could you split to unpartitioned table support only?

Thank you, this is possible. And I'm also thinking to separate the partition name and commit message.

[PingLiuPing]

@mbasmanova @jinchengchenghh Thank you very much for the review comments, I split the PR into multiple smaller PRs. The first one is #14723. In following PRs, I will add

1. Iceberg partition spec (new files).
2. Add new option for Timestamp::tmToStringView to meet Iceberg spec (completely standalone code).
3. Customise iceberg writer options.
4. Iceberg file name generator.
5. Iceberg partition ID generator (new files).
6. Identity partition transform.