Support iceberg reader for hive catalog and data stored on HDFS

[caneGuy]

Co-authored-by: caneGuy [email protected]
Co-authored-by: Ielihs [email protected]

This is the first PR for #1030 which add support for Iceberg table stored on HDFS and use HiveCatalog for catalog.
Goals:

• basic query ability for Iceberg table
• support hive catalog
• support iceberg data compressed by snappy
• support parquet data format
• support v1 format

Non Goals:

• add iceberg statistic for planner
• support hadoop catalog
• support v2 format
• metadata cache
• support gzip
• write iceberg table

Example:

CREATE DATABASE external_db;
USE external_db;

CREATE EXTERNAL RESOURCE "iceberg0"
PROPERTIES (
  "type" = "iceberg",
  "starrocks.catalog-type"="HIVE",
  "iceberg.catalog.hive.metastore.uris"="thrift://xxx:9083"
);
CREATE EXTERNAL TABLE `iceberg_tbl_snappy` (
  `id` bigint NULL,
  `data` varchar(200) NULL
) ENGINE=ICEBERG
PROPERTIES (
  "resource" = "iceberg0",
  "database" = "iceberg",
  "table" = "iceberg_table_snappy"
);

select * from iceberg_tbl_snappy;

We have run the tpcds queries for correctness check.

[caneGuy]

cc @openinx could you help review the logic related with iceberg？thanks

[openinx]

Thanks for pinging me, @caneGuy . I'd like to take a look when I have a chance.

[caneGuy]

run starrocks_clang-format
run starrocks_be_unittest
run starrocks_fe_unittest
run starrocks_tscannode

[dirtysalt]

run starrocks_fe_unittest

[caneGuy]

run starrocks_clang-format

[dirtysalt]

run starrocks_fe_unittest

[caneGuy]

i have resolved comments from @openinx thanks PTAL

[openinx]

Here we use the default new Configuration() to initialize the HiveCatalog, could we access the iceberg table that backed by a hadoop filesystem ? I rise this question because in my view the HiveCatalog will initialize its filesystem FileIO by using this hadoop configuration. https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java#L88

If we here use a default hadoop configuration, then how could we access the customized hadoop fs ?

[caneGuy]

There is a premise that we must put hadoop conf on classpath like hive table.
We will refactor for this use a common PR for hive external table and iceberg table

[openinx]

Here we don't handle the nested schema correctly , right ? Because I see the nested schema won't be indexed in this icebergColumns map. Maybe you can try the iceberg' TypeUtil#indexByName method to generate the name -> fieldId index.

[openinx]

Looks like the starrocks does not support nested fields, right ?

[caneGuy]

we do not support nested fields in this version @openinx

[caneGuy]

I will submit an other PR for nested schema

[openinx]

I see those tasks are FileScanTask, will starrocks provides any bin-pack algorithm to balance the splits between different parallelism ?

[openinx]

Besides, I'm thinking that we may need to introduce an extra data structure to handle the read process for iceberg v2 table because its split will contains both data file split and delete file split.

[caneGuy]

The statistics here are now useless, I will delete these codes and submit another pr.

[openinx]

The div between task.length() and file.fileSizeInBytes() will always be 0 because it's a long-long division and the task.length() will always be less than file.fileSizeInBytes(). I will suggest to cast the task.length() to a double type.

[caneGuy]

The statistics here are now useless, I will delete these codes and submit another pr.

[openinx]

Seems we currently only support expression like this pattern var op literal , actually there are more complex expression like AND, OR, NOT etc, is there any plan to support in the next version ?

In fact, I don't suggest to add the filter push down in this PR ( Because we are implementing an incomplete filter pushdown in this PR). It's good to focus a feature in one PR.

[openinx]

If plan to add filter push down for starrocks, the iceberg's SparkFilters class is a good example to follow.

[caneGuy]

As discussed with @imay we will submit an other pr to optimize this.
I will keep these codes in this pr for some users to try first thanks @openinx

[imay]

LGTM

[openinx]

Looks good to me overall. Give my +1.

[imay]

LGTM

[Seaven]

Nice work, and I suggest add some UT cases to cover query iceberg table later, like PlanFragmentTest

[caneGuy]

Nice work, and I suggest add some UT cases to cover query iceberg table later, like PlanFragmentTest

i will add some test case in PlanFragmentTest thanks @Seaven

[caneGuy]

run starrocks_be_unittest

[Seaven]

run starrocks_be_unittest