[Iceberg]Support setting warehouse data directory for Hadoop catalog

[hantangwangd]

Description

This PR enable Presto Iceberg Hadoop catalog to specify an independent warehouse data directory to store table data files, in this way, we can manage metadata files on HDFS and store data files on Object Stores in a formal production environment.

See issue: #24383

Motivation and Context

Enabling Presto Iceberg to leverage the powerful capabilities of object storages

Impact

Hadoop catalog has the capability of leveraging object stores

Test Plan

• Build an object storage environment base on MinIO docker, configure iceberg.catalog.warehouse to a local file path, and iceberg.catalog.hadoop.warehouse.datadir to a s3 path, fully run IcebergDistributedTestBase, IcebergDistributedSmokeTestBase, and TestIcebergDistributedQueries in CI test

Contributor checklist

• Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

== RELEASE NOTES ==

Iceberg Connector Changes
* Add table properties ``write.data.path`` to specify independent data write paths for Iceberg tables
* Add connector configuration property ``iceberg.catalog.hadoop.warehouse.datadir`` for Hadoop catalog to specify root data write path for its new created tables

[steveburnett]

Thanks for the doc! Looks good, only a few nits only of punctuation and capitalization.

[steveburnett]

LGTM! (docs)

Pulled updated branch, new local doc build, looks good. Thanks!

[hantangwangd]

Thanks @steveburnett for your suggestion, fixed!

[agrawalreetika]

Thanks for the change. LGTM

[imjalpreet]

@hantangwangd Thanks for the PR! I took a first pass and had some minor comments.

[ZacBlanco]

I just have a few high level comments. At the core, I understand what we're trying to solve, but I'm not sure yet if this is the right solution. What happens when we have a table which already exists at the warehouse directory but has a data directory which doesn't align with the new datadir property? Should we respect the property or the table in the case of an insert? This could be confusing for users.

And if the table already exists and doesn't doesn't have a metadata folder which is in a "safe" filesystem, should we error, or warn the user? Do we even have a way of knowing that a filesystem is "safe" (supports atomic renames)?

Correct me if I'm wrong, but in theory we could have already supported this use case within the iceberg connector by using SetTableProperty procedure and just setting "write.data.path" on the individual tables, right? From my understanding, all this change does is provide a default for new tables and makes it a viewable table property. I'm wondering if it might be better providing this as schema-level property that users can set, similar to how the hive connector has a schema-level "location" property. Then, we can set defaults for the data path on schema creation, but override it on a table-level if we prefer.

However, I don't believe the metadata path could be set that way since the HadoopCatalog relies on listing the warehouse metadata directories to get the schemas and tables.

Just some thoughts for discussion. I think I just want to refine our approach and make sure there isn't any ambiguous behavior from the user perspective.

[hantangwangd]

@ZacBlanco Thanks for your feedback, very glad to be able to clarify and refine the approch through discussion. Overall, I think the core point of this change is to separate the data write path and metadata path of Iceberg tables in Hadoop catalog, and provide a simpler and more convenient way for users to specify the data write path of tables. I will reply item by item below. Please let me know if you have any comments.

What happens when we have a table which already exists at the warehouse directory but has a data directory which doesn't align with the new datadir property? Should we respect the property or the table in the case of an insert?

Undoubtedly, when inserting data, the iceberg table's table property write.data.path will be respected. In general, iceberg.catalog.warehouse.datadir is a Hadoop catalog level configuration property used to define the default data write path on table creating. It will not affect the data write path of existing tables.

• We can explicitly specify the write data path of a table when creating it, and later change the write data path at any time through the SetTableProperty procedure, so the write data path of a table does not need to be aligned with the warehouse data dir.

• We can change the location of the warehouse data dir at any time, which will only affect newly created tables, and will not affect the data write path of existing tables.

• When we want to change the data write path of a table, we should explicitly use the SetTableProperty procedure (which needs to support property write.data.location). And the strategy of the Iceberg table itself is also similar, that is, the existing data files will not be affected by the later modified write.data.location.

Do you think the above behaviors make sense?

And if the table already exists and doesn't doesn't have a metadata folder which is in a "safe" filesystem, should we error, or warn the user? Do we even have a way of knowing that a filesystem is "safe" (supports atomic renames)?

I think we could specify the requirements for the file system where the metadata is located in the document. But I don't think Presto engine is responsible for identifying and distinguishing specific file systems and reminding users. This is the user's own responsibility, because they have a better understanding of their own usage scenarios. For example, for a considerable number of users, they do not need to consider transaction consistency issues caused by cross engine interoperability.

in theory we could have already supported this use case within the iceberg connector by using SetTableProperty procedure and just setting "write.data.path" on the individual tables

SetTableProperty procedure and CreateTable can both specify and set table properties, but their usage scenarios are different. I don't think SetTableProperty can replace CreateTable to specify table properties, its main purpose is to alter table properties. And I think users may also not be able to accept the requirement to execute set properties procedure after creating a table in order to specify the data write path. What do you think?

I'm wondering if it might be better providing this as schema-level property that users can set, similar to how the hive connector has a schema-level "location" property. Then, we can set defaults for the data path on schema creation, but override it on a table-level if we prefer.

Firstly, for Hadoop catalog, we cannot arbitrarily set the value of location which also maintains metadata data like hive. Therefore, we should only set a property like data.location which only contains data files. Secondly, I don't think this schema-level concept you mentioned conflicts with the catalog level data dir. Their roles are similar, just at different level. As described above, they all used to define the default data write path on table creating, they will not affect the data write path of existing tables, and can be overridden on a table-level if we prefer.

Therefore, if necessary, we can add this schema-level data dir property in the future. I think it can work well with catalog-level data dir. What's your opinion?

[steveburnett]

New release note guidelines as of last week: PR #24354 automatically adds links to this PR to the release notes. Please remove the manual PR link in the following format from the release note entries for this PR.

:pr:`12345`

I have updated the Release Notes Guidelines to remove the examples of manually adding the PR link.

[ZacBlanco]

Thanks for your response. I did some more thinking around this and concluded that we don't really have a consistent way of specifying location for each connector. But I think we should strive to do so.

Some examples:

1. In the hive connector, we allow schemas to set the location property. It can also be set through a external_location table property, otherwise it is controlled by the HMS configuration (external to Presto).
2. In the iceberg connector when using the Hive catalog, the location is determined by either the HMS configuration (external to presto), or configured by the user specifying a location property.
3. In the Iceberg REST connector, table location is specified via the catalog-level iceberg.catalog.warehouse property.

My thoughts are that

1. We should definitely provide the ability to set both the iceberg write.data.location and write.metadata.location through the table properties in the CREATE TABLE statements. I think that we should use the iceberg property names directly rather than doing something like location or write_data_location and write_metadata_location.
2. I think providing this configuration in Presto's iceberg catalog configuration for the HadoopCatalog inthis PR is probably fine because the difference with the HMS and Hive connector is that it's an external process with its own configuration. With the hadoop catalog, we don't really have that option since it's all filesystem-based.
   a. Though this makes me wonder whether the RestCatalog should even utilize iceberg.catalog.warehouse as I might expect RestCatalog implementations to have some kind of similar default data and metadata locations. I looked at Nessie's docs and saw that they have the ability to specify default locations. I think we might want to ensure that those configs are respected over any Presto configurations. We may even want to remove the ability to specify default locations for the Presto iceberg catalog when configured with Rest/Nessie. The Hive connector doesn't really allow it, and I think it wouldn't be clear to users as to which configuration is the correct one to respect.

[hantangwangd]

Hi @ZacBlanco @imjalpreet @agrawalreetika @steveburnett, the comments have been addressed, please take a look when convenient, thanks a lot!

[steveburnett]

LGTM! (docs)

Pull updated branch, new local doc build, looks good. Thanks!