[Feature Proposal] support setting custom metadata location prefix from catalog level

[puchengy]

Currently, we have write.metadata.location table property to customize metadata location for each table.

However, it is not possible to make all table metadata write to a common prefix. This feature is desirable as the platform maintainer would want to put all metadata into a metadata specific bucket and implement some lock down mechanism to protect the metadata.

Methodology proposal:

1. have a MetadataLocationProvider (similar to LocationProvider)
2. have a catalog property to specify a config name metadata.prefix and it will used by this MetadataLocationProvider.

[singhpk234]

can this feature come in handy to your use-case :

• Allow table defaults to be configured and/ or enforced at catalog level using catalog properties. #4011

[puchengy]

@singhpk234 Thanks for sharing. Yes, I think this PR will provide some help to my user-case to certain extent, but can not fully fulfill my goal.

[puchengy]

@rdblue Hi I want to share some findings after trying to figure out the way to support this based on today's Iceberg community sync https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit# So to recap, we agreed that:

• we want to deprecate metadataFileLocation method in TableOperations interface
• we want to provide a new method in LocationProvider interface (maybe call newMetadataLocation)

I did some investigations and it seems doesn't work because the implementations of metadataFileLocation function in catalogs' table operation are different, for example:

• BaseMetastoreTableOperations & RestTableOpeartions use custom metadata location if set, otherwise fall back to table_location/metadata/filename.
• However, (1) HadoopTableOperations use its own way to define metadata location; (2) StaticTableOperations on the other hand will throw exceptions.
• Because of above, it seems not straightforward to me to add newMetadataLocation function in LocationProvider interface and deprecate metadataFileLocation function. Because we either need to change HadoopTableOperations and StaticTableOperations metadata location behavior or we need to somehow let the LocationProvider to be aware of different type of table operations so that we can return the location providers properly.

Given the way we discussed doesn't seem working to me, I have this alternative option: add newMetadataLocation function in the LocationProvider interface and let it throw unimplemented exception by default. Then we still keep the metadataFileLocation function, so that we can first try to run newMetadataLocation and catch unimplemented exception to execute fallback logic.

Could you share your thoughts if you have better way to achieve what we discussed this morning? or how do you think about my alternative solution, thanks.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'