Spark: Session level Iceberg table config defaults

[karuppayya]

An organization may want all iceberg tables created to have certain TBLPROPERTIES.
Currently, such properties have to be specified in every CREATE command. There is high possibilty of it getting missed
There is no way to specify such properties, such that it gets applied to all table creation.
Couple of questions.

1. Should we introduce a mechanism to add some default properties that affects all tables.(Say spark.iceberg,defaults.prop1=value1, all properties with prefix spark.iceberg.defaults will go to TBLPROPERTIES).
2. Should such functionality be introduced in Spark(can be useful to other datasources as well) or in iceberg?
   Thoughts? @aokolnychyi @RussellSpitzer @rdblue

[aokolnychyi]

I think it will be useful. For example, I've frequently seen projects where folks create a number of tables and want to use more or less the same properties across tables within that project. Instead of specifying the same props over and over again, they could set them in their session config template. In another project, they may have a set of different props.

I believe Spark has does that in some cases already. For instance, properties like spark.datasource.ds_name.key are propagated into sources during some operations. I'd vote for a catalog-native solution in Spark 3. That way, it will become a standard way of representing this in Spark.

The session conf can look like this:

spark.sql.catalog.cat_name.table_defaults.key1 = value1
spark.sql.catalog.cat_name.table_defaults.key2 = value2

Then individual property values in create statements will override what is in the session conf.

CREATE TABLE ...
TBLPROPERTIES (
  'k2' 'custom_value'
)

In the future, we can also support read and write options.

// the name is arbitrary
interface SupportsSessionConfigDefaults extends TableCatalog {
    String tablePropertiesPrefix();
}

[RussellSpitzer]

I know we implemented something like this within the Spark Cassandra Connector, where we allowed a user to set any property that the underlying SCC could use at the Catalog level and therefor set it for all operations carried out by that Catalog.

In our use case I didn't have any hope of doing this in Spark itself so we just implemented it in the "Initialize" method of the catalog but it might be nice if Spark supported this as a first class feature as well.

[aokolnychyi]

@holdenk @dongjoon-hyun @viirya @rdblue @sunchao, do you think it is worth trying in Spark? Seems all catalog implementations would benefit from this.

[rdblue]

This sounds useful, kind of like the site configuration in Hadoop. I like Russell's idea to do this at the catalog level, but I'm already a little worried that catalog configuration is going to end up being quite large and difficult to maintain across Flink, Spark, Hive, and others since each has a separate way to configure catalogs.

Would we actually set these values in tables or would we just change the default so that precedence is table config, then administrator config, then hard-coded defaults?

Where should we put the administrator config?

[aokolnychyi]

In the example above, I'd imagine Spark would load defaults from the session config, combine it with properties provided in the create table statement and pass to the catalog implementation. Meaning we would get a map with k1 -> v1 (comes from the session conf) and k2 -> custom_value (comes from the statement directly), which will be persistent in the metadata.

The idea of not persisting props and simply overriding at runtime seems interesting but it will be way harder to implement, I guess.

[kuldeepsinghchauhan]

I am aligned with @rdblue ,
This is a very critical feature, In an actual production scenario where we are planning to roll out the iceberg format for operations and analytics users. we are currently stuck and cannot decide how to set up iceberg table-level properties defaults in a central config file. I know we are talking about setting up this at the spark session level however, it would be best if Iceberg can leverage properties from spark-defaults. conf or default configs of each engine like hive or Trino, presto, etc so that each table that is created using one of these engines can behave as per their default values unless the user decides to override it at session level or table definition level. This feature will also help in admin-level centralized snapshot and metadata maintenance, where the admin can set up the data lake level default values for such properties in one or more central config level files, and users can override as required.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[szehon-ho]

Im curious, is this solved by #4011 ?

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'