Spark 3.4: Allow write mode (copy-on-write/merge-on-read) to be specified in SQLConf

[wypoon]

Currently, the write delete/update/merge mode for a table (copy-on-write or merge-on-read) is specified as a table property in the table. For some use cases, it makes sense to want to use copy-on-write for some Spark jobs writing to a table and merge-on-read for other jobs writing to the same table. Constantly toggling the table property is not a usable workaround.

In this change, we allow the mode to be specified in Spark's SQLConf. If it is specified in the SQLConf, that setting overrides what is set in the corresponding table property.

[pan3793]

I opened another PR also aims to expose some configuration to SQL conf.

Seems that Iceberg only allows a small set of configurations to be overwritten by SQL session configuration. I'm wondering if we can expose all configurations to SQL conf.

[wypoon]

@pan3793 I think it would be on a case-by-case basis depending on the use case for each configuration.

cc: @rdblue @aokolnychyi @RussellSpitzer @szehon-ho

[manuzhang]

I think read configs like read.split.* should also be configurable in SQL session. @wypoon can you initiate a discussion on dev list?

[wypoon]

I think read configs like read.split.* should also be configurable in SQL session. @wypoon can you initiate a discussion on dev list?

@manuzhang you can add your voice to the thread I already started on the dev list.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.