-
Notifications
You must be signed in to change notification settings - Fork 3k
Spark 3.4: Create non-existing Tag/Branch when using CREATE OR REPLACE #8086
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spark 3.4: Create non-existing Tag/Branch when using CREATE OR REPLACE #8086
Conversation
| } | ||
|
|
||
| @Test | ||
| public void replaceBranch() throws NoSuchTableException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there was a test gap for branches where a pure replace and a replace on a non-existing branch wasn't exercised, therefore I added replaceBranch() / replaceBranchDoesNotExist() / createOrReplace() (similar tests already exist in TestTagDDL)
...src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceBranchExec.scala
Outdated
Show resolved
Hide resolved
e9e31d0 to
b501628
Compare
|
|
||
| val manageSnapshot = iceberg.table.manageSnapshots() | ||
| if (!replace) { | ||
| if (create && replace && null == iceberg.table().refs().get(tag)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a little hard to reason through, what about simplifying to something more like:
val exists = null == iceberg.table().refs().get(tag)
if (create && !exists) {
if (exists && ifNotExists) {
return Nil
}
manageSnapshot.createTag(tag, snapshotId)
} else if (replace && exists) {
manageSnapshot.replaceTag(tag, snapshotId)
}There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that will change the expected behavior unfortunaly. Tests are expecting ALTER TABLE x REPLACE TAG non-existing to throw an exception, therefore we need to distinguish whether CREATE OR REPLACE is used here vs REPLACE
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would removing the exists variable from the second if statement achieve that? Then it would always call replaceTag if it's not a create and result in the expected throw?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, this would restore the behavior I mentioned above, but it would change a different behavior where ALTER TABLE x CREATE TAG t is expected to fail when t already exists. The conditions are quite tricky unfortunately and from what I've seen is that we need to explicitly handle create && replace && !refExists
fe5a55f to
763dfcd
Compare
Currently, executing `ALTER TABLE x CREATE OR REPLACE TAG xyz` will fail with `Tag does not exist: xyz`. As a user I'd expect this to create the tag due to the `CREATE OR REPLACE` usage. The same issue happens with branches.
763dfcd to
bc2addc
Compare
This is based on apache#7097, and back-ports bug fixes apache#7652 and apache#8086
Currently, executing
ALTER TABLE x CREATE OR REPLACE TAG xyzwill fail withTag does not exist: xyz.As a user I'd expect this to create the tag due to the
CREATE OR REPLACEusage. The same issue happens with branches.