Core: ValidationException on metadata-only delete of certain delete files #4304
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…only delete with presence of certain delete files
szehon-ho
force-pushed
the
delete_metadata_bug
branch
from
b401647 to
b213236
Compare
core/src/main/java/org/apache/iceberg/ManifestFilterManager.java
Outdated
Show resolved
Hide resolved
spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java
Show resolved
Hide resolved
spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java
Show resolved
Hide resolved
szehon-ho
force-pushed
the
delete_metadata_bug
branch
from
c07770e to
c82bae4
Compare
Member
Author
|
@RussellSpitzer thanks for review, unnested the new case (though it seems still breaks the "cyclomatic complexity" sadly). Also added the new test asserts for metadata delete. |
Member
Yeah to fix that the code needs less "exits" I just thought it reads easier |
szehon-ho
changed the title
szehon-ho
changed the title
RussellSpitzer
approved these changes
Mar 15, 2022
Member
Author
|
Thanks @RussellSpitzer for review, might look at reducing the method's cyclomatic complexity in a follow up if its possible |
Member
Author
|
Note: this fix does not fix all the issues without #4370 |
9 tasks
sunchao
pushed a commit
to sunchao/iceberg
that referenced
this pull request
May 9, 2023
…te files (apache#4304) (cherry picked from commit b64aa0d)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The presence of some V2 delete files lead to following error for a subsequent delete that tries to use metadata.
Specifically, these are delete files that cannot be removed by the file-metrics alone, such as if all values of the column of the delete statement's expression are null (as the test demonstrates).
This because there is a mismatch. The optimizer chooses metadata-only delete if all DataFiles can be deleted using metadata, and does not check if all DeleteFiles can be removed only by metadata: https://github.com/apache/iceberg/blob/apache-iceberg-0.13.1/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java#L241
But the actual metadata-delete logic in ManifestFIlterManager tries to delete even DeleteFiles, and it throws ValidationException when it finds a DeleteFile that cannot be cleanlly deleted because the metrics-bound do not match or are not available.
This change attempts to reconcile these two, by having the ManifestFilterManager skip DeleteFiles that do not have tight enough metrics to be safely removed. These will get cleaned up eventually by sequenceNumber expiration, or later if we build another action to more aggressively drop dangling DeleteFiles (there are already instances today where they are left behind, beyond this).
The other options I considered:
These may be cleaner, but the first one will be a huge performance hit as it means we skip the metadata-only delete unnecessarily if such a delete file exists.
The second is also too conservative, as 90% of the time the delete files can be removed opportunistically by metadata-only deletes (certain tests actually asserting that they are removed, so it would be an unexpected change).
Hence this more specific check.