Core: Validate concurrently added delete files in RowDelta #3195
Conversation
| * @param conflictDetectionFilter an expression on rows in the table | ||
| * @return this for method chaining | ||
| */ | ||
| RowDelta conflictDetectionFilter(Expression conflictDetectionFilter); |
There was a problem hiding this comment.
This is actually different from the implementation in PR #3069. I think this one is slightly better as we set the conflict detection filter only once and then enable data/delete file validation.
There was a problem hiding this comment.
Otherwise, we would need to reason about what happens if the filters are different, which one to use for validating referenced data files, etc.
| return (globalDeletes == null || globalDeletes.length == 0) && sortedDeletesByPartition.isEmpty(); | ||
| } | ||
|
|
||
| public List<DeleteFile> referencedDeleteFiles() { |
There was a problem hiding this comment.
Would Iterable<DeleteFile> work instead? That would make this lazy.
| protected void validateNoNewDeleteFiles(TableMetadata base, Long startingSnapshotId, | ||
| Expression dataFilter, boolean caseSensitive) { | ||
| // if there is no current table state, no files have been added | ||
| if (base.currentSnapshot() == null) { |
There was a problem hiding this comment.
Can you add the check for base.formatVersion() < 2?
There was a problem hiding this comment.
Added to both methods. Could you check, @rdblue?
|
Looks good. Thanks for adding that check! |
|
Thanks for reviewing, @rdblue! |
2 participants
This PR validates concurrently added delete files in
BaseRowDelta.