Skip to content

Spark 3.2: Cleanup DataFrame overwritePartitions isolation level handling #4310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
szehon-ho merged 1 commit into from
Mar 14, 2022
Merged

Spark 3.2: Cleanup DataFrame overwritePartitions isolation level handling #4310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions ...-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestConflictValidation.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,7 @@ public void testOverwritePartitionSerializableIsolation() throws Exception {
Table table = validationCatalog.loadTable(tableIdent);
final long snapshotId = table.currentSnapshot().snapshotId();

List<SimpleRecord> records = Lists.newArrayList(
new SimpleRecord(1, "a"));
List<SimpleRecord> records = Lists.newArrayList(new SimpleRecord(1, "a"));
spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();

// Validating from previous snapshot finds conflicts
Expand Down Expand Up @@ -165,6 +164,22 @@ public void testOverwritePartitionSnapshotIsolation2() throws Exception {
.overwritePartitions();
}

@Test
public void testOverwritePartitionSnapshotIsolation3() throws Exception {
Table table = validationCatalog.loadTable(tableIdent);
final long snapshotId = table.currentSnapshot().snapshotId();

List<SimpleRecord> records = Lists.newArrayList(new SimpleRecord(1, "a"));
spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();

// Validation should not find conflicting data file in snapshot isolation mode
Dataset<Row> conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
conflictingDf.writeTo(tableName)
.option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
.option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
.overwritePartitions();
}

@Test
public void testOverwritePartitionNoSnapshotIdValidation() throws Exception {
Table table = validationCatalog.loadTable(tableIdent);
Expand All @@ -173,7 +188,7 @@ public void testOverwritePartitionNoSnapshotIdValidation() throws Exception {
new SimpleRecord(1, "a"));
spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();

// Validating from previous snapshot finds conflicts
// Validating from null snapshot is equivalent to validating from beginning
Dataset<Row> conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
AssertHelpers.assertThrowsCause("Conflicting deleted data files should throw exception",
ValidationException.class,
Expand Down
10 changes: 4 additions & 6 deletions spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,17 +277,15 @@ public void commit(WriterCommitMessage[] messages) {
IsolationLevel isolationLevel = writeConf.isolationLevel();
Long validateFromSnapshotId = writeConf.validateFromSnapshotId();

if (isolationLevel != null && validateFromSnapshotId != null) {
dynamicOverwrite.validateFromSnapshot(validateFromSnapshotId);
}

if (isolationLevel == SERIALIZABLE) {
if (validateFromSnapshotId != null) {
dynamicOverwrite.validateFromSnapshot(validateFromSnapshotId);
}
dynamicOverwrite.validateNoConflictingData();

@kbendick kbendick Mar 13, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: It seems like this one is repeated in both cases. Since there's no existing else clause, it's probably not worth reducing the duplication as we might need more if statements but thought I'd call it out.

@szehon-ho szehon-ho Mar 14, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks yea good catch, I was thinking to leave it here to make it more clear in each validation mode what we are validating, and maybe there are future modes where we do not do this as you said. As opposed to the validationSnapshotId which is the basis of all validations.

dynamicOverwrite.validateNoConflictingDeletes();

} else if (isolationLevel == SNAPSHOT) {
if (validateFromSnapshotId != null) {
dynamicOverwrite.validateFromSnapshot(validateFromSnapshotId);
}
dynamicOverwrite.validateNoConflictingDeletes();
}

Expand Down