Skip to content

[HUDI-3735] TestHoodieSparkMergeOnReadTableRollback is flaky #5874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nsivabalan merged 2 commits into from
Jun 24, 2022
Merged

[HUDI-3735] TestHoodieSparkMergeOnReadTableRollback is flaky #5874

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d0f3612
[HUDI-3735] TestHoodieSparkMergeOnReadTableRollback is flaky
xicm Jun 15, 2022
e43fe0a
Merge branch 'master' into HUDI-3735
xicm Jun 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions ...c/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableRollback.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,8 +227,8 @@ void testRollbackWithDeltaAndCompactionCommit(boolean rollbackUsingMarkers) thro
allFiles = listAllBaseFilesInPath(hoodieTable);
// After rollback, there should be no base file with the failed commit time
List<String> remainingFiles = Arrays.stream(allFiles).filter(file -> file.getPath().getName()
.contains(commitTime1)).map(fileStatus -> fileStatus.getPath().toString()).collect(Collectors.toList());
assertEquals(0, remainingFiles.size(), "There files should have been rolled-back "
.contains("_" + commitTime1)).map(fileStatus -> fileStatus.getPath().toString()).collect(Collectors.toList());
assertEquals(0, remainingFiles.size(), "These files should have been rolled-back "
+ "when rolling back commit " + commitTime1 + " but are still remaining. Files: " + remainingFiles);
inputPaths = tableView.getLatestBaseFiles()
.map(baseFile -> new Path(baseFile.getPath()).getParent().toString())
Expand All @@ -240,7 +240,7 @@ void testRollbackWithDeltaAndCompactionCommit(boolean rollbackUsingMarkers) thro
/*
* Write 3 (inserts + updates - testing successful delta commit)
*/
final String commitTime2 = "003";
final String commitTime2 = "000000003";
try (SparkRDDWriteClient thirdClient = getHoodieWriteClient(getHoodieWriteConfigWithSmallFileHandlingOff(true));) {
thirdClient.startCommitWithTime(commitTime2);

Expand All @@ -265,8 +265,10 @@ void testRollbackWithDeltaAndCompactionCommit(boolean rollbackUsingMarkers) thro
thirdClient.rollback(commitTime2);
allFiles = listAllBaseFilesInPath(hoodieTable);
// After rollback, there should be no base file with the failed commit time
assertEquals(0, Arrays.stream(allFiles)
.filter(file -> file.getPath().getName().contains(commitTime2)).count());
List<String> remainingFiles = Arrays.stream(allFiles).filter(file -> file.getPath().getName()
.contains("_" + commitTime2)).map(fileStatus -> fileStatus.getPath().toString()).collect(Collectors.toList());
assertEquals(0, remainingFiles.size(), "These files should have been rolled-back "
+ "when rolling back commit " + commitTime2 + " but are still remaining. Files: " + remainingFiles);

metaClient = HoodieTableMetaClient.reload(metaClient);
hoodieTable = HoodieSparkTable.create(cfg, context(), metaClient);
Expand All @@ -282,7 +284,7 @@ void testRollbackWithDeltaAndCompactionCommit(boolean rollbackUsingMarkers) thro
/*
* Write 4 (updates)
*/
newCommitTime = "004";
newCommitTime = "000000004";
thirdClient.startCommitWithTime(newCommitTime);

writeStatusJavaRDD = thirdClient.upsert(writeRecords, newCommitTime);
Expand Down