Skip to content

HBASE-29604 BackupHFileCleaner uses flawed time based check #7360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ndimiduk merged 1 commit into from
Oct 10, 2025
Merged

HBASE-29604 BackupHFileCleaner uses flawed time based check #7360

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/BackupHFileCleaner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,13 @@ public class BackupHFileCleaner extends BaseHFileCleanerDelegate implements Abor
private boolean stopped = false;
private boolean aborted = false;
private Connection connection;
// timestamp of most recent read from backup system table
private long prevReadFromBackupTbl = 0;
// timestamp of 2nd most recent read from backup system table
private long secondPrevReadFromBackupTbl = 0;
// timestamp of most recent completed cleaning run
private volatile long previousCleaningCompletionTimestamp = 0;
Copy link
Member

@ndimiduk ndimiduk Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the first run of the chore during the process lifecycle will always do nothing? Is there something else we can base it's initialization value on?

Copy link
Contributor Author

@DieterDP-ng DieterDP-ng Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. That was also the (intended, but not actual) behavior of this cleaner before this PR I think.

I guess we can just initialize this to X time before the current time. Could pick X rudimentary, I think a minute is sufficient, or we could set X to the interval in which the cleaner runs, if I trace the code correctly, that would be the value of hbase.master.cleaner.interval.


@Override
public void postClean() {
Copy link

@krconv krconv Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that this definitely looks like an oversight from the original implementation, and that this change corrects it to work as it seems it was originally intended

previousCleaningCompletionTimestamp = EnvironmentEdgeManager.currentTime();
}

@Override
public Iterable<FileStatus> getDeletableFiles(Iterable<FileStatus> files) {
Expand All @@ -79,12 +82,12 @@ public Iterable<FileStatus> getDeletableFiles(Iterable<FileStatus> files) {
return Collections.emptyList();
}

secondPrevReadFromBackupTbl = prevReadFromBackupTbl;
prevReadFromBackupTbl = EnvironmentEdgeManager.currentTime();
// Pin the threshold, we don't want the result to change depending on evaluation time.
final long recentFileThreshold = previousCleaningCompletionTimestamp;

return Iterables.filter(files, file -> {
// If the file is recent, be conservative and wait for one more scan of the bulk loads
if (file.getModificationTime() > secondPrevReadFromBackupTbl) {
if (file.getModificationTime() > recentFileThreshold) {
LOG.debug("Preventing deletion due to timestamp: {}", file.getPath().toString());
return false;
}
Expand Down
13 changes: 10 additions & 3 deletions hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupHFileCleaner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,11 +108,11 @@ protected Set<TableName> fetchFullyBackedUpTables(BackupSystemTable tbl) {
Iterable<FileStatus> deletable;

// The first call will not allow any deletions because of the timestamp mechanism.
deletable = cleaner.getDeletableFiles(List.of(file1, file1Archived, file2, file3));
deletable = callCleaner(cleaner, List.of(file1, file1Archived, file2, file3));
assertEquals(Set.of(), Sets.newHashSet(deletable));

// No bulk loads registered, so all files can be deleted.
deletable = cleaner.getDeletableFiles(List.of(file1, file1Archived, file2, file3));
deletable = callCleaner(cleaner, List.of(file1, file1Archived, file2, file3));
assertEquals(Set.of(file1, file1Archived, file2, file3), Sets.newHashSet(deletable));

// Register some bulk loads.
Expand All @@ -125,10 +125,17 @@ protected Set<TableName> fetchFullyBackedUpTables(BackupSystemTable tbl) {
}

// File 1 can no longer be deleted, because it is registered as a bulk load.
deletable = cleaner.getDeletableFiles(List.of(file1, file1Archived, file2, file3));
deletable = callCleaner(cleaner, List.of(file1, file1Archived, file2, file3));
assertEquals(Set.of(file2, file3), Sets.newHashSet(deletable));
}

private Iterable<FileStatus> callCleaner(BackupHFileCleaner cleaner, Iterable<FileStatus> files) {
cleaner.preClean();
Iterable<FileStatus> deletable = cleaner.getDeletableFiles(files);
cleaner.postClean();
return deletable;
}

private FileStatus createFile(String fileName) throws IOException {
Path file = new Path(root, fileName);
fs.createNewFile(file);
Expand Down
4 changes: 4 additions & 0 deletions ...-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ public void init(Map<String, Object> params) {

/**
* Should the master delete the file or keep it?
* <p>
* This method can be called concurrently by multiple threads. Implementations must be thread
* safe.
* </p>
* @param fStat file status of the file to check
* @return <tt>true</tt> if the file is deletable, <tt>false</tt> if not
*/
Expand Down
4 changes: 4 additions & 0 deletions hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ public interface FileCleanerDelegate extends Configurable, Stoppable {

/**
* Determines which of the given files are safe to delete
* <p>
* This method can be called concurrently by multiple threads. Implementations must be thread
* safe.
* </p>
* @param files files to check for deletion
* @return files that are ok to delete according to this cleaner
*/
Expand Down