Skip to content

HDDS-9636. Fix NoSuchElement in OMSnapshotPurgeRequest and exit loop early in SnapshotDeletingService #5554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aswinshakil merged 2 commits into from
Nov 14, 2023
Merged

HDDS-9636. Fix NoSuchElement in OMSnapshotPurgeRequest and exit loop early in SnapshotDeletingService #5554

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dadfb7d
HDDS-9636. Fix NoSuchElement in OMSnapshotPurgeRequest and exit loop …
aswinshakil Nov 6, 2023
ffa67c3
Merge Master.
aswinshakil Nov 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,6 +405,12 @@ private long handleDirectoryCleanUp(
while (deletedDirIterator.hasNext()) {
Table.KeyValue<String, OmKeyInfo> deletedDir =
deletedDirIterator.next();
String deletedDirKey = deletedDir.getKey();

// Exit for dirs out of snapshot scope.
if (!deletedDirKey.startsWith(dbBucketKeyForDir)) {
Copy link
Contributor

@hemantk-12 hemantk-12 Nov 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use Table#iterator(prefix)?

Copy link
Member Author

@aswinshakil aswinshakil Nov 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not aware this existed. Yes, we can do it, but we need to change this in multiple places and test it. I can do it as a part of a larger separate patch.

break;
}

if (isDirReclaimable(deletedDir, previousDirTable,
renamedTable, renamedList)) {
Expand Down
37 changes: 24 additions & 13 deletions ...-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@

import java.io.File;
import java.io.IOException;
import java.util.NoSuchElementException;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
Expand Down Expand Up @@ -143,23 +144,33 @@ public static void checkSnapshotActive(SnapshotInfo snapInfo,
public static SnapshotInfo getNextActiveSnapshot(SnapshotInfo snapInfo,
SnapshotChainManager chainManager, OmSnapshotManager omSnapshotManager)
throws IOException {
while (chainManager.hasNextPathSnapshot(snapInfo.getSnapshotPath(),
snapInfo.getSnapshotId())) {

UUID nextPathSnapshot =
chainManager.nextPathSnapshot(
snapInfo.getSnapshotPath(), snapInfo.getSnapshotId());
// If the snapshot is deleted in the previous run, then the in-memory
// SnapshotChainManager might throw NoSuchElementException as the snapshot
// is removed in-memory but OMDoubleBuffer has not flushed yet.
try {
while (chainManager.hasNextPathSnapshot(snapInfo.getSnapshotPath(),
snapInfo.getSnapshotId())) {

String tableKey = chainManager.getTableKey(nextPathSnapshot);
SnapshotInfo nextSnapshotInfo =
omSnapshotManager.getSnapshotInfo(tableKey);
UUID nextPathSnapshot =
Copy link
Contributor

@swamirishi swamirishi Nov 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aswinshakil When would getNextActiveSnapshot be called and the current snapshot also deleted in parallel. I guess this should not happen right?

Copy link
Member Author

@aswinshakil aswinshakil Nov 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is called when the current snapshot is deleted. This is to set the deep clean flag.

Copy link
Contributor

@swamirishi swamirishi Nov 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are deleting the current snapshot from the chain then why call this function at the first place, it would return null anyhow right? So do you mean to say we are deleting snapshots which are not deep cleaned? So are we not taking any lock on the portion of the chain.

Copy link
Member Author

@aswinshakil aswinshakil Nov 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #5339, KeyDeletingService runs every 100ms. So even before the 1st request is flushed by the OMDoubleBuffer, The second request(same) comes again. At that time the chain already removed the current deleted snapshot (from the 1st request). When the 2nd request (same as 1st) comes this will throw NoSuchElementException

chainManager.nextPathSnapshot(
snapInfo.getSnapshotPath(), snapInfo.getSnapshotId());

if (nextSnapshotInfo.getSnapshotStatus().equals(
SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE)) {
return nextSnapshotInfo;
}
String tableKey = chainManager.getTableKey(nextPathSnapshot);
SnapshotInfo nextSnapshotInfo =
omSnapshotManager.getSnapshotInfo(tableKey);

snapInfo = nextSnapshotInfo;
if (nextSnapshotInfo.getSnapshotStatus().equals(
SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE)) {
return nextSnapshotInfo;
}

snapInfo = nextSnapshotInfo;
}
} catch (NoSuchElementException ex) {
LOG.error("The snapshot {} is not longer in snapshot chain, It " +
"maybe removed in the previous Snapshot purge request.",
snapInfo.getTableKey());
}
return null;
}
Expand Down