Skip to content

HDDS-13009. Background snapshot defrag service #9227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
smengcl wants to merge 26 commits into from
Closed

HDDS-13009. Background snapshot defrag service #9227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
e6c6b02
HDDS-13009. Background snapshot defrag service
smengcl Oct 30, 2025
29379ea
Patch needsDefrag flag from HDDS-13785. The flag wasn't correctly set.
smengcl Oct 30, 2025
8dc2e5d
Update snapshot local metadata after defrag
smengcl Oct 30, 2025
d52c774
Change full defrag impl: create checkpoint then delete, rather than c…
smengcl Oct 31, 2025
6a46b7a
HDDS-13860. RocksDatabase#open leaks column family handles when faili…
smengcl Oct 31, 2025
648aa7e
Use OmMetadataManagerImpl to open defragged DB; snapshotLocalData.set…
smengcl Oct 31, 2025
cf74dfc
Use getDeltaFiles() for incremental defrag. https://github.com/apache…
smengcl Oct 31, 2025
01f38f2
Move const; findbugs; address copilot comments
smengcl Oct 31, 2025
b5de4e3
pmd
smengcl Oct 31, 2025
62fd5f4
Add version suffix to defragged DB dirs; cleanup
smengcl Oct 31, 2025
0c2f696
delete old DB; fix append version suffix to defrag DB dir names
smengcl Oct 31, 2025
aaaf0c0
Merge remote-tracking branch 'asf' into HDDS-13009-defrag-service-reb…
smengcl Oct 31, 2025
4fba9fa
improve error handling
smengcl Nov 1, 2025
6dd3876
Use FileUtils.deleteDirectory for cleanup
smengcl Nov 1, 2025
c07898f
logging
smengcl Nov 1, 2025
b12d41c
throw when failing to ingest SST file
smengcl Nov 1, 2025
108f35d
fix previous defragmented DB existence check
smengcl Nov 1, 2025
aa90ce8
add debug logging in OmMetadataManagerImpl
smengcl Nov 1, 2025
2dc5309
add debug logging in SnapshotCache#invalidate
smengcl Nov 1, 2025
553acdb
throw exception when previous snapshot is not defragged in incrementa…
smengcl Nov 1, 2025
24fbb1c
fix checkpoint debug logging
smengcl Nov 1, 2025
1689158
Invalidate snapshot cache before deleting old DB
smengcl Nov 1, 2025
edc1eb6
Update log message order in OmMetadataManagerImpl
smengcl Nov 1, 2025
dae1162
Update snapshot defrag test configuration and assertions
smengcl Nov 1, 2025
1766882
cleanup
smengcl Nov 1, 2025
76d5bc8
An attempt to fix test error
smengcl Nov 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -521,9 +521,13 @@ public final class OzoneConsts {
public static final String OM_SNAPSHOT_DIR = "db.snapshots";
public static final String OM_SNAPSHOT_CHECKPOINT_DIR = OM_SNAPSHOT_DIR
+ OM_KEY_PREFIX + "checkpointState";
public static final String OM_SNAPSHOT_CHECKPOINT_DEFRAGGED_DIR = "checkpointStateDefragged";
public static final String OM_SNAPSHOT_CHECKPOINT_DEFRAGGED_DIR = OM_SNAPSHOT_DIR
+ OM_KEY_PREFIX + "checkpointStateDefragged";
public static final String TEMP_DIFF_SST_FILES_DIR = OM_SNAPSHOT_CHECKPOINT_DEFRAGGED_DIR
+ OM_KEY_PREFIX + "tempDiffSstFiles";
public static final String OM_SNAPSHOT_DIFF_DIR = OM_SNAPSHOT_DIR
+ OM_KEY_PREFIX + "diffState";
public static final String SNAPSHOT_DEFRAG_VERSION_SUFFIX_PREFIX = "-v";

public static final String OM_SNAPSHOT_INDICATOR = ".snapshot";
public static final String OM_SNAPSHOT_DIFF_DB_NAME = "db.snapdiff";
Expand Down
13 changes: 13 additions & 0 deletions hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import static org.apache.hadoop.ozone.OzoneConsts.DB_COMPACTION_SST_BACKUP_DIR;
import static org.apache.hadoop.ozone.OzoneConsts.OM_CHECKPOINT_DIR;
import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_CHECKPOINT_DEFRAGGED_DIR;
import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_CHECKPOINT_DIR;
import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_DIFF_DIR;
import static org.apache.hadoop.ozone.OzoneConsts.SNAPSHOT_INFO_TABLE;
Expand Down Expand Up @@ -66,6 +67,7 @@ public class RDBStore implements DBStore {
private final RDBCheckpointManager checkPointManager;
private final String checkpointsParentDir;
private final String snapshotsParentDir;
private final String defraggedSnapshotsParentDir;
private final RDBMetrics rdbMetrics;
private final RocksDBCheckpointDiffer rocksDBCheckpointDiffer;

Expand Down Expand Up @@ -133,6 +135,7 @@ public class RDBStore implements DBStore {
if (!createCheckpointDirs) {
checkpointsParentDir = null;
snapshotsParentDir = null;
defraggedSnapshotsParentDir = null;
} else {
Path checkpointsParentDirPath =
Paths.get(dbLocation.getParent(), OM_CHECKPOINT_DIR);
Expand All @@ -143,6 +146,12 @@ public class RDBStore implements DBStore {
Paths.get(dbLocation.getParent(), OM_SNAPSHOT_CHECKPOINT_DIR);
snapshotsParentDir = snapshotsParentDirPath.toString();
Files.createDirectories(snapshotsParentDirPath);

// TODO: Put this behind a feature flag
Copy link

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creating the defraggedSnapshotsParentDir directory unconditionally may not be desired in all environments. As the TODO indicates, this should be behind a feature flag or configuration option to allow administrators to disable defragmentation if needed.

Copilot uses AI. Check for mistakes.
Path defraggedSnapshotsParentDirPath =
Paths.get(dbLocation.getParent(), OM_SNAPSHOT_CHECKPOINT_DEFRAGGED_DIR);
defraggedSnapshotsParentDir = defraggedSnapshotsParentDirPath.toString();
Files.createDirectories(defraggedSnapshotsParentDirPath);
}

if (enableCompactionDag) {
Expand Down Expand Up @@ -193,6 +202,10 @@ public String getSnapshotsParentDir() {
return snapshotsParentDir;
}

public String getDefraggedSnapshotsParentDir() {
return defraggedSnapshotsParentDir;
}

@Override
public RocksDBCheckpointDiffer getRocksDBCheckpointDiffer() {
return rocksDBCheckpointDiffer;
Expand Down
6 changes: 3 additions & 3 deletions hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ public boolean isClosed() {
*
* @see ManagedCheckpoint
*/
final class RocksCheckpoint implements Closeable {
public final class RocksCheckpoint implements Closeable {
private final ManagedCheckpoint checkpoint;

private RocksCheckpoint() {
Expand Down Expand Up @@ -614,7 +614,7 @@ public List<LiveFileMetaData> getLiveFilesMetaData() throws RocksDatabaseExcepti
}
}

RocksCheckpoint createCheckpoint() {
public RocksCheckpoint createCheckpoint() {
return new RocksCheckpoint();
}

Expand Down Expand Up @@ -665,7 +665,7 @@ public Collection<ColumnFamily> getExtraColumnFamilies() {
return Collections.unmodifiableCollection(columnFamilies.values());
}

byte[] get(ColumnFamily family, byte[] key) throws RocksDatabaseException {
public byte[] get(ColumnFamily family, byte[] key) throws RocksDatabaseException {
try (UncheckedAutoCloseable ignored = acquire()) {
return db.get().get(family.getHandle(), key);
} catch (RocksDBException e) {
Expand Down
4 changes: 4 additions & 0 deletions .../integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServlet.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
import static org.apache.hadoop.ozone.OzoneConsts.MULTIPART_FORM_DATA_BOUNDARY;
import static org.apache.hadoop.ozone.OzoneConsts.OM_DB_NAME;
import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_CHECKPOINT_DEFRAGGED_DIR;
import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_DIFF_DIR;
import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_DIR;
import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_INCLUDE_SNAPSHOT_DATA;
Expand Down Expand Up @@ -497,6 +498,9 @@ private void testWriteDbDataToStream() throws Exception {

Set<String> initialFullSet =
getFiles(Paths.get(metaDir.toString(), OM_SNAPSHOT_DIR), metaDirLength);
// TODO: OM bootstrapping is not aware of defrag dir yet. Remove the workaround workaround when implemented
initialFullSet.remove(OM_SNAPSHOT_CHECKPOINT_DEFRAGGED_DIR);

assertThat(finalFullSet).contains(expectedLogStr);
assertThat(finalFullSet).contains(expectedSstStr);
assertEquals(initialFullSet, finalFullSet, "expected snapshot files not found");
Expand Down
Loading