HDDS-12561. Reclaimable Rename entry filter for reclaiming renaming entries #8054

swamirishi · 2025-03-11T23:38:22Z

What changes were proposed in this pull request?

Currently rename entries are never reclaimed and these entries can be reclaimed on snapshot deep cleaning by just peeking into the previous snapshot if the object exists either in the directoryTable/FileTable/KeyTable. While performing this operation we should ensure read locks are taken on the previous snapshot and the previous snapshot is not deleted. As part of this patch we implement ReclaimableRenameEntryFilter which extends abstract class ReclaimableFilter and check whether this rename entry corresponding to the object exists in the previous snapshot.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12561

How was this patch tested?

Unit tests

…napshots Change-Id: If1a980e5550dda6775c3af53b14a1eab71206a43

Change-Id: Ia8cd0c367e54c978d7534e7bd3702803c8cdf38e

…ntries Change-Id: Id17a31bcb09f676c120652502fb2fc897973821e

Change-Id: I772783d23bf250ca6d468bf6326a09a49c614eb9

Change-Id: I0a0c2a198870d1254c42fd0d7dcaa0b7b7346ae8

Change-Id: I1c1ad2f536ead10a92caa7eec0b8d891289a0257

Change-Id: Ib70ac6c3feade6dc3c24b34c537aa551bbf31062

Change-Id: I0257e9e9100b2e810cca27450f0ada3aec617914

Change-Id: I5b4eedefd85881933572df1a6c8b2960a2cc9278

Change-Id: I6a5e9ab92c10164288c090a8b748383a9430ee69

hemantk-12 · 2025-04-01T06:11:09Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+import org.rocksdb.RocksDBException;
+
+/**
+ * Test class for ReclaimableDirFilter.


nit: ReclaimableRenameEntryFilter not ReclaimableDirFilter.

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

Change-Id: I411b2d0a6f3fa0551ec2f306317440f8ed7387c0

Change-Id: Ie7575e9b7ccb1eb9c2b6d875e9d88e4005d12026

Change-Id: I53fa063a7268e23870b3a7a6851e9920b81ed241

Change-Id: I298fc6d96335b5cb9c79f46901f0d89dc2829362

Change-Id: I9c80b73b6b72d7214edb3e92ae32799a4308f223

Change-Id: I20a877c63a968d9bf912e12cfd10c7aebc7a25bb

Change-Id: I7f5ab7eccbade1f71515e86c9667ab9c1948189e

Change-Id: I62416787dbc587f393f795766d555ea536b8cda5 # Conflicts: # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestMultiSnapshotLocks.java

Change-Id: I64e291f29c19d5c36bc9a5ef3fcf7f416a1149fa

Change-Id: I3c25eaaad8d8471c96f80113f136d628fa1389c8

aswinshakil · 2025-04-16T15:58:55Z

...r/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableRenameEntryFilter.java

+  }
+
+  @SafeVarargs
+  private final boolean isRenameEntryReclaimable(Table.KeyValue<String, String> renameEntry,


Just to have an understanding on this. Are we just checking if the renamed entry is in the previous snapshot's keyTable and dirTable? Should the check be separate of keys and directories?

We are checking if a rename entry points to an object in the previous snapshot. Since objectId is unique in the system the entry would be either in the directoryTable or fileTable or keyTable

aswinshakil · 2025-04-16T16:21:49Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+    String value = UUID.randomUUID().toString();
+    Table<String, OmKeyInfo> keyTable = getMockedTable(ImmutableMap.of(value, mock(OmKeyInfo.class)));
+    Table<String, OmDirectoryInfo> directoryTable = getFailingMockedTable();
+    testReclaimableRenameEntryFilter(volume, bucket, index, value, keyTable, directoryTable, index == 0);


I'm trying to understand these test suites. What does index here mean and why should it be this condition?

We should only be able to reclaim the entry if the snapshot is the first snapshot in the chain.

also reclaimable if no snapshot at all? index = Math.min(index, actualNumberOfSnapshots);

Change-Id: I4e57e43596a9d90ecb2dad62883b746d8cdf992e

…ne/om/snapshot/filter/TestAbstractReclaimableFilter.java Co-authored-by: Doroszlai, Attila <[email protected]>

Change-Id: I7114755259f35362cc7a5db527663e39e38c8b7a

Change-Id: Ife615fc19c66ec5db1558d8ea930cc45f65f41b6

Change-Id: If5159bfd6cb61e3c8ab00e8322de290d27d41940

Change-Id: I18a97d75efb94cb2aec91fbdb1aef57ff11fb1a7

…ne/om/snapshot/filter/ReclaimableFilter.java Co-authored-by: Doroszlai, Attila <[email protected]>

Change-Id: I864704aa4ef0140eb2d3005795cfeb89c721246d

Change-Id: If60fe12bf3a5bc111c62bc86889027fe291c8baa

Change-Id: I443fd757524cfcd35780f06631f06112e33b0f04

Change-Id: Ic1c1920826f5462cb38d36075cffa498814c96c7

Change-Id: Ie1c9dfa79bde12aba9d8e8cec647025f074a414a

Change-Id: Iaae2379e6289e88abc90771f2af932574d25a1c8 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java

adoroszlai · 2025-05-01T05:41:47Z

Please wait for clean CI run in fork before marking ready for review.

Change-Id: I91138a5797daa758479c188127c66f6bd6d39ee4

Change-Id: I89c1cfe51ac0810127bc80e0ad9ee2dfcdd2dda6

swamirishi · 2025-05-01T12:25:33Z

@aswinshakil @hemantk-12 Can we go ahead and merge this change?

jojochuang · 2025-05-02T19:09:56Z

hadoop-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/OMMetadataManager.java

+  /**
+   * Given renameKey, return the volume, bucket and objectID from the key.
+   */
+  String[] splitRenameKey(String renameKey);


we're getting into the habit of creating helper methods inside OMMetadataManager that are unrelated to OMMetadataManager internals. IMO it (and others) should go into a utility class, like SnapshotUtil

We should not do it as part of a util function as this is completely dependent on how the metadata has been layed out. Rather having it in the snapshotUtil is a bad idea as we are mixing up metadata layout internal stuff outside of this metdata layer.

@jojochuang I don't like the whole idea of Snapshots using MetadataManager directly, right now all of our code is tightly coupled with metadatamanager. Like other flows snapshots are supposed to use KeyManager & BucketManager etc. These set of patches would cleanup these shortcomings in our implementation. I @prashantpogde & @hemantk-12 had discussed this and aligned to eventually get there. If we start using MetadataManager we are completely defeating the purpose of higher level abstractions.

jojochuang · 2025-05-02T19:11:30Z

...r/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableRenameEntryFilter.java

+public class ReclaimableRenameEntryFilter extends ReclaimableFilter<String> {
+
+  public ReclaimableRenameEntryFilter(OzoneManager ozoneManager,
+                                      OmSnapshotManager omSnapshotManager, SnapshotChainManager snapshotChainManager,


extra indentation. our checkstyle rule allow this?

jojochuang · 2025-05-02T19:15:26Z

...r/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableRenameEntryFilter.java

+
+  @Override
+  protected String getVolumeName(Table.KeyValue<String, String> keyValue) throws IOException {
+    return getKeyManager().getMetadataManager().splitRenameKey(keyValue.getKey())[0];


IMO it should be just

Suggested change

return getKeyManager().getMetadataManager().splitRenameKey(keyValue.getKey())[0];

return SnapshotUtil.splitRenameKey(keyValue.getKey())[0];

jojochuang · 2025-05-02T19:20:04Z

...r/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableRenameEntryFilter.java

+      if (previousTable != null) {
+        String prevDbKey = renameEntry.getValue();
+        WithObjectID withObjectID = previousTable.getIfExist(prevDbKey);
+        if (withObjectID != null) {


if the rename entry is found in any of the previous tables, return false because it cannot be reclaimed.

yeah we cannot delete the rename entry

jojochuang · 2025-05-02T19:55:03Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+    index = Math.min(index, actualNumberOfSnapshots);
+    String value = UUID.randomUUID().toString();
+    Table<String, OmKeyInfo> keyTable = getMockedTable(ImmutableMap.of(value, mock(OmKeyInfo.class)));
+    Table<String, OmDirectoryInfo> directoryTable = getFailingMockedTable();


why make directory table throw exception?

We want to ensure keyTable if the entry exists in the keyTable then it shouldn't even call directory table existence just checking if it existed in keyTable then it shouldn't go to directory table

jojochuang · 2025-05-02T19:56:08Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+    String value = UUID.randomUUID().toString();
+    Table<String, OmKeyInfo> keyTable = getMockedTable(Collections.emptyMap());
+    Table<String, OmDirectoryInfo> directoryTable = getFailingMockedTable();
+    testReclaimableRenameEntryFilter(volume, bucket, index, value, keyTable, directoryTable, true);


so always reclaimable if the key is not in the snapshot (key table is empty)?

jojochuang · 2025-05-02T20:00:54Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+    String value = UUID.randomUUID().toString();
+    Table<String, OmKeyInfo> keyTable = getMockedTable(ImmutableMap.of(value, mock(OmKeyInfo.class)));
+    Table<String, OmDirectoryInfo> directoryTable = getFailingMockedTable();
+    testReclaimableRenameEntryFilter(volume, bucket, index, value, keyTable, directoryTable, index == 0);


also reclaimable if no snapshot at all? index = Math.min(index, actualNumberOfSnapshots);

jojochuang · 2025-05-02T20:06:50Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+    String value = UUID.randomUUID().toString();
+    Table<String, OmKeyInfo> keyTable = getMockedTable(Collections.emptyMap());
+    Table<String, OmDirectoryInfo> directoryTable = getMockedTable(Collections.emptyMap());
+    testReclaimableRenameEntryFilter(volume, bucket, index, value, keyTable, directoryTable, true);


always reclaimable if both key table and directory table are empty in a FSO bucket.

jojochuang · 2025-05-02T20:07:50Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+    String value = UUID.randomUUID().toString();
+    Table<String, OmKeyInfo> keyTable = getMockedTable(ImmutableMap.of(value, mock(OmKeyInfo.class)));
+    Table<String, OmDirectoryInfo> directoryTable = getMockedTable(Collections.emptyMap());
+    testReclaimableRenameEntryFilter(volume, bucket, index, value, keyTable, directoryTable, index == 0);


reclaimable if key is in the key table and its the first snapshot (or there's no snapshot) in a FSO bucket.

yeah since there is no previous snapshot

jojochuang · 2025-05-02T20:08:22Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+    String value = UUID.randomUUID().toString();
+    Table<String, OmKeyInfo> keyTable = getMockedTable(Collections.emptyMap());
+    Table<String, OmDirectoryInfo> directoryTable = getMockedTable(ImmutableMap.of(value, mock(OmDirectoryInfo.class)));
+    testReclaimableRenameEntryFilter(volume, bucket, index, value, keyTable, directoryTable, index == 0);


reclaimable if key is in the key table or directory table and its the first snapshot (or there's no snapshot) in a FSO bucket.

jojochuang · 2025-05-02T21:00:35Z

...r/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableRenameEntryFilter.java

+  }
+
+  @Override
+  protected Boolean isReclaimable(Table.KeyValue<String, String> renameEntry) throws IOException {


a comment explaining how it works would be idea.

jojochuang · 2025-05-02T21:10:56Z

...c/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java

+                                                Boolean expectedValue)
+      throws IOException {
+    List<SnapshotInfo> snapshotInfos = getLastSnapshotInfos(volume, bucket, 1, index);
+    SnapshotInfo prevSnapshotInfo = snapshotInfos.get(0);


look at just one nearest previous snapshot.
Could you explain why just look at one snapshot?

We just have to look if a key corresponding to the object is present in the previous snapshot or not. That is what the rename table tracks. It just keeps a note of the reference of the object in the previous snapshot's keyTable/fileTable/directoryTable. If the entry doesn't exist then we can very well delete this entry

smengcl · 2025-05-03T03:50:52Z

...r/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableRenameEntryFilter.java

+
+  @Override
+  protected Boolean isReclaimable(Table.KeyValue<String, String> renameEntry) throws IOException {
+    ReferenceCounted<OmSnapshot> previousSnapshot = getPreviousOmSnapshot(0);


Is the index always supposed to be 0 ?

yeah. There is going to be only 1 snapshot in the previous snapshot chain.

swamirishi · 2025-05-03T04:28:25Z

#8054 (comment)
yeah

Change-Id: Iee91900723476e99f37f850ebda289acb6971187

swamirishi · 2025-05-05T16:07:17Z

If there are no more review comments. Can we merge this @jojochuang ?

jojochuang

let's get it merged and proceed with the next batch of PRs.

swamirishi · 2025-05-05T18:00:24Z

thank you for reviewing the PR @hemantk-12 , @jojochuang @aswinshakil @smengcl

…ntries (apache#8054)

…239-container-reconciliation Commits: 113 commits 8ceb5c3 Revert "HDDS-11232. Spare InfoBucket RPC call for the FileSystem#getFileStatus calls for the general case. (apache#6988)" (apache#8358) 8ecf4a8 HDDS-12993. Use DatanodeID in ReconNodeManager. (apache#8456) 3e09a11 HDDS-13024. HTTP connect to 0.0.0.0 failed (apache#8439) c3cb14f HDDS-12914. Bump awssdk to 2.31.40, test ResumableFileDownload (apache#8455) c6b47e9 HDDS-4677. Document Ozone Ports and Connection End Points (apache#8226) ae0d757 HDDS-13044. Remove DatanodeDetails#getUuid usages from hdds-common/client/container-service (apache#8462) 2bf3f16 HDDS-12568. Implement MiniOzoneCluster.Service for Recon (apache#8452) e59d251 HDDS-13046. Add .vscode to .gitignore (apache#8461) 50e6d61 HDDS-13022. Split up exclusive size tracking for key and directory cleanup in SnapshotInfo (apache#8433) 6db647d HDDS-12966. DBDefinitionFactory should not throw InvalidArnException (apache#8454) fe2875c HDDS-12948. [Snapshot] Increase `ozone.om.fs.snapshot.max.limit` default value to 10k (apache#8376) 215bdc2 HDDS-11904. Fix HealthyPipelineSafeModeRule logic. (apache#8386) dba7557 HDDS-12989. Throw CodecException for the Codec byte[] methods (apache#8444) d9c73f0 HDDS-11298. Support owner field in the S3 listBuckets request (apache#8315) 7a5129d HDDS-12810. Check and reserve space atomically in VolumeChoosingPolicy (apache#8360) d51ccf7 HDDS-13016. Add a getAllNodeCount() method to NodeManager. (apache#8445) 8b27fb9 HDDS-12299. Merge OzoneAclConfig into OmConfig (apache#8383) 96f4f79 HDDS-12501. Remove OMKeyInfo from SST files in backup directory. (apache#8214) 8adb500 HDDS-13021. Make callId unique for each request in AbstractKeyDeletingService (apache#8432) faeadd6 HDDS-13000. [Snapshot] listSnapshotDiffJobs throws NPE. (apache#8418) 7c957cc HDDS-11954. List keys command gets wrong info from BasicOmKeyInfo (apache#8401) 0697383 HDDS-13027. Workaround for GitHub connection failures (apache#8443) 0d5f933 HDDS-12919. Remove redundant FileLock from ChunkWrite (apache#8435) 395892a HDDS-13012. [Snapshot] Add audit logs for listSnapshotDiffJobs, snapshotDiff and cancelSnapshotDiff. (apache#8425) d92f0ed HDDS-12653. Add option in `ozone debug log container list` to filter by Health State (apache#8415) 78f8a02 HDDS-12645. Improve output of `ozone debug replicas chunk-info` (apache#8146) f32470c HDDS-12995. Split ListOptions for better reuse (apache#8413) cdd1b13 HDDS-13019. Bump jline to 3.30.0 (apache#8428) 1651408 HDDS-13018. Bump common-custom-user-data-maven-extension to 2.0.2 (apache#8429) 8f6222e HDDS-12949. Cleanup SCMSafeModeManager (apache#8390) adb385f HDDS-13002. Use DatanodeID in ContainerBalancerTask (apache#8421) 2b2a796 HDDS-12969. Use DatanodeID in XceiverClientGrpc (apache#8430) 3c0bafa HDDS-13001. Use DatanodeID in SCMBlockDeletingService. (apache#8420) 8448bc2 HDDS-12187. Use hadoop-dependency-client as parent, not dependency (apache#8416) 582fd03 HDDS-13015. Bump zstd-jni to 1.5.7-3 (apache#8426) ec12f98 HDDS-12970. Use DatanodeID in Pipeline (apache#8414) 3f48a18 HDDS-12179. Improve Container Log to also capture the Volume Information (apache#7858) 034b497 HDDS-12964. Add minimum required version of Docker Compose to RunningViaDocker page (apache#8385) c0a6b42 HDDS-12947. Add CodecException (apache#8411) 70949c5 HDDS-11603. Reclaimable Key Filter for Snapshots garbage reclaimation (apache#8392) 08283f3 HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time (apache#8347) f47df78 HDDS-12958. [Snapshot] Add ACL check regression tests for snapshot operations. (apache#8419) 1cc8445 HDDS-12207. Unify output of ozone debug replicas verify checks (apache#8248) f087d0b HDDS-12994. Use DatanodeID in ReconSCMDBDefinition. (apache#8417) d6a7723 HDDS-12776. ozone debug CLI command to list all Duplicate open containers (apache#8409) c78aeb0 HDDS-12951. EC: Log when falling back to reconstruction read (apache#8408) 412f22d HDDS-12959. Eliminate hdds-hadoop-dependency-server (apache#8384) df701dc HDDS-12996. Workaround for Docker Compose concurrent map writes (apache#8412) e6daae4 HDDS-12972. Use DatanodeID in ContainerReplica. (apache#8396) c1103ae HDDS-12877. Support StorageClass field in the S3 HeadObject request (apache#8351) 4135384 HDDS-12973. Add javadoc for CompactionNode() and make getCompactionNodeGraph return ConcurrentMap (apache#8395) 4f2e13c HDDS-12954. Do not throw IOException for checksum. (apache#8387) 49b8fbd HDDS-12971. Use DatanodeID in Node2PipelineMap (apache#8403) d2da18f HDDS-12346. Reduce code duplication among TestNSSummaryTask classes (apache#8287) 82b73e3 HDDS-11856. Set DN state machine thread priority higher than command handler thread. (apache#8253) d4f2734 HDDS-12689. Import BOM for AWS SDK, declare dependencies (apache#8406) 4775e76 HDDS-12975. Fix percentage of blocks deleted in grafana dashboard (apache#8398) ac0d696 HDDS-12968. [Recon] Fix column visibility issue in Derby during schema upgrade finalization. (apache#8393) e71dcf6 HDDS-11981. Add annotation for registering feature validator based on a generic version (apache#7603) 254297c HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries (apache#8055) 4f467c8 HDDS-12978. Remove TestMultipartObjectGet (apache#8400) a99f207 HDDS-12967. Skip CommonChunkManagerTestCases.testFinishWrite if fuser cannot be started (apache#8389) d29d76b HDDS-12697. Ozone debug CLI to display details of a single container (apache#8264) 1d1bc88 HDDS-12974. Docker could not parse extra host IP (apache#8397) 7e675d7 HDDS-12053. Make print-log-dag command run locally and offline (apache#8016) d3faab3 HDDS-12561. Reclaimable Rename entry filter for reclaiming renaming entries (apache#8054) af1f98c HDDS-10822. Tool to omit raft log in OM. (apache#8154) 3201ca4 HDDS-12952. Make OmSnapshotManager#snapshotLimitCheck thread-safe and consistent (apache#8381) 522c88d HDDS-12963. Clean up io.grpc dependencies (apache#8382) fa8bd9d HDDS-12916. Support ETag in listObjects response (apache#8356) fdc77db HDDS-12300. Merge OmUpgradeConfig into OmConfig (apache#8378) 623e144 HDDS-12956. Bump vite to 4.5.14 (apache#8375) 8c8eaf1 HDDS-12944. Reduce timeout for integration check (apache#8374) 40d2e00 HDDS-11141. Avoid log flood due due pipeline close in XceiverServerRatis (apache#8325) 9fe1dba HDDS-12942. Init layout version config should not be public (apache#8373) 452e7aa HDDS-12596. OM fs snapshot max limit is not enforced (apache#8377) 8b095d5 HDDS-12795. Rename heartbeat and first election configuration name (apache#8249) bee8164 HDDS-12920. Configure log4j to gzip rolled over service log files (apache#8357) e16a50f HDDS-12934. Split submodule for Freon. (apache#8367) b1e9511 HDDS-12925. Update datanode volume used space on container deletion (apache#8364) 440bc82 Revert "HDDS-12596. OM fs snapshot max limit is not enforced (apache#8157)" f345492 HDDS-12596. OM fs snapshot max limit is not enforced (apache#8157) ee7b1dc HDDS-12901. Introduce EventExecutorMetrics instead of setting the metrics props unsafely (apache#8371) 810e148 HDDS-12939. Remove UnknownPipelineStateException. (apache#8372) 560fcdf HDDS-12728. Add Ozone 2.0.0 to compatibility acceptance tests (apache#8361) 5815a47 HDDS-12933. Remove the table names declared in OmMetadataManagerImpl (apache#8370) ee32fa5 HDDS-12560. Reclaimable Filter for Snaphost Garbage Collections (apache#8053) 45374ea HDDS-12932. Rewrite OMDBDefinition (apache#8362) 5cb6dd8 HDDS-12575. Set default JUnit5 timeout via property (apache#8348) 8efc0cd HDDS-11633. Delete message body too large, causing SCM to fail writing raft log (apache#8354) ac9d9fd HDDS-12915. Intermittent failure in testCreatePipelineThrowErrorWithDataNodeLimit (apache#8359) 86039e8 HDDS-12848. Create new submodule for ozone admin (apache#8292) 2d0f8cb HDDS-12833. Remove the CodecRegistry field from DBStoreBuilder (apache#8327) c71b393 HDDS-12921. UnusedPrivateField violations in tests (apache#8353) 9f3dd01 HDDS-12917. cp: option '--update' doesn't allow an argument (apache#8346) a14b395 HDDS-12922. Use OMDBDefinition in GeneratorOm and FSORepairTool (apache#8355) c8a98d6 HDDS-12892. OM Tagging Request incorrectly sets full path as key name for FSO (apache#8345) ade69e3 HDDS-12649. Include name of volume or bucket in length validation error (apache#8322) c68308d HDDS-12599. Create an ozone debug CLI command to list all the containers based on final state (apache#8282) 319d5a4 HDDS-12773. bad substitution in bats test (apache#8290) 6f5e02a HDDS-12900. (addendum: fix pmd) Use OMDBDefinition in OmMetadataManagerImpl (apache#8337) 2a7000d HDDS-12900. Use OMDBDefinition in OmMetadataManagerImpl (apache#8337) 9c0c66c HDDS-12915. Mark testCreatePipelineThrowErrorWithDataNodeLimit as flaky a73e052 HDDS-12907. Enable FieldDeclarationsShouldBeAtStartOfClass PMD rule (apache#8344) d083f82 HDDS-12905. Move field declarations to start of class in ozone-common (apache#8342) 3f90e1c HDDS-12906. Move field declarations to start of class in ozone-manager module (apache#8343) 403fb97 HDDS-12878. Move field declarations to start of class in tests (apache#8308) 63d5c73 HDDS-12912. Remove deprecated `PipelineManager#closePipeline(Pipeline, boolean)` (apache#8340) cf1fb88 HDDS-12902. Shutdown executor in CloseContainerCommandHandler and ECReconstructionCoordinator (apache#8341) 825ba02 HDDS-9585. Improve import/export log in ContainerLogger (apache#8330) b70d35a HDDS-12889. Enable AppendCharacterWithChar PMD rule (apache#8324) cd308ea HDDS-12904. Move field declarations to start of class in other hdds modules (apache#8336) 4905286 HDDS-12899. Move field declarations to start of class in hdds-server-scm (apache#8332) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerLogger.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplica.java hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java hadoop-ozone/recon-codegen/src/main/java/org/apache/ozone/recon/schema/ContainerSchemaDefinition.java

…g renaming entries (apache#8054) Change-Id: I84c1be708f3589ad2d975292757093f00d81d030

HDDS-12559. Implement Bulk Ozone Locks for taking locks on multiple s…

7327b47

…napshots Change-Id: If1a980e5550dda6775c3af53b14a1eab71206a43

swamirishi requested review from hemantk-12 and prashantpogde March 11, 2025 23:38

swamirishi force-pushed the HDDS-12561 branch from 30caf21 to c79560b Compare March 12, 2025 00:26

swamirishi added 2 commits March 11, 2025 18:27

HDDS-12560. Reclaimable Filter for Snaphost Garbage Collections

bd5b0c6

Change-Id: Ia8cd0c367e54c978d7534e7bd3702803c8cdf38e

HDDS-12561. Reclaimable Rename entry filter for reclaiming renaming e…

d4a2f07

…ntries Change-Id: Id17a31bcb09f676c120652502fb2fc897973821e

swamirishi force-pushed the HDDS-12561 branch from c79560b to d4a2f07 Compare March 12, 2025 01:30

swamirishi added 5 commits March 12, 2025 05:51

HDDS-12560. Mock SnapshotDiffManager construction

f1c85fd

Change-Id: I772783d23bf250ca6d468bf6326a09a49c614eb9

Merge remote-tracking branch 'origin/HDDS-12560' into HEAD

01c52b9

Change-Id: I0a0c2a198870d1254c42fd0d7dcaa0b7b7346ae8

HDDS-12559. Revert unintended change in method signature

43ab7b7

Change-Id: I1c1ad2f536ead10a92caa7eec0b8d891289a0257

Merge remote-tracking branch 'origin/HDDS-12559' into HEAD

51c88f1

Change-Id: Ib70ac6c3feade6dc3c24b34c537aa551bbf31062

Merge remote-tracking branch 'origin/HDDS-12560' into HEAD

d27bd24

Change-Id: I0257e9e9100b2e810cca27450f0ada3aec617914

ivandika3 added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Mar 21, 2025

swamirishi added 2 commits March 31, 2025 18:10

HDDS-12559. Address review comments

b901166

Change-Id: I5b4eedefd85881933572df1a6c8b2960a2cc9278

Merge remote-tracking branch 'apache/master' into HEAD

690eae9

Change-Id: I6a5e9ab92c10164288c090a8b748383a9430ee69

hemantk-12 reviewed Apr 1, 2025

View reviewed changes

swamirishi added 4 commits April 1, 2025 01:04

HDDS-12559. Add javadoc

865f3a5

Change-Id: I411b2d0a6f3fa0551ec2f306317440f8ed7387c0

HDDS-12560. Address review comments

5743edb

Change-Id: Ie7575e9b7ccb1eb9c2b6d875e9d88e4005d12026

HDDS-12559. Move acquireLock to another private function

abcfaff

Change-Id: I53fa063a7268e23870b3a7a6851e9920b81ed241

HDDS-12560. Address review comments

a2127a4

Change-Id: I298fc6d96335b5cb9c79f46901f0d89dc2829362

hemantk-12 mentioned this pull request Apr 1, 2025

HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries #8055

Merged

swamirishi added 6 commits April 2, 2025 00:20

HDDS-12559. Address review comments

9f6d2a0

Change-Id: I9c80b73b6b72d7214edb3e92ae32799a4308f223

HDDS-12560. Address review comments

9d6bae3

Change-Id: I20a877c63a968d9bf912e12cfd10c7aebc7a25bb

Merge remote-tracking branch 'origin/HDDS-12559' into HEAD

7d785ab

Change-Id: I7f5ab7eccbade1f71515e86c9667ab9c1948189e

Merge remote-tracking branch 'apache/master' into HEAD

e1d2317

Change-Id: I62416787dbc587f393f795766d555ea536b8cda5 # Conflicts: # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestMultiSnapshotLocks.java

Merge remote-tracking branch 'origin/HDDS-12560' into HEAD

5eca81d

Change-Id: I64e291f29c19d5c36bc9a5ef3fcf7f416a1149fa

HDDS-12561. Address review comments

c4312b1

Change-Id: I3c25eaaad8d8471c96f80113f136d628fa1389c8

adoroszlai mentioned this pull request Apr 8, 2025

HDDS-11603. Reclaimable Filter for Snapshots garbage reclaimation #7345

Closed

aswinshakil reviewed Apr 16, 2025

View reviewed changes

HDDS-12560. Address review comments

93ba939

Change-Id: I4e57e43596a9d90ecb2dad62883b746d8cdf992e

swamirishi and others added 12 commits April 17, 2025 08:24

Update hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozo…

3935bf8

…ne/om/snapshot/filter/TestAbstractReclaimableFilter.java Co-authored-by: Doroszlai, Attila <[email protected]>

HDDS-12560. Fix method folding

6d638a0

Change-Id: I7114755259f35362cc7a5db527663e39e38c8b7a

Merge remote-tracking branch 'apache/master' into HEAD

681bb3d

Change-Id: Ife615fc19c66ec5db1558d8ea930cc45f65f41b6

HDDS-12560. Remove Checked Function

8fd746c

Change-Id: If5159bfd6cb61e3c8ab00e8322de290d27d41940

Merge remote-tracking branch 'origin/HDDS-12560' into HEAD

b69182e

Change-Id: I18a97d75efb94cb2aec91fbdb1aef57ff11fb1a7

Update hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozo…

d32bcf0

…ne/om/snapshot/filter/ReclaimableFilter.java Co-authored-by: Doroszlai, Attila <[email protected]>

HDDS-12560. Fix compilation issue

f8a2b6e

Change-Id: I864704aa4ef0140eb2d3005795cfeb89c721246d

Merge remote-tracking branch 'origin/HDDS-12560' into HEAD

f323e3e

Change-Id: If60fe12bf3a5bc111c62bc86889027fe291c8baa

HDDS-12560. Address review comments

29a26d2

Change-Id: I443fd757524cfcd35780f06631f06112e33b0f04

HDDS-12560. Fix test case

1ebba73

Change-Id: Ic1c1920826f5462cb38d36075cffa498814c96c7

Merge remote-tracking branch 'origin/HDDS-12560' into HEAD

0010107

Change-Id: Ie1c9dfa79bde12aba9d8e8cec647025f074a414a

Merge remote-tracking branch 'apache/master' into HEAD

9317970

Change-Id: Iaae2379e6289e88abc90771f2af932574d25a1c8 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java

swamirishi marked this pull request as ready for review May 1, 2025 05:38

swamirishi added 2 commits May 1, 2025 01:49

HDDS-12561. Fix test case

52b8864

Change-Id: I91138a5797daa758479c188127c66f6bd6d39ee4

HDDS-12561. fix checkstyle

6436166

Change-Id: I89c1cfe51ac0810127bc80e0ad9ee2dfcdd2dda6

jojochuang reviewed May 2, 2025

View reviewed changes

smengcl reviewed May 3, 2025

View reviewed changes

HDDS-12561. Add commment

591eceb

Change-Id: Iee91900723476e99f37f850ebda289acb6971187

jojochuang approved these changes May 5, 2025

View reviewed changes

jojochuang merged commit d3faab3 into apache:master May 5, 2025
42 checks passed

vtutrinov pushed a commit to vtutrinov/ozone that referenced this pull request May 13, 2025

HDDS-12561. Reclaimable Rename entry filter for reclaiming renaming e…

9175ae1

…ntries (apache#8054)

swamirishi added a commit to swamirishi/ozone that referenced this pull request Dec 3, 2025

CDPD-80569. HDDS-12561. Reclaimable Rename entry filter for reclaimin…

f9906ea

…g renaming entries (apache#8054) Change-Id: I84c1be708f3589ad2d975292757093f00d81d030

	return getKeyManager().getMetadataManager().splitRenameKey(keyValue.getKey())[0];
	return SnapshotUtil.splitRenameKey(keyValue.getKey())[0];

HDDS-12561. Reclaimable Rename entry filter for reclaiming renaming entries #8054

HDDS-12561. Reclaimable Rename entry filter for reclaiming renaming entries #8054

Uh oh!

Conversation

swamirishi commented Mar 11, 2025

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented May 1, 2025

Uh oh!

swamirishi commented May 1, 2025

Uh oh!

jojochuang May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swamirishi May 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang May 2, 2025 •

edited

Loading

swamirishi May 3, 2025 •

edited

Loading