HDDS-11603. Reclaimable Key Filter for Snapshots garbage reclaimation #8392
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change-Id: I34b91e11970c9cd785ba863dd22a4081244cede6
Change-Id: I5dbb017343b5a666c56d53af2839dcb340fdf0ba
Change-Id: I67de73438c7ebc37c3d6e44bfe6e09188fb537e1
Change-Id: Ie80cd2191504e3a5ebd1cb21d04d300dc0c22ef9
Change-Id: If027d4ac4c5171ded3499b900909347d59406f36
Change-Id: I7fe6d4029abe5917e20bb82ae1b2494e2319da70
Change-Id: Ie7c8950b6511919a195cc31b677286621df4a198 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotDiffManager.java
Change-Id: I7409c5f4b3ce256cdccfdf80aec0e439fc3fa48a
Change-Id: Icd0bb1a62e0435f4329cd1c0abdab52dce13827e
Change-Id: Ib8617005bbe1ac29835db8dcd6a27909cf08e57f
Change-Id: I8bab35cc6dfef7fcac5e69f806904049ff584e32
Change-Id: I81496bcab937444954a0c50d2790812c5412c14d
Change-Id: I2198239de160d7868758f6e06dc9658de10017ab # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java
Change-Id: Ifc9ccb406af0aaf8f1282dfb2d612383dca102fb
Change-Id: I47390045a950428c29cc13469bc9639875b5f2e0
Change-Id: Id38661072466b08ab24140f13f3310f480711bd0
Change-Id: Ia1027f55775864c2353a3e67cc56b76cff5f0727
Change-Id: I6e4e5efe70366e45f15361f34914b4f313f8dd1a
Change-Id: If1a2f7670b2492a9d7295607e50f11c3bf92fb2f
Change-Id: I6cae66db1a6f7f07d8a91849d1287567a75de633 # Conflicts: # hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/IOzoneManagerLock.java # hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java # hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/lock/TestOzoneManagerLock.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/MultiSnapshotLocks.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableFilter.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestMultiSnapshotLocks.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableFilter.java
Change-Id: I95079e15e23db210691345404be0b7ee8a576607 # Conflicts: # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableRenameEntryFilter.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableRenameEntryFilter.java
Change-Id: I225cbdb0224fee194fb59560c2d954e8697f7c2f
Change-Id: Id8f2bb8351ca84703cd4c604c1e4b3b655c6ba46 # Conflicts: # hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java # hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java # hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableDirFilter.java
Change-Id: Ic5078558eabbf45c3c08db674c5c63c364c3e004
Change-Id: Ic1a43d8734682a836cc8783b7d9360654172b905
Change-Id: I4c11891d2d3e3fa190e0ba87baf29026c449ccbc
Change-Id: I14cb747155e8db45ad107bfbcf55fdaae6c16006
jojochuang
reviewed
May 7, 2025
Contributor
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
I'd like to continue tomorrow but this is for today.
| !OmSnapshotManager.areSnapshotChangesFlushedToDB(getOzoneManager().getMetadataManager(), snapInfo); | ||
| } | ||
|
|
||
| // TODO: Move this util class. |
| } | ||
| } | ||
|
|
||
| public static boolean isBlockLocationInfoSame(OmKeyInfo prevKeyInfo, |
Contributor
There was a problem hiding this comment.
please add a javadoc explaining what it does.
| OmSnapshotManager omSnapshotManager, SnapshotChainManager snapshotChainManager, | ||
| SnapshotInfo currentSnapshotInfo, KeyManager keyManager, | ||
| IOzoneManagerLock lock) { | ||
| super(ozoneManager, omSnapshotManager, snapshotChainManager, currentSnapshotInfo, keyManager, lock, 2); |
| return keyValue.getValue().getBucketName(); | ||
| } | ||
|
|
||
| @Override |
Contributor
There was a problem hiding this comment.
Suggested change
| @Override | |
| /** | |
| * Determines whether a deleted key entry is reclaimable by checking its presence in prior snapshots. | |
| * | |
| * This method validates the existence of the deleted key in the previous snapshot's key table or file table. | |
| * If the key is not found in the previous snapshot, it is marked as reclaimable. Otherwise, additional checks | |
| * are performed using the previous-to-previous snapshot to confirm its eligibility for reclamation. | |
| * | |
| * @param deletedKeyInfo The key-value pair representing the deleted key information. | |
| * @return {@code true} if the key is reclaimable (not present in prior snapshots), {@code false} otherwise. | |
| * @throws IOException If an error occurs while accessing snapshot data or key information. | |
| */ | |
| @Override |
| * previousToPrevSnapshot - Snapshot which is used to check | ||
| * if key is exclusive to previousSnapshot. | ||
| */ | ||
| private void calculateExclusiveSize(SnapshotInfo previousSnapshotInfo, |
Contributor
There was a problem hiding this comment.
does this fix the exclusive size bug? related: HDDS-10051 https://issues.apache.org/jira/browse/HDDS-11528 https://issues.apache.org/jira/browse/HDDS-11151 https://issues.apache.org/jira/browse/HDDS-9431
Contributor
There was a problem hiding this comment.
if so, shall we add checks in the test to verify exclusive size is expected
Contributor
Author
There was a problem hiding this comment.
No this doesn't fix that issue. Exclusive size calculation will be fixed as part of the key deleting service change
Contributor
Author
There was a problem hiding this comment.
There is a test case already here.
|
|
||
| } | ||
|
|
||
| private Optional<OmKeyInfo> getPreviousSnapshotKey(long volumeId, OmBucketInfo bucketInfo, |
Contributor
There was a problem hiding this comment.
return the OmKeyInfo object in the previous snapshot.
Change-Id: Ib06b31f57fe2b9b570023277d0df1a84f4771857
jojochuang
reviewed
May 7, 2025
Contributor
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
Ok i think this is good. Just a few suggest on javadocs and more test coverage.
| * Compares the block location info of 2 key info. | ||
| * @return true if block locations are same else false. | ||
| */ | ||
| public static boolean isBlockLocationInfoSame(OmKeyInfo prevKeyInfo, |
Contributor
There was a problem hiding this comment.
this is a relatively complex helper method. A test for this helper for various corner cases would be ideal.
Contributor
Author
There was a problem hiding this comment.
Since this change doesn't pertain to this PR as I have simply moved this function to SnapshotUtils I can create a follow up task for adding a test case for this.
Comment on lines
48
to
50
| /** | ||
| * Test class for ReclaimableKeyFilter. | ||
| */ |
Contributor
There was a problem hiding this comment.
Suggested change
| /** | |
| * Test class for ReclaimableKeyFilter. | |
| */ | |
| /** | |
| * Test class for verifying the behavior of the ReclaimableKeyFilter. | |
| * | |
| * <p>The ReclaimableKeyFilter is responsible for determining if a deleted key entry | |
| * can be reclaimed based on its presence in previous snapshots. In essence, if a key | |
| * does not appear in a previous snapshot—or if the snapshot chain's state indicates that | |
| * the key is no longer valid—the key is considered reclaimable. These tests simulate a | |
| * variety of snapshot chain scenarios to validate that: | |
| * <ul> | |
| * <li>Keys are not reclaimed if they exist in prior snapshots.</li> | |
| * <li>Keys are reclaimed if they are removed from the previous snapshot.</li> | |
| * <li>Exclusive size metrics, including data and replicated sizes, are correctly updated | |
| * for snapshots where keys are not reclaimed.</li> | |
| * </ul> | |
| * | |
| * <p>This class extends {@link AbstractReclaimableFilterTest} to leverage common setup, snapshot | |
| * chain preparation, and utility methods. | |
| */ |
Contributor
Author
There was a problem hiding this comment.
done added a javadoc for AbstractReclaimableFilterTest as well.
jojochuang
reviewed
May 7, 2025
| private OmKeyInfo getMockedOmKeyInfo(long objectId) { | ||
| return getMockedOmKeyInfo(objectId, 0, 0); | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| /** | |
| * Tests that a key not present in prior snapshots is not reclaimable. | |
| * | |
| * @param actualNumberOfSnapshots the total number of snapshots in the chain. | |
| * @param index the snapshot chain index used for testing. | |
| * @throws IOException if an I/O error occurs during the test. | |
| * @throws RocksDBException if RocksDB encounters an error. | |
| */ |
| testReclaimableKeyFilter(volume, bucket, index, keyInfo, prevKeyInfo, prevPrevKeyInfo, | ||
| prevKeyInfo == null, size, size); | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| /** | |
| * Tests the filter when the object IDs differ between current and previous snapshot key entries. | |
| * | |
| * @param actualNumberOfSnapshots the total number of snapshots in the chain. | |
| * @param index the snapshot chain index used for testing. | |
| * @throws IOException if an I/O error occurs during the test. | |
| * @throws RocksDBException if RocksDB encounters an error. | |
| */ |
| testReclaimableKeyFilter(volume, bucket, index, keyInfo, prevKeyInfo, prevPrevKeyInfo, | ||
| true, Optional.empty(), Optional.empty()); | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| /** | |
| * Tests the filter behavior when block location information differs between snapshots. | |
| * | |
| * @param actualNumberOfSnapshots the total number of snapshots in the chain. | |
| * @param index the snapshot chain index used for testing. | |
| * @throws IOException if an I/O error occurs during the test. | |
| * @throws RocksDBException if RocksDB encounters an error. | |
| */ |
| testReclaimableKeyFilter(volume, bucket, index, keyInfo, prevKeyInfo, prevPrevKeyInfo, | ||
| true, Optional.empty(), Optional.empty()); | ||
| } | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| /** | |
| * Tests the exclusive size calculation for a non-reclaimable key. | |
| * | |
| * <p>This test verifies that for a key which is not reclaimable, | |
| * the filter correctly updates the exclusive size and exclusive replicated size | |
| * stored in the filter's maps. | |
| * | |
| * @param actualNumberOfSnapshots the total number of snapshots in the chain. | |
| * @param index the snapshot chain index used for testing size calculations. | |
| * @throws IOException if an I/O error occurs during the test. | |
| * @throws RocksDBException if RocksDB encounters an error. | |
| */ |
Change-Id: I44837d815d0c6246adfe6c660d02bf1861add4f0
Change-Id: I733db471146334555ece4089f248b3f1b20b7e27
Contributor
Author
|
@jojochuang Can this be merged? |
jojochuang
approved these changes
May 9, 2025
Contributor
Author
|
Thanks @jojochuang for reviewing the pr |
vtutrinov
pushed a commit
to vtutrinov/ozone
that referenced
this pull request
May 13, 2025
aswinshakil
added a commit
to aswinshakil/ozone
that referenced
this pull request
May 16, 2025
…239-container-reconciliation Commits: 113 commits 8ceb5c3 Revert "HDDS-11232. Spare InfoBucket RPC call for the FileSystem#getFileStatus calls for the general case. (apache#6988)" (apache#8358) 8ecf4a8 HDDS-12993. Use DatanodeID in ReconNodeManager. (apache#8456) 3e09a11 HDDS-13024. HTTP connect to 0.0.0.0 failed (apache#8439) c3cb14f HDDS-12914. Bump awssdk to 2.31.40, test ResumableFileDownload (apache#8455) c6b47e9 HDDS-4677. Document Ozone Ports and Connection End Points (apache#8226) ae0d757 HDDS-13044. Remove DatanodeDetails#getUuid usages from hdds-common/client/container-service (apache#8462) 2bf3f16 HDDS-12568. Implement MiniOzoneCluster.Service for Recon (apache#8452) e59d251 HDDS-13046. Add .vscode to .gitignore (apache#8461) 50e6d61 HDDS-13022. Split up exclusive size tracking for key and directory cleanup in SnapshotInfo (apache#8433) 6db647d HDDS-12966. DBDefinitionFactory should not throw InvalidArnException (apache#8454) fe2875c HDDS-12948. [Snapshot] Increase `ozone.om.fs.snapshot.max.limit` default value to 10k (apache#8376) 215bdc2 HDDS-11904. Fix HealthyPipelineSafeModeRule logic. (apache#8386) dba7557 HDDS-12989. Throw CodecException for the Codec byte[] methods (apache#8444) d9c73f0 HDDS-11298. Support owner field in the S3 listBuckets request (apache#8315) 7a5129d HDDS-12810. Check and reserve space atomically in VolumeChoosingPolicy (apache#8360) d51ccf7 HDDS-13016. Add a getAllNodeCount() method to NodeManager. (apache#8445) 8b27fb9 HDDS-12299. Merge OzoneAclConfig into OmConfig (apache#8383) 96f4f79 HDDS-12501. Remove OMKeyInfo from SST files in backup directory. (apache#8214) 8adb500 HDDS-13021. Make callId unique for each request in AbstractKeyDeletingService (apache#8432) faeadd6 HDDS-13000. [Snapshot] listSnapshotDiffJobs throws NPE. (apache#8418) 7c957cc HDDS-11954. List keys command gets wrong info from BasicOmKeyInfo (apache#8401) 0697383 HDDS-13027. Workaround for GitHub connection failures (apache#8443) 0d5f933 HDDS-12919. Remove redundant FileLock from ChunkWrite (apache#8435) 395892a HDDS-13012. [Snapshot] Add audit logs for listSnapshotDiffJobs, snapshotDiff and cancelSnapshotDiff. (apache#8425) d92f0ed HDDS-12653. Add option in `ozone debug log container list` to filter by Health State (apache#8415) 78f8a02 HDDS-12645. Improve output of `ozone debug replicas chunk-info` (apache#8146) f32470c HDDS-12995. Split ListOptions for better reuse (apache#8413) cdd1b13 HDDS-13019. Bump jline to 3.30.0 (apache#8428) 1651408 HDDS-13018. Bump common-custom-user-data-maven-extension to 2.0.2 (apache#8429) 8f6222e HDDS-12949. Cleanup SCMSafeModeManager (apache#8390) adb385f HDDS-13002. Use DatanodeID in ContainerBalancerTask (apache#8421) 2b2a796 HDDS-12969. Use DatanodeID in XceiverClientGrpc (apache#8430) 3c0bafa HDDS-13001. Use DatanodeID in SCMBlockDeletingService. (apache#8420) 8448bc2 HDDS-12187. Use hadoop-dependency-client as parent, not dependency (apache#8416) 582fd03 HDDS-13015. Bump zstd-jni to 1.5.7-3 (apache#8426) ec12f98 HDDS-12970. Use DatanodeID in Pipeline (apache#8414) 3f48a18 HDDS-12179. Improve Container Log to also capture the Volume Information (apache#7858) 034b497 HDDS-12964. Add minimum required version of Docker Compose to RunningViaDocker page (apache#8385) c0a6b42 HDDS-12947. Add CodecException (apache#8411) 70949c5 HDDS-11603. Reclaimable Key Filter for Snapshots garbage reclaimation (apache#8392) 08283f3 HDDS-12087. TransactionToDNCommitMap too large causes GC to pause for a long time (apache#8347) f47df78 HDDS-12958. [Snapshot] Add ACL check regression tests for snapshot operations. (apache#8419) 1cc8445 HDDS-12207. Unify output of ozone debug replicas verify checks (apache#8248) f087d0b HDDS-12994. Use DatanodeID in ReconSCMDBDefinition. (apache#8417) d6a7723 HDDS-12776. ozone debug CLI command to list all Duplicate open containers (apache#8409) c78aeb0 HDDS-12951. EC: Log when falling back to reconstruction read (apache#8408) 412f22d HDDS-12959. Eliminate hdds-hadoop-dependency-server (apache#8384) df701dc HDDS-12996. Workaround for Docker Compose concurrent map writes (apache#8412) e6daae4 HDDS-12972. Use DatanodeID in ContainerReplica. (apache#8396) c1103ae HDDS-12877. Support StorageClass field in the S3 HeadObject request (apache#8351) 4135384 HDDS-12973. Add javadoc for CompactionNode() and make getCompactionNodeGraph return ConcurrentMap (apache#8395) 4f2e13c HDDS-12954. Do not throw IOException for checksum. (apache#8387) 49b8fbd HDDS-12971. Use DatanodeID in Node2PipelineMap (apache#8403) d2da18f HDDS-12346. Reduce code duplication among TestNSSummaryTask classes (apache#8287) 82b73e3 HDDS-11856. Set DN state machine thread priority higher than command handler thread. (apache#8253) d4f2734 HDDS-12689. Import BOM for AWS SDK, declare dependencies (apache#8406) 4775e76 HDDS-12975. Fix percentage of blocks deleted in grafana dashboard (apache#8398) ac0d696 HDDS-12968. [Recon] Fix column visibility issue in Derby during schema upgrade finalization. (apache#8393) e71dcf6 HDDS-11981. Add annotation for registering feature validator based on a generic version (apache#7603) 254297c HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries (apache#8055) 4f467c8 HDDS-12978. Remove TestMultipartObjectGet (apache#8400) a99f207 HDDS-12967. Skip CommonChunkManagerTestCases.testFinishWrite if fuser cannot be started (apache#8389) d29d76b HDDS-12697. Ozone debug CLI to display details of a single container (apache#8264) 1d1bc88 HDDS-12974. Docker could not parse extra host IP (apache#8397) 7e675d7 HDDS-12053. Make print-log-dag command run locally and offline (apache#8016) d3faab3 HDDS-12561. Reclaimable Rename entry filter for reclaiming renaming entries (apache#8054) af1f98c HDDS-10822. Tool to omit raft log in OM. (apache#8154) 3201ca4 HDDS-12952. Make OmSnapshotManager#snapshotLimitCheck thread-safe and consistent (apache#8381) 522c88d HDDS-12963. Clean up io.grpc dependencies (apache#8382) fa8bd9d HDDS-12916. Support ETag in listObjects response (apache#8356) fdc77db HDDS-12300. Merge OmUpgradeConfig into OmConfig (apache#8378) 623e144 HDDS-12956. Bump vite to 4.5.14 (apache#8375) 8c8eaf1 HDDS-12944. Reduce timeout for integration check (apache#8374) 40d2e00 HDDS-11141. Avoid log flood due due pipeline close in XceiverServerRatis (apache#8325) 9fe1dba HDDS-12942. Init layout version config should not be public (apache#8373) 452e7aa HDDS-12596. OM fs snapshot max limit is not enforced (apache#8377) 8b095d5 HDDS-12795. Rename heartbeat and first election configuration name (apache#8249) bee8164 HDDS-12920. Configure log4j to gzip rolled over service log files (apache#8357) e16a50f HDDS-12934. Split submodule for Freon. (apache#8367) b1e9511 HDDS-12925. Update datanode volume used space on container deletion (apache#8364) 440bc82 Revert "HDDS-12596. OM fs snapshot max limit is not enforced (apache#8157)" f345492 HDDS-12596. OM fs snapshot max limit is not enforced (apache#8157) ee7b1dc HDDS-12901. Introduce EventExecutorMetrics instead of setting the metrics props unsafely (apache#8371) 810e148 HDDS-12939. Remove UnknownPipelineStateException. (apache#8372) 560fcdf HDDS-12728. Add Ozone 2.0.0 to compatibility acceptance tests (apache#8361) 5815a47 HDDS-12933. Remove the table names declared in OmMetadataManagerImpl (apache#8370) ee32fa5 HDDS-12560. Reclaimable Filter for Snaphost Garbage Collections (apache#8053) 45374ea HDDS-12932. Rewrite OMDBDefinition (apache#8362) 5cb6dd8 HDDS-12575. Set default JUnit5 timeout via property (apache#8348) 8efc0cd HDDS-11633. Delete message body too large, causing SCM to fail writing raft log (apache#8354) ac9d9fd HDDS-12915. Intermittent failure in testCreatePipelineThrowErrorWithDataNodeLimit (apache#8359) 86039e8 HDDS-12848. Create new submodule for ozone admin (apache#8292) 2d0f8cb HDDS-12833. Remove the CodecRegistry field from DBStoreBuilder (apache#8327) c71b393 HDDS-12921. UnusedPrivateField violations in tests (apache#8353) 9f3dd01 HDDS-12917. cp: option '--update' doesn't allow an argument (apache#8346) a14b395 HDDS-12922. Use OMDBDefinition in GeneratorOm and FSORepairTool (apache#8355) c8a98d6 HDDS-12892. OM Tagging Request incorrectly sets full path as key name for FSO (apache#8345) ade69e3 HDDS-12649. Include name of volume or bucket in length validation error (apache#8322) c68308d HDDS-12599. Create an ozone debug CLI command to list all the containers based on final state (apache#8282) 319d5a4 HDDS-12773. bad substitution in bats test (apache#8290) 6f5e02a HDDS-12900. (addendum: fix pmd) Use OMDBDefinition in OmMetadataManagerImpl (apache#8337) 2a7000d HDDS-12900. Use OMDBDefinition in OmMetadataManagerImpl (apache#8337) 9c0c66c HDDS-12915. Mark testCreatePipelineThrowErrorWithDataNodeLimit as flaky a73e052 HDDS-12907. Enable FieldDeclarationsShouldBeAtStartOfClass PMD rule (apache#8344) d083f82 HDDS-12905. Move field declarations to start of class in ozone-common (apache#8342) 3f90e1c HDDS-12906. Move field declarations to start of class in ozone-manager module (apache#8343) 403fb97 HDDS-12878. Move field declarations to start of class in tests (apache#8308) 63d5c73 HDDS-12912. Remove deprecated `PipelineManager#closePipeline(Pipeline, boolean)` (apache#8340) cf1fb88 HDDS-12902. Shutdown executor in CloseContainerCommandHandler and ECReconstructionCoordinator (apache#8341) 825ba02 HDDS-9585. Improve import/export log in ContainerLogger (apache#8330) b70d35a HDDS-12889. Enable AppendCharacterWithChar PMD rule (apache#8324) cd308ea HDDS-12904. Move field declarations to start of class in other hdds modules (apache#8336) 4905286 HDDS-12899. Move field declarations to start of class in hdds-server-scm (apache#8332) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ContainerLogger.java hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplica.java hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java hadoop-ozone/recon-codegen/src/main/java/org/apache/ozone/recon/schema/ContainerSchemaDefinition.java
swamirishi
added a commit
to swamirishi/ozone
that referenced
this pull request
Dec 3, 2025
…reclaimation (apache#8392) (cherry picked from commit 70949c5) Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotUtils.java Change-Id: I2db0766b3e0708dbee2f3196fa5d87f5101f10b3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently deleted key entries are reclaimed by just peeking into the previous snapshot without taking locks. Whenever we are processing these deletes locks should be taken before deleting so that these don't work on inconsistent data.
As part of this patch we implement ReclaimableKeyFilter which extends abstract class ReclaimableFilter and check whether this key entry corresponding to the object exists in the previous snapshot's key table/fileTable.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11603
How was this patch tested?
Existing and additional Unit tests