HDDS-7883. [Snapshot] Accommodate FSO, key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService #4407

aswinshakil · 2023-03-16T03:27:20Z

What changes were proposed in this pull request?

To support the following for Snapshot Garbage Collection.

To handle renamed keys between snapshots.
To accommodate FSO bucket keys.
Implement OMSnapshotPurgeRequest cleaning up SnapshotChain, checkpoint directory.
Set limit for number of deletedKeys, similar to KeyDeletingService

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7883

How was this patch tested?

Unit tests.

Change-Id: I6fe3de2e9409757bb2871bfbcbac8abd7e11dd53

…apshotPurgeRequest for SnapshotDeletingService

smengcl · 2023-03-16T19:45:20Z

.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java

              for (OmKeyInfo keyInfo: repeatedOmKeyInfo.getOmKeyInfoList()) {
                splitRepeatedOmKeyInfo(toReclaim, toNextDb,
-                    keyInfo, previousKeyTable);
+                    keyInfo, previousKeyTable, renamedKeyTable,
+                    bucketInfo, volumeId);
              }


It should be fine as long as we are reading volume/bucket info directly from active DB. Since we don't allow deletion of buckets with snapshots in the first place. (And this is the important assumption here.)

Originally I thought we are going to get volume and bucket info from snapshot DB, in which case we would have to retain these two tables in SstFilteringService (which is not the case at the moment).

smengcl · 2023-03-17T07:02:11Z

.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java

+            while (deletedIterator.hasNext() &&
+                deletionCount <= keyLimitPerSnapshot) {


We could add a block comment right above the loop explaining that we are trying to gather as much key blocks as possible in one SCM block deletion request here.

We can add it once we are sending the actual request to SCM

.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java

smengcl · 2023-03-17T07:17:51Z

.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java


      // These are uncommitted blocks wrapped into a pseudo KeyInfo
      if (deletedKeyInfo.getObjectID() == OBJECT_ID_RECLAIM_BLOCKS) {
        return false;


This means we won't reclaim those blocks (e.g. added by Hsync) in deletedTable? while actually the blocks could be deleted if the block ID doesn't match.

This may deserve a TODO here (Add block ID checking). Also, add more comments regarding the desired GC behavior.

Make sure when deferring this entry to the next snapshot or active DB, it is merged properly into the same OmKeyInfo. The blocks cannot be thrown away without reclamation (leaking blocks on SCMs/DNs otherwise).

We will reclaim these blocks on the deletedTable, here these uncommitted blocks are put in reclaimList which will be reclaimed by KeyDeletingService.

.../ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java

smengcl

+1 pending findbugs in test

smengcl · 2023-03-21T19:02:40Z

Thanks @aswinshakil for the patch

* master: (43 commits) HDDS-8148. Improve log for Pipeline creation failure (apache#4385) HDDS-7853. Add support for RemoveSCM in SCMRatisServer. (apache#4358) HDDS-8042. Display certificate issuer in cert list command. (apache#4429) HDDS-8189. [Snapshot] renamedKeyTable should only track keys in buckets that has at least one active snapshot. (apache#4436) HDDS-8154. Perf: Reuse Mac instances in S3 token validation (apache#4433) HDDS-8245. Info log for keyDeletingService when nonzero number of keys are deleted. (apache#4451) HDDS-8233. ReplicationManager: Throttle delete container commands from over-replication handlers (apache#4447) HDDS-8220. [Ozone-Streaming] Trigger volume check on IOException in StreamDataChannelBase (apache#4428) HDDS-8173. Fix to remove enrties from RocksDB after container gets deleted. (apache#4445) HDDS-7975. Rebalance acceptance tests (apache#4437) HDDS-8152. Reduce S3 acceptance test setup time (apache#4393) HDDS-8172. ECUnderReplicationHandler should consider commands already sent when processing the container (apache#4435) HDDS-7883. [Snapshot] Accommodate FSO, key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService (apache#4407) HDDS-8168. Make deadlines inside MoveManager for move commands configurable (apache#4415) HDDS-7918. EC: ECBlockReconstructedStripeInputStream should check for spare replicas before failing an index (apache#4441) HDDS-8222. EndpointBase#getBucket should handle BUCKET_NOT_FOUND (apache#4431) HDDS-8068. Fix Exception: JMXJsonServlet, getting attribute RatisRoles of Hadoop:service=OzoneManager. (apache#4352) HDDS-8139. Datanodes should not drop block delete transactions based on transaction ID (apache#4384) HDDS-8216. EC: OzoneClientConfig is overwritten in ECKeyOutputStream (apache#4425) HDDS-8054. Fix NPE in metrics for failed volume (apache#4340) ...

aswinshakil and others added 21 commits February 2, 2023 23:59

HDDS-7740. [Snapshot] Implement SnapshotDeletingService

d10ba4b

Merge master and fix findbugs.

81cad24

HDDS-7740: Fix tests.

bbd106b

Merge remote-tracking branch 'asf/master' into HDDS-7740

d2e5750

Change-Id: I6fe3de2e9409757bb2871bfbcbac8abd7e11dd53

HDDS-7740: Address review comments.

fb7b7ae

Merge apache/master to HDDS-7740

31de5bc

HDDS-7740. Fix Unit test.

a31bd8c

HDDS-7740. Add Unit test.

2470c11

HDDS-7740. Add suggestion.

a0f58d3

HDDS-7740. Change put to putWithBatch.

c36f0fd

Trigger Build

e79b931

Merge branch 'master' of https://github.com/apache/ozone into HDDS-7740

e79910e

HDDS-7740. Fix UT.

b37e860

HDDS-7740. Address review comments.

0bc1e14

HDDS-7740. Update current snapshot deletedTable.

9ab9373

HDDS-7740. Disable test.

ff78d47

Merge branch 'master' of https://github.com/apache/ozone into HDDS-7740

54c88d5

HDDS-7740. Change variable name and adjust condition.

2144e16

HDDS-7740. Add comment and update variable name.

0d9f773

HDDS-7883. [Snapshot] Accommodate FSO, Key renames and implement OMSn…

5f17166

…apshotPurgeRequest for SnapshotDeletingService

HDDS-7883. Fix findbugs.

8dfc27e

aswinshakil changed the base branch from master to 4440 March 16, 2023 03:27

aswinshakil changed the base branch from 4440 to master March 16, 2023 03:27

Merge master

090f428

aswinshakil force-pushed the HDDS-7883 branch from 2780348 to 090f428 Compare March 16, 2023 03:36

Merge master

006b3cc

smengcl added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Mar 16, 2023

smengcl reviewed Mar 16, 2023

View reviewed changes

smengcl mentioned this pull request Mar 16, 2023

HDDS-8122. [Snapshot] Disable RocksDB column family auto compaction in SstFilteringService #4400

Merged

HDDS-7883. Remove unused declaration.

f8180d6

smengcl reviewed Mar 17, 2023

View reviewed changes

HDDS-7883. Add test and address comments.

fa1118d

smengcl approved these changes Mar 21, 2023

View reviewed changes

Fix Findbugs

eeaa9f7

smengcl changed the title ~~HDDS-7883. [Snapshot] Accommodate FSO, Key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService~~ HDDS-7883. [Snapshot] Accommodate FSO, key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService Mar 21, 2023

smengcl merged commit 184bfc2 into apache:master Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-7883. [Snapshot] Accommodate FSO, key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService #4407

HDDS-7883. [Snapshot] Accommodate FSO, key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService #4407

Uh oh!

aswinshakil commented Mar 16, 2023

Uh oh!

smengcl Mar 16, 2023

Uh oh!

smengcl Mar 17, 2023

Uh oh!

aswinshakil Mar 17, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smengcl Mar 17, 2023

Uh oh!

aswinshakil Mar 17, 2023

Uh oh!

Uh oh!

smengcl left a comment

Uh oh!

smengcl commented Mar 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		while (deletedIterator.hasNext() &&
		deletionCount <= keyLimitPerSnapshot) {

HDDS-7883. [Snapshot] Accommodate FSO, key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService #4407

HDDS-7883. [Snapshot] Accommodate FSO, key renames and implement OMSnapshotPurgeRequest for SnapshotDeletingService #4407

Uh oh!

Conversation

aswinshakil commented Mar 16, 2023

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

smengcl Mar 16, 2023

Choose a reason for hiding this comment

Uh oh!

smengcl Mar 17, 2023

Choose a reason for hiding this comment

Uh oh!

aswinshakil Mar 17, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smengcl Mar 17, 2023

Choose a reason for hiding this comment

Uh oh!

aswinshakil Mar 17, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

smengcl commented Mar 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants