HDDS-11183. Keys from DeletedTable and DeletedDirTable of Active Object Store should be deleted on batch operation while creating a snapshot

[swamirishi]

What changes were proposed in this pull request?

Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot and not directly from the table. The transactions have to be flushed to delete the keys. If the transaction is not flushed then the deleted keys wouldn't be present in the next transaction replay leading to having unreferenced blocks in the system.

Scenario:
Snapshot S1 create is fired.
create S1 transaction has not been flushed yet, but it has created the snapshot directory and deleted the entries from the table.
This OM has received a bootstrap request from another OM. The bootstrap command will only copy the OM instance and the snapshots that have been flushed to the AOS. S1 will not be a part of it.
The boostrapped OM will replay unflushed transactions(Create S1 will be a part of it). Now it will create a checkpoint out of the bootstrapped AOS which doesn’t have the delete entries. Now the snapshot instance will also not have the deletedKey entries

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11183

How was this patch tested?

Existing unit tests and addition of unit tests.

[smengcl]

Thanks @swamirishi for the patch. The idea makes sense to me. It should have used deleteWithBatch() in the first place.

[hemantk-12]

Thanks for the patch and good catch @swamirishi

LGTM.

[smengcl]

https://github.com/apache/ozone/actions/runs/9964095152/job/27532079656

Error:  Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 180.071 s <<< FAILURE! - in org.apache.hadoop.ozone.om.snapshot.TestSnapshotBackgroundServices
Error:  org.apache.hadoop.ozone.om.snapshot.TestSnapshotBackgroundServices.testSnapshotAndKeyDeletionBackgroundServices  Time elapsed: 41.556 s  <<< ERROR!
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: Cannot invoke "org.apache.ratis.server.protocol.TermIndex.toProto()" because the return value of "org.apache.ratis.server.raftlog.RaftLog.getTermIndex(long)" is null
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
	at org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupInfo(AdminProtocolServiceGrpc.java:496)
...

[hemantk-12]

Thanks @swamirishi for the patch and @smengcl for the review.