HDDS-11498. Improve SCM deletion efficiency.

[slfan1989]

What changes were proposed in this pull request?

Background

During our use of deletion, I noticed that it can be very slow, especially after we switched to the EC policy.

Our Ozone01 cluster currently has about 1K machines. Initially, we chose to use a Ratis-3Replica strategy, but for cost
considerations, we gradually switched to the EC-6-3 strategy in July.

The following chart shows the deletion speed for Ratis-3Replica .

The following chart shows the deletion speed for EC-6-3.

By reviewing the code and analyzing the logs, we found that the following situation can cause deletion to be very slow. We will illustrate this with an example.

Example

We want to delete data from an EC container with ContainerId = 1000. Since it is EC-6-3, there are 9 replicas (DN1, DN2, DN3, ... DN9).

Before deletion, we first select a batch of DNs; at this time, we may only select DN1 to DN6. We then send the deletion command to these 6 DNs, and the command executes normally, successfully deleting 6 blocks. However, if DN7 to DN9 are not selected, our deletion process will get stuck.

I came up with a possible solution to eliminate this stuck situation. We require that all replicas of the container to be deleted must be present in the selected DN list simultaneously. Otherwise, we will skip that container.

private void getTransaction(DeletedBlocksTransaction tx,
      DatanodeDeletedBlockTransactions transactions,
      Set<DatanodeDetails> dnList, Set<ContainerReplica> replicas,
      Map<UUID, Map<Long, CmdStatus>> commandStatus) {
    DeletedBlocksTransaction updatedTxn =
        DeletedBlocksTransaction.newBuilder(tx)
            .setCount(transactionStatusManager.getOrDefaultRetryCount(
              tx.getTxID(), 0))
            .build();
    
     // Requiring that replicas must be present in the DN list simultaneously ensures that the deletion commands for all 
     // replicas of the same container can be issued at once, avoiding situations where some replicas of the container are 
     // deleted while others are not.
    for (ContainerReplica replica : replicas) {
      DatanodeDetails datanodeDetails = replica.getDatanodeDetails();
      if (!dnList.contains(datanodeDetails)) {
        return;
      }
    }

    for (ContainerReplica replica : replicas) {
      DatanodeDetails details = replica.getDatanodeDetails();
      if (!dnList.contains(details)) {
        continue;
      }
      if (!transactionStatusManager.isDuplication(
          details, updatedTxn.getTxID(), commandStatus)) {
        transactions.addTransactionToDN(details.getUuid(), updatedTxn);
      }
    }
  }

Internal deployment results.

We rolled out this improvement internally in the SCM around 7 PM on September 26th, and we observed a significant enhancement in deletion efficiency, with 50 million blocks being fully processed within 5 hours.

The core aspect of this improvement is to ensure that all DNs within the same container receive the delete command simultaneously. When they send their ACKs, they can reach the SCM at approximately the same time, which facilitates the confirmation of block deletions.

What is the link to the Apache JIRA

JIRA: HDDS-11498. Improve SCM deletion efficiency.

How was this patch tested?

Production environment validation.

[slfan1989]

@adoroszlai @xichen01 Could you help review this PR? Thank you very much! Based on actual usage, it indeed improves the overall deletion speed. I also made some changes to the logs and removed a piece of code that will not be called.

[xichen01]

@slfan1989 Thanks for your patch, some comments for you refer.

[slfan1989]

@sodonnel Could you spare some time to help review this PR? Thank you very much! This PR improves the deletion efficiency of SCM to some extent.

[slfan1989]

@sumitagrawl Could you help review this PR? Thank you very much! The purpose of this PR is to enhance the efficiency of SCM deletion.

[slfan1989]

@xichen01 Do you have any other comments? If so, I will further improve this PR. Thank you very much!

[xichen01]

LGTM +1

[slfan1989]

LGTM +1

Thank you for your response! I would like to provide an update on the execution status. We have achieved significant efficiency improvements in both EC-6-3 and Ratis-3 replicas. We are in the process of converting the cluster's data from Ratis-3 replicas to EC-6-3, primarily due to server cost considerations. EC-6-3 allows us to reduce our logical replication rate to 1.5 replicas, which necessitates a large-scale deletion of Ratis-3 replicas. After implementing this PR, we are able to delete 4PB of logical data each day, and the deletion efficiency is in line with our expectations.

I sincerely hope the community accepts this change, as I believe this PR will help more users achieve better deletion performance.

Ratis-3 Cluster

EC-6-3 Cluster

cc: @adoroszlai

[adoroszlai]

@ashishkumar50 @sumitagrawl please review

[slfan1989]

@xichen01 @adoroszlai Could we continue to move forward with this PR? I'm concerned that other team members might not have noticed it. Our core logic ensures that the deleted container is in the initially selected DN list, while the other changes mainly involve improving logs and cleaning up some unused code.

[slfan1989]

We have already provided relevant explanations regarding the core logic, which clarifies the purpose of this PR.
Line 289-307

[slfan1989]

@sodonnel @sumitagrawl @ashishkumar50 I wonder if you could help review this PR? Thank you very much!

[sumitagrawl]

@slfan1989 Thanks for working over this. The case as getting fix refers case where dnList do not have the DNs. This is possible if DN is not healthy OR overloaded, and hence not considered.
Earlier thought was If replica is there means DNs are healthy, but with overload condition, this might not be true.

Please check if this is same scenario where this is happening?

Overall changes looks good with few minor comments.

[ashishkumar50]

@slfan1989 Thanks for the improvement.

We then send the deletion command to these 6 DNs, and the command executes normally, successfully deleting 6 blocks. However, if DN7 to DN9 are not selected, our deletion process will get stuck.

In your case whether any DN was slow or unhealthy? causing not to be selected for deletion.
With current behaviour SCM will keep track of remaining DN7 to DN9, until it waits to be successfully deleted.
After your change it requires all DN1 to DN9 to be in healthy and normal state, so that SCM will send deletion request in one time and process it.
But in certain case(may have happened in your environment as well) when some DNs are unhealthy or slow. In this case SCM will keep waiting and space reclaim from all DNs will halt for some time because of particular unhealthy DN. Do we need to consider this?

[slfan1989]

@slfan1989 Thanks for working over this. The case as getting fix refers case where dnList do not have the DNs. This is possible if DN is not healthy OR overloaded, and hence not considered. Earlier thought was If replica is there means DNs are healthy, but with overload condition, this might not be true.

Please check if this is same scenario where this is happening?

Overall changes looks good with few minor comments.

@sumitagrawl @ashishkumar50

Thank you very much for your feedback and suggestions !

The main issue we are facing is the uncertainty of the deletion rate: sometimes it is relatively fast, but most of the time it is very slow, particularly during certain periods.

Regarding the "dnList do not have the DNs" situation, it may be due to some DNs being unhealthy or under heavy load. In distributed systems, it is common for some DNs to be in an unhealthy state. While the replicas in these unhealthy DNs cannot be deleted, this should not affect the deletion efficiency of data in other healthy DNs.

Based on our monitoring metrics, here is the deletion monitoring data from October 4, 2024, 00:00 to October 7, 2024, 00:00:

Overall monitoring shows that during the periods of October 4, 2024, from 00:00 to 04:00 and from 04:30 to 08:30, the deletion operations were relatively fast. However, during other time periods, the deletion efficiency significantly decreased, making it nearly impossible to complete deletion tasks.

To better compare the deletion efficiency, I will provide two additional monitoring reports for the periods of October 4, 2024, from 04:30 to 08:30 and October 6, 2024, from 04:30 to 08:30.

We can observe that during the period of October 4, 2024, from 04:30 to 08:30, the ratio of NumBlockDeletionTransactionSuccess to NumBlockDeletionTransactionCompleted is 3:1. In contrast, during the period of October 6, 2024, from 04:30 to 08:30, this ratio is significantly higher.

2024-10-04 04:30:00 ~ 2024-10-04 08:30:00

2024-10-06 04:30:00 ~ 2024-10-06 08:30:00

From the monitoring data for October 6, 2024, from 04:30 to 08:30, we can infer that a significant number of container replicas within the SCM were partially deleted, preventing the SCM from completing the deletion confirmation.

Our cluster consists of 1,500 machines, and the number of malfunctioning machines does not exceed 10. If more than 10 machines fail, we receive an alert and begin manual intervention for these DNs. In my personal view, the slow deletion issue arises from a combination of multiple factors, and it should not be attributed solely to unhealthy DNs or high load.

The contributing factors I personally speculate are as follows:

1. The OM continuously issues deletion requests for blocks to the SCM, which alters the order in which the SCM selects transaction iterators.
2. The replicas of a certain container are only sent to the DNs included in the DN list.
3. Due to some unhealthy DNs or high load on DNs, the DN list is constantly changing.

Based on the current monitoring observations, once some DNs in the replicas of a certain container (A) do not receive the deletion command, even if those DNs later recover, they may struggle to quickly receive the deletion command for container (A) due to potential changes in the SCM's transaction iterator (as the OM continuously issues new deletion requests). As a result, these DNs may take a long time to receive the deletion command, leading to a slow deletion process.

The actual purpose of this PR is to make deletions orderly. If a certain container does not meet the conditions for deletion, we will ignore it until it satisfies the deletion conditions (i.e., when all DNs are healthy).

[slfan1989]

@slfan1989 Thanks for the improvement.

We then send the deletion command to these 6 DNs, and the command executes normally, successfully deleting 6 blocks. However, if DN7 to DN9 are not selected, our deletion process will get stuck.

In your case whether any DN was slow or unhealthy? causing not to be selected for deletion. With current behaviour SCM will keep track of remaining DN7 to DN9, until it waits to be successfully deleted. After your change it requires all DN1 to DN9 to be in healthy and normal state, so that SCM will send deletion request in one time and process it. But in certain case(may have happened in your environment as well) when some DNs are unhealthy or slow. In this case SCM will keep waiting and space reclaim from all DNs will halt for some time because of particular unhealthy DN. Do we need to consider this?

@ashishkumar50 Thank you for your question! In my previous response, I provided some detailed thoughts on this matter. Overall, I believe that the slow deletion might be related to some unhealthy DNs, but their unhealthiness is just one of the factors involved. In this PR, we will skip containers that do not meet the conditions, meaning that if a certain DN is unhealthy, no deletions will be executed for all replicas of the containers above it.

[ashishkumar50]

The actual purpose of this PR is to make deletions orderly. If a certain container does not meet the conditions for deletion, we will ignore it until it satisfies the deletion conditions (i.e., when all DNs are healthy).

This makes sense when we have enough containers to delete(during SCM delete iteration) for which all DNs are healthy, my only point is at some point when when there are only containers left which doesn't satisfy this condition, whether we should wait for all DNs to become healthy or just proceed with the deletion for the available healthy DNs at that point?

[slfan1989]

The actual purpose of this PR is to make deletions orderly. If a certain container does not meet the conditions for deletion, we will ignore it until it satisfies the deletion conditions (i.e., when all DNs are healthy).

This makes sense when we have enough containers to delete(during SCM delete iteration) for which all DNs are healthy, my only point is at some point when when there are only containers left which doesn't satisfy this condition, whether we should wait for all DNs to become healthy or just proceed with the deletion for the available healthy DNs at that point?

@ashishkumar50 I believe waiting for all data nodes (DNs) to be ready before proceeding with deletion is a better choice. In the case of 3 replicas, it's acceptable for SCM to wait, but for EC-6-3, it’s less friendly. If one DN has an issue, the deletion of containers on the other 8 DNs will be affected. Issuing delete commands to these 8 DNs will occupy their deletion queues, leading to slower deletion of other containers.

[ashishkumar50]

@slfan1989 Thanks for the explanation, LGTM.

[sumitagrawl]

@slfan1989 two of comment is still pending...

[slfan1989]

@slfan1989 two of comment is still pending...

@sumitagrawl Could you please help review this PR again? Thank you very much!

[slfan1989]

@slfan1989 Thanks for the explanation, LGTM.

@ashishkumar50 Thank you very much for helping to review the code!

[sumitagrawl]

@slfan1989 Plz find comment for still open cases

[sumitagrawl]

LGTM

[adoroszlai]

Thanks @slfan1989 for the patch, @ashishkumar50, @sumitagrawl, @xichen01 for the review.

[slfan1989]

@ashishkumar50, @sumitagrawl, @xichen01 Thank you very much for helping to review the code! I appreciate your suggestions.

@adoroszlai Thank you very much for merging this PR and for helping to find members to review it!