HDDS-8627. Recon - API for Count of deletePending directories

[ArafatKhan2198]

What changes were proposed in this pull request?

1. This PR does two main things :-

• It lets the OmTableInsightTask figure out the total count of directories marked for deletion.
• It makes the complex OmTableInsightTask easier to work with by separating different handlers for each table.

2. Finding the Count of Deleted Directories: :-

• It iterates through the DeletedDirectory table and gets the total count of deleted directories.

3. Refactoring the OmTableInsightTask :-

• Noticing that we need to find sizes for keys and other entities in various tables, which has made the process() and reprocess() methods long and complicated.
• To simplify things, we've created separate Insight-Handler classes (like OpenKeysInsightHandler, DeletedKeysInsightHandler) that handle specific tasks for each table.
• This change makes the code easier to manage and paves the way for adding more tables or operations in the future by creating and connecting specific handlers.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8627

How was this patch tested?

Unit Testing and Manual Testing

1. Deleted 3 directories :- dir1, dir2, dir3 :-

2. Summary printing out the count of total deleted directories :-

[ArafatKhan2198]

@devmadhuu @dombizita @ashishkumar50 can you please take a look !

[ashishkumar50]

@ArafatKhan2198, Thanks for working on this. Please find few comments inline.

[devmadhuu]

@ArafatKhan2198 thanks for working on this patch, can you pls elaborate the purpose of this method since you added tableName as extra parameter now. Is this method only for counting records of tables which are having keys as OMKeyInfo and RepeatedOmKeyInfo only ?. Also if GlobalStatsTable will not have counts of OpenFile and OpenKey table ?

[ArafatKhan2198]

Thanks for the comment @devmadhuu

The purpose of the getTableSizeAndCount method is to calculate the total count of records, total unreplicated size, and total replicated size in a given iterator over a table, based on the specified tableName. It is only used for tables which want both the size as well as the count to be calculated hence in our case it will only work for OPEN_KEY_TABLE, OPEN_FILE_TABLE, DELETED_TABLE & DELETED_DIR_TABLE.

Table Name	Object Associated
OPEN_KEY_TABLE	OmKeyInfo
OPEN_FILE_TABLE	OmKeyInfo
DELETED_DIR_TABLE	OmKeyInfo
DELETED_TABLE	RepeatedOmKeyInfo

In our case, the OPEN_KEY_TABLE, OPEN_FILE_TABLE, and DELETED_DIR_TABLE all have OmKeyInfo associated with them. Therefore, we need to differentiate between them based on the table name. The reason for this differentiation is because the calculation of data differs for the OPEN_KEY_TABLE and OPEN_FILE_TABLE compared to the DELETED_DIR_TABLE.

To summarize, when calculating data for the OPEN_KEY_TABLE and OPEN_FILE_TABLE, we consider the dataSize of the keys stored in the OmKeyInfo objects. However, for the DELETED_DIR_TABLE, we retrieve the size of the deleted directory from the NSSummary table. It's important to note that the replicatedSize value is not applicable to the DELETED_DIR_TABLE.

So, to clarify, the globalStats table stores both the sizes and counts of tables, and the getTableSizeAndCount method calculates these values and updates the globalStats table accordingly.

[ArafatKhan2198]

@devmadhuu @adoroszlai @ashishkumar50
Could you please review this.

[devmadhuu]

Thanks @ArafatKhan2198 for continued effort on this. Mostly looks good to me except few minor comments.

[sumitagrawl]

@ArafatKhan2198 Thanks for working over this, IMO, this insight is not required as its very difficult to handle dynamic scenario, and further can not see further use case as already we have implemented api to give "pending deletion size".

[ArafatKhan2198]

Initially, we attempted to implement the DU space calculation of deleted directories using NSSummaries. However, Sumit highlighted the difficulty in determining the total size of all deleted directories using our original approach. Currently, we iterate through the DeletedDirectoryTable, calculate the size of each deleted directory, and then find the sum total size. The issue arises when both the parent and child entries are encountered in the DeletedDirectoryTable, resulting in the same size being calculated twice.

To address the challenge of calculating the size of deleted directories, especially when encountering both parent and child entries in the DeletedDirectoryTable, we are adjusting our approach. To obtain the count of deletePending directories and their corresponding data size, we can derive the data size from the keys of the deleted directories. These keys are already being calculated and are an integral part of the deleted directories.

cc: @devmadhuu @dombizita

[devmadhuu]

Thanks @ArafatKhan2198 for continued effort on this patch. Few comments, Pls check.

[sumitagrawl]

LGTM

[dombizita]

Please adjust the Jira (HDDS-8627) title and description to the changes discussed during the PR review @ArafatKhan2198 @sumitagrawl

[ArafatKhan2198]

Please adjust the Jira (HDDS-8627) title and description to the changes discussed during the PR review @ArafatKhan2198 @sumitagrawl

Thanks for pointing it out will do!