HDDS-8415. Recon UI:OM DB Insights Changes

[smitajoshi12]

What changes were proposed in this pull request?

https://docs.google.com/document/d/1H4VjvNNHwKRac3XDSUCrBnfMOWhqNzxOIlbqwbcw6dI/edit?pli=1#
New left nav under "Insights" should come as "OM DB Insights" where a two tabs will be added on page:
Container Level Info - Refer backend API PR #4509
Key Level Info - Refer backend API PR - #4516

Final Output Document
https://docs.google.com/document/d/1leWPiv6dEuL3jcKfCjd90YH34_oCL67zWJ0WNJ7FDoc/edit

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8415

How was this patch tested?

Manually
Summary Information on Overview Page

/api/v1/keys/open?limit=0

/api/v1/keys/deletePending?limit=1
Response
{
"keysSummary": {
"totalUnreplicatedDataSize": 4610000000,
"totalReplicatedDataSize": 13830000000,
"totalDeletedKeys": 461
},
"lastKey": "/volume/bucket1/e1w46pj5xo/0/-9223372036854772223",
"replicatedDataSize": 30000000,
"unreplicatedDataSize": 10000000,
"deletedKeyInfo": [

],
"status": "OK"
}

---

1. Container Mismatch Info

RestEndPoint:- /api/v1/containers/mismatch?limit=10&prevKey=0&missingIn=OM
RestEndPoint:- /api/v1/containers/mismatch?limit=10&prevKey=0&missingIn=SCM
End Point :- /api/v1/containers/2/keys
Actual Cluster Response

---

2. Open Keys
   End Point /api/v1/keys/open?includeFso=true&includeNonFso=false&limit=10&prevKey
   Actual Cluster Response

Path :- is mapped to path in fso array
key :- is mapped to key.
Response:
{
"keysSummary": {

    "totalUnreplicatedDataSize": 17289,
    "totalReplicatedDataSize": 51867,
    "totalOpenKeys": 1
},

"lastKey": "/-4611686018427388160/-9223372036854775552/-9223372036854770943/dir4-key1-fso/110627551865339924",
"replicatedDataSize": 51867,
"unreplicatedDataSize": 17289,
"fso": [
{
"key": "/-4611686018427388160/-9223372036854775552/-9223372036854770943/dir4-key1-fso/110627551865339924",
"path": "dir4-key1-fso",
"inStateSince": 1688041920657,
"size": 17289,
"replicatedSize": 51867,
"replicationInfo": {
"replicationFactor": "THREE",
"requiredNodes": 3,
"replicationType": "RATIS"
}
}
}}}

---

3. Keys Pending for Deletion
   End Point :- /api/v1/keys/deletePending?limit=10&prevKey
   Actual Cluster Response

---

4. Deleted Container Keys
   End Point - /api/v1/containers/mismatch/deleted?limit=10&prevKey=0
   Dummy Data Response
   

[devabhishekpal]

Just had a few nits.

[devmadhuu]

@smitajoshi12 Thanks for working on this patch, Few comments:

1. Container Mismatch Info screenshot is not inline with what is provided in PR description.
2. In all the pages, "keys" column header label should be "Count Of Keys".
3. Container Mismatch Info page data showing key count as 1, however on expansion of respective container id shows multiple keys, so looks like pages based on test data in db.json is not correct.
4. Key value shown on OpenKeys page doesn't seem correct.
5. Path shown in "Keys Pending For Deletion" seems not correct.
6. "Keys" label in table of page "Deleted Container Keys" should be "Count Of Keys".

[devmadhuu]

@ArafatKhan2198 - can you check the backend API response for points #3, #4, #5 ?

[ArafatKhan2198]

@devmadhuu @smitajoshi12

Regarding point number 3, the data shown in the earlier screenshots on the UI was just dummy data and not from an actual cluster. However, I have now verified the API calls with real cluster data, and they are functioning correctly. The getKeysForContainer() API called on the expansion box accurately lists the keys present in the specific container.

For point number 4, dummy data has once again been used.

Regarding point number 5, I have suggested some changes in the table structure, including additions and modifications. The table now should display the following columns mapped to their respective JSON attributes:

JSON Response Attribute	Table Column Name
fileName	Key Name
keyName	Path
creationTime	Creation Time
Sum of dataSize attribute of each object in omKeyInfoList	Total Data Size
Count of all keys inside omKeyInfoList	Total Keys

Furthermore, I have suggested an expansion button where the details of the keys present in each omKeyInfoList (unique to every record in the table) will be shown.

Expanded Table for Each Row ➖

Key Data Size	Replicated Data Size	Creation Time	Modification Time
dataSize	replicatedSize	creationTime	modificationTime

[ArafatKhan2198]

When I click on OM for the ExistsAt operation the page does not get refreshed for mismatch endpoint.

[ArafatKhan2198]

Could you check the filter of FSO and NonFso as well, it's not getting propely refreshed when I click one

[ArafatKhan2198]

Across all the tables where we display the amount of data, it is currently shown in bytes. However, I would suggest displaying it in kilobytes (KB) instead, as it would be more suitable.

[devmadhuu]

@smitajoshi12 - pls update the screenshots in PR description with correct data mapping of UI fields with backend API response.

[ArafatKhan2198]

Thank you for incorporating the final changes into the PR, @smitajoshi12. I have a few more minor comments remaining. Please address them, and then we should be all set to proceed :-

• FSO and NON-FSO filter still not working - bit flaky
• Need to remove Total deleted blocks from overview page as we will not be persuing it for now
• Need to change the limit for the summary of deleted keys to 1 for the overview page
• See if can reduce the font size of numbers in the overview page summary cards

[smitajoshi12]

Thank you for incorporating the final changes into the PR, @smitajoshi12. I have a few more minor comments remaining. Please address them, and then we should be all set to proceed :-

• FSO and NON-FSO filter still not working - bit flaky
• Need to remove Total deleted blocks from overview page as we will not be persuing it for now
• Need to change the limit for the summary of deleted keys to 1 for the overview page
• See if can reduce the font size of numbers in the overview page summary cards

@ArafatKhan2198
Completed Changes in latest commit.

[smitajoshi12]

When I click on OM for the ExistsAt operation the page does not get refreshed for mismatch endpoint.

@ArafatKhan2198 Please check and let us know

[smitajoshi12]

/ defaultSelectedKeys={this.state.mismatchMissingState}

@ArafatKhan2198 yes Completed Change and attached latest screenshots.

[smitajoshi12]

@smitajoshi12 Thanks for working on this patch, Few comments:

1. Container Mismatch Info screenshot is not inline with what is provided in PR description.
2. In all the pages, "keys" column header label should be "Count Of Keys".
3. Container Mismatch Info page data showing key count as 1, however on expansion of respective container id shows multiple keys, so looks like pages based on test data in db.json is not correct.
4. Key value shown on OpenKeys page doesn't seem correct.
5. Path shown in "Keys Pending For Deletion" seems not correct.
6. "Keys" label in table of page "Deleted Container Keys" should be "Count Of Keys".

@devmadhuu
Completed Changes and added latest screenshots.

[ArafatKhan2198]

When I click on OM for the ExistsAt operation the page does not get refreshed for mismatch endpoint.

@ArafatKhan2198 Please check and let us know

Thank you for incorporating the changes, @smitajoshi12. I have tested them on my local machine, and they appear to be functioning correctly. You might want to consider clearing your browser cache; doing so should resolve the issue.

[devmadhuu]

@smitajoshi12 thanks for working on this patch. LGTM +1

[ArafatKhan2198]

@smitajoshi12 Thanks for working on this. LGTM +1

[smitajoshi12]

When I click on OM for the ExistsAt operation the page does not get refreshed for mismatch endpoint.

@ArafatKhan2198 Please check and let us know

Thank you for incorporating the changes, @smitajoshi12. I have tested them on my local machine, and they appear to be functioning correctly. You might want to consider clearing your browser cache; doing so should resolve the issue.

@ArafatKhan2198
Yes Filters will work fine with latest changes and removed asynchronous code execution.

[sumitagrawl]

@smitajoshi12 Thanks for working over this, have few query, please check

[sumitagrawl]

@smitajoshi12 Few more cases in UI to handle,

1. "Container mismatchInfo' --> ExistAt --> Tip to be provided,

• "SCM: Container exist at SCM but missing at OM"
• "OM: Container exist at OM but missing at SCM"

[sumitagrawl]

Also, do keyName with 0, 1, ... is valid?

[sumitagrawl]

Also, do path value is ok?

[sumitagrawl]

how do summay comes for below?

[smitajoshi12]

• OM: Container exist at OM but missing at SCM"

@sumitagrawl

[smitajoshi12]

Also, do keyName with 0, 1, ... is valid?
May be with test data.

Actual Screenshots with this command ozone freon dfsg --path=ofs://om/volume/bucket1 -s=10000000 -n=100000 in local cluster.

[smitajoshi12]

how do summay comes for below?

In Local Cluster Testing time we used this command ozone fs -rm -r -skipTrash ofs://om/volume/ got response in
/api/v1/keys/deletePending?limit=1 wiith this endpoint.

{
"keysSummary": {
"totalUnreplicatedDataSize": 4610000000,
"totalReplicatedDataSize": 13830000000,
"totalDeletedKeys": 461
},
"lastKey": "/volume/bucket1/e1w46pj5xo/0/-9223372036854772223",
"replicatedDataSize": 30000000,
"unreplicatedDataSize": 10000000,
"deletedKeyInfo": [

],
"status": "OK"
}