HDDS-9988. SCM UI shows storage usage percentage #5882

jianghuazhu · 2023-12-28T16:31:26Z

What changes were proposed in this pull request?

Now, the SCM web ui does not show how much storage percentage is used by each DataNode node, the purpose of this PR is to make up for it.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9988

How was this patch tested?

The SCM web ui can display the storage percentage used by each DataNode node.

jianghuazhu · 2023-12-28T16:36:49Z

ci: https://github.com/jianghuazhu/ozone/actions/runs/7348062465
New web ui:

Can you help review this PR, @adoroszlai @hemantk-12 .
Thanks.

jianghuazhu · 2023-12-31T11:31:01Z

Can you help review this PR, @aswinshakil @fapifta @szetszwo .
Thanks.

xBis7

@jianghuazhu Thanks for the patch. I've left a comment.

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

jianghuazhu · 2024-01-07T04:28:26Z

Can you help review this pr again, @xBis7 .
Thanks.

xBis7

@jianghuazhu Thanks for updating the patch. I have some suggestions for optimizing the test.

hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm-overview.html

hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java

whbing · 2024-01-08T12:48:41Z

How about also display Capacity and ScmUsed ?
similar HDFS UI like this ?

IMO, it's also best to have a sort function

jianghuazhu · 2024-01-09T11:59:39Z

Thanks @whbing .
The DataNode UI already displays the total storage size and used size. Does it still need to be displayed in the SCM UI?
What do you think, @xBis7 .

xBis7 · 2024-01-09T12:34:12Z

IMO, it's also best to have a sort function

There is a sort function. I don't know if it works for every column though. That needs to be checked.

The DataNode UI already displays the total storage size and used size. Does it still need to be displayed in the SCM UI?
What do you think, @xBis7 .

It can be useful to have such information in one place, but if it's out of scope then it should be done as part of another jira. Tickets with narrow scope are easier for people to review.

How about also display Capacity and ScmUsed ?

Capacity seems related and can be part of this PR but ScmUsed doesn't.

As long as the PR changes are within scope, I'm ok with them.

xBis7

@jianghuazhu Thanks for making the changes. I went over the code once more.

I also tested it locally in docker. It looks good but Recon displays a different percentage than SCM UI. It could be due to the docker dev env and I don't have an available cluster at the moment, to load the patch. If you can verify that there is no issue.

Can you check the CI failure?
https://github.com/jianghuazhu/ozone/actions/runs/7461358714

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

jianghuazhu · 2024-01-09T16:42:11Z

Thanks @xBis7 and @whbing .
I've updated it.
ci:
https://github.com/jianghuazhu/ozone/actions/runs/7462830986
New SCM UI:

Newly added columns can be sorted.

jianghuazhu · 2024-01-11T04:07:46Z

Can you help me take a look at this PR again, @xBis7 .
Thanks.

xBis7

I also tested it locally in docker. It looks good but Recon displays a different percentage than SCM UI. It could be due to the docker dev env and I don't have an available cluster at the moment, to load the patch. If you can verify that there is no issue.

@jianghuazhu Can you also verify this? If it's an issue we will file a new jira.

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java

jianghuazhu · 2024-01-11T11:53:54Z

I found some problems.
Recon UI:

Here's an explanation of some important data:
Ozone Used: 1.7TB
Non Ozone Used: 1.6TB
Remaining: 127.6 TB
As can be seen here, the total storage capacity size is 130.9TB.

In Recon UI, the calculation formula of percentage is used for storage: (1.7 + 1.6) / 130.9. However, the output result here does not retain decimal places, so the final result is 3%.

SCM UI:

In the SCM UI, the calculation formula for storage usage percentage is: 1.7 / 130.9. In the final output result, we retain 1 decimal place, so the final result is 1.3%. It should be pointed out that Non Ozone Used is not the actual storage capacity used by Ozone.
@xBis7, what are your thoughts?

xBis7 · 2024-01-11T12:15:59Z

It should be pointed out that Non Ozone Used is not the actual storage capacity used by Ozone.

Where does Non Ozone Used come from?

jianghuazhu · 2024-01-11T12:50:49Z

In our test cluster, each node is configured with 12 disks, and each disk stores Ozone data and HDFS data respectively. For example, one of the disk directories:
/data5:
131G ./hadoopdata
167G ./ozonedata

I understand that Non Ozone Used here refers to the storage capacity used by HDFS and the data used by other programs.

xBis7 · 2024-01-11T13:04:38Z

Thanks for testing it out. Capacity refers to the entire disk space, right? It's not just Ozone disk space.

If there is a lot of space from the total capacity, occupied by non Ozone files, and the SCM UI doesn't report it, then the Used Space Percentage will be misleading.

e.g.

We have 100 TB of disk space
Ozone takes up to 5TB
Other non Ozone files take another 2TB

In that scenario, you have

7% space used
SCM UI reports 5%?
You still don't have 95% available but only 93%

Is there an easy way to include that number in the SCM UI?

jianghuazhu · 2024-01-11T15:26:36Z

As you can see now, the percentages shown in the Recon UI represent node disk usage, not just Ozone usage, which is an extension of Ozone.
However, I believe that the storage usage shown by the SCM should be the data Used by the real Ozone datanodes, and there is no need to show 'Non Ozone Used' data.
Therefore, we can use 'SCM Used Space Percent' in the SCM UI to indicate how much Ozone data is managed by the DataNode service. The data shown here should correspond to the storage capacity of the DataNode UI.

@xBis7 , what do you think?

xBis7 · 2024-01-11T15:33:14Z

That's why I asked about capacity. Is it referring to the entire disk or just Ozone?

jianghuazhu · 2024-01-11T16:04:42Z

I think the percentage of storage usage here is relative to the entire disk.

jianghuazhu · 2024-01-12T12:40:55Z

I have an idea.
In the SCM UI, the storage usage percentages of Ozone Used and Non Ozone Used are displayed respectively.
New SCM UI:

Here’s an explanation of some key data:
scmUsed:
This refers to Ozone Used. Calculation formula = scmUsed / capacity, and the result value retains 2 decimal places.
nonScmUsed:
This refers to Non Ozone Used.
Calculation method:
1.scmNonUsed = capacity - scmUsed - remaining
2.nonScmUsed= scmNonUsed / capacity
In Recon UI, the method of calculating Non Ozone Used is the same as the method of calculating nonScmUsed.
@xBis7 @adoroszlai , do you have any suggestions?

xBis7 · 2024-01-12T13:29:28Z

@jianghuazhu The latest approach looks good. Displaying Total as well could be practical.

jianghuazhu · 2024-01-13T01:03:25Z

@xBis7 , I've updated.
ci: https://github.com/jianghuazhu/ozone/actions/runs/7505648725
New SCM UI:

xBis7 · 2024-01-13T08:13:06Z

@jianghuazhu Sorry for not been clear. With Total, I was referring to the total used space. Can you rename that column back to Capacity?

Otherwise, changes look good. After that change, I'll open up the PR Ci.

jianghuazhu · 2024-01-14T00:17:06Z

Thanks @xBis7 .
I've updated it.
Latest SCM UI:

xBis7

@jianghuazhu Thanks for the changes, LGTM!

jianghuazhu · 2024-01-18T11:13:33Z

@xBis7 @adoroszlai , do you have any new suggestions for this PR?
If possible, help merge it into the master branch.
In addition, I will submit some new features later, which are based on this PR.
Thanks.

xBis7 · 2024-01-18T11:25:48Z

@jianghuazhu Let's wait to see if @adoroszlai has any comments. Otherwise, it can be merged.

adoroszlai

Thanks @jianghuazhu for working on this.

adoroszlai · 2024-01-18T11:52:51Z

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

+      usedPercentage = Double.valueOf(decimalFormat.format(usedPerc));
+      nonUsedPercentage = Double.valueOf(decimalFormat.format(nonUsedPerc));


Here we format the value, then parse it as double. The caller then simply concatenates them into the final text being displayed. I think formatting may be lost this way.

I think calculateStoragePercentage() should return the formatted String instead.

If there are no reports, return "N/A", since we have no information.

I will update soon.

adoroszlai · 2024-01-18T11:55:01Z

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

+      double scmUsedPerc = storagePercentage[0];
+      double nonScmUsedPerc = storagePercentage[1];
+      map.put(usedSpacePercent,
+          "scmUsed: " + scmUsedPerc + "%, nonScmUsed: " + nonScmUsedPerc + "%");


I'm not sure "scmUsed" and "nonScmUsed" are clear enough for users. Also, the table header already says "Used Space Percent".

How about "Ozone: ... , other: ..."?

Is there a definite way of expressing this?
Then we display 'Ozone: ... , other: ...'?

Can you please clarify the question?

My suggestion is to change the value displayed in table cells:

-scmUsed: 12.34%, nonScmUsed: 1.23% +Ozone: 12.34%, other: 1.23%

(... are just placeholders in my previous comment)

Okay, I will update it.

adoroszlai · 2024-01-18T12:04:00Z

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

+  private final String usedSpacePercent = "USEDSPACEPERCENT";
+  private final String totalCapacity = "CAPACITY";


These should be constants. (The existing ones, too, but those are being fixed in HDDS-10152.)

@adoroszlai , do we need to wait for HDDS-10152 to be resolved?

No, I don't think we need to wait.

(However, I'd like to get HDDS-10157 fixed before merging anything else, since CI is failing in native check.)

adoroszlai · 2024-01-18T12:05:15Z

hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java

+      String capacityResult = nodeManager.calculateStorageCapacity(dni);
+      assertEquals(capacityResult, totalCapacity);


Please follow assertEquals(expected, actual) argument order.

adoroszlai · 2024-01-18T12:10:18Z

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

+   * @param dni DataNode node that needs to be calculated.
+   * @return
+   */
+  public String calculateStorageCapacity(DatanodeInfo dni) {


Both new calculate methods can be static and take List<StorageReportProto> as parameter instead of DatanodeInfo.

This allows the test to be much simpler (see other comment on the test for details).

adoroszlai · 2024-01-18T12:12:25Z

hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java

+    OzoneConfiguration conf = getConf();
+    conf.setTimeDuration(OZONE_SCM_HEARTBEAT_PROCESS_INTERVAL, 1000,
+        MILLISECONDS);
+    try (SCMNodeManager nodeManager = createNodeManager(conf)) {
+      EventQueue eventQueue = (EventQueue) scm.getEventQueue();
+      LayoutVersionManager versionManager =
+          nodeManager.getLayoutVersionManager();
+      LayoutVersionProto layoutInfo = toLayoutVersionProto(
+          versionManager.getMetadataLayoutVersion(),
+          versionManager.getSoftwareLayoutVersion());
+      DatanodeDetails dn = MockDatanodeDetails.randomDatanodeDetails();
+      UUID dnId = dn.getUuid();
+      List<StorageReportProto> reports = volumeCount > 0 ?
+          generateStorageReportProto(volumeCount, dnId, perCapacity,
+              used, remaining) : null;
+      nodeManager.register(dn, reports != null ?
+          HddsTestUtils.createNodeReport(reports, emptyList()) : null, null);
+      nodeManager.processHeartbeat(dn, layoutInfo);
+      eventQueue.processAll(8000L);
+      NodeStateManager nodeStateManager = nodeManager.getNodeStateManager();
+      List<DatanodeInfo> dataNodeInfoList = nodeStateManager.getAllNodes();
+      DatanodeInfo dni = dataNodeInfoList.get(0);


If calculate... methods are changed as suggested, all we need here is:

List<StorageReportProto> reports = volumeCount > 0 ? generateStorageReportProto(volumeCount, dnId, perCapacity, used, remaining) : null;

jianghuazhu · 2024-01-19T14:06:28Z

@adoroszlai , I've updated it.
Can you help review again?
ci: https://github.com/jianghuazhu/ozone/actions/runs/7583986252
SCM UI:

patch updated

adoroszlai · 2024-01-19T14:19:58Z

Thanks @jianghuazhu for improving the patch.

adoroszlai · 2024-01-19T14:20:47Z

@dombizita would you like to take a look, too?

dombizita

Thanks for working on this @jianghuazhu, it looks good to me! I also tested it locally and it looks as expected.

jianghuazhu · 2024-01-22T10:00:19Z

Thanks @dombizita .

adoroszlai · 2024-01-22T10:02:22Z

Thanks @jianghuazhu for the patch, @dombizita, @xBis7 for the review.

adoroszlai requested review from dombizita and xBis7 January 5, 2024 20:18

xBis7 reviewed Jan 5, 2024

View reviewed changes

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java Outdated Show resolved Hide resolved

HDDS-9988. SCM UI shows storage usage percentage

11a0564

jianghuazhu force-pushed the HDDS-9988 branch from b4c1a26 to 11a0564 Compare January 6, 2024 16:51

jianghuazhu added 2 commits January 7, 2024 02:01

Fix findbugs

0a40dd1

Fix findbugs

a1f4c71

xBis7 reviewed Jan 8, 2024

View reviewed changes

Improve some unit tests.

a1a969b

xBis7 reviewed Jan 9, 2024

View reviewed changes

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java Outdated Show resolved Hide resolved

Improve some unit tests.

3562920

xBis7 reviewed Jan 11, 2024

View reviewed changes

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java Outdated Show resolved Hide resolved

hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java Outdated Show resolved Hide resolved

adoroszlai added the UI label Jan 12, 2024

Improve some unit tests.

31f2b87

Improve some UI display styles.

403e2e2

xBis7 approved these changes Jan 14, 2024

View reviewed changes

adoroszlai previously requested changes Jan 18, 2024

View reviewed changes

Improve some UI display styles.

2b07acb

dombizita approved these changes Jan 22, 2024

View reviewed changes

Merge remote-tracking branch 'origin/master' into HDDS-9988

75da57d

adoroszlai merged commit 9afd1a1 into apache:master Jan 22, 2024

Tejaskriya pushed a commit to Tejaskriya/ozone that referenced this pull request Jan 24, 2024

HDDS-9988. Show used storage percent in SCM UI (apache#5882)

ed2488c

		usedPercentage = Double.valueOf(decimalFormat.format(usedPerc));
		nonUsedPercentage = Double.valueOf(decimalFormat.format(nonUsedPerc));

		private final String usedSpacePercent = "USEDSPACEPERCENT";
		private final String totalCapacity = "CAPACITY";

		String capacityResult = nodeManager.calculateStorageCapacity(dni);
		assertEquals(capacityResult, totalCapacity);

HDDS-9988. SCM UI shows storage usage percentage #5882

HDDS-9988. SCM UI shows storage usage percentage #5882

Uh oh!

Conversation

jianghuazhu commented Dec 28, 2023

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

jianghuazhu commented Dec 28, 2023

Uh oh!

jianghuazhu commented Dec 31, 2023

Uh oh!

xBis7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jianghuazhu commented Jan 7, 2024

Uh oh!

xBis7 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

whbing commented Jan 8, 2024

Uh oh!

jianghuazhu commented Jan 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xBis7 commented Jan 9, 2024

Uh oh!

xBis7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jianghuazhu commented Jan 9, 2024

Uh oh!

jianghuazhu commented Jan 11, 2024

Uh oh!

xBis7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jianghuazhu commented Jan 11, 2024

Uh oh!

xBis7 commented Jan 11, 2024

Uh oh!

jianghuazhu commented Jan 11, 2024

Uh oh!

xBis7 commented Jan 11, 2024

Uh oh!

jianghuazhu commented Jan 11, 2024

Uh oh!

xBis7 commented Jan 11, 2024

Uh oh!

jianghuazhu commented Jan 11, 2024

Uh oh!

jianghuazhu commented Jan 12, 2024

Uh oh!

xBis7 commented Jan 12, 2024

Uh oh!

jianghuazhu commented Jan 13, 2024

Uh oh!

xBis7 commented Jan 13, 2024

Uh oh!

jianghuazhu commented Jan 14, 2024

Uh oh!

xBis7 left a comment

Choose a reason for hiding this comment

Uh oh!

jianghuazhu commented Jan 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xBis7 commented Jan 18, 2024

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

xBis7 left a comment •

edited

Loading

jianghuazhu commented Jan 9, 2024 •

edited

Loading

jianghuazhu commented Jan 18, 2024 •

edited

Loading