HDDS-6244. ContainerBalancer metrics don't show updated values in JMX

[siddhantsangwan]

What changes were proposed in this pull request?

The ContainerBalancerMetrics class is responsible for recording metrics. These metrics always show 0 when accessed through JMX, while expected values are greater than 0.

I deleted some metrics of the type MutableGaugeLong and instead used the type MutableCounterLong. Introduced a new metric countIterations that keeps count of the number of iterations that balancer has run for.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6244

How was this patch tested?

Updated TestContainerBalancer#testMetrics()

[siddhantsangwan]

@lokeshj1703 @JacksonYao287 please take a look!

[lokeshj1703]

@siddhantsangwan Thanks for working on this! The changes look good to me. I have a few comments inline.

[siddhantsangwan]

@lokeshj1703 I've replaced the current metrics and included more testing.

[JacksonYao287]

thanks @siddhantsangwan for the work ! the change looks good.
I also suggest that we can add aggregate metrics(total containers and total datasize). but this can be implemented in a seperate jira.

[lokeshj1703]

Thanks @siddhantsangwan for updating the PR! I have some minor comments.

[siddhantsangwan]

Thanks for reviewing @lokeshj1703 @JacksonYao287
Addressed your comments and opened a new jira for aggregate metrics: https://issues.apache.org/jira/browse/HDDS-6362

[lokeshj1703]

Thanks for updating the PR! The changes look good to me. +1.
I have one minor comment, let me know if you would like to address it in this or new PR.

[lokeshj1703]

I will also recommend changing names of other metrics. Like datanodesNum -> numDatanodes and movedContainersNum -> numMovedContainers.
We can do it in a separate PR as well.

[siddhantsangwan]

Oh, I remember we had decided on this naming convention in a previous PR: #2230 (comment)

But this does conflict with numIterations where "num" is the prefix.

[lokeshj1703]

I see. My idea is that it should be easy to search these metrics in prometheus. Maybe we need to have common prefix for balancer then. Can we check some naming guide here?

[siddhantsangwan]

I looked into some examples. We've used "num" as a prefix in other places, such as numCacheHits, numCacheMisses, numInFlightReplications, numWriteStateMachineOps etc.

[siddhantsangwan]

@lokeshj1703 I've fixed the naming.

[lokeshj1703]

@siddhantsangwan Thanks for the contribution! @JacksonYao287 Thanks for the reviews! I have committed the PR to master branch.