HDDS-9874. Introduce Metrics for listKeys Dashboard

[muskan1012]

What changes were proposed in this pull request?

Introduced following metrics for listKey operations:

• listKeysAveragePagination
• listKeysOpsPerSec

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9874

How was this patch tested?

manually tested it on a cluster

[adoroszlai]

@muskan1012 There are checkstyle and findbugs failures. It would be nice to wait with PR creation until a clean CI run in your fork is completed.

https://github.com/muskan1012/ozone/actions/runs/7128100497

[xichen01]

We have the Latency metric listKeysLatencyNs, maybe we just need to add a cumulative number of listKeys metric(such as: listKeysSize), which we can calculate to get listKeysOpsPerSec and listKeysAveragePagination.

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java

Lines 92 to 93 in 0e91b2a

	@Metric(about = "listKeys latency in nanoseconds")
	private MutableRate listKeysLatencyNs;

[muskan1012]

I implemented the cumulative number of listKeys using keyCount, which is set to maxKeys if truncated or cacheKeyMap.size() otherwise.
listKeysAveragePagination is set using keyCount.
listKeysOpsPerSec is calculated as keyCount / ((Time.monotonicNowNanos() - startNanos) / 1_000_000_000.0f).
Added latency tracking with addListKeysReadFromRocksDbLatencyNs.
This approach captures the metrics as you suggested.

[kerneltime]

cc @tanvipenumudy can you please take a look as well?

[tanvipenumudy]

Thank you @muskan1012 for the patch, please find a few comments.

[kerneltime]

@muskan1012 can you please rebase the change and address the conflicts?

[tanvipenumudy]

Thank you @muskan1012 for updating the patch, could you please take a look at the build failure?

[tanvipenumudy]

@muskan1012, the failed checks appear related to the changes (NPE while initializing a new OmMetadataManager). Could you please address this issue similarly to how the previous test failures were handled?

Could you also please rebase the patch? Thanks!

[tanvipenumudy]

Thank you @muskan1012 for rebasing and fixing the test failures.

Reran the workflow; could you please take a look at the integration test failure on TestListKeys? Thanks!

[tanvipenumudy]

@adoroszlai, @xichen01 could you please take another look at the patch? Thanks

[tanvipenumudy]

How was this patch tested?
manually tested it on a cluster

@muskan1012, it would help if you could also please share the updated list keys dashboard (or prometheus/graphana metrics - whichever works!) once all changes are incorporated, thanks.

[adoroszlai]

Thanks @muskan1012 for updating the patch.

[adoroszlai]

Thanks @muskan1012 for updating the patch. Mostly looks good, some unnecessary changes are still left over.

Also, can you please consider @kerneltime's suggestion to rename averagePagination to keyCount?

[muskan1012]

Hi @adoroszlai, thank you for reviewing it. Fixed minor nits as well.

patch updated

[tanvipenumudy]

Thank you @muskan1012 for working on the patch iteratively, thank you @adoroszlai, @kerneltime, @xichen01 for the reviews.