HDDS-6170. Add metrics to replication manager to track container health states

[sodonnel]

What changes were proposed in this pull request?

Replication Manager processes all the containers in SCM periodically, and so has a view of the health of the system. However it does not count up the replicas in each state and hence that overview of the system health is not easily visible.

This Jira adds a ReplicationManagerReport object, which ReplicationManager can populate by incrementing various counters as it processes the containers. The report allows the number of containers in each lifecycle state to be counted, while also counting the number of containers in various health states, eg under replicated, over replicated etc. The report also allows a sample of the container IDs in the state are stored in the report (max of 100 per state) and these can be extracted later for debugging (a later Jira will provide the extra feature, probably via an Ozone Admin command).

The report is integrated with the ReplicationManagerMetrics class, and will add metrics like the following:

{
    "name" : "Hadoop:service=StorageContainerManager,name=ReplicationManagerMetrics",
    "modelerType" : "ReplicationManagerMetrics",
    "tag.Hostname" : "3212ea4aecc5",
    "InflightReplication" : 0,
    "InflightDeletion" : 0,
    "InflightMove" : 0,

## New metrics from here
    "NumOpenContainers" : 1,
    "NumClosingContainers" : 0,
    "NumQuasiClosedContainers" : 2,
    "NumClosedContainers" : 0,
    "NumDeletingContainers" : 0,
    "NumDeletedContainers" : 0,
    "NumUnderReplicatedContainers" : 0,
    "NumMisReplicatedContainers" : 0,
    "NumOverReplicatedContainers" : 0,
    "NumMissingContainers" : 0,
    "NumUnhealthyContainers" : 0,
    "NumEmptyContainers" : 0,
    "NumOpenUnhealthyContainers" : 0,
    "NumStuckQuasiClosedContainers" : 0,
## end of new metrics    
    "NumReplicationCmdsSent" : 0,
    "NumReplicationCmdsCompleted" : 0,
    "NumReplicationCmdsTimeout" : 0,
    "NumDeletionCmdsSent" : 0,
    "NumDeletionCmdsCompleted" : 0,
    "NumDeletionCmdsTimeout" : 0,
    "NumReplicationBytesTotal" : 0,
    "NumReplicationBytesCompleted" : 0,
    "NumDeletionBytesTotal" : 0,
    "NumDeletionBytesCompleted" : 0,
    "ReplicationTimeNumOps" : 0,
    "ReplicationTimeAvgTime" : 0.0,
    "DeletionTimeNumOps" : 0,
    "DeletionTimeAvgTime" : 0.0
  }

Note that some of the above metrics (the ones for container LifeCycle state) duplicate others available from a different source in SCM:

{
    "name" : "Hadoop:service=StorageContainerManager,name=SCMContainerMetrics",
    "modelerType" : "SCMContainerMetrics",
    "tag.Hostname" : "92503dfbd5db",
    "OpenContainers" : 0,
    "ClosingContainers" : 0,
    "QuasiClosedContainers" : 0,
    "ClosedContainers" : 0,
    "DeletingContainers" : 0,
    "DeletedContainers" : 0,
    "TotalContainers" : 0
  }

I am open to removing these duplicates in the new ReplicationManager metrics, but it may be useful to keep them, as the ReplicationManager counts are captured at a point in time, and they are calculated differently, and hence may be helpful in debugging some problems. For that reason, it is still good to capture them in the Report object, but it is debatable on whether they should be in the metrics or not.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6170

How was this patch tested?

New unit tests and validated in docker-compose environment.

[adoroszlai]

Thanks @sodonnel for the patch. It looks great. The only thing I spotted is a RuntimeException, which is too generic.

[sodonnel]

@adoroszlai Thanks for the review. I have committed your suggested change, so I will let the checks run again, although I don't expect that change to affect the result of them.