HDDS-10941. Add a few interesting ContainerStateMachine metrics in CSMMetrics

[jojochuang]

What changes were proposed in this pull request?

HDDS-10941. CSMMetrics to record number of started transactions

(1)
Add a label RaftGroupId so that Prometheus can identify different CSM instances by pipeline id. Previously, only one Container State Machine metrics instance is exported, which is confusing.

Add a few metrics for ContainerStateMachine operations.

• opsQueueingDelay -- the queueing latency of ContainerOp thread pool.
• pendingApplyTransactions -- count the number of transactions between startTransaction and applyTransaction
• untilApplyTransactionNs -- measures the latency between startTransaction and applyTransaction
• startTransactionCompleteNs -- measures the time spent in startTransaction callback method.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10941

How was this patch tested?

Existing unit tests passed. https://github.com/jojochuang/ozone/actions/runs/9409277999/job/25918878051
Deployed a version of this PR on a real cluster and observed the metrics were exported properly.

[jojochuang]

[xichen01]

@jojochuang Thanks for the patch, you can refer to some comments

[duongkame]

Thanks @jojochuang for the patch, LGTM.

[duongkame]

Suggested change

	ContainerCommandResponseProto proto = dispatchCommand(request, context);
	return proto;
	return dispatchCommand(request, context);
	;

[duongkame]

As discussed offline, we'd also be interested in queueing delay and execution latency of ContainerStateMachine.write (the submitted task in writeStateMachineData). As Ratis has to wait for the future returned by ContainerStateMachine.write to complete to mark transaction committed, those latencies will determine how many transactions per second we can put to datanodes for a given block/container.

[jojochuang]

Rebased the PR.
TODO: incorporate Duong's suggestion.

[jojochuang]

Is this good to go? @xichen01 @duongkame

[xichen01]

LGTM+1

[jojochuang]

Merged. Thanks @xichen01 @duongkame