HDDS-7394. OM RPC FairCallQueue decay decision metrics list caller username in the metric

[xBis7]

What changes were proposed in this pull request?

On the Prometheus endpoint for the OM, in the DecayRpcScheduler summary for users, the username is exposed in the metric name. It makes almost impossible to monitor these values as every time a new user shows up we need to register a new metrics name.

The metric name from org_apache_hadoop_ipc_decay_rpc_scheduler_volume becomes org_apache_hadoop_ipc_decay_rpc_scheduler_caller_hadoop_volume for a user with hadoop username.

The proposed solution is to filter the logs and remove the username from the metric and add it in a username tag.

This metric comes from hadoop-common-3.3.4.jar/DecayRpcScheduler and more specifically

Metrics2Util.NameValuePair entry = (Metrics2Util.NameValuePair)topNCallers.poll();
String topCaller = "Caller(" + entry.getName() + ")";
String topCallerVolume = topCaller + ".Volume";
String topCallerPriority = topCaller + ".Priority";
rb.addCounter(Interns.info(topCallerVolume, topCallerVolume), entry.getValue());

The name is in the format Caller(username).MetricType eg. Caller(hadoop).Volume. The username might exist in the metric name for a purpose. We don't want to change the way the metric works but the way it's presented. The cleanest way to deal with this seems to filter the metric in PrometheusMetricsSink. We are not making any functional changes but we are only filtering what is presented to the user.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7394

How was this patch tested?

New unit tests were added for this patch. It was also tested manually, with a docker cluster and the OM /prom endpoint.

To test it in a docker environment:

in compose/ozone add in docker-config

CORE-SITE.XML_ipc.9862.callqueue.impl=org.apache.hadoop.ipc.FairCallQueue
CORE-SITE.XML_ipc.9862.scheduler.impl=org.apache.hadoop.ipc.DecayRpcScheduler
CORE-SITE.XML_ipc.9862.scheduler.priority.levels=2
CORE-SITE.XML_ipc.9862.backoff.enable=true
CORE-SITE.XML_ipc.9862.faircallqueue.multiplexer.weights=99,1
CORE-SITE.XML_ipc.9862.decay-scheduler.thresholds=90
OZONE-SITE.XML_ozone.om.address=0.0.0.0:9862

then

$ export COMPOSE_FILE=docker-compose.yaml:monitoring.yaml
$ docker-compose up --scale datanode=3 -d
$ docker exec -it ozone_s3g_1 bash
bash-4.2$ export AWS_ACCESS_KEY=test AWS_SECRET_KEY=pass
bash-4.2$ ozone freon s3bg -t 1 -n 10

on your browser go to http://localhost:9874/prom and you should see

# TYPE org_apache_hadoop_ipc_decay_rpc_scheduler_priority counter
org_apache_hadoop_ipc_decay_rpc_scheduler_priority{context="ipc.9862",hostname="e32f2e3bddb9",username="hadoop"} 1
...
...
...
# TYPE org_apache_hadoop_ipc_decay_rpc_scheduler_volume counter
org_apache_hadoop_ipc_decay_rpc_scheduler_volume{context="ipc.9862",hostname="e32f2e3bddb9",username="hadoop"} 21

[kerneltime]

CC @duongkame

[xBis7]

@kerneltime any updates on this?

[duongkame]

Thanks @xBis7 for the patch.

The name is in the format Caller(username).MetricType eg. Caller(hadoop).Volume. The username might exist in the metric name for a purpose. We don't want to change the way the metric works but the way it's presented. The cleanest way to deal with this seems to filter the metric in PrometheusMetricsSink.

I still think this should rather be a change in hadoop-common. To keep it backward compatible, we may just introduce a new monitorable metric without changing the existing one. Capturing and modifying metrics in Ozone PrometheusMetricsSink looks pretty hacky, but if we have to do so, we'd rather copy the metrics to a new one without changing the original metric.

[kerneltime]

@xBis7 I understand the need, but I think there was intent in the original metric, and this comes from being able to diagnose which application/user is seeing what metric.
From: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FairCallQueue.html

The implementation of RpcScheduler used with FairCallQueue by default is DecayRpcScheduler, which maintains a count of requests received for each user.

Is this for ofs use cases or s3 based access?

[xBis7]

@kerneltime @duongkame Thanks for taking a look at the patch.

I think there was intent in the original metric, and this comes from being able to diagnose which application/user is seeing what metric.

I agree, that's why there are no functional changes. The metric that gets passed back and forth is exactly the same and we are only filtering how the endpoint presents it, like here with the RocksDb metrics.

Is this for ofs use cases or s3 based access?

It was used with S3 but it could be for both use cases.

I still think this should rather be a change in hadoop-common.

That was my first thought but it seems too complex for what we need. We don't want to change the metric or reduce the times it gets registered. We need the name to be consistent, so that it's easier to track with a wildcard search or something like that, but still want to be able to see the user for every registry.
If you think that's the best, I'm happy to move forward with it. We don't have to make the change to hadoop-commons but instead create a class in Ozone, that overrides the one from the jar.

[neils-dev]

Is this for ofs use cases or s3 based access?

This applies to both @kerneltime. This prometheus metric, org_apache_hadoop_ipc_decay_rpc_scheduler_caller_* includes the user in the metric name on all rpc metric events.

[neils-dev]

Thanks @xBis7 for this important patch. Thanks @duongkame and @kerneltime for reviewing this PR and for your comments.