HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics #6070

aierate · 2024-01-23T04:07:29Z

What changes were proposed in this pull request?

Use the Performance replica the MutableRate XceiverClient Metrics of XceiverClientMetrics for Client and S3G.
The Performance including MutableQuantiles and MutableMinMax and MutableStat Metrics, the MutableStat is same with original MutableRate Metrics.
Replacing MutableRate with Performance ensures that the original MutableRate remains unchanged, and adds configurable MutableQuantiles and MutableMinMax metrics.
Reference from the following link：
HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics #5627

Before

http://ozone:9878/jmx?qry=Hadoop:service=S3Gateway,name=XceiverClientMetrics

 {
    "name" : "Hadoop:service=S3Gateway,name=XceiverClientMetrics",
    "modelerType" : "XceiverClientMetrics",
    "tag.Context" : "dfs",
    "tag.Hostname" : "conway-hadoop3",
    "numPendingCreateContainer" : 0,
    "opCountCreateContainer" : 0,
    "CreateContainerLatencyNumOps" : 0,
    "CreateContainerLatencyAvgTime" : 0.0,
    
    .
    .
    .

    "opCountWriteChunk" : 0,
    "WriteChunkLatencyNumOps" : 0,
    "WriteChunkLatencyAvgTime" : 0,
    
    .
    . 
    .
   
    "numPendingStreamWrite" : 0,
    "opCountStreamWrite" : 0,
    "StreamWriteLatencyNumOps" : 0,
    "StreamWriteLatencyAvgTime" : 0.0,
    "EcReconstructionFailsTotal" : 0,
    "EcReconstructionTotal" : 0,
    "PendingOps" : 0,
    "TotalOps" : 0
  },

After

http://ozone:9878/jmx?qry=Hadoop:service=S3Gateway,name=XceiverClientMetrics
Default (Add MutableMinMax Metrics):

 {
    "name" : "Hadoop:service=S3Gateway,name=XceiverClientMetrics",
    "modelerType" : "XceiverClientMetrics",
    "tag.Context" : "dfs",
    "tag.Hostname" : "conway-hadoop3",
    "numPendingCreateContainer" : 0,
    "opCountCreateContainer" : 0,
    "CreateContainerLatencyNumOps" : 0,
    "CreateContainerLatencyAvgTime" : 0.0,
    "CreateContainerLatencyIMinTime" : 0,
    "CreateContainerLatencyIMaxTime" : 0,
    
    .
    .
    .

    "opCountWriteChunk" : 1,
    "WriteChunkLatencyNumOps" : 1,
    "WriteChunkLatencyAvgTime" : 701.0,
    "WriteChunkLatencyIMinTime" : 0,
    "WriteChunkLatencyIMaxTime" : 0,
    
    .
    . 
    .
   
    "numPendingStreamWrite" : 0,
    "opCountStreamWrite" : 0,
    "StreamWriteLatencyNumOps" : 0,
    "StreamWriteLatencyAvgTime" : 0.0,
    "StreamWriteLatencyIMinTime" : 0,
    "StreamWriteLatencyIMaxTime" : 0,
    "EcReconstructionFailsTotal" : 0,
    "EcReconstructionTotal" : 0,
    "PendingOps" : 0,
    "TotalOps" : 2
  },

Enable quantiles Metrics:

<property>
    <name>ozone.xceiver.metrics.percentiles.intervals.seconds</name>
    <value>60,300</value>
  </property>

http://ozone:9878/jmx?qry=Hadoop:service=S3Gateway,name=XceiverClientMetrics

 {
    "name" : "Hadoop:service=S3Gateway,name=XceiverClientMetrics",
    "modelerType" : "XceiverClientMetrics",
    "tag.Context" : "dfs",
    "tag.Hostname" : "conway-hadoop3",
    "numPendingCreateContainer" : 0,
    "opCountCreateContainer" : 0,
    "CreateContainerLatencyNumOps" : 0,
    "CreateContainerLatencyAvgTime" : 0.0,
    "CreateContainerLatency60sNumOps" : 0,
    "CreateContainerLatency60s50thPercentileTime" : 0,
    "CreateContainerLatency60s75thPercentileTime" : 0,
    "CreateContainerLatency60s90thPercentileTime" : 0,
    "CreateContainerLatency60s95thPercentileTime" : 0,
    "CreateContainerLatency60s99thPercentileTime" : 0,
    "CreateContainerLatency300sNumOps" : 0,
    "CreateContainerLatency300s50thPercentileTime" : 0,
    "CreateContainerLatency300s75thPercentileTime" : 0,
    "CreateContainerLatency300s90thPercentileTime" : 0,
    "CreateContainerLatency300s95thPercentileTime" : 0,
    "CreateContainerLatency300s99thPercentileTime" : 0,
    .
    .
    .
    "numPendingWriteChunk" : 0,
    "opCountWriteChunk" : 1,
    "WriteChunkLatencyNumOps" : 1,
    "WriteChunkLatencyAvgTime" : 722.0,
    "WriteChunkLatency60sNumOps" : 1,
    "WriteChunkLatency60s50thPercentileTime" : 722,
    "WriteChunkLatency60s75thPercentileTime" : 722,
    "WriteChunkLatency60s90thPercentileTime" : 722,
    "WriteChunkLatency60s95thPercentileTime" : 722,
    "WriteChunkLatency60s99thPercentileTime" : 722,
    "WriteChunkLatency300sNumOps" : 0,
    "WriteChunkLatency300s50thPercentileTime" : 0,
    "WriteChunkLatency300s75thPercentileTime" : 0,
    "WriteChunkLatency300s90thPercentileTime" : 0,
    "WriteChunkLatency300s95thPercentileTime" : 0,
    "WriteChunkLatency300s99thPercentileTime" : 0,
    "WriteChunkLatencyIMinTime" : 0,
    "WriteChunkLatencyIMaxTime" : 0,
    .
    .
    .
    "numPendingStreamWrite" : 0,
    "opCountStreamWrite" : 0,
    "StreamWriteLatencyNumOps" : 0,
    "StreamWriteLatencyAvgTime" : 0.0,
    "StreamWriteLatency60sNumOps" : 0,
    "StreamWriteLatency60s50thPercentileTime" : 0,
    "StreamWriteLatency60s75thPercentileTime" : 0,
    "StreamWriteLatency60s90thPercentileTime" : 0,
    "StreamWriteLatency60s95thPercentileTime" : 0,
    "StreamWriteLatency60s99thPercentileTime" : 0,
    "StreamWriteLatency300sNumOps" : 0,
    "StreamWriteLatency300s50thPercentileTime" : 0,
    "StreamWriteLatency300s75thPercentileTime" : 0,
    "StreamWriteLatency300s90thPercentileTime" : 0,
    "StreamWriteLatency300s95thPercentileTime" : 0,
    "StreamWriteLatency300s99thPercentileTime" : 0,
    "StreamWriteLatencyIMinTime" : 0,
    "StreamWriteLatencyIMaxTime" : 0,
    "EcReconstructionFailsTotal" : 0,
    "EcReconstructionTotal" : 0,
    "PendingOps" : 0,
    "TotalOps" : 2
  }

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10182

How was this patch tested?

Manually Test

aierate · 2024-01-23T11:25:38Z

@xichen01 Can you help me review the code? Thank you.

adoroszlai

Thanks @aierate for the patch.

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java

adoroszlai · 2024-01-24T12:40:33Z

@tanvipenumudy please take a look

xichen01 · 2024-01-24T14:51:17Z

@tanvipenumudy Thanks for you Patch.

You need to have XceiverClientMetrics implement MetricsSource and override getMetrics.
just like:

public final class XceiverClientMetrics implements MetricsSource {
  @Override
  public void (MetricsCollector collector, boolean all) {
    //....
  }
}

This is because Hadoop Metrics only supports the automatic retrieval of values from Hadoop's built-in classes. MutableMinMax is a custom class, so if getMetrics is not overridden, then MutableMinMax will not be updated.
You can test that the Min/Max metric will always be 0 in the current implementation.

aierate · 2024-01-25T04:10:35Z

@tanvipenumudy Thanks for you Patch.

You need to have XceiverClientMetrics implement MetricsSource and override getMetrics. just like:
public final class XceiverClientMetrics implements MetricsSource {
  @Override
  public void (MetricsCollector collector, boolean all) {
    //....
  }
}
This is because Hadoop Metrics only supports the automatic retrieval of values from Hadoop's built-in classes. MutableMinMax is a custom class, so if getMetrics is not overridden, then MutableMinMax will not be updated. You can test that the Min/Max metric will always be 0 in the current implementation.

@xichen01 Thanks for your suggestion very much. I am fixed this bug, please take a look.

aierate · 2024-01-25T04:40:02Z

@tanvipenumudy Thanks for you Patch.
You need to have XceiverClientMetrics implement MetricsSource and override getMetrics. just like:
public final class XceiverClientMetrics implements MetricsSource {
  @Override
  public void (MetricsCollector collector, boolean all) {
    //....
  }
}
This is because Hadoop Metrics only supports the automatic retrieval of values from Hadoop's built-in classes. MutableMinMax is a custom class, so if getMetrics is not overridden, then MutableMinMax will not be updated. You can test that the Min/Max metric will always be 0 in the current implementation.
@xichen01 Thanks for your suggestion very much. I am fixed this bug, please take a look.

Here is the jmx after fixed this bug.
http://ozone:9878/jmx?qry=Hadoop:service=S3Gateway,name=XceiverClientMetrics

{
    "name" : "Hadoop:service=S3Gateway,name=XceiverClientMetrics",
    "modelerType" : "XceiverClientMetrics",
    "tag.Context" : "dfs",
    "tag.Hostname" : "conway-hadoop3",
    "PendingOps" : 0,
    "TotalOps" : 4,
    "EcReconstructionTotal" : 0,
    "EcReconstructionFailsTotal" : 0,
    "numPendingCreateContainer" : 0,
    "opCountCreateContainer" : 0,
    "CreateContainerLatencyNumOps" : 0,
    "CreateContainerLatencyAvgTime" : 0.0,
    "CreateContainerLatency60sNumOps" : 0,
    "CreateContainerLatency60s50thPercentileTime" : 0,
    "CreateContainerLatency60s75thPercentileTime" : 0,
    "CreateContainerLatency60s90thPercentileTime" : 0,
    "CreateContainerLatency60s95thPercentileTime" : 0,
    "CreateContainerLatency60s99thPercentileTime" : 0,
    "CreateContainerLatency300sNumOps" : 0,
    "CreateContainerLatency300s50thPercentileTime" : 0,
    "CreateContainerLatency300s75thPercentileTime" : 0,
    "CreateContainerLatency300s90thPercentileTime" : 0,
    "CreateContainerLatency300s95thPercentileTime" : 0,
    "CreateContainerLatency300s99thPercentileTime" : 0,
    "CreateContainerLatencyIMinTime" : 3.4028234663852886E38,
    "CreateContainerLatencyIMaxTime" : 1.401298464324817E-45,
   .
   .
   .
   "numPendingPutBlock" : 0,
    "opCountPutBlock" : 2,
    "PutBlockLatencyNumOps" : 2,
    "PutBlockLatencyAvgTime" : 30.0,
    "PutBlockLatency60sNumOps" : 1,
    "PutBlockLatency60s50thPercentileTime" : 30,
    "PutBlockLatency60s75thPercentileTime" : 30,
    "PutBlockLatency60s90thPercentileTime" : 30,
    "PutBlockLatency60s95thPercentileTime" : 30,
    "PutBlockLatency60s99thPercentileTime" : 30,
    "PutBlockLatency300sNumOps" : 0,
    "PutBlockLatency300s50thPercentileTime" : 0,
    "PutBlockLatency300s75thPercentileTime" : 0,
    "PutBlockLatency300s90thPercentileTime" : 0,
    "PutBlockLatency300s95thPercentileTime" : 0,
    "PutBlockLatency300s99thPercentileTime" : 0,
    "PutBlockLatencyIMinTime" : 30.0,
    "PutBlockLatencyIMaxTime" : 30.0,
   . 
   .
   .
   "numPendingStreamWrite" : 0,
    "opCountStreamWrite" : 0,
    "StreamWriteLatencyNumOps" : 0,
    "StreamWriteLatencyAvgTime" : 0.0,
    "StreamWriteLatency60sNumOps" : 0,
    "StreamWriteLatency60s50thPercentileTime" : 0,
    "StreamWriteLatency60s75thPercentileTime" : 0,
    "StreamWriteLatency60s90thPercentileTime" : 0,
    "StreamWriteLatency60s95thPercentileTime" : 0,
    "StreamWriteLatency60s99thPercentileTime" : 0,
    "StreamWriteLatency300sNumOps" : 0,
    "StreamWriteLatency300s50thPercentileTime" : 0,
    "StreamWriteLatency300s75thPercentileTime" : 0,
    "StreamWriteLatency300s90thPercentileTime" : 0,
    "StreamWriteLatency300s95thPercentileTime" : 0,
    "StreamWriteLatency300s99thPercentileTime" : 0,
    "StreamWriteLatencyIMinTime" : 3.4028234663852886E38,
    "StreamWriteLatencyIMaxTime" : 1.401298464324817E-45
  }

aierate · 2024-01-25T11:11:06Z

All unit tests have passed. https://github.com/aierate/ozone/actions

aierate · 2024-01-30T09:38:05Z

@xBis7 Can you help me review the code? Thank you.

…Performance Metrics

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java

xichen01 · 2024-02-01T11:04:18Z

LGTM

adoroszlai · 2024-02-01T13:41:04Z

@duongkame @kerneltime @tanvipenumudy please take a look

xichen01 · 2024-03-19T07:37:40Z

Thanks @aierate for the patch, @kerneltime @xichen01 for the review.

aierate changed the title ~~HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics for S3G and Client~~ HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics Jan 23, 2024

adoroszlai reviewed Jan 24, 2024

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java Outdated Show resolved Hide resolved

adoroszlai requested review from duongkame and kerneltime January 24, 2024 12:40

adoroszlai added the performance label Jan 24, 2024

aierate force-pushed the HDDS-10182 branch from 87fb0ef to 339d180 Compare January 25, 2024 09:44

aierate force-pushed the HDDS-10182 branch from 339d180 to 4368f96 Compare January 30, 2024 03:40

HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client …

8aad675

…Performance Metrics

aierate force-pushed the HDDS-10182 branch from 4368f96 to 8aad675 Compare February 1, 2024 03:06

xichen01 reviewed Feb 1, 2024

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java Outdated Show resolved Hide resolved

HDDS-10182. change config name

6dbf772

xichen01 changed the title ~~HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics~~ HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics Feb 1, 2024

adoroszlai added the metrics label Mar 13, 2024

Merge branch 'master' into HDDS-10182

611b409

kerneltime approved these changes Mar 14, 2024

View reviewed changes

xichen01 merged commit d4606e1 into apache:master Mar 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics #6070

HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics #6070

Uh oh!

aierate commented Jan 23, 2024 •

edited

Loading

Uh oh!

aierate commented Jan 23, 2024

Uh oh!

adoroszlai left a comment

Uh oh!

Uh oh!

adoroszlai commented Jan 24, 2024

Uh oh!

xichen01 commented Jan 24, 2024

Uh oh!

aierate commented Jan 25, 2024 •

edited

Loading

Uh oh!

aierate commented Jan 25, 2024

Uh oh!

aierate commented Jan 25, 2024

Uh oh!

aierate commented Jan 30, 2024

Uh oh!

Uh oh!

xichen01 commented Feb 1, 2024

Uh oh!

adoroszlai commented Feb 1, 2024

Uh oh!

xichen01 commented Mar 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics #6070

HDDS-10182. Add P99 quantiles and Min/Max Metrics for Xceiver Client Performance Metrics #6070

Uh oh!

Conversation

aierate commented Jan 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Before

After

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

aierate commented Jan 23, 2024

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai commented Jan 24, 2024

Uh oh!

xichen01 commented Jan 24, 2024

Uh oh!

aierate commented Jan 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aierate commented Jan 25, 2024

Uh oh!

aierate commented Jan 25, 2024

Uh oh!

aierate commented Jan 30, 2024

Uh oh!

Uh oh!

xichen01 commented Feb 1, 2024

Uh oh!

adoroszlai commented Feb 1, 2024

Uh oh!

xichen01 commented Mar 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aierate commented Jan 23, 2024 •

edited

Loading

aierate commented Jan 25, 2024 •

edited

Loading