Skip to content

Aligning metrics with OTel semantic conventions #4482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Oct 23, 2023
Merged

Aligning metrics with OTel semantic conventions #4482

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
151488a
Aligning metrics with OTel semantic convention
xakep139 Sep 27, 2023
0e76509
Merge branch 'release/8.0' into xakep139/4432-metric-names-convention
xakep139 Oct 4, 2023
eb3189c
fix the build
xakep139 Oct 4, 2023
3ca7aa2
convert WindowsCounters
xakep139 Oct 4, 2023
bd7a57c
resilience enrichment tags
xakep139 Oct 4, 2023
7cab4c8
again Windows counters
xakep139 Oct 4, 2023
98e3f0e
Update src/Libraries/Microsoft.AspNetCore.HeaderParsing/Metric.cs
xakep139 Oct 5, 2023
2967ec9
fixes + improve mutation score
xakep139 Oct 5, 2023
2fc604f
CR + fixes
xakep139 Oct 6, 2023
fa01223
remove redundant dimension
xakep139 Oct 6, 2023
3a66a5d
CR
xakep139 Oct 6, 2023
fa88043
Update src/Libraries/Microsoft.Extensions.Diagnostics.Testing/Metrics…
xakep139 Oct 6, 2023
3613c4d
fixes
xakep139 Oct 6, 2023
717678c
minor fix
xakep139 Oct 6, 2023
bca9575
Update src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitor…
xakep139 Oct 9, 2023
1887075
CR (Reiley)
xakep139 Oct 9, 2023
d036ccc
Merge branch 'release/8.0' into xakep139/4432-metric-names-convention
xakep139 Oct 10, 2023
43a73fe
CR (James)
xakep139 Oct 10, 2023
e10ccda
fixes in tests
xakep139 Oct 10, 2023
a8af544
fix tests
xakep139 Oct 10, 2023
fc8c3e0
CR
xakep139 Oct 17, 2023
fac0bb4
Update src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitor…
xakep139 Oct 17, 2023
ddd86c0
CR
xakep139 Oct 18, 2023
dde8e31
Merge branch 'release/8.0' into xakep139/4432-metric-names-convention
xakep139 Oct 23, 2023
c61c8d3
align Resilience attribute names
xakep139 Oct 23, 2023
0d244ba
fix test coverage
xakep139 Oct 23, 2023
42ac60e
Update src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitor…
xakep139 Oct 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/Libraries/Microsoft.AspNetCore.HeaderParsing/Metric.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ namespace Microsoft.AspNetCore.HeaderParsing;

internal static partial class Metric
{
[Counter("HeaderName", "Kind", Name = @"HeaderParsing.ParsingErrors")]
[Counter("header.name", "error.kind", Name = "aspnetcore.header_parsing.parse_errors")]
public static partial ParsingErrorCounter CreateParsingErrorCounter(Meter meter);

[Counter("HeaderName", "Type", Name = @"HeaderParsing.CacheAccess")]
[Counter("header.name", "access.type", Name = "aspnetcore.header_parsing.cache_accesses")]
public static partial CacheAccessCounter CreateCacheAccessCounter(Meter meter);
}
6 changes: 4 additions & 2 deletions src/Libraries/Microsoft.Extensions.Diagnostics.HealthChecks.Common/Metric.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@ namespace Microsoft.Extensions.Diagnostics.HealthChecks;

internal static partial class Metric
{
[Counter("healthy", "status", Name = @"R9\\HealthCheck\\Report")]
// TODO: should we rename "healthy" to "is_healthy"? Or maybe remove the attribute at all?
// To klauco: do we really need this true/false dimension? We already have "health.status" dimension.
[Counter("healthy", "health.status", Name = "health_check.reports")]
public static partial HealthCheckReportCounter CreateHealthCheckReportCounter(Meter meter);

[Counter("name", "status", Name = @"R9\\HealthCheck\\UnhealthyHealthCheck")]
[Counter("health_check.name", "health.status", Name = "health_check.unhealthy_checks")]
Copy link
Member

@noahfalk noahfalk Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fyi @timmydo - A little while back you were looking for a counter on health checks. Perhaps this functionality matches what you wanted?

public static partial UnhealthyHealthCheckCounter CreateUnhealthyHealthCheckCounter(Meter meter);

public static void RecordMetric(this HealthCheckReportCounter counterMetric, bool isHealthy, HealthStatus status)
Expand Down
4 changes: 2 additions & 2 deletions ...ies/Microsoft.Extensions.Diagnostics.HealthChecks.Common/TelemetryHealthCheckPublisher.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ public Task PublishAsync(HealthReport report, CancellationToken cancellationToke
Log.Healthy(_logger, report.Status);
}

_metrics.HealthCheckReportCounter.RecordMetric(true, report.Status);
_metrics.HealthCheckReportCounter.RecordMetric(isHealthy: true, report.Status);
}
else
{
Expand Down Expand Up @@ -80,7 +80,7 @@ public Task PublishAsync(HealthReport report, CancellationToken cancellationToke
Log.Unhealthy(_logger, report.Status, stringBuilder);
PoolFactory.SharedStringBuilderPool.Return(stringBuilder);

_metrics.HealthCheckReportCounter.RecordMetric(false, report.Status);
_metrics.HealthCheckReportCounter.RecordMetric(isHealthy: false, report.Status);
}

return Task.CompletedTask;
Expand Down
21 changes: 11 additions & 10 deletions ...ies/Microsoft.Extensions.Diagnostics.ResourceMonitoring/Linux/LinuxUtilizationProvider.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ namespace Microsoft.Extensions.Diagnostics.ResourceMonitoring.Linux;

internal sealed class LinuxUtilizationProvider : ISnapshotProvider
{
private const float Hundred = 100.0f;
private const long Hundred = 100L;

private readonly object _cpuLocker = new();
private readonly object _memoryLocker = new();
private readonly LinuxUtilizationParser _parser;
Expand Down Expand Up @@ -48,7 +49,7 @@ public LinuxUtilizationProvider(IOptions<ResourceMonitoringOptions> options, Lin
var hostCpus = _parser.GetHostCpuCount();
var availableCpus = _parser.GetCgroupLimitedCpus();

_scale = hostCpus * Hundred / availableCpus;
_scale = hostCpus / availableCpus;
_scaleForTrackerApi = hostCpus / availableCpus;

#pragma warning disable CA2000 // Dispose objects before losing scope
Expand All @@ -58,13 +59,13 @@ public LinuxUtilizationProvider(IOptions<ResourceMonitoringOptions> options, Lin
var meter = meterFactory.Create("Microsoft.Extensions.Diagnostics.ResourceMonitoring");
#pragma warning restore CA2000 // Dispose objects before losing scope

_ = meter.CreateObservableGauge(name: ResourceUtilizationCounters.CpuConsumptionPercentage, observeValue: CpuPercentage);
_ = meter.CreateObservableGauge(name: ResourceUtilizationCounters.MemoryConsumptionPercentage, observeValue: MemoryPercentage);
_ = meter.CreateObservableGauge(name: ResourceUtilizationCounters.CpuUtilization, observeValue: CpuUtilization, unit: "1");
_ = meter.CreateObservableGauge(name: ResourceUtilizationCounters.MemoryUtilization, observeValue: MemoryUtilization, unit: "1");

Resources = new SystemResources(1, hostCpus, _totalMemoryInBytes, hostMemory);
}

public double CpuPercentage()
public double CpuUtilization()
{
var now = _timeProvider.GetUtcNow();
bool needUpdate = false;
Expand All @@ -91,7 +92,7 @@ public double CpuPercentage()

if (deltaHost > 0 && deltaCgroup > 0)
{
var percentage = Math.Min(Hundred, deltaCgroup / deltaHost * _scale);
var percentage = Math.Min(1.0, deltaCgroup / deltaHost * _scale);

_cpuPercentage = percentage;
_refreshAfterCpu = now.Add(_cpuRefreshInterval);
Expand All @@ -105,7 +106,7 @@ public double CpuPercentage()
return _cpuPercentage;
}

public double MemoryPercentage()
public double MemoryUtilization()
{
var now = _timeProvider.GetUtcNow();
bool needUpdate = false;
Expand All @@ -126,7 +127,7 @@ public double MemoryPercentage()
{
if (now >= _refreshAfterMemory)
{
var memoryPercentage = Math.Min(Hundred, (double)memoryUsed / _totalMemoryInBytes * Hundred);
var memoryPercentage = Math.Min(1.0, (double)memoryUsed / _totalMemoryInBytes);

_memoryPercentage = memoryPercentage;
_refreshAfterMemory = now.Add(_memoryRefreshInterval);
Expand All @@ -149,9 +150,9 @@ public Snapshot GetSnapshot()
var memoryUsed = _parser.GetMemoryUsageInBytes();

return new Snapshot(
totalTimeSinceStart: TimeSpan.FromTicks(hostTime / (long)Hundred),
totalTimeSinceStart: TimeSpan.FromTicks(hostTime / Hundred),
kernelTimeSinceStart: TimeSpan.Zero,
userTimeSinceStart: TimeSpan.FromTicks((long)(cgroupTime / (long)Hundred * _scaleForTrackerApi)),
userTimeSinceStart: TimeSpan.FromTicks((long)(cgroupTime / Hundred * _scaleForTrackerApi)),
memoryUsageInBytes: memoryUsed);
}
}
10 changes: 5 additions & 5 deletions ...raries/Microsoft.Extensions.Diagnostics.ResourceMonitoring/ResourceUtilizationCounters.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,24 +7,24 @@ namespace Microsoft.Extensions.Diagnostics.ResourceMonitoring;
/// Represents the names of instruments published by this package.
/// </summary>
/// <remarks>
/// These counters are currently only published on Linux.
/// These metrics are currently only published on Linux.
/// </remarks>
/// <seealso cref="System.Diagnostics.Metrics.Instrument"/>
public static class ResourceUtilizationCounters
{
/// <summary>
/// Gets the CPU consumption of the running application in percentages.
/// Gets the CPU consumption of the running application in range <c>[0, 1]</c>.
Copy link

@reyang reyang Oct 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a stepping back question regarding all utilizations measured by %.

Related discussions and PRs:

FYI @noahfalk IIRC for garbage collector metrics we decided to use absolute value (e.g. total time spent in GC since the beginning of the process lifecycle), and encourage users to derive utilization based on their actual need (so the user can choose whatever sliding window and sampling frequency).

Copy link
Member

@noahfalk noahfalk Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I recall this component has a sampling mechanism and sliding window built into it and the app developer can configure some of those parameters or accept defaults. So when this component reports an instantaneous utilization measurement of 73%, it really means that over the last X minutes some aggregation of Y CPU usage samples had an average CPU utilization of 73% for the app developer's chosen X and Y. Its not the pattern I'd expect most metrics to follow as there is extra implementation complexity and it can't accurately be re-aggregated to compute other sliding windows later. However because the app developer configures a sampling window of known fixed size it doesn't suffer from the issues that those other metrics had where they inferred a sample window from the polling frequency or from some unpredictable process behavior.

A while back I encouraged the engineers who were working on this to simplify and not maintain their own sliding window sampling reservoir, but I feel like what is there now is still well defined and likely useful as long as the sampling window is configured well up-front. I didn't object to this even though I expect metrics of this style will be an outlier in the long run. Having something like this shouldn't block us from later creating an alternate metric that emits absolute measures of CPU and memory usage with more aggregation flexibility.

@reyang do you feel like that addresses your concerns or you are still worried here?

Copy link

@reyang reyang Oct 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too concerned about having alternative metrics that customers can conveniently use - as long as there is clarity:

  1. What exactly does this metric mean. Refer to https://github.com/dotnet/extensions/pull/4482/files#r1347628433.
  2. What's the recommendation if there are multiple ways to get similar things - e.g. pros/cons, pitfalls, best practices. For example, https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Process.

Copy link

@reyang reyang Oct 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. What's the recommendation if there are multiple ways to get similar things - e.g. pros/cons, pitfalls, best practices. For example, https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Process.

@noahfalk given OpenTelemetry.Instrumentation.Process is still in Preview, I think we should make a decision about the direction - e.g. all CPU/Memory metrics will be exposed by Microsoft.Extensions.Diagnostics.ResourceMonitoring, and we will deprecate OpenTelemetry.Instrumentation.Process and point customers to Microsoft.Extensions.Diagnostics.ResourceMonitoring. Meanwhile we'll make sure Microsoft.Extensions.Diagnostics.ResourceMonitoring works for all supported version of the runtime + operating systems. Does this make sense and do you feel we can make a call before .NET 8 GA?

@Yun-Ting FYI since you've worked on OpenTelemetry.Instrumentation.Process.

Copy link
Member

@noahfalk noahfalk Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thought would be:
In .NET 8 - Point devs to these ResourceMonitoring metrics. Either deprecate OTel.Instrumentation.Process or adjust guidance so that we only suggest it only as a fallback for devs that need the metric in a specific form not already handled by ResourceMonitoring.

In the future (.NET 9?) - Implement the core CPU + memory metrics currently in OTel.Instrumentation.Process inside System.Diagnostic.DiagnosticSource. I think metrics like process.cpu.time or process.memory.virtual are so commonly desired that we shouldn't be asking devs to add extra library dependencies or new background polling services (serviceCollection.AddResourceMonitoring()) in order to enable them. At this point ResourceMonitoring would serve two roles going forward: (1) support for networking metrics which are a little more involved to configure (2) back-compat for projects already using the other process metrics defined here and wanting to continue doing so.

What does everyone think? 

Copy link

@reyang reyang Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thought would be: In .NET 8 - Point devs to these ResourceMonitoring metrics.

Just confirming - point devs that are using any supported version of .NET (core/framework), not just limited to folks who use .NET 8?

Copy link
Member

@noahfalk noahfalk Oct 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I meant as of November 2023 (or whenever this assembly releases as stable), all .NET devs should be able to use ResourceMonitoring to satisfy their need for CPU/Virtual Memory metrics.

NOTE: After I wrote that I saw that this library has a weirdly inconsistent approach to how it measures memory. I'm presuming that gets resolved. If it didn't then I wouldn't feel comfortable recommending it.

/// </summary>
/// <remarks>
/// The type of an instrument is <see cref="System.Diagnostics.Metrics.ObservableGauge{T}"/>.
/// </remarks>
public static string CpuConsumptionPercentage => "cpu_consumption_percentage";
public static string CpuUtilization => "process.cpu.utilization";

/// <summary>
/// Gets the memory consumption of the running application in percentages.
/// Gets the memory consumption of the running application in range <c>[0, 1]</c>.
/// </summary>
/// <remarks>
/// The type of an instrument is <see cref="System.Diagnostics.Metrics.ObservableGauge{T}"/>.
/// </remarks>
public static string MemoryConsumptionPercentage => "memory_consumption_percentage";
public static string MemoryUtilization => "process.memory.utilization";
}
Loading