[k8s] Promote k8s.pod.cpu.time, k8s.node.cpu.time and container.cpu.time metrics to release_candidate

[ChrsMark]

Related to #3001

Changes

This PR aims to advertise the intention for stabilising the K8s/container metrics:

• k8s.pod.cpu.time
• k8s.node.cpu.time
• container.cpu.time

These metrics have been in use for long time now in the OpenTelemetry Collector and specifically the kubeletstatsreceiver:

• container.cpu.time
• k8s.pod.cpu.time
• k8s.node.cpu.time

Requirement levels have been discussed at #3001 (comment)

TODOS:

• Stabilise cpu.mode which is conditionally required for container.cpu.time.

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• Links to the prototypes or existing instrumentations (when adding or changing conventions)

[ChrsMark]

@open-telemetry/specs-semconv-approvers it's still unclear if for metrics to be promoted through stability levels their respective Entitites should be first promoted. Discussing this in the System SemConv SIG last week, @dmitryax mentioned that Entities should not block metrics' graduation while they are still being worked. But we need to verify this.

Other than this, we still need to stabilise cpu.mode which is conditionally required for container.cpu.time.

[dashpole]

lgtm, assuming other blockers are resolved

[ChrsMark]

@dashpole I spent some time verifying the container.cpu.time metric and cpu.mode attribute. I'm sharing my findings here along with some concerns that I want to check before proceeding with stabilising these.

Issue

container.cpu.time of OpenTelemetry's Semantic Conventions define the cpu.mode attribute as conditionally required:

Required if mode is available, i.e. metrics coming from the Docker Stats API. with values stated as
Following states SHOULD be used: user, system, kernel

(note that system is actually kernel and we should remove kernel only keeping system. That is explained below)

This means that dockerstats receiver can produce the container.cpu.time metric with the cpu.mode attribute but the kubeletstats receiver only provides the total without the cpu.mode attribute where total != user + system

Kuberentes/kubeletstats path

Opencontainers cgroup library calcalates the CPU stats at (for cgroupd v1) https://github.com/opencontainers/cgroups/blob/v0.0.6/fs/cpuacct.go#L37:

totalUsage, err := fscommon.GetCgroupParamUint(path, "cpuacct.usage")
// → stats.CpuStats.CpuUsage.TotalUsage

userModeUsage, kernelModeUsage, err := getCpuUsageBreakdown(path)
// reads "cpuacct.stat" (user / system lines)
// → stats.CpuStats.CpuUsage.UsageInUsermode
// → stats.CpuStats.CpuUsage.UsageInKernelmode

From https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpuacct.html:

cpuacct.usage gives the CPU time (in nanoseconds) obtained by this group which is essentially the CPU time obtained by all the tasks in the system.

cpuacct.stat file lists a few statistics which further divide the CPU time obtained by the cgroup into user and system times. Currently the following statistics are supported:
user: Time spent by tasks of the cgroup in user mode. system: Time spent by tasks of the cgroup in kernel mode.

In cgroup V2 the calculation happens in https://github.com/opencontainers/cgroups/blob/v0.0.6/fs2/cpu.go#L79-L101
and is based on the cpu.stat(usage_usec, user_usec, system_usec).

CPU Stats struct: https://github.com/opencontainers/cgroups/blob/v0.0.6/stats.go#L22-L41

cAdvisor

cAdvisor derives the above metrics from the library and sets them at https://github.com/google/cadvisor/blob/v0.56.2/container/libcontainer/handler.go#L769-L772.

cAdvisor's internal structs: https://github.com/google/cadvisor/blob/v0.56.2/info/v1/container.go#L303-L319

cAdvisor to Prometheus

Those stats are exposed eventually as Prometheus metrics at https://github.com/google/cadvisor/blob/v0.56.2/metrics/prometheus.go#L151-L188:

• container_cpu_user_seconds_total: Cumulative user cpu time consumed in seconds
• container_cpu_system_seconds_total: Cumulative system cpu time consumed in seconds
• container_cpu_usage_seconds_total: Cumulative cpu time consumed in seconds

Notes

1. user + system ≠ total
2. system==kernel (same applies for docker stats so we can remove the kernel mode and only use system)

Why user + system ≠ total? On cgroup v1 they come from two different kernel accounting mechanisms — cpuacct.usage (nanosecond-resolution, high-fidelity) vs cpuacct.stat (jiffie-based, 10ms resolution). This discrepancy persists on cgroup v2 but is smaller (microsecond vs microsecond, but usage_usec includes more accounting than just user+system).

cAdvisor to Kubelet' /stats/summary API

At cadvisorInfoToCPUandMemoryStats() Kubelet retrieves the Cpu.Usage.Total of the v1.CpuStats and sets it to the UsageCoreNanoSeconds. This the only metric that the Kubelet stats API exposes and what we use in the kubeletstats receiver to set the container.cpu.time.

Hence the sequence for the K8s path is the following:

cpuacct.usage 
 -> CpuStats.CpuUsage.TotalUsage (opencontainers/cgroups lib) 
     -> v1.CpuStats.CpuUsage.Total (cAdvisor)
         -> container_cpu_user_seconds_total (prometheus metric of cAdvisor)
         -> UsageCoreNanoSeconds (kubelet's stats api) -> container.cpu.time (kubeletstats receiver)

Docker stats receiver path

The component uses the Moby library and retrieves the data from the CPUUsage struct.

In cgroup v2 the library uses the UsageUsec, SystemUsec and UserUsec from the kernel-> https://github.com/moby/moby/blob/v28.5.2/daemon/stats_unix.go#L184

In cgroup v1 the library uses the cpuacct.usage and cpuacct.stat.

Summary

1. cAdvisor and Docker/Moby provide the cpu time broken down by total, user, system/kernel
2. kubelet only exposes the total cpu which is not equal to user+system
3. We define the container.cpu.time as the common metric regardless of how it is collected (kubelet/k8s, docker, cadvisor etc).

However the cpu.mode would only be populated if user and system CPU times are available, which can happen only when collecting from Docker/Moby or cAdvisor. Kubelet only provides the total CPU time.

So all the underneath libraries and implementations align and we just need to ensure that our model in Otel SemConv is correct:

1. I suggest we change the kernel mode to system which already exists in the enum (-> remove the kernel).
2. I want to ensure that having the cpu.mode as optional is correct and will not break Prometheus compatability.

An example is that on the same system we can emit the container.cpu.time{} collected from kubeletstats receiver (total is implied) but at the same time dockerstats receiver can collect container.cpu.time{}, container.cpu.time{cpu.mode=user} and container.cpu.time{cpu.mode=system}.

In that case, is the following valid Prometheus metrics?

container.cpu.time{}
container.cpu.time{cpu.mode=user}
container.cpu.time{cpu.mode=system}

Note that container.cpu.time{cpu.mode=user} + container.cpu.time{cpu.mode=system} != container.cpu.time{}.

If I'm not mistaken container.cpu.time{} and container.cpu.time{cpu.mode=user|system} would be 2 different time-series families however to my eyes this looks confusing, and we should either add a cpu.mode=total option which is also weird or split the total into a different metric?

container.cpu.total_time{}
container.cpu.time{cpu.mode=user|system}

@dashpole @jsuereth @open-telemetry/semconv-k8s-approvers thoughts?

[dashpole]

Why user + system ≠ total? On cgroup v1 they come from two different kernel accounting mechanisms — cpuacct.usage (nanosecond-resolution, high-fidelity) vs cpuacct.stat (jiffie-based, 10ms resolution). This discrepancy persists on cgroup v2 but is smaller (microsecond vs microsecond, but usage_usec includes more accounting than just user+system).

Do we think this is a big enough difference to warrant a new metric? Is it possible to have the docker version populate the attribute, but kubelet doesn't populate it?

[ChrsMark]

Is it possible to have the docker version populate the attribute, but kubelet doesn't populate it?

That was the original idea but I think we need 3 different metrics/time-series here anyway. Dockerstats receiver can populate all 3 while kubeletstats only the total.

One option here is to keep container.cpu.time as is to not break k8s and remove the cpu.mode attribute entirely. This will be populated both in k8s and docker retrieved by the total/cpuacct.usage/etc.

Then to cover the 2 granular metrics/modes we can introduce a new different metric i.e. container.cpu.stat.time{cpu.mode=user|system} available for environments that interact directly with runtime APIs or even cAdvisor.

This is also close to the notion of croup v1:

cpuacct.stat file lists a few statistics which further divide the CPU time obtained by the cgroup into user and system times.

While cpuacct.usage -> total.

Open to other names/options here too but overall I think we can't avoid the split.

[dashpole]

That was the original idea but I think we need 3 different metrics/time-series here anyway. Dockerstats receiver can populate all 3 while kubeletstats only the total.

Sorry, i'm not following. If you only have the total, you just expose the metric without the attribute that splits into user/system. If user + system < total, then we can add an other category (or name it if we can identify it).

[ChrsMark]

Sorry, i'm not following. If you only have the total, you just expose the metric without the attribute that splits into user/system.

It's not clear to me what is the complete suggestion here. Could you share how you think about it both in the kubeletstats and the dockerstats receiver?

Note that today the dockerstats receiver provides 3 different metrics:

• container.cpu.usage.kernelmode
• container.cpu.usage.total
• container.cpu.usage.usermode

From my perspective it makes sense to be able to provide all these 3 if they are available. So how are we going to cover this given that total != user + system (for cgroup v1) based on what kernel's docs for cgroup stats mention?

(verifying this locally (for cgroup v2) with docker stats API, I found that total is occasionally 1 µs higher than user + kernel)

If user + system < total, then we can add an other category (or name it if we can identify it).

Would other mode fit in cpu.mode 🤔 ? I don't think we can identify the missing state, cgroup stats only give total, user and system:

• https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpuacct.html#cpu-accounting-controller
• https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files

[dashpole]

Expose one metric: container.cpu.usage

It has an attribute: cpu.mode, with user, system, other. User and system are fetched from cgroups. other = total - user - system (and is not allowed to be negative). The total of the metric stays correct. And user and system are also correct. Other is defined as something like "CPU usage which is part of the total, but does not fall into any defined category".

The kubeletstats receiver omits the cpu.mode attribute, and just provides container.cpu.usage as the total it has access to.

[ChrsMark]

Expose one metric: container.cpu.usage

I guess you mean container.cpu.time :) ?

It has an attribute: cpu.mode, with user, system, other. User and system are fetched from cgroups. other = total - user - system (and is not allowed to be negative). The total of the metric stays correct. And user and system are also correct. Other is defined as something like "CPU usage which is part of the total, but does not fall into any defined category".

The kubeletstats receiver omits the cpu.mode attribute, and just provides container.cpu.usage as the total it has access to.

Ok, so in that case the dockerstats receiver would only emit the 3 states (and not the total explicitely):

container.cpu.time{cpu.mode=user}
container.cpu.time{cpu.mode=system}
container.cpu.time{cpu.mode=other}

where the container.cpu.time{cpu.mode=other} is calculated by the receiver. And in that case the sum(container.cpu.time) is correct.

While the kubeletstats only emits the container.cpu.time{}.

I think that would be fine, if we accept other as a valid cpu.mode. I can verify this with @open-telemetry/semconv-system-approvers

[TylerHelmuth]

hype

[dashpole]

BTW @ChrsMark my original question from #3693 (comment), is: Is the other category actually significant enough that it is worth giving to users. I.e. if it always incredibly tiny, maybe we could just skip it, and accept that the total of kernel + user can be off by tiny amounts.

[ChrsMark]

BTW @ChrsMark my original question from #3693 (comment), is: Is the other category actually significant enough that it is worth giving to users. I.e. if it always incredibly tiny, maybe we could just skip it, and accept that the total of kernel + user can be off by tiny amounts.

I tried on my end the following (cgroup v2):

for i in $(seq 1 100); do
  curl -s --unix-socket /var/run/docker.sock \
    "http://localhost/containers/kind-control-plane/stats?stream=false" \
  | jq '.cpu_stats.cpu_usage'
done | jq -s '.'

and fed the results to an llm to analyse. The delta is always exactly 0 or +1000 ns. In approximately 45% of samples, the discrepancy always being exactly 1 µs. Might be a rounding "noise" in the cgroup's v2 accounting or sth. Given that container.cpu.time unit is seconds we talk about 0.000001 diff. Could we just skip it and probably mention something in the notes (trying that at https://github.com/open-telemetry/semantic-conventions/pull/3700/changes#diff-f75a3538a566f4f5da34963c1c365c1b4522e438e1a781572571a1cfcc4e644cL26, let me know what you think)?

[dashpole]

Yep! Thats what I was hoping for. We won't need the other case after all then

[ChrsMark]

Yep! Thats what I was hoping for. We won't need the other case after all then

@dashpole Ok! I have tried to update the cpu.mode part at #3700 to clarify things a bit, PTAL