From 1e0cadb19a9cc51777eb344ff36b3ab066eab7aa Mon Sep 17 00:00:00 2001 From: ChrsMark Date: Mon, 11 May 2026 10:19:54 +0300 Subject: [PATCH 1/4] Remove kernel from cpu.mode enum and clarify container cpu states Signed-off-by: ChrsMark --- .chloggen/fix_container_cpu_mode.yaml | 22 +++++++++++++++++ docs/registry/attributes/cpu.md | 1 - docs/runtime/dotnet-metrics.md | 1 - docs/system/container-metrics.md | 17 +++++++------ docs/system/process-metrics.md | 2 -- docs/system/system-metrics.md | 2 -- model/container/metrics.yaml | 35 ++++++++++++++++++++++----- model/cpu/registry.yaml | 4 --- 8 files changed, 60 insertions(+), 24 deletions(-) create mode 100644 .chloggen/fix_container_cpu_mode.yaml diff --git a/.chloggen/fix_container_cpu_mode.yaml b/.chloggen/fix_container_cpu_mode.yaml new file mode 100644 index 0000000000..f010ac0482 --- /dev/null +++ b/.chloggen/fix_container_cpu_mode.yaml @@ -0,0 +1,22 @@ +# Use this changelog template to create an entry for release notes. +# +# If your change doesn't affect end users you should instead start +# your pull request title with [chore] or use the "Skip Changelog" label. + +# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' +change_type: bug_fix + +# The name of the area of concern in the attributes-registry, (e.g. http, cloud, db) +component: container + +# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). +note: Remove kernel from cpu.mode enum and clarify container cpu states + +# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists. +# The values here must be integers. +issues: [3700] + +# (Optional) One or more lines of additional information to render under the primary note. +# These lines will be padded with 2 spaces and then inserted directly into the document. +# Use pipe (|) for multiline entries. +subtext: diff --git a/docs/registry/attributes/cpu.md b/docs/registry/attributes/cpu.md index 8c559fa969..3a3b48fa7e 100644 --- a/docs/registry/attributes/cpu.md +++ b/docs/registry/attributes/cpu.md @@ -23,7 +23,6 @@ Attributes specific to a cpu instance. | `idle` | Idle | ![Development](https://img.shields.io/badge/-development-blue) | | `interrupt` | Interrupt | ![Development](https://img.shields.io/badge/-development-blue) | | `iowait` | IO Wait | ![Development](https://img.shields.io/badge/-development-blue) | -| `kernel` | Kernel | ![Development](https://img.shields.io/badge/-development-blue) | | `nice` | Nice | ![Development](https://img.shields.io/badge/-development-blue) | | `steal` | Steal | ![Development](https://img.shields.io/badge/-development-blue) | | `system` | System | ![Development](https://img.shields.io/badge/-development-blue) | diff --git a/docs/runtime/dotnet-metrics.md b/docs/runtime/dotnet-metrics.md index bfe08dcf07..1e3371d2b1 100644 --- a/docs/runtime/dotnet-metrics.md +++ b/docs/runtime/dotnet-metrics.md @@ -96,7 +96,6 @@ This metric reports the same values as accessing the corresponding processor tim | `idle` | Idle | ![Development](https://img.shields.io/badge/-development-blue) | | `interrupt` | Interrupt | ![Development](https://img.shields.io/badge/-development-blue) | | `iowait` | IO Wait | ![Development](https://img.shields.io/badge/-development-blue) | -| `kernel` | Kernel | ![Development](https://img.shields.io/badge/-development-blue) | | `nice` | Nice | ![Development](https://img.shields.io/badge/-development-blue) | | `steal` | Steal | ![Development](https://img.shields.io/badge/-development-blue) | | `system` | System | ![Development](https://img.shields.io/badge/-development-blue) | diff --git a/docs/system/container-metrics.md b/docs/system/container-metrics.md index 27d578dd97..22414d6cc1 100644 --- a/docs/system/container-metrics.md +++ b/docs/system/container-metrics.md @@ -61,9 +61,9 @@ This metric is [recommended][MetricRecommended]. | Name | Instrument Type | Unit (UCUM) | Description | Stability | Entity Associations | | -------- | --------------- | ----------- | -------------- | --------- | ------ | -| `container.cpu.time` | Counter | `s` | Total CPU time consumed. [1] | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | [`container`](/docs/registry/entities/container.md#container) | +| `container.cpu.time` | Counter | `s` | CPU time consumed. [1] | ![Release Candidate](https://img.shields.io/badge/-rc-mediumorchid) | [`container`](/docs/registry/entities/container.md#container) | -**[1]:** Total CPU time consumed by the specific container on all available CPU cores +**[1]:** CPU time consumed by the specific container on all available CPU cores **Attributes:** @@ -71,9 +71,12 @@ This metric is [recommended][MetricRecommended]. | --- | --- | --- | --- | --- | --- | | [`cpu.mode`](/docs/registry/attributes/cpu.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Conditionally Required` [1] | string | The CPU mode for this data point. A container's CPU metric SHOULD be characterized _either_ by data points with no `mode` labels, _or only_ data points with `mode` labels. [2] | `user`; `system` | -**[1] `cpu.mode`:** Required if mode is available, i.e. metrics coming from the Docker Stats API. +**[1] `cpu.mode`:** Required if mode is available, i.e. metrics coming from the Docker Stats API, containerd stats or cAdvisor. -**[2] `cpu.mode`:** Following states SHOULD be used: `user`, `system`, `kernel` +**[2] `cpu.mode`:** Following modes SHOULD be used for containers if available: `user`, `system`. +In implementations like [`opencontainers/cgroup`](https://pkg.go.dev/github.com/opencontainers/cgroups@v0.0.6#CpuUsage), [Moby/Docker](https://pkg.go.dev/github.com/Moby/Moby@v28.5.2+incompatible/api/types/container#CPUUsage) or [cAdvisor](https://pkg.go.dev/github.com/google/cadvisor@v0.56.2/info/v1#CpuUsage), the respective states are retrieved from the cgroup stats directly: In cgroup v1 the user/system modes come from the [cpuacct.stat](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpuacct.html#cpu-accounting-controller) file and the `user`/`system` stats respectively In cgroup v2 the user/system modes come from the [cpu.stat](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files) file and the `user_usec`/`system_usec` stats respectively +Kubernetes only exposes the total CPU time ([cpu.stat.usage_usec](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files)) through Kubelet's stats API, hence the metric should be reported without the mode attribute implying it is the total of all modes. +In pure container environments the CPU's mode is usually available and cpu.mode should be reported. In that case summarising the CPU time over the different modes gives the total CPU time. --- @@ -84,7 +87,6 @@ This metric is [recommended][MetricRecommended]. | `idle` | Idle | ![Development](https://img.shields.io/badge/-development-blue) | | `interrupt` | Interrupt | ![Development](https://img.shields.io/badge/-development-blue) | | `iowait` | IO Wait | ![Development](https://img.shields.io/badge/-development-blue) | -| `kernel` | Kernel | ![Development](https://img.shields.io/badge/-development-blue) | | `nice` | Nice | ![Development](https://img.shields.io/badge/-development-blue) | | `steal` | Steal | ![Development](https://img.shields.io/badge/-development-blue) | | `system` | System | ![Development](https://img.shields.io/badge/-development-blue) | @@ -115,9 +117,9 @@ This metric is [opt-in][MetricOptIn]. | --- | --- | --- | --- | --- | --- | | [`cpu.mode`](/docs/registry/attributes/cpu.md) | ![Development](https://img.shields.io/badge/-development-blue) | `Conditionally Required` [1] | string | The CPU mode for this data point. A container's CPU metric SHOULD be characterized _either_ by data points with no `mode` labels, _or only_ data points with `mode` labels. [2] | `user`; `system` | -**[1] `cpu.mode`:** Required if mode is available, i.e. metrics coming from the Docker Stats API. +**[1] `cpu.mode`:** Required if mode is available, i.e. metrics coming from the Docker Stats API, containerd stats or cAdvisor. -**[2] `cpu.mode`:** Following states SHOULD be used: `user`, `system`, `kernel` +**[2] `cpu.mode`:** Following modes SHOULD be used for containers if available: `user`, `system`. --- @@ -128,7 +130,6 @@ This metric is [opt-in][MetricOptIn]. | `idle` | Idle | ![Development](https://img.shields.io/badge/-development-blue) | | `interrupt` | Interrupt | ![Development](https://img.shields.io/badge/-development-blue) | | `iowait` | IO Wait | ![Development](https://img.shields.io/badge/-development-blue) | -| `kernel` | Kernel | ![Development](https://img.shields.io/badge/-development-blue) | | `nice` | Nice | ![Development](https://img.shields.io/badge/-development-blue) | | `steal` | Steal | ![Development](https://img.shields.io/badge/-development-blue) | | `system` | System | ![Development](https://img.shields.io/badge/-development-blue) | diff --git a/docs/system/process-metrics.md b/docs/system/process-metrics.md index bd12e9a8c9..eb3c7ff079 100644 --- a/docs/system/process-metrics.md +++ b/docs/system/process-metrics.md @@ -76,7 +76,6 @@ This metric is [recommended][MetricRecommended]. | `idle` | Idle | ![Development](https://img.shields.io/badge/-development-blue) | | `interrupt` | Interrupt | ![Development](https://img.shields.io/badge/-development-blue) | | `iowait` | IO Wait | ![Development](https://img.shields.io/badge/-development-blue) | -| `kernel` | Kernel | ![Development](https://img.shields.io/badge/-development-blue) | | `nice` | Nice | ![Development](https://img.shields.io/badge/-development-blue) | | `steal` | Steal | ![Development](https://img.shields.io/badge/-development-blue) | | `system` | System | ![Development](https://img.shields.io/badge/-development-blue) | @@ -116,7 +115,6 @@ This metric is [opt-in][MetricOptIn]. | `idle` | Idle | ![Development](https://img.shields.io/badge/-development-blue) | | `interrupt` | Interrupt | ![Development](https://img.shields.io/badge/-development-blue) | | `iowait` | IO Wait | ![Development](https://img.shields.io/badge/-development-blue) | -| `kernel` | Kernel | ![Development](https://img.shields.io/badge/-development-blue) | | `nice` | Nice | ![Development](https://img.shields.io/badge/-development-blue) | | `steal` | Steal | ![Development](https://img.shields.io/badge/-development-blue) | | `system` | System | ![Development](https://img.shields.io/badge/-development-blue) | diff --git a/docs/system/system-metrics.md b/docs/system/system-metrics.md index 51e698f31e..624ecfcfb4 100644 --- a/docs/system/system-metrics.md +++ b/docs/system/system-metrics.md @@ -175,7 +175,6 @@ This metric is [recommended][MetricRecommended]. | `idle` | Idle | ![Development](https://img.shields.io/badge/-development-blue) | | `interrupt` | Interrupt | ![Development](https://img.shields.io/badge/-development-blue) | | `iowait` | IO Wait | ![Development](https://img.shields.io/badge/-development-blue) | -| `kernel` | Kernel | ![Development](https://img.shields.io/badge/-development-blue) | | `nice` | Nice | ![Development](https://img.shields.io/badge/-development-blue) | | `steal` | Steal | ![Development](https://img.shields.io/badge/-development-blue) | | `system` | System | ![Development](https://img.shields.io/badge/-development-blue) | @@ -239,7 +238,6 @@ This metric is [opt-in][MetricOptIn]. | `idle` | Idle | ![Development](https://img.shields.io/badge/-development-blue) | | `interrupt` | Interrupt | ![Development](https://img.shields.io/badge/-development-blue) | | `iowait` | IO Wait | ![Development](https://img.shields.io/badge/-development-blue) | -| `kernel` | Kernel | ![Development](https://img.shields.io/badge/-development-blue) | | `nice` | Nice | ![Development](https://img.shields.io/badge/-development-blue) | | `steal` | Steal | ![Development](https://img.shields.io/badge/-development-blue) | | `system` | System | ![Development](https://img.shields.io/badge/-development-blue) | diff --git a/model/container/metrics.yaml b/model/container/metrics.yaml index 56e948dd6c..27cb1bdd15 100644 --- a/model/container/metrics.yaml +++ b/model/container/metrics.yaml @@ -23,9 +23,9 @@ groups: code_generation: metric_value_type: double stability: release_candidate - brief: "Total CPU time consumed." + brief: "CPU time consumed." note: > - Total CPU time consumed by the specific container on all available CPU cores + CPU time consumed by the specific container on all available CPU cores instrument: counter unit: "s" entity_associations: @@ -33,9 +33,30 @@ groups: attributes: - ref: cpu.mode brief: "The CPU mode for this data point. A container's CPU metric SHOULD be characterized _either_ by data points with no `mode` labels, _or only_ data points with `mode` labels." - note: "Following states SHOULD be used: `user`, `system`, `kernel`" + note: > + Following modes SHOULD be used for containers if available: `user`, `system`. + + In implementations like [`opencontainers/cgroup`](https://pkg.go.dev/github.com/opencontainers/cgroups@v0.0.6#CpuUsage), + [Moby/Docker](https://pkg.go.dev/github.com/Moby/Moby@v28.5.2+incompatible/api/types/container#CPUUsage) + or [cAdvisor](https://pkg.go.dev/github.com/google/cadvisor@v0.56.2/info/v1#CpuUsage), the respective states + are retrieved from the cgroup stats directly: + In cgroup v1 the user/system modes come from the + [cpuacct.stat](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpuacct.html#cpu-accounting-controller) + file and the `user`/`system` stats respectively + In cgroup v2 the user/system modes come from the + [cpu.stat](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files) file + and the `user_usec`/`system_usec` stats respectively + + Kubernetes only exposes the total CPU time + ([cpu.stat.usage_usec](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files)) + through Kubelet's stats API, hence the metric should be + reported without the mode attribute implying it is the total of all modes. + + In pure container environments the CPU's mode is usually available and cpu.mode should be reported. + In that case summarising the CPU time over the different modes gives the total CPU time. requirement_level: - conditionally_required: Required if mode is available, i.e. metrics coming from the Docker Stats API. + conditionally_required: > + Required if mode is available, i.e. metrics coming from the Docker Stats API, containerd stats or cAdvisor. - id: metric.container.cpu.usage type: metric @@ -54,9 +75,11 @@ groups: attributes: - ref: cpu.mode brief: "The CPU mode for this data point. A container's CPU metric SHOULD be characterized _either_ by data points with no `mode` labels, _or only_ data points with `mode` labels." - note: "Following states SHOULD be used: `user`, `system`, `kernel`" + note: > + Following modes SHOULD be used for containers if available: `user`, `system`. requirement_level: - conditionally_required: Required if mode is available, i.e. metrics coming from the Docker Stats API. + conditionally_required: > + Required if mode is available, i.e. metrics coming from the Docker Stats API, containerd stats or cAdvisor. # container.memory.* metrics and attribute group - id: metric.container.memory.usage diff --git a/model/cpu/registry.yaml b/model/cpu/registry.yaml index 9d6e527b0e..a0ae7a310c 100644 --- a/model/cpu/registry.yaml +++ b/model/cpu/registry.yaml @@ -37,10 +37,6 @@ groups: value: 'steal' brief: Steal stability: development - - id: kernel - value: 'kernel' - brief: Kernel - stability: development stability: development examples: [ "user", "system" ] - id: cpu.logical_number From 430633434138bc157c85a65a78d1c938095ac443 Mon Sep 17 00:00:00 2001 From: Christos Markou Date: Thu, 14 May 2026 10:28:01 +0300 Subject: [PATCH 2/4] Update model/container/metrics.yaml Co-authored-by: Roger Coll --- model/container/metrics.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/model/container/metrics.yaml b/model/container/metrics.yaml index 27cb1bdd15..a49d4197ac 100644 --- a/model/container/metrics.yaml +++ b/model/container/metrics.yaml @@ -39,13 +39,13 @@ groups: In implementations like [`opencontainers/cgroup`](https://pkg.go.dev/github.com/opencontainers/cgroups@v0.0.6#CpuUsage), [Moby/Docker](https://pkg.go.dev/github.com/Moby/Moby@v28.5.2+incompatible/api/types/container#CPUUsage) or [cAdvisor](https://pkg.go.dev/github.com/google/cadvisor@v0.56.2/info/v1#CpuUsage), the respective states - are retrieved from the cgroup stats directly: + are retrieved from the cgroup stats directly. In cgroup v1 the user/system modes come from the [cpuacct.stat](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpuacct.html#cpu-accounting-controller) - file and the `user`/`system` stats respectively + file and the `user`/`system` stats respectively. In cgroup v2 the user/system modes come from the [cpu.stat](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files) file - and the `user_usec`/`system_usec` stats respectively + and the `user_usec`/`system_usec` stats respectively. Kubernetes only exposes the total CPU time ([cpu.stat.usage_usec](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files)) From 004b94f2b938ce6090d1e9562ce7c2b189e6f2e9 Mon Sep 17 00:00:00 2001 From: ChrsMark Date: Thu, 14 May 2026 10:28:55 +0300 Subject: [PATCH 3/4] make fix Signed-off-by: ChrsMark --- docs/system/container-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/system/container-metrics.md b/docs/system/container-metrics.md index 22414d6cc1..3a43aff317 100644 --- a/docs/system/container-metrics.md +++ b/docs/system/container-metrics.md @@ -74,7 +74,7 @@ This metric is [recommended][MetricRecommended]. **[1] `cpu.mode`:** Required if mode is available, i.e. metrics coming from the Docker Stats API, containerd stats or cAdvisor. **[2] `cpu.mode`:** Following modes SHOULD be used for containers if available: `user`, `system`. -In implementations like [`opencontainers/cgroup`](https://pkg.go.dev/github.com/opencontainers/cgroups@v0.0.6#CpuUsage), [Moby/Docker](https://pkg.go.dev/github.com/Moby/Moby@v28.5.2+incompatible/api/types/container#CPUUsage) or [cAdvisor](https://pkg.go.dev/github.com/google/cadvisor@v0.56.2/info/v1#CpuUsage), the respective states are retrieved from the cgroup stats directly: In cgroup v1 the user/system modes come from the [cpuacct.stat](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpuacct.html#cpu-accounting-controller) file and the `user`/`system` stats respectively In cgroup v2 the user/system modes come from the [cpu.stat](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files) file and the `user_usec`/`system_usec` stats respectively +In implementations like [`opencontainers/cgroup`](https://pkg.go.dev/github.com/opencontainers/cgroups@v0.0.6#CpuUsage), [Moby/Docker](https://pkg.go.dev/github.com/Moby/Moby@v28.5.2+incompatible/api/types/container#CPUUsage) or [cAdvisor](https://pkg.go.dev/github.com/google/cadvisor@v0.56.2/info/v1#CpuUsage), the respective states are retrieved from the cgroup stats directly. In cgroup v1 the user/system modes come from the [cpuacct.stat](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cpuacct.html#cpu-accounting-controller) file and the `user`/`system` stats respectively. In cgroup v2 the user/system modes come from the [cpu.stat](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files) file and the `user_usec`/`system_usec` stats respectively. Kubernetes only exposes the total CPU time ([cpu.stat.usage_usec](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files)) through Kubelet's stats API, hence the metric should be reported without the mode attribute implying it is the total of all modes. In pure container environments the CPU's mode is usually available and cpu.mode should be reported. In that case summarising the CPU time over the different modes gives the total CPU time. From 95302f51a98a1edb12025901de0d262de717b26f Mon Sep 17 00:00:00 2001 From: ChrsMark Date: Thu, 14 May 2026 10:54:09 +0300 Subject: [PATCH 4/4] deprecate instead of removing Signed-off-by: ChrsMark --- model/cpu/registry.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/model/cpu/registry.yaml b/model/cpu/registry.yaml index a0ae7a310c..d459cee6d1 100644 --- a/model/cpu/registry.yaml +++ b/model/cpu/registry.yaml @@ -37,6 +37,13 @@ groups: value: 'steal' brief: Steal stability: development + - id: kernel + value: 'kernel' + stability: development + brief: "Deprecated. Use `system` instead." + deprecated: + reason: renamed + renamed_to: "system" stability: development examples: [ "user", "system" ] - id: cpu.logical_number