Double amount of memory reported used by Pod when a pod has multiple containers.

[gerardnico]

Hallo,

I'm coming from Kubernetes Dashboard and I see a discrepancy in memory reported (double).

The time in the snapshot images is not aligned but I see roughly a double amount of memory reported by Headlamp versus Kubernetes Dashboard and Kubectl

Headlamp:

Kubernetes Dashboard:

Kubectl

kubectl top pod -n monitoring prometheus-kube-x-0

NAME                  CPU(cores)   MEMORY(bytes)   
prometheus-kube-x-0   411m         345Mi

A pod can have multiple containers and the metrics container_memory_working_set_bytes reports them.

In the snapshot above, this is the prometheus pod.

• the main container starts itself a image
• the pod has 2 containers.

It reports:

• 1 line for the total it seems
• 1 line for the image started by prometheus
• and 2 lines for each container

The pod has a total of 346714112 (first line) and you can see that this is also seem to be equal roughly to the total of the containers.

The actual query does not filter out the containers memory, making it double?:

memoryQuery={`sum(container_memory_working_set_bytes{namespace='${resource.jsonData.metadata.namespace}',pod=~'${resource.jsonData.metadata.name}'}) by (pod,namespace)`}

I propose to:

• filter the containers memory out with the container label.
• delete the sum as the name and namespace identifies the pod uniquely ?

container_memory_working_set_bytes{container!='', namespace='${resource.jsonData.metadata.namespace}',pod=~'${resource.jsonData.metadata.name}'})

This is already the case for the pod CPU query by the way.

cpuQuery={`sum(rate(container_cpu_usage_seconds_total{container!='',namespace='${resource.jsonData.metadata.namespace}',pod='${resource.jsonData.metadata.name}'}[1m])) by (pod,namespace)`}

[illume]

Hello. Thank you for the great report. (lots of details!)

I'm going to transfer it to the main headlamp-k8s/headlamp repo issues board. This one is for plugins.

[viceice]

• prometheus: fix doubled memory usage headlamp-k8s/plugins#184
  🤗

[gerardnico]

The reason of my silence is that I got notified just on close. Weird but I gave up long ago this fight.

Thanks !