Missing containerID info EKS 1.21.2 ContainerD

[paologallinaharbur]

In some scenarios it was reported that the containerID reported by the agent was missing

Description

In EKS 1.21.2 having the ContainerD as runtime, the reported container label is always empty causing the integration to fail when fetching the containerID.

Example

container_tasks_state{container="",id="/kubepods/burstable/podb9bac7ec-7264-4c8b-a6e9-cf177ffcdd7c/21241cbb20ab776dc87b05bb27c9912ba70f5fcfdfaadbbc400c5b4da89576a5",image="",name="",namespace="",pod="",state="iowaiting"}

Expected Behavior

containerID always present

In an identical EKS 1.21.2 having Docker as container runtime the everything works as expected

Example

container_threads_max{container="newrelic-logging",id="/kubepods/burstable/pod88a4c57f-5619-47b4-9c9b-70f4a28b8aec/83ce33548d81da5aaa4154568cde74c9d736932a6a79abf44d4a74e9b59b04a8",image="newrelic/newrelic-fluentbit-output@sha256:e64b37e642f233f20abd1655ace5655d419d4ca34e14d6c84a59506c14d1bd74",name="k8s_newrelic-logging_newrelic-bundle-newrelic-logging-p6x5l_newrelic_88a4c57f-5619-47b4-9c9b-70f4a28b8aec_0",namespace="newrelic",pod="newrelic-bundle-newrelic-logging-p6x5l"}

Steps to Reproduce

Create a cluster having:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ${CLUSTER_NAME}
  region: ${AWS_REGION}
  version: "${EKS_VERSION}"
managedNodeGroups:
  - name: containerd
    ami: ${AMI_ID}
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh ${CLUSTER_NAME} --container-runtime containerd

Install the integration and run:
kubectl exec [newrelic-infra-pod] curl -- -kH "Authorization: Bearer "$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"" https://localhost:10250/metrics/cadvisor > logs.txt

[nadiamoe]

causing the integration to fail when fetching the containerID

Other metrics derived from cAdvisor will also be missing, namely:

nri-kubernetes/src/metric/definition.go

Lines 844 to 857 in 92e4f3b

	var CadvisorQueries = []prometheus.Query{
	{
	MetricName: "container_memory_usage_bytes",
	Labels: prometheus.QueryLabels{
	Operator: prometheus.QueryOpNor,
	Labels: prometheus.Labels{
	"container_name": "",
	},
	},
	},
	{MetricName: "container_cpu_cfs_periods_total"},
	{MetricName: "container_cpu_cfs_throttled_periods_total"},
	{MetricName: "container_cpu_cfs_throttled_seconds_total"},
	{MetricName: "container_memory_mapped_file"},

[paologallinaharbur]

Currently we are not sure if it is an issue on kubelet/cadvisor or if we cannot assume that the container label is populated

[paologallinaharbur]

This might be related
awslabs/amazon-eks-ami#764