Amazon EKS AMI creates containerd socket at diferent path

[rafaelgaspar]

Describe what happened:

Amazon EKS AMI is currently creating the containerd socket at /run/dockershim.sock, the reason for that is that aws-cni-plugin still points to that path, so they cannot at the moment just change to /run/containerd/containerd.sock.

Setting criSocketPath: /run/dockershim.sock works for CRI metrics, but not for containerd metrics.

Describe what you expected:

containerd metrics to be available, maybe have a way to set a different mountPath for the criSocketPath inside datadog container so that it recognises it is a containerd socket.

Steps to reproduce the issue:

Spin up cluster on EKS and set --container-runtime containerd for bootstrap.sh

Additional environment details (Operating System, Cloud provider, etc):

EKS AMI v1.21.2-13+d2965f0db10712

[rafaelgaspar]

BTW, there is also awslabs/amazon-eks-ami#724

[cw-sakamoto]

After making the following settings, it worked even if dockershim.sock for containerd's cri path.

agents:
  volumes:
    - hostPath:
        path: /var/run/dockershim.sock
      name: crisocketpath
  volumeMounts:
    - name: crisocketpath
      mountPath: /var/run/containerd/containerd.sock
      readOnly: true

[bubalush]

After making the following settings, it worked even if dockershim.sock for containerd's cri path.

agents:
  volumes:
    - hostPath:
        path: /var/run/dockershim.sock
      name: crisocketpath
  volumeMounts:
    - name: crisocketpath
      mountPath: /var/run/containerd/containerd.sock
      readOnly: true

What about default kubernetes.* metrics of the pods and etc.?

[vboulineau]

Hello,

Setting criSocketPath: /run/dockershim.sock is indeed the right thing to do, however you don't get containerd automatically activated simply because the auto-detection (which is based on socket name) fails to recognize this as a containerd socket.

The proper way to force activate a feature that's not autodetected is to use:
autoconfig_include_features: ["containerd"] or DD_ AUTOCONFIG_INCLUDE_FEATURES=containerd

In the Helm chart, this can be set through a custom datadog.yaml or through datadog.env in values.yaml

This is available since Agent 7.29. We will update the Kubernetes distribution documentation with this information.

[cw-sakamoto]

@bubalush
#338 (comment)

Sorry, I certainly couldn't check the kubernetes.* metrics with this config.

@vboulineau

autoconfig_include_features: ["containerd"] or DD_ AUTOCONFIG_INCLUDE_FEATURES=containerd

I can check containerd metrics with this setting, but I can't seem to check kubernetes.*(ex. kubernetes.cpu.usage.total) metrics.

[vboulineau]

@cw-sakamoto These metrics are coming from the kubelet check, which should always be automatically scheduled inside Kubernetes, it does not depend on container runtime.

Check the output of agent status, you should have something like:

    kubelet (7.0.0)
    ---------------
      Instance ID: kubelet:5bbc63f3938c02f4 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 1,032
      Metric Samples: Last Run: 1,456, Total: 1,142,551
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 4,128
      Average Execution Time : 818ms
      Last Execution Date : 2021-08-26 11:45:16 UTC (1629978316000)
      Last Successful Execution Date : 2021-08-26 11:45:16 UTC (1629978316000)

If that's the case but you still don't see kubernetes.* metrics, there's probably something wrong, I invite you to open a support ticket as we'll need more information to debug it.

[cw-sakamoto]

@vboulineau

    kubelet (7.0.0)
    ---------------
      Instance ID: kubelet:5bbc63f3938c02f4 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 1
      Metric Samples: Last Run: 1,211, Total: 1,211
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 4
      Average Execution Time : 1.921s
      Last Execution Date : 2021-08-26 12:34:37 UTC (1629981277000)
      Last Successful Execution Date : 2021-08-26 12:34:37 UTC (1629981277000)

Looking at agent status, there seems to be no problem, but Total Runs is very low.

I can confirm metrics for kubernetes.kubelet.*, but I can't confirm metrics for kubenetes.{cpu,memory,filesystem,network}.*.

[vboulineau]

@cw-sakamoto It's indeed not expected to only have 1 run. The issue is probably specific to your scenario, so a support ticket would be a good place to solve it.

[saiindrakaran]

I also face similar issue as @cw-sakamoto , although my agent status looks [OK], I can't confirm metrics for kubenetes.{cpu,memory,filesystem,network}.*.

containerd
    ----------
      Instance ID: containerd [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/containerd.d/conf.yaml.default
      Total Runs: 16
      Metric Samples: Last Run: 363, Total: 5,808
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 16
      Average Execution Time : 323ms
      Last Execution Date : 2021-08-28 00:57:15 UTC (1630112235000)
      Last Successful Execution Date : 2021-08-28 00:57:15 UTC (1630112235000)

kubelet (7.0.0)
    ---------------
      Instance ID: kubelet:5bbc63f3938c02f4 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/kubelet.d/conf.yaml.default
      Total Runs: 12
      Metric Samples: Last Run: 754, Total: 9,048
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 4, Total: 48
      Average Execution Time : 992ms
      Last Execution Date : 2021-08-28 00:57:04 UTC (1630112224000)
      Last Successful Execution Date : 2021-08-28 00:57:04 UTC (1630112224000)