Fix: Optimize single_process_type execution

[svteb]

Description

The single_process_type / process_check spec tests took on average 1:30:00 to execute (in github actions), this was reduced to around 10-15 minutes. It was caused by incorrect use of libraries which checked unnecessary processes multiple times.

These two changes need to be merged first:
cluster_tools change: #27
k8s_kernel_introspection change: #4

Because this pull request makes changes in two libraries, it is rather difficult to verify it through actions. Currently there are changes to the shards.yaml and shards.lock files which will be removed once the reviewers approve the change and the individual library pull requests are merged.

For reviewers:
There are two big changes, the first is the replacement of workload_resource_test by cnf_workload_resources, both yield resource names, but workload_resource_test also yields all the container names per resource, thus we could get resources like this (it also did not return all the containers for some reason?):

CNFManager.workload_resource_test(args, config) do |resource, container, initialized|

resource: {kind: "Deployment", name: "nginx-webapp", namespace: "cnfspace"}
resource: {kind: "Deployment", name: "nginx-webapp", namespace: "cnfspace"}
resource: {kind: "Deployment", name: "nginx-webapp", namespace: "cnfspace"}
resource: {kind: "Pod", name: "sidecar-container-demo", namespace: "cnfspace"}
resource: {kind: "Pod", name: "sidecar-container-demo", namespace: "cnfspace"}

optimized version:

CNFManager.cnf_workload_resources(args, config) do |resource|

resource: {kind: "Deployment", name: "nginx-webapp", namespace: "cnfspace"}
resource: {kind: "Pod", name: "sidecar-container-demo", namespace: "cnfspace"}

With this change, we do not iterate over every resource multiple times. The second big change is the addition of pid filtering by cgroups. The previous version was incorrectly checking processes on node instead of a container. This lead to something like this (which makes no sense considering we want verify to processes of specific container).

for each container:
         container_node = get_container_node()
         get_all_pids_on_node(container_node)

optimized version:

for each container:
         container_node = get_container_node()
         get_all_pids_on_container(container, container_node)

The cgroup filtering currently depends on ctr purely because it was easier to write than runc filtering. Considering the fact that the current testsuite depends on containerd runtime (ctr) it should not be too problematic and can be changed in the future. Finally, I noticed that the cluster_tools library provided a function that was a copy of the code in single_process_check. That same function is also shared by the zombie task which has some unique quirks I will not get too deep into here. That shared function (ClusterTools.all_containers_by_resource?) had to be changed slightly so it would not break the zombie task. A better refactor can be considered in the future but it is outside the scope of this pull request.

Issues:

Refs: #2084

How has this been tested:

• Covered by existing integration testing
• Added integration testing to cover
• Verified all A/C passes
  • develop
  • master
  • tag/other branch
• Test environment
  • Shared Packet K8s cluster
  • New Packet K8s cluster
  • Kind cluster
• Have not tested

Types of changes:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist:

Documentation

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• No updates required.

Code Review

• Does the test handle fatal exceptions, ie. rescue block

Issue

• Tasks in issue are checked off

[kosstennbl]

Nice change, optimizations for this test are very welcome.
Few additional points:

• Maybe we could use "crictl" for k8s introspection change for easier support of [Feature] Support multiple container engines #2103 in case it will be implemented?
• Operation of workload_resource_test and cnf_workload_resources is complicated and feels that workload_resource_test design could be a cause for multiple slowdowns in our tests. Probably we should look into fixing this at a higher scale. Not a scope of this PR though.

[svteb]

Nice change, optimizations for this test are very welcome. Few additional points:

• Maybe we could use "crictl" for k8s introspection change for easier support of [Feature] Support multiple container engines #2103 in case it will be implemented?
• Operation of workload_resource_test and cnf_workload_resources is complicated and feels that workload_resource_test design could be a cause for multiple slowdowns in our tests. Probably we should look into fixing this at a higher scale. Not a scope of this PR though.

Yes, after some discussions we have come to the conclusion that ctr might not be generic enough. The issue is that every container cli tool has some nuances that prevent easy access to cgroups/filtering pids. The issue with crictl is that it is too high-level and does not provide access to cgroups. We've decided to use runc instead which is a lower level cli tool.

Edit:
After further thought I've decided to forego using any container cli tool and piped a few bash commands together which work like this:

"find /proc -maxdepth 1 -regex '/proc/[0-9]+' -exec grep -l '#{container_id}' {}/cgroup \\; 2>/dev/null | sed -e 's,/proc/\\([0-9]*\\)/cgroup,\\1,'\""

1. Get all process directories from /proc
2. Grep through their /proc/<pid>/cgroup file to find if they are matching the container_id
3. Return matching pids.

The spec tests are passing.

[martin-mat]

lgtm

[kosstennbl]

lgtm

[kosstennbl]

That could be changed to stdout_error, but as we have CI fully successful now, we can probably do that on the scope of the whole testsuite with another PR