node-kubelet-benchmark fails with OOM

[vpickard]

I'm wary of "just increase resources" fixes, it could be that we're hiding a legitimate problem of performance/resource-usage regressions

/cc @karan
I would be curious to get some input from folks with COS expertise

Originally posted by @spiffxp in #17743 (comment)

[vpickard]

This PR is in progress, to reduce the number of pods to get the benchmark tests running again.

kubernetes/kubernetes#91813

But, it would be good to get to the root cause of the increase in memory consumption. From the debug in 91813, the top suspect is containerd.

[BenTheElder]

Any updates?

[MHBauer]

https://testgrid.k8s.io/sig-node-kubelet#node-kubelet-benchmark
Looks to be running consistently but with either a timeout on a test or

W0813 10:37:20.435] Trying to find master named 'bootstrap-e2e-master'
W0813 10:37:20.435] Looking for address 'bootstrap-e2e-master-ip'
W0813 10:37:21.534] ERROR: (gcloud.compute.addresses.describe) Could not fetch resource:
W0813 10:37:21.534]  - The resource 'projects/k8s-jkns-gke-ubuntu-updown/regions/us-west1/addresses/bootstrap-e2e-master-ip' was not found
W0813 10:37:21.534] 
W0813 10:37:21.800] Could not detect Kubernetes master node.  Make sure you've launched a cluster with 'kube-up.sh'
I0813 10:37:21.901] Master not detected. Is the cluster up?
I0813 10:37:21.901] Dumping logs from nodes locally to '/workspace/_artifacts'
W0813 10:37:28.520]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env
W0813 10:37:28.520]     subprocess.check_call(cmd, env=env)
W0813 10:37:28.520]   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
W0813 10:37:28.521]     raise CalledProcessError(retcode, cmd)
W0813 10:37:28.522] subprocess.CalledProcessError: Command '('kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--test', '--deployment=node', '--provider=gce', '--cluster=bootstrap-e2e', '--gcp-network=bootstrap-e2e', '--gcp-project-type=node-e2e-project', '--gcp-zone=us-west1-b', '--node-args=--image-config-file=/workspace/test-infra/jobs/e2e_node/benchmark-config.yaml', '--node-test-args=--feature-gates=DynamicKubeletConfig=true --kubelet-flags="--cgroups-per-qos=true --cgroup-root=/"', '--node-tests=true', '--test_args=--nodes=1 --skip="\\[Flaky\\]"', '--timeout=65m')' returned non-zero exit status 1
E0813 10:37:28.546] Command failed
I0813 10:37:28.547] process 315 exited with code 1 after 34.6m 

Which I do not understand how to interpret.

[BenTheElder]

the actual failure is probably in other logs, but that log is saying that it can't find the static IP for the control plane node ("master") in GCP so the cluster must not be up

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.