Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k3s server create 100% CPU load #11246

Closed
allnightlong opened this issue Nov 6, 2024 · 0 comments
Closed

k3s server create 100% CPU load #11246

allnightlong opened this issue Nov 6, 2024 · 0 comments

Comments

@allnightlong
Copy link

Environmental Info:
K3s Version:
k3s -v
k3s version v1.30.6+k3s1 (1829eaa)
go version go1.22.8

Node(s) CPU architecture, OS, and Version:
Linux ***-master 6.8.0-48-generic #48-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 14:04:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server
6 agents

Describe the bug:
100% CPU usage of k3s processes on master node. Can't diagnose and find out root of the problem.
HTOP screenshot
Screenshot 2024-11-06 at 18 19 12

NewRelic CPU usage for 3 days screenshot
Screenshot 2024-11-06 at 18 20 41

Steps To Reproduce:
I'm not sure exactly, how to reproduce the problem. My application creates a lot of short-living kube jobs.
Could it be the cleanup problem?

As 3 days graph suggests, the CPU load increases day-by-day (while the application is having the same load), so k3s finally arrives at the point, where CPU load is unbearable.

I've tried to run:

k3s-killall.sh
systemctl start k3s

from this topic #6095 (comment), but it never helped.

So I've ended up reinstalling the whole cluster on my new machine, just to find myself in the same situation 5 days after.

Expected behavior:
CPU load shouldn't reach over 100%

Actual behavior:
CPU load reaches 100%, and as a result it takes too long for kubernetes to launch pods, so the jobs falls out of the deadline.

Additional context / logs:
I can provide any necessary logs on the request

@k3s-io k3s-io locked and limited conversation to collaborators Nov 6, 2024
@brandond brandond converted this issue into discussion #11251 Nov 6, 2024
@github-project-automation github-project-automation bot moved this from New to Done Issue in K3s Development Nov 6, 2024

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
None yet
Projects
Status: Done Issue
Development

No branches or pull requests

1 participant