Graceful Node shutdown Single Node Cluster

[containerpope]

With kubernetes version 1.22, graceful node shutdown is becoming GA. Will there be a way to use it, while running just a single node cluster?

The background is, that we have single node edge devices, that receive an external signal to power them off (30sec after signal arrival) I would like to send sigterm to every process running on the system, to ensure everything stops properly.
How would the k3s service react to this? Would the graceful node shutdown happen and all pods are ended cleanly or would it not stop the containers?

Really appreciate any feedback on this :)

[dweomer]

Per https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/:

Note that by default, both configuration options described above, shutdownGracePeriod and shutdownGracePeriodCriticalPods are set to zero, so you will need to configure them as appropriate for your environment to activate graceful node shutdown functionality.

So, something like k3s server --kubelet-arg shutdownGracePeriod=30s --kubelet-arg shutdownGracePeriodCriticalPods=30s

[brandond]

Stopping the service still stops the service. Graceful node shutdown involves external signalling between systemd and the kubelet to communicate that the node will shut down soon, at which point the service will be stopped as usual. It does not change service stop behavior in any way.

[containerpope]

@brandond thank you for the clarification. If I run systemctl poweroff, from my understanding systemd tries to stop all services, resulting in a sigterm to all of them. In k3s, the kubelet should be running under the k3s service. Does systemd at first stop the child processes by forwarding the sigterm and then shutting down the parent k3s service? If this is not the case, is there a way to target the kubelet as a subprocess of k3s? In case it is stopped first, should the stopping of the k3s service take the duration, configured for the graceful node shutdown?
I want to avoid loosing/corrupting data, because I have around 30 seconds after a specific signal to close everything, before power is killed hard.

[brandond]

Upstream has a pretty good description of how it works in their blog post.
https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/#how-does-it-work

[imranrazakhan]

So, something like k3s server --kubelet-arg shutdownGracePeriod=30s --kubelet-arg shutdownGracePeriodCriticalPods=30s

@dweomer your suggested configuration not working, I am getting following error

k3s[25706]: E0110 21:05:18.152605 25706 server.go:157] "Failed to parse kubelet flag" err="unknown flag: --shutdownGracePeriod"

[brandond]

@imranrazakhan please read the next thread down... the one that contains the selected answer.

[containerpope]

Thank you for all the help, I finally figured out to configure it properly:

1. Create a kubelet.config (it only overrides what you set) and add the graceful node settings of your choice (seems like not possible via simple CLI args):

# /data/rancher/kubelet.config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 15s,
shutdownGracePeriodCriticalPods: 10s

2. Reference the kubelet.config in the k3s systemd service:

--kubelet-arg="config=/data/rancher/kubelet.config"

3. Optional: Cleanup Pods in Phase shutdown, by either a CronJob or by an additional systemd service.

# /etc/systemd/system/k3s-cleanup.service
[Unit]
Description=k3s-cleanup
StartLimitInterval=200
StartLimitBurst=5
Wants=k3s.service

[Service]
Type=oneshot
ExecStart=kubectl delete pods --field-selector status.phase=Failed -A --ignore-not-found=true
RemainAfterExit=true
User=root
StandardOutput=journal
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

After you then do a systemctl poweroff, the poweroff is delayed until all pods are shutdowned or time is up.

[ehlesp]

Thanks @containerpope for this setup, but I think I must leave a warning here about your kubelet.config file. In my experience, after installing a server node following the configuration proposed here, the K3s server process had an unmarshalling problem with the ',' character at the end of the shutdownGracePeriod parameter line. In my log the error showed as follows:

E1127 20:35:49.150342    3684 server.go:205] "Failed to load kubelet config file" err="failed to load Kubelet config file /etc/rancher/k3s/kubelet.config, error failed to d
ecode: v1beta1.KubeletConfiguration.ShutdownGracePeriodCriticalPods: ShutdownGracePeriod: unmarshalerDecoder: time: unknown unit \"s,\" in duration \"30s,\", error found in
 #10 byte of ...|od\":\"30s,\",\"shutdown|..., bigger context ...|KubeletConfiguration\",\"shutdownGracePeriod\":\"30s,\",\"shutdownGracePeriodCriticalPods\":\"10s\"}|..."
path="/etc/rancher/k3s/kubelet.config"

I checked the official documentation about the KubeletConfiguration object to see if anything was wrong with the kubelet.config file shown in this thread but, in the end, I had to remove the ',' to fix the error. So this is just a warning for anyone applying a similar configuration and being surprised by the same odd issue as me.

This has happened to me while using k3s v1.22.3+k3s1, and below I leave my now working server node's configuration for reference.

/etc/rancher/k3s/config.yaml

# k3sserver01

cluster-domain: "deimos.cluster.io"
tls-san:
    - "k3sserver01.deimos.cloud"
flannel-backend: host-gw
flannel-iface: "ens2"
bind-address: "0.0.0.0"
https-listen-port: 6443
advertise-address: "10.0.0.1"
advertise-port: 6443
node-ip: "10.0.0.1"
node-external-ip: "200.0.1.5"
node-taint:
    - "k3s-controlplane=true:NoExecute"
log: "/var/log/k3s.log"
kubelet-arg: "config=/etc/rancher/k3s/kubelet.config"
disable:
    - metrics-server
    - servicelb
protect-kernel-defaults: true
secrets-encryption: true
agent-token: "someStrangePassword"

/etc/rancher/k3s/kubelet.config

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s

[containerpope]

@ehlesp interesting, in my setup it works nicely as described in the docs. Did you test it without blank lines in the config? Anyway, good to know, that a comma can cause problems here.

[ehlesp]

To be sure that I wasn't imagining things yesterday, I've tested it today again a couple of times with exactly the same format you've used (no blank line and the comma in place) but the issue happens the same. Unless I remove the comma, the k3s process is unable to load my kubelet.config. I don't know what could be different in this regard between your setup and mine, @containerpope , but I'll add that I install my server node with the following command:

wget -qO - https://get.k3s.io | INSTALL_K3S_VERSION="v1.22.3+k3s1" sh -s - server

Notice that I set the kubelet-arg as a parameter inside my config.yaml file rather than as a command line argument, although it doesn't make any difference (I tried that too). I think I may have accidentaly bumped into some weird K3s parsing issue here.

[containerpope]

@ehlesp very interesting to hear, I remember having some problems to until I got this working in our setup. I assume, as you already wrote, that there might be some parsing issues. The difference in our setup could be, that we install the binary manually and then control everything through a modified systemd service.
The next time I touch one of our devices I have to check again. Anyway, thank you for taking the time to check my proposal. I think anyone reading this should follow your approach, because of the unusual way we setup our system.

[ehlesp]

I thought this issue deserved a proper bug report, so I did here. Anyone meeting the same issue as me while applying a KubeletConfiguration as shown in this thread may leave a comment there.