Node stuck at deletion due to finalizer - regression

[ShylajaDevadiga]

Environment
k3s version v1.19.4+k3s-989c9369 (989c936)
Issue
Deletion of node is stuck due to presence of finalizer. This seems to be a regression

Steps to reproduce

1. Create a 3 node cluster
2. Verify nodes are Ready and pods are Running
3. Delete node3 using kubectl delete node
   Deletion of a node is stuck, unless finalizer is removed.

--
    finalizers:
    - wrangler.cattle.io/managed-etcd-controller
    - wrangler.cattle.io/node
    labels:
--
          f:finalizers:
            .: {}
            v:"wrangler.cattle.io/managed-etcd-controller": {}
            v:"wrangler.cattle.io/node": {}

Related issue rancher/rke2#401

[davidnuzik]

The issue was also reproduced in v1.19.3+k3s3. Since the issue is pre-existing we don't want it to block our imminent releases. We'll mention this in the release notes. I've bumped the milestone to the next set of patch releases. This will need to get fixed in 1.19, 1.20, and master.
We'll see if there is a workaround so that this can also be indicated in the release notes.

The issue seems to happen intermittently and will not always be observed. The release notes will briefly explain the issue and workaround to fix the finalizers so the node(s) can be deleted.

[ShylajaDevadiga]

Note: To delete the node that is stuck during delete

kubectl get node -o name <nodename> | xargs -i kubectl patch {} -p '{"metadata":{"finalizers":[]}}' --type=merge

[xiaods]

@davidnuzik do you know what reason caused this issue? curiously want to know the root cause.

[brandond]

etcd node controller

[xiaods]

etcd node controller

Opps, where is etcd node controller?

[brandond]

https://github.com/k3s-io/k3s/blob/master/pkg/etcd/controller.go

[xiaods]

@brandond hi, with my testing, i have not found the stable reproduce the node stuck for this bug.
the above issue describe issue base on v1.19.4
I am not sure the latest version 1.19.10 can resolve the bug.

[brandond]

I'm not sure how to reproduce it on demand either. My understanding is that something on the rancher/wrangler side has an issue where onDelete handlers leave stuck finalizers. For this reason, onDelete handlers are to be avoided in general, and the code needs to be rewritten to periodically reconcile the Kubernetes and etcd cluster member lists, instead of relying on watching v1.Node.

[xiaods]

I'm not sure how to reproduce it on demand either. My understanding is that something on the rancher/wrangler side has an issue where onDelete handlers leave stuck finalizers. For this reason, onDelete handlers are to be avoided in general, and the code needs to be rewritten to periodically reconcile the Kubernetes and etcd cluster member lists, instead of relying on watching v1.Node.

thanks for your clarify. can't wait to fix it asap.

[mitchellmaler]

I have been testing IaC code to rebuild rke2 nodes and ran into this issue. From what I can tell it is only the control-plane nodes which points as noted above is probably the etcd controller. When recreating the issue I checked that the etcd member was removed which it always was so the controller didn't fail to complete that task. One thing I have started checking is if the current node being deleted is the current etcd leader. I have added some logic to call 'move-leader' if the current node being deleted is the etcd leader and have yet to run into the same issue.

[rancher-max]

Validated on master branch using commitid ad1a40a96c400c17780cf9455f5da330d690194c

As this is an intermittent issue, I spun up multiple clusters and deleted a node and none of them were stuck. This has been fixed in all of our currently released versions as well (1.19.14, 1.20.10, and 1.21.4) so any users tracking this thread should be able to upgrade to one of those versions to see this fixed.

[ozbillwang]

Note: To delete the node that is stuck during delete

kubectl get node -o name <nodename> | xargs -i kubectl patch {} -p '{"metadata":{"finalizers":[]}}' --type=merge

Thanks so much. I stuck at the same issue for more than 1 hour, until saw this

[CihatDinc]

Note: To delete the node that is stuck during delete

kubectl get node -o name <nodename> | xargs -i kubectl patch {} -p '{"metadata":{"finalizers":[]}}' --type=merge

it worked, thanks 👍