[BUG] Node restart breaks CNI when kube-ovn is deployed with Multus

[cnvergence]

Kube-OVN Version

latest

Kubernetes Version

v1.29.1

Operation-system/Kernel Version

Ubuntu 22.04

Description

Having deployed kube-ovn with multus on the cluster with configured multiple external subnets, we are running sometimes into the issue after node restarts Multus with kube-ovn does not start properly, probably due to the race condition mentioned in this issue.

Current workaround is to delete multus shim on node, restart multus pod and ovn-ovs pod.

Steps To Reproduce

1. Deploy kube-ovn with multus on the cluster
2. Create multiple external networks with network-attachment definitions
3. Restart the node
4. Observe the inability to start in ovn-ovs pod

Current Behavior

Node restart in a cluster with kube-ovn and multus sometimes results in CNI failure.

Expected Behavior

Node restart should be handled differently in kube-ovn/multus stack.

[oilbeater]

@cnvergence, could you provide detailed error logs? The issue you mentioned in Multus should not cause Kube-OVN to fail to start. If Kube-OVN failed to start due to a node restart, we need to investigate if something went wrong on the Kube-OVN side.

[cnvergence]

@oilbeater it does not happen all the time, just sometimes, I will try to collect the logs
I wonder if anyone of the community had observed the same issue, maybe there is a way to mitigate it from kube-ovn side?