K0s vcluster ends up in crash loop if it ever fails

[ananace]

What happened?

$ kubectl logs vcluster-0 
Defaulted container "syncer" out of: syncer, vcluster (init)
2023-12-07 15:33:42     INFO    setup/initialize.go:86  k0s config secret detected, syncer will ensure that it contains service CIDR    {"component": "vcluster"}
2023-12-07 15:33:42     INFO    k0s/k0s.go:58   Starting k0s    {"component": "vcluster", "args": "/k0s-binary/k0s controller --config=/etc/k0s/config.yaml --data-dir=/data/k0s --status-socket=/run/k0s/status.sock --disable-components=konnectivity-server,kube-scheduler,csr-approver,kube-proxy,coredns,network-provider,helm,metrics-server,kubelet-config"}
2023-12-07 15:33:43     INFO    setup/controller_context.go:221 couldn't retrieve virtual cluster version (Get "https://localhost:6443/version": dial tcp [::1]:6443: connect: connection refused), will retry in 1 seconds     {"component": "vcluster"}
2023-12-07 15:33:43     INFO    setup/controller_context.go:221 couldn't retrieve virtual cluster version (Get "https://localhost:6443/version": dial tcp [::1]:6443: connect: connection refused), will retry in 1 seconds     {"component": "vcluster"}
2023-12-07 15:33:44     INFO    loghelper/klog.go:24    Usage of deprecated component name      {"component": "vcluster", "component": "k0s", "time": "2023-12-07 15:33:44", "level": "warning"}
2023-12-07 15:33:44     INFO    loghelper/klog.go:24            {"component": "vcluster", "component": "k0s", "time": "2023-12-07 15:33:44", "level": "warning", "pre-flight-check": "os/NET", "property": "no kernel config found"}
2023-12-07 15:33:44     INFO    loghelper/klog.go:24    Error: failed to initialize runtime config: an instance of k0s is already running       {"component": "vcluster", "component": "k0s"}
2023-12-07 15:33:44     INFO    setup/controller_context.go:221 couldn't retrieve virtual cluster version (Get "https://localhost:6443/version": dial tcp [::1]:6443: connect: connection refused), will retry in 1 seconds     {"component": "vcluster"}
2023-12-07 15:33:44     INFO    setup/initialize.go:107 Error running k0s: exit status 1        {"component": "vcluster"}

You probably want to remove an existing k0s socket if one exists when starting.

What did you expect to happen?

An error should not leave the vcluster in a crash loop

How can we reproduce it (as minimally and precisely as possible)?

Run a 0.18.0 k0s vcluster, crash the cluster somehow

Anything else we need to know?

No response

Host cluster Kubernetes version

$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.11

Host cluster Kubernetes distribution

Bare-metal plain Kubernetes install

vlcuster version

$ vcluster --version
vcluster version 0.18.0

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k0s

OS and Arch

OS: Red Hat Enterprise Linux release 8.9 (Ootpa)
Arch: amd64

[FabianKramm]

@ananace thanks a lot for creating this issue! We will take a look!

[ananace]

As a temporary workaround - to keep our vclusters functioning - I've been patching the statefulset with;

containers:
  - name: syncer
    command:
      - /bin/sh
      - -c
      - |
        rm -rf /run/k0s/*
        /vcluster start $@

[FabianKramm]

@ananace thanks for the workaround we have a fix on the way!