Hetzner Cloud Volumes support and problems with cluster extend after manual updates of nodes

[Edke]

Hi everyone.

I'm a K8s newbie and evaluating different k8s solutions, came upon your awesome hetzner-kube as I'm a Hetzner Online really satisfied user.

Of course usage of Hetzner volumes would be welcome, I investigated and found https://github.com/hetznercloud/csi-driver and was surprised that no mention about project and it's integration.

After some hacking I got it working according to it's guidelines. Only info not clear was a feature-gates part. That I managed with these steps, found here hetznercloud/csi-driver#17 (comment):

1. ssh to each node
2. kubeadm alpha kubelet config download
3. sudo systemctl restart kubelet

After this I got Volumes working nicely.

BUt now and I'm not sure what is a cause, can't add more workers to cluster:

$ hetzner-kube cluster add-worker --datacenters nbg1-dc3  --name he1                                                           ────────────────────────
2019/03/09 11:24:18 creating server 'he1-worker-06'...
  --- [======================================] 100%
2019/03/09 11:24:22 Created node 'he1-worker-06' with IP 116.203.115.180                                                          
2019/03/09 11:24:22 creating server 'he1-worker-07'...
  --- [======================================] 100%
2019/03/09 11:24:25 Created node 'he1-worker-07' with IP 116.203.115.156                                                          
2019/03/09 11:24:25 sleep for 30s...
he1-master-01        : wireguard configured                7.7% [--------------]                                                  
he1-worker-01        : configure wireguard                 5.9% [--------------]                                                  
he1-worker-02        : wireguard configured               11.8% [--------------]                                                  
he1-worker-03        : configure wireguard                 5.9% [--------------]                                                  
he1-worker-04        : configure wireguard                 5.9% [--------------]                                                  
he1-worker-05        : configure wireguard                 5.9% [--------------]                                                  
he1-worker-06        : configure wireguard                64.7% [========>-----]                                                  
he1-worker-07        : wireguard configured               70.6% [=========>----]                                                  
2019/03/09 11:29:16 run failed
command:systemctl enable wg-quick@wg0 && systemctl restart wg-quick@wg0 && systemctl enable overlay-route.service && systemctl restart overlay-route.service
stdout:Created symlink /etc/systemd/system/multi-user.target.wants/[email protected] → /lib/systemd/system/[email protected].  
Created symlink /etc/systemd/system/multi-user.target.wants/overlay-route.service → /etc/systemd/system/overlay-route.service.    
Job for overlay-route.service failed because the control process exited with error code.                                          
See "systemctl status overlay-route.service" and "journalctl -xe" for details.                                                    

err:Process exited with status 1

So I have questions:

1. Is Cloud Volumes planned as feature so no manual tweaking of nodes is required ? As addon it would be great.
2. Does my current problem with new worker nodes has anything to do with manual nodes modification ?

PS: Sorry, I know this isn't a proper issue more it's a question, I will create new propers ones from this one after discussion.

[mavimo]

@Edke should be fixed here #258
most relevant changes to make it available for new nodes & c c4d78d9#diff-65af547d7ac35df5808831c9bf26e0f1R32

[mavimo]

@Edke fixed in 0.4.1

[Edke]

@mavimo I updated hetzner-kube and tried to recreate new cluster with it. Can't get hcloud-volumes to work though

hcloud-csi-nodes keep crashing

hcloud-csi-controller-0                      4/4     Running            0          12m
hcloud-csi-node-j62ml                        1/2     CrashLoopBackOff   7          12m
hcloud-csi-node-xmwq5                        1/2     CrashLoopBackOff   7          12m
hcloud-csi-node-z8rtk                        1/2     CrashLoopBackOff   7          12m