Block storage support?

[klausenbusk]

Hello

I'm just wondering if there is any ETA for block storage support?

The README says In the future, it may implement:

Regards Kristian Klausen

[andrewsykim]

@klausenbusk volume support on external cloud controller manager requires some work to be done in both the kube-controller-manager and the kubelets since both are responsible for attaching/detaching/mounting/unmounting volumes. From my understanding we're waiting on CSI (container storage interface) to be ready to integrate into kubelets before adding that feature to external cloud controller managers.

I have brought this up before if you want to follow up on those discussions:
kubernetes/enhancements#88 (comment)
kubernetes/kubernetes#48690 (comment)

[andrewsykim]

To clarify, there's no set ETA, but you can follow up with sig-storage to see where the CSI project is

[klausenbusk]

Thanks for the fast response, I will stick with my "bash" flexvolume script solution for the time being.

Just a side question: Is DigitalOcean working on any anything else related to k8s? I just saw that you are working on DO support in kops.

DO support in https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler could also be nice (mostly if DO lose a region, auto-scaling in the remaining region could be nice), maybe I should take a stab at that myself.

[andrewsykim]

kops support for DO is more of a side project for me.

Re: cluster-autoscaler I would love to see it working with DO, I haven't had a chance to work on it since I figured supporting cloud controller manager is probably more important. Do let me know if you take a stab at it, would love to give feedback where I can :)

[klausenbusk]

Would it make sense to add a flexvolume plugin to this repo?

I have a 167 line bash script currently which work as a flexvolume plugin, and I think that can be stripped down even further. As we can just use the default mountdev/unmountdevice call (I think).
Reimplementing it in Go shouldn't be that hard. In the end I just need to implement the init/attach/detach/waitforattach and isattached call. It could properly work without the last 2.

I did a bit of research, and with this proposal kubernetes/community#833 , installing the flexvolume plugin will be pretty easy, the needed changes was just merged a few days ago: kubernetes/kubernetes#50031 .

With the flexvolume in place, we/I could then create a external provisioner with https://github.com/kubernetes-incubator/external-storage . I the end it would properly would at least as good as kubernetes/kubernetes#50044 , one thing I can't figure out through, is how to ensure that k8s does not schedule more than 5 volumes to a node.

[andrewsykim]

@klausenbusk thanks for doing some research and digging into this! I didn't know the dynamic discovery of flex volume plugins was in already, that's going to be super helpful.

For flex volumes plugins, there's a project called rexray that integrates DO block storage with Kubernetes flex volumes written in Go. You can read up on how to setup here. One of my biggest issues with rexray's flexvolume plugin is that there's some host configuration required to make it work with Kubernetes, but I think we can do some work to make that easier.

As you mentioned there isn't a good story right now to provision the volumes. So even though we can use rexray (or another plugin) for attach/detach/mount/unmount, we still have to create those volumes manually. I think creating an external provisioner is going in the right direction and something we'll need eventually to better integrate with flex volumes plugins. I'm not sure if external storage provisioner should be a separate project or outside of CCM, that's something I'd have to discuss with sig cluster lifecycle.

For the scheduling concern regarding having more than 5 volumes to a node, I think there are some ways to work around this, but it's something we can work that out afterwards. I think it's more important to have a better story for the install process of flex volume plugins and an external provisioner.

[andrewsykim]

cc @baldwinSPC @odacremolbap

[klausenbusk]

For flex volumes plugins, there's a project called rexray that integrates DO block storage with Kubernetes flex volumes written in Go.

I have used rexray before (back when my cluster was at 1.5), but rexray does not currently support the attach/detach controller, which is one of the reason I wrote my own flexvolume plugin in bash.

I took a stab on implementing the same plugin in Go: https://gist.github.com/klausenbusk/422e2a9ebc2ac0856cc5ffab2fe90334 , all the logic is there, but the code quality could be better.
Note: I haven't tested it.

[andrewsykim]

@klausenbusk I'm not sure I follow you completely. When you say rexray does not support attach/detach controller you mean it doesn't support controller managed attach/detach right? When using rexray you are enabling attach/detach controller on the individual kubelets in which case rexray's flex volume plugin does have the ability to attach/detach volumes.

[klausenbusk]

When you say rexray does not support attach/detach controller you mean it doesn't support controller managed attach/detach right?

Correct (--enable-controller-attach-detach).

When using rexray you are enabling attach/detach controller on the individual kubelets in which case rexray's flex volume plugin does have the ability to attach/detach volumes.

You mean --enable-controller-attach-detach=false? The flexrex plugin should work without issue if --enable-controller-attach-detach=false is set.

[andrewsykim]

Gotcha, so what you're requesting originally is to have a flexvolume plugin that is controller managed, which rexray does not currently support.

[klausenbusk]

I'm not sure if external storage provisioner should be a separate project or outside of CCM

The code can go into the external-storage project, per kubernetes-retired/external-storage#318 .

[andrewsykim]

@klausenbusk cool, looking forward to that!

[klausenbusk]

With 1.9 just released with a alpha implementation of CSI, does DigitalOcean plan on providing a CSI plugin?

[baldwinSPC]

@klausenbusk over at stackpoint.io we built the official DO flex volume, so likely we'll look at re-creating it as a CSI plugin, but probably won't get going on that until March.

[klausenbusk]

@klausenbusk over at stackpoint.io we built the official DO flex volume, so likely we'll look at re-creating it as a CSI plugin, but probably won't get going on that until March.

Cool :)

BTW: digitalocean-provisoner was just merged upstream: kubernetes-retired/external-storage#470

[klausenbusk]

Lets close this issue. @fatih is working on a CSI driver (and provisioner?)

[fatih]

@klausenbusk yeap, I've officially started working on it. Follow the repo or the csi folder for more info. Eventually we're going to split it out into it's own repository later.

[fatih]

for anyone that somehow end up here, the CSI driver is moved to https://github.com/digitalocean/csi-digitalocean