New Storage/Device APIs to handle device-less storages

[harche]

Storages that do not have a backing device (such as ephemeral) need to be handled properly at the sandbox level. There are few suggestions by @bergwolf and @WeiZhang555 that need to be sorted out.
We can use this issue as a placeholder to keep the discussion public.

Requesting @bergwolf and @WeiZhang555 and whoever else is interested to put their thoughts here so that a larger community can participate.

The result of this discussion will lead to refactoring the support for ephemeral PR

[WeiZhang555]

I'll copy some text from our previous discussion in Slack:

From me:

For discussion, I want to share something in mind though it's not very clear either, so feel free to correct me.

type Mount struct {                                                                 
    Source      string                                                              
    Destination string                                                              
                                                                                    
    // Type specifies the type of filesystem to mount.                              
    Type string                                                                     
                                                                                    
    // Options list all the mount options of the filesystem.                        
    Options []string                                                                
                                                                                    
    // HostPath used to store host side bind mount path                             
    HostPath string                                                                 
                                                                                    
    // ReadOnly specifies if the mount should be read only or not                   
    ReadOnly bool                                                                   
                                                                                    
    // BlockDevice represents block device that is attached to the                  
    // VM in case this mount is a block device file or a directory                  
    // backed by a block device.                                                    
    BlockDevice *drivers.BlockDevice                                                
}```            

For a Mount struct, besides normal information for a mountpoint, there're two additional fields: HostPath for 9p and BlockDevice for block:
1. 9pfs volume: There's a real existing dir on host, you can pass to many kata containers with multi bind mount, and unbind when one container leaves.
2. virtio-blk or virio-scsi backed file:  in use case "-v /dev/sda:/dev/sda",  we need to hotplug a real block device to hypervisor. Sometimes user would want to "bind mount" a block device to different containers, but from kata-runtime/hypervisor's perspective, it doesn't need to hotplug /dev/sda to hypervisor many times, it can hotplug once but reference it to many containers. Then how to know when should we unplug the block? --> we need to remember which container is the last reference and unplug the device when nobody is using it any more. This part is what I am working towards and not fully finished(see https://github.com/kata-containers/runtime/pull/301)

So what @bergwolf suggested  is that ephemeral volume can share the same "reference count" logic of devices, from this perspective, I can understand his suggestion and choice. But on the other side, the ephemeral volume doesn't have a "backend"(a dir or a block) on host though it also has a source on host(totally ignored in ephemeral volume), everything happens only inside the VM which is the scope of kata-agent, so I think another choice is kata-agent can handle the reference count logic in its "ephemeralStorageHandler", and kata-runtime doesn't need to create a new global sandbox level Mount persistent file. (edited)

from kata-agent:
 
```191 // storageHandlerList lists the supported drivers.                               
192 var storageHandlerList = map[string]storageHandler{                              
193     driver9pType:        virtio9pStorageHandler,                                 
194     driverBlkType:       virtioBlkStorageHandler,                                
195     driverSCSIType:      virtioSCSIStorageHandler,                               
196     driverEphemeralType: ephemeralStorageHandler,                                
197 }```                                                                                         

That's why I say, it's not about right or wrong, only different implementation choice and design.

[WeiZhang555]

From @bergwolf

hey guys, we were discussing pr-307 here that @WeiZhang555 and I have a bit different opinions. To summary, @WeiZhang555 doesn't want to handle fake devices in the device infrastructure and I wanted to allow for it because that would allow a storage with fake devices to be maintained at the sandbox level and thus be shared by different containers in the same sandbox.

I think there are a few functionality requirements:

1. we need ephemeral storage support
2. we want to share it among containers within the sandbox

If we do not support it through the device infrastructure, I think we'll need to define a new sandbox (fake?) storage data structure to track it at the sandbox level, also possibly a new set of APIs to expose the functionality to frakti. WDYT?

[bergwolf]

@WeiZhang555 @harche Sorry for the late reply. I was OOO last week. I am fine if you want to go with agent handling volume reference counting. It somewhat duplicates with runtime but I can live with it as long as we do not make it ephemeral volume specific -- other volume types might use it as well.

[sboeuf]

Are you going to rework #307 based on comments from @bergwolf and @WeiZhang555 ?

[WeiZhang555]

I think we need more opinions from others, if we decide to go with @bergwolf 's suggestion then the #307 can keep it is and nothing need rework, otherwise we need to modify the kata agent and rework the kata-runtime PR #307

[harche]

@WeiZhang555 so should I wait to update the PR with your suggestion until we hear from others? If yes, who specifically? Please tag them explicitly so that we know when to proceed, otherwise others is an arbitrary guess.

[harche]

@bergwolf @WeiZhang555 I have created a PR that doesn't use devices, #458

Let me know what do you think.