support config options through annotations

[bergwolf]

As we were discussing the agent hook pr (kata-containers/agent#365 (comment)), we might want to support specifying all kata configs through labels or annotations, and then there is no dependency on a fixed cli configuration and each pod can be customized. The issue is created to track the feature/discussion.

A simple suggestion can be that we define a format like io.kata-containers.config.<subsystem>.<key>. Then we can have labels/annotations:

1. io.kata-containers.config.hypervisor.qemu.path: the qemu path in hypervisor.qemu section
2. io.kata-containers.config.shim.kata.enable_debug: the enable_debug option in shim.kata section

etc.

We can still have a default config in configuration.toml, and the per pod annotation configs takes a higher priority and override the values in configuration.toml.

The use case is to support very flexible controllers that can customize any config options we support on a per pod basis.

/cc @jodh-intel @amshinde @eguzman3 @egernst

[jodh-intel]

This is in fact an old topic. The problem is that docker labels are not passed to the container and (atleast the last time I looked) docker doesn't provide a way to inject OCI annotations into the config.json it generates.

The OCI spec of course does now support the specification of a hypervisor, kernel and image as first-class objects.

Also, virtcontainers already provides a way to specify those same details as annotations:

• https://github.com/kata-containers/runtime/blob/master/virtcontainers/asset.go

If we want to add kata config as annotations it would need a prefix like io.katacontainers.${subsystem}.${option_key} or we could just stuff everything into a single annotation:

io.katacontainers.config = {
    "version": "1.2.3",
    "runtime": {
        "enable_debug": true
            :
    },
    "shim": {
    },
    :
}

If we are going to consider adding another set of config via annotations we need to:

• Think very carefully about which configuration sources we want to maintain.
• Ideally simplify what we already support (might require deprecation, rather than immediate removal).
• Determine how users can provide such config when using docker/k8s/etc.
• Document the ordering and priority that such "configuration sources" will take extremely carefully.
• Test all possible combinations extremely carefully.
• Evaluate https://github.com/spf13/viper as a potential replacement for https://github.com/urfave/cli since viper seems well thought out with respect to how multiple configuration sources are handled.

[bergwolf]

@jodh-intel The cli config options are already an API for kata containers, even w/o being able to be set via annotations. I've updated the description to focus on providing the set-via-annotation capability so that k8s controllers become the main focus. And we don't need to introduce case-by-case annotations as being done in #1695.

/cc @mcastelino @egernst

[jodh-intel]

Nice! Thanks @bergwolf.

[egernst]

/cc @amshinde

[egernst]

@amshinde @bergwolf is this handled now?