Support file-based plugin configuration in catalog with hot reload

[azdagron]

Plugin configuration today happens once, when the plugin is loaded by the catalog. The plugin configuration data is provided statically by the contents of the plugin_data section of the plugin config.

Several use cases exist where it would be convenient to change plugin configuration at runtime (#4390).

We can support this generically in the catalog system by introducing a new plugin_data_file setting. This setting would be mutually exclusive with plugin_data. It would point to a file on disk where the plugin configuration would live.

For example, this is how a plugin configuration looks today:

Plugins {
    NodeAttestor "k8s_psat" {
         plugin_data = {
            some_setting = "foo"
            ...
        }
    }
    ...
}

And this is how a plugin configuration would look using this option:

Plugins {
    NodeAttestor "k8s_psat" {
         plugin_data_file = "k8s_psat.conf"
    }
    ...
}

And the plugin.conf file:

some_setting = "foo"

Either on some interval, or in response to a signal or something, the catalog would:

1. reload the plugin configuration from the file
2. if changed, push the new configuration into the plugin via the config service
3. Log the results.

The Configure RPC is already documented to indicate that implementations should be able to respond to Configure calls executed concurrently with other RPCs. Built-in plugins are all written to expect this.

[azdagron]

Outstanding question is how do we want to trigger reloading. Signal? Timer?

[azdagron]

I have a prototype implementation I'll dust off for this. Just need to write up some tests.

[edwbuck]

For clarification, the "k8s_psat.conf" file in the example above would be HCL (version 1) correct?

If we are establishing plugin loading policy, I propose that such files match whatever the server loads (HCL 1.0 at the moment).

[azdagron]

By convention, that's correct. And certainly what will be expected for all built-in plugins for the foreseeable future. Technically, SPIRE Core would no longer care as long as the configuration was utf-8 text based, since this data isn't parsed alongside the main HCL config. The data would be loaded from file, turned into a string, and then otherwise passed unmodified/unparsed into the plugin. So external plugin authors could do what they wanted 🤷