[RFC] Dynamic Documentation

[mattdurham]

This covers only the documentation which broadly defines what happens, and the feature flag itself.

This is to address issues #1121, #869, and #868.

The goal is to

• Allow the loading of all configuration files locally and remotely from a variety of sources.
• Allow the templating of configuration files from a variety of sources, for instance pull secrets from Consul/Vault other sources
• Allow the configuration file itself to be broken down into logical sections that are recombined in a set order.

The major mechanism it will be using to do this is by a custom fork of the gomplate tool, that allows one to embed it has a library.

[mattdurham]

Requesting a review of the concept, more so than the actual code, since it is all just stubs in this PR.

[rfratto]

Is there an example folder/set of configs that uses this? From the docs alone I'm having a hard time envisioning how this ties together.

[mattdurham]

I will translate some unit tests into a structure that helps lay it out.

[mattdurham]

Uploaded some documentation, all these are legit examples I did with my fork with dynamic configuration. Should give a better overview, there are a lot more I want to add but didn't want to spend forever on the examples. Its much easier to view if you download or look at the repo and not the diff.

@rfratto @rlankfo @tpaschalis

[rlankfo]

Looks good to me for the most part, I like the cookbook style recipes. The naming conventions proposed seem a bit wonky to me though vs just importing all files in a directory for example.

[rlankfo]

This naming convention seems foreign to me -- I haven't seen other projects that work this way. Usually projects with conf.d style configurations will have a main config file (say an agent.yaml in our case) and then a config directory (like conf.d/ or agent.d/) where all *.conf or *.yaml files are in the directory are imported. Would something like this be possible instead of relying on naming prefixes?

[mattdurham]

It's definitely possible, we would need to run all possible config types loaded on each file to see what it actually contains. They would be in memory at that point so the cost should be limited.

[rlankfo]

is the file:// scheme required here?

[mattdurham]

I can change this here in the template_paths and datasources to default to file:// if none specified BUT datasources embedded in templates would still need need file:// and that seems inconsistent.

[rlankfo]

Is -config.file no longer required with -dynamic-config? I'm not a fan of the naming prefixes and to me it makes more sense to pass the main config file to -config.file and have a -config.directory flag to import additional *.yaml files. (or configure this directory in the main config).

[mattdurham]

If you are using dynamic configuration then -config-file points to the config file that handles the dynamic configuration, basically a list of folders to scan and any datasources you want to define. We could have -config.directory but you would either need to allow an array of them and also additional flags to specify datasources.

You also do not need to specify an agent-*.yml at all, only import the parts you need.

[rlankfo]

Oh I see. IMO overriding the -config.file flag seems strange -- could a dynamic config block be included in the main agent config (what we're loading as agent-*.yml according to the proposed design) ?

[mattdurham]

The dynamic config block tells the system where to find the agent-*.yml if we didn't want to reuse -config.file then a new flag could be added. I resued it for simplicity but if it makes it clearer then that's worth it. The dynamic config tells the system where to load the agent-*.yml so it needs to be added first.

Example /etc/grafana/dynamic-config.yml

template_paths:
  - "file:///etc/grafana/config"

The directory /etc/grafana/config has one file agent-1.yml.

Running ./agent -config.file=/etc/grafana/dynamic-config.yml -enable-features=dynamic-config,integrations-next will

1. Load the configuration at /etc/grafana/dynamic-config.yml
2. Search the directory at /etc/grafana/config
3. Find the agent-1.yml
4. Unmarshal the config and start the agent

[rlankfo]

My thinking is that it makes more sense to do the following:

1. Load the agent.yaml config by passing it to -config.file. This would contain the contents of agent-*.yml as well as the contents of dynamic-config.yaml
2. If the dynamic-config feature is enabled, parse the template paths.

This seems more "conventional" to me. It would mean we don't need a separate config when using dynamic config, and we don't have to discover the base agent config file. I may be missing something here, but does this seem sane to you?

[mattdurham]

My thought is you will want to have a base agent config file that itself will be templated/re-used. The benefit of having them separate is allowing you to configure a fleet of agent instances.

If you have a dozen agent instances running and want to have a base agent that has organization defaults (endpoints, username, password, agent metrics enabled), then you can have a folder with just your agent-1.yml and in the dynamic configuration, template_paths you have the directory where your base agent-1.yml resides then the individual instances would have different directories they load the specific details of what they want to scrape/observe. The dynamic-configuration template_paths directories could even be templated with environment vars.

template_paths:
  - "s3://agent_configs/base" # contains agent-1.yml
  - "s3://agent_configs/{{agent_purpose}} # contains specific overrides for this agents purpose
  - "s3://agent_configs/{{hostname}} # contains specific overrides for this host

If you needed to globally update a value or setting you could do it in the base and when the reload timer (as of yet undiscussed) triggered it would propagate.

This might be a use case that is not useful or I am overcomplicating it =)

[rlankfo]

Why can't the base still be templated or re-used if you combine the dynamic config? I think you'd still be able to accomplish what you've described here (e.g. env vars in the template_paths section of the agent.yml) if you combine the dynamic config with the base config. It does sound a bit overcomplicated to me 😅

[rlankfo]

Are these features going to be tightly coupled? e.g. will users be able to make use of multiple config files w/o enabling the dynamic config feature that parses the gomplate templates? In my mind these seem like they should be separate.

[mattdurham]

The templating is optional, if not needed inside the config files then you don't include the template markup. I do think it is integral to the feature though, ie if they were separate PRs to keep the code review easier that's fine but releasing together is important.

[tpaschalis]

Nice work man! Thanks for providing nice the cookbook, it was great to see how we envision that it would work in action! 🚀
And this looks something that's going to really help with maintaining complicated configs.

Some thoughts:

Dynamic Configuration will find the first file matching pattern agent-*.yml and load that as the base. You can only have one agent template.

First off, it sounded a little weird that we'll only accept the first result of the wildcard (I suppose in lexicographic order) and silently discard the rest. ​I'm not sure which is the best solution since we have both files that are additive and ones that are not. Have we given any thought to erroring out when there are more than one matching files, or being able to add a matcher for what's in *?

Secondly, do we explicitly fill in with every available field and add empty values as default? Is that how for example 01_Basics/02_assets/final.yml came to be larger than its inputs? Where do we get these default values from?

Finally, nice one with forking gomplate to query EC2; much simpler than I thought it would be! In order to be able to query URL-based datasources (eg. GCP Buckets or Azure Blobs) with authentication, will we need something similar?

[mattdurham]

@tpaschalis in regards to the final.yaml, it is the result of our unmarshaling of the config which includes all the fields and defaults. It will error out if it finds multiple files that match: agent, server, metrics, traces, or logs. Will update the documentation to reflect that. The cloud provider queries will use the default auth which is generally environment variables or profiles (AWs/GCP, not used Azure much so unsure).

[rfratto]

Nice! I'm supportive of exploring the general idea of using templates to support dynamic configuration.

I have some high-level concerns about relying on templates; I'm worried it might be really hard to debug problems and add a lot of overhead to the mental map you build of your configs. That being said, it's a good idea to start with this anyway. It's easier to implement than the alternatives I can think of, and that'll be really useful to collect feedback and use cases. I'd also love to be wrong about my concerns and see this graduate out of a feature flag :)

My only "big" concern is making sure that you can have more than one file per specific instantiation of an multiplexed integration (i.e., one file for mysql_exporter against server-a, another for server-b). Splitting up integrations across multiple files aligns with the CRD concept from the operator, and also makes it easier to write instructions that users can copy and paste into new files.

(I have some language / styling nits that I've left out of the review, I don't think they're important right now as an RFC)

[rfratto]

Maybe give a link to issues/discussions to point people where to share feedback? This will be rendered on the docs site so it might not be immediately clear where people should be sharing their experiences

[rfratto]

Can we manually trim out zero values from the final.yml files so it's not overwhelming to readers? (even if that means it's not synced with the actual output of /-/config anymore)

[mattdurham]

I struggled with this, partly due to laziness and partly since I wanted to match exactly what agentctl dryrun outputs. I tried to call out what should be in the final. I may see if I can add an option to dryrun to compare a default config with the generated and only output the differences.

[rfratto]

👍 might be a good opportunity for us to add missing omitempty all over the place (though it doesn't help with all our dependencies that don't have it)

[rfratto]

Nice! I think we should make it clear what happens if you have metrics instances defined in the base config file and also in templates. (IMO, it should be valid to have both and just append the arrays, if that's not already what happens)

[mattdurham]

It is appended, but I will add more verbiage on what is additive vs what is replaced.

[mattdurham]

I feel the only large outstanding concern is whether to use file-names or have a header on each file to determine the config type it should load.

Filenames would be the existing agent-*.yml,
Header would be adding the "root" element back into the config, so

integrations:
  redis_exporter_configs:
  - redis_addr: localhost

then scan the top element to find which configuration to load. (Or at least something similar) The reason I went with filenames is that it enforces figuring out what the file contains when looking in a directory. Else people can name them config1.yml, config2.yml etc. Granted they still can name them in a way that makes it easy to understand in the header case.

Thoughts? @rfratto @rlankfo

[mattdurham]

@rfratto thoughts on removing the server loading and embedding it into the dynamic configuration block. That way it is NOT reloaded.

[rfratto]

I feel the only large outstanding concern is whether to use file-names or have a header on each file to determine the config type it should load.

I'm not opposed to the header so that all files have the same schema, but FWIW my suggestion was instead of

template_paths:
   - "/etc/grafana/01_assets" 

to have a user explicitly specify:

base_template_path: /etc/grafana/01_assets/agent-1.yml 
metrics_instance_template_paths: 
  - /etc/grafana/01_assets/metrics-instance-*.yml 
integration_template_paths: 
  - /etc/grafana/01_assets/integrations-*.yml 
traces_template_path: /etc/grafana/01_assets/traces.yml
logs_template_path: /etc/grafana/01_assets/logs.yml

That way they're in control of the naming and folder structure conventions, and we still know what the contents of the files are expected to be without having to add the header.

[mattdurham]

It's a bit more config but solves the singleton problem and allows them to override the pattern. Let me prototype the changes and see how it feels.

[mattdurham]

Prototype this and handling some of the nonfile providers is difficult as a single string, so would need to be the below, which feels verbose.

base_template_path:
  url: s3:///mybucket/
  filter: agent-*.yml

Alternative would be to have the template paths as is BUT allow the override of filter, so by default would use the ones listed above but allow override for each filter.

template_paths:
   - "/etc/grafana/01_assets" 
integrations_filter:  myspecific_filter_integrations-*.yml