Proposal feature: Support multiple template resources per file

[crandles]

This will break the template config syntax, so it may be necessary to alter in some way to provide backwards compatibility. That being said, this PR would allow for a single Template Resource Config file to contain multiple template definitions.

• Use case(s):
  This would allow for the dynamic generation of confd templates in the following manner
  Example: Initial confd template resouce is defined, which then dynamically creates a second confd resource template based on data retrieved from backend.

/etc/confd/conf.d/default.toml

[[template]]
src = "nginx.toml.tmpl"
dest = "/etc/confd/conf.d/nginx.toml"
keys = [
    "/nginx/",
]

/etc/confd/templates/nginx.toml.tmpl

# In this example, every domain configured under "/nginx/*" gets its own config file.
{{range gets "/nginx/*}}
{{$data := json .Value}}
[[template]]
src = "nginx.conf.tmpl"
dest = "/etc/nginx/conf.d/{{$data.domain}}.conf"
keys = [
    "/nginx/{{$data.domain}}/",
    "/other/path/{{$data.other}}/",
]
{{end}}

I've left out the final nginx.conf.tmpl, that would be a standard confd template used to generate a single nginx/conf.d/$domain.conf file in this example.

I think this could solve for #310 and #256 (and other dynamic-config generation use cases) in a clean manner.

[crandles]

The goal here is to allow a simple install of confd to bootstrap itself based on backend-configured data.

With this, I can create a small RPM to install confd with a simple config and an initial template which will instruct confd to connect to the backend and pull the rest of its configs dynamically. With #314, the templates could also be sourced from the backend, and I would end up with a very dynamic system with one place to make configuration changes.

Without a way to do this, I would need to enforce additional configs and templates via a separate mechanism (Puppet, Ansible, another etcd backed process), and introduce another layer of complexity.

[crunchywelch]

I have a similar use case, which could maybe be solved in a similar fashion with a variable in the "dest" value of the config file. We are using uuids to generate service config files for multiple docker containers on the same host.

Starting loosely from the nginx example here https://github.com/kelseyhightower/confd/blob/master/docs/quick-start-guide.md, this:

/0491f260-d089-4263-a7f7-74c0144b6874/template
/0491f260-d089-4263-a7f7-74c0144b6874/subdomain
/0491f260-d089-4263-a7f7-74c0144b6874/upstream/app1
/0491f260-d089-4263-a7f7-74c0144b6874/upstream/app2
/b6f0764e-5fb0-4d3c-b559-f76ac2edf8d3/template
/b6f0764e-5fb0-4d3c-b559-f76ac2edf8d3/subdomain
/b6f0764e-5fb0-4d3c-b559-f76ac2edf8d3/upstream/app1
/b6f0764e-5fb0-4d3c-b559-f76ac2edf8d3/upstream/app2

would use the value in the "template" key to source /etc/confd/conf.d/{{ template }}.toml, and could use the same variable to define the config file location or name:

[template]
prefix = "/nginx"
src = "nginx.tmpl"
dest = "/tmp/{{ template }}.conf"
owner = "nginx"
mode = "0644"
keys = [
  "/subdomain",
  "/upstream",
]
check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/service nginx reload"

This isn't perfect, since it only lets you essentially have one template per UUID, but one could in theory prefix both the UUID and template name with a service name if you wanted (nginx-0491f260-d089-4263-a7f7-74c0144b6874)

The advantage to this is that you are not loading a potentially large number of keys every time with {{range gets "/nginx/*}} when, at least in our case, only one service will really be changing at a time, so this should scale a little better.

Thoughts on this?

[bsingr]

+1

I do not need all of the proposed capabilites. I am looking for a way to render multiple templates but then call the "reload_cmd" only once.

My goal is to update multiple configuration files for a single application and then restart that application. It's a huge JVM-thing were the restart takes minutes, so I want to call restart only once.

[sruon]

+1, any progress?

[kelseyhightower]

It seems people want a way to run templates in a specific order. Say run this 10 templates then call reload_cmd at the end. Is that the right way to think about this?

[crandles]

I think that's part of it.

The other piece is that those 10 templates are very similar -- either the
confd configs describing them are the same, minus a single identifier, or
the templates are the same as well.

In this sort of a use case it would simplify things if one confd
config/template could be used to write to many destinations.

On Sat, Feb 20, 2016 at 10:55 AM Kelsey Hightower [email protected]
wrote:

It seems people want a way to run templates in a specific order. Say run
this 10 templates then call reload_cmd at the end. Is that the right way to
think about this?

—
Reply to this email directly or view it on GitHub
#315 (comment)
.

[crandles]

The other piece is that those 10 templates are very similar -- either the
confd configs describing them are the same, minus a single identifier, or
the templates are the same as well.

In this sort of a use case it would simplify things if one confd
config/template could be used to write to many destinations.

[kelseyhightower]

@crandles What about providing a way to provide a list of src/dest mappings? Something like this:

[template]

[[targets]]
src = "nginx.conf.tmpl"
dest = "/etc/nginx/nginx.conf"

[[targets]]
src = "www.example.com.conf.tmpl"
dest = "/etc/nginx/conf.d/www.example.com.conf"

keys = [
    "/nginx/",
]

check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/service nginx reload"

[crandles]

I think that might be helpful.

In that case would the "keys" be utilized across all targets?

[kelseyhightower]

Now that I had more time to think about this I don't like where this is headed. confd is a simple tool designed to watch some keys in a config database and generate a single template. I know it's limiting, but it solves the 80% use case.

While I like this idea, I think it's the wrong direction for confd. I'm closing this PR out of the sake of simplicity.

[crandles]

Understood. I think I've found somewhere else to manage the complexity -- I appreciate the consideration.