Create multiple config files from templates

[lorenz]

I want to run tinc with confd, tinc requires you to have a configuration file for every host. These can all be built from the same config. I'd like to have the host list and the host-specific configuration stored in etcd and afterwards turned into files by confd. Any chance that this is getting added?

[kelseyhightower]

@lorenz What would that look like? What data would each template receive?

[lorenz]

@kelseyhightower Basically there would be a go template which is the same for every file. The data would be equal (structure-wise, not data-wise) but grouped up in directories, one for every file. Does that make sense?

[mborho]

+1

The possibility to generate config files programmatically would be very handy.

[kelseyhightower]

@mborho Can you give me an idea of what an example resource template and template file would look like?

[mborho]

@kelseyhightower Good question :)
Let me start with the use case first. Having some keys in the following way:

/auth/foo/user1
/auth/foo/user2
/auth/bla/user1

Which should result in multiple files after processing:

foo.htpasswd
bla.htpasswd

Currently I'm using an intermediate file, which gets cut by some awk magic added to the restart_cmd.

But perhaps a custom tag is possible in the templates, something like this:

{{configfile "filename" "fileprefix" "content"}}

And in the template resource something like:

dest_dir_multiple = /path/for/configfiles/

So the config-files would be saved as /path/for/configfiles/[filename].[fileprefix]

Makes this any sense for you?

[crunchywelch]

FWIW, this maybe addresses this issue: #332

[kelseyhightower]

After thinking about this for a while, I'm going to reject this feature in order to keep confd simple. confd is really designed to map one template resource to a single output file. This keeps confd simple, but as you pointed out it does limit the types of problems confd can solve.

I'm willing to make this trade off though in order to keep confd simple.

[openxxs]

Hi, @bacongobbler I need to confd a nginx.conf file and a ./conf.d/upstream.conf file, which nginx.conf includes ./conf.d/upstream.conf. Because these two files are determined by different etcd clusters, I have to start a confd process for each.
toml file for nginx.conf:

[template]
prefix = "/domeos/loadbalancer/nginx/nginx03"
src = "nginx.conf.tmpl"
dest = "/etc/nginx/nginx.conf"
keys = [
    "/conf",
]
check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/nginx -s reload"

toml file for ./conf.d/upstream.conf:

[template]
src = "upstream.conf.tmpl"
dest = "/etc/nginx/conf.d/upstream.conf"
keys = [
    "/registry/pods/default",
]
check_cmd = "/usr/sbin/nginx -t -c /etc/nginx/nginx.conf"
reload_cmd = "/usr/sbin/nginx -s reload"

When I change the /registry/pods/default value and generate a fault ./conf.d/upstream.conf, confd tells me:

INFO /etc/nginx/conf.d/servers.conf has md5sum 457c221119f29b5eeaa0a45ddae63d52 should be 46d8305693d13c7fbaf8d509bb284c94
2016-06-28T02:40:19Z 26c17fae16ff ./confd[426]: INFO Target config /etc/nginx/conf.d/upstream.conf out of sync
2016-06-28T02:40:19Z 26c17fae16ff ./confd[426]: ERROR "2016/06/28 02:40:19 [emerg] 436#436: "upstream" directive is not allowed here in /etc/nginx/conf.d/upstream.conf:4\nnginx: [emerg] "upstream" directive is not allowed here in /etc/nginx/conf.d/upstream.conf:4\n"
2016-06-28T02:40:19Z 26c17fae16ff ./confd[426]: ERROR exit status 1

It means check_cmd works. But the ./conf.d/upstream.conf has been updated, which makes nginx product ERROR.
Dose check_cmd have to be /usr/sbin/nginx -t -c {{ .src }}? What if check_cmd = "bash myCmd.sh"?

[bacongobbler]

check_cmd can be whatever you want it to be. It could literally be /bin/true if you don't care about validating your templates.

[waszi]

I am using confd to configure nginx on multiple containeres, domain config/maps etc it is easy to split config without multiple destination but there is small exception. SSL Certificates requires separate files with name based on domain name. Any ideas how to solve this ?

[ghost]

Hi @kelseyhightower , do we support multiple destinations for config files now?

[okushchenko]

@ricky3dec there is no support for multiple destinations yet. Can you share your use case?

[godefroi]

I have a use case. I use confd to write out rules files for Prometheus. I want each group in a separate .yaml file, and I have each group in a separate key in Consul. I would like to use a single resource to write each key into a separate file, since I don't know what rules groups (keys) will exist ahead of time. It might look like this:

[template]
src = "prometheus_rule.yaml.tmpl"
dest = "/prometheus/rules/{{$key}}.yaml"
keys = [
        "/prometheus/rules/*",
]

[okushchenko]

@godefroi I think that this way of maintaining templates is convenient for an operator, but only for manual management. If you are using configuration management tool, like Confd, there should be no justification to maintain that approach of separate config files.

Will a single config file with a loop inside work for your use case?

[godefroi]

Yes, and that's ultimately what I did. It may not be possible in the general case, however.

[okushchenko]

@godefroi I'll reopen this issue for now. I think this issue needs more discussion. Let's wait for more use-cases to be shared by a community.

[fiatflux]

Wireguard also requires a separate file per device. I would like to use confd to generate these configuration files so that I can dynamically add and remove tunnels in a VPN mesh. The workaround of breaking up one generated file into multiple files isn't hard, but feels a bit clunky.

[hubo1016]

It could be implemented with some hack:

1. introduce a new option multi_dest or variable_dest
2. introduce a new template function setdestination(path, prefix='/')
3. generate an UUID (or some random hash) to create a boundary
4. when setdestination is called, output something like --<UUID>--dest=/newpath--<EOL> to separate the output file
5. split the file and copy each part to different file
6. run check_cmd and reload_cmd

The check_cmd and reload_cmd should support more parameters in this mode.

[okushchenko]

@hubo1016 it's a hack. I've tried to find a simple solution to this problem, but I just can't find one. If someone will propose a decent solution I'll gladly consider it.

[pantierra]

Thanks a lot for this nice piece of Software!

My two cents, with a use case: A multi-domain prosody (jabber server) I want to run through docker and to generate its configuration on the fly based on a list of urls. prosody creates vhosts - similar to a web server - ideally with different files for each domain. confd with the here requested feature would be perfect for this. I guess loops are enough for me, it is just not that elegant.

[metanovii]

Hi, my case: auto generate certs for some domains. Without this func i must create template for each domain.

[okushchenko]

@xamanu @metanovii thanks for sharing your use-cases!

[jinxmcg]

I also need this for generating multiple interface files at setup. The workaround I did was to use confd to generate a .sh script (parameters functions etc) which is executed with "reload_comd" and generates the multiple config files I need.

[sdwire]

My use case: provisioning dashboards for grafana from a central source. From what I can tell in the grafana docs, each provisioned dashboard must be a separate JSON file in, for example, /var/lib/grafana/dashboards. I'd like to use confd to pull those dashboards from consul's kv store (where all the rest of our config lives) and create one JSON file for each key in a consul path.

[ykfq]

Its really useful to seperate the dest files as our nginx configuration file is growing very large with thousands of lines.

[poppyred]

+1

[Yullin]

Hi, my case: auto generate certs for some domains. Without this func i must create template for each domain.

so, about this case, any one had best practise case?

[metanovii]

Hi, my case: auto generate certs for some domains. Without this func i must create template for each domain.

so, about this case, any one had best practise case?

hi, now i use consul-template. One template for all certificates.

[tigerfsh]

@metanovii Can you share the link how you use consul-template?

[metanovii]

@metanovii Can you share the link how you use consul-template?

Now i use hashicorp vault + consul-template. Some cronjob generate certificates and put into vault and consul-template get certificate from vault.

example config for consul-template:

reload_signal = "SIGHUP"
kill_signal = "SIGINT"
max_stale = "10m"
log_level = "warn"
pid_file = "/var/run/consul-template.pid"

wait {
    min = "5s"
    max = "10s"
}

vault {
  address = " "
  token=" "
  grace = "10m"
  unwrap_token = false
  renew_token = true
}
syslog {
  enabled = true
  facility = "LOCAL5"
}

template {
  contents="{{ with secret \"kv/somepath/infra/security/x.509/somedomain\" }}{{ .Data.data.crt }}{{ end }}"
  destination="/etc/nginx/certs/somedomain.crt"
  perms = 0600
}
template {
  contents="{{ with secret \"kv/somepath/infra/security/x.509/somedomain\" }}{{ .Data.data.key }}{{ end }}"
  destination="/etc/nginx/certs/somedomain.key"
  perms = 0600
}

.....

exec {
  command = "systemctl reload nginx"
}

[wurenny]

multiple dests are very useful, e.g:
i want add some nginx proxy servers to nginx conf.d by rest api, i can do this:

• invoke consul api to add some nginx proxy server key, value include tls key and crt
• confd read consul key, mass generate nginx conf file and tls key and tls crt
• reload nginx

maybe it is not sure how many proxy servers, and dest file of confd template must be dynamic