Add support for dynamic keys

[diegomarangoni]

This was mentioned before by @duffqiu in this issue.

Would be nice to prioritize the services running in the same machine as my nginx with different weight.
But, for that works, I need to watch a key in a dynamic path that contains a dynamic token, like the machine id.

[template]
src = "nginx.tmpl"
dest = "/etc/nginx/conf.d/example.com"
keys = [ "/announce/services/hhvm", "/announce/machines/${CURRENT_MACHINE_ID}/services/hhvm" ]
owner = "nginx"
mode = "0644"
reload_cmd = "service nginx reload"

And at the template, loop the both keys, but excluding elements of first loop that are in second one, like that:

upstream hhvm_backend {
    least_conn;

    server 127.0.0.1 down;

{{ range getvs "/announce/machines/${CURRENT_MACHINE_ID}/services/hhvm/*" }}
    server {{ . }}:9000 weight=5;
{{ end }}

{{ range getvs "/announce/services/hhvm/*" }}
    ### at this loop, exclude the elements of the previous one ###
    server {{ . }}:9000;
{{ end }}

    keepalive 8;
}

server {
    listen 80 default_server;

    server_name .example.com;
    root /var/www/example.com;

    location / {
        try_files $uri /index.php$is_args$args;
    }

    location ~ \.(hh|php)$ {
        include fastcgi_params;

        fastcgi_param     REQUEST_METHOD $request_method;
        fastcgi_param     SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_index     index.php;
        fastcgi_pass      hhvm_backend;
        fastcgi_keep_conn on;
    }
}

This could be achieved by doing some kind of named keys, like:

[template]
keys = {
    "all" => "/announce/services/hhvm",
    "local" => "/announce/machines/${CURRENT_MACHINE_ID}/services/hhvm"
}

and then:

upstream hhvm_backend {
    least_conn;

    server 127.0.0.1 down;

{{ range keys("local") }}
    server {{ .Value }}:9000 weight=5;
{{ end }}

{{ range keys("all") }}
{{ if nin(.Key, keys("local")) }}
    server {{ .Value }}:9000;
{{ end }}
{{ end }}

    keepalive 8;
}

This would be very useful, I gave the example of prioritizing local services, but I can go further, like prioritizing services at the same datacenter, or even cloud provider.

Thanks.

[danhowitt]

+1 for dynamic keys. In the template I can reference dynamic keys based on environment variables:

{{$service := getenv "SERVICE_NAME"}}
{{range gets (print "/config/" $service "/*")}}
  export {{.Key}="{{.Value}}"
{{end}}

But I'm unable to use environment variables to define the keys to be watched:

[template]
src = "env.tmpl"
dest = "/usr/src/app/environment"
keys = [
    "/config/{{ getEnv "SERVICE_NAME" }}/",
]

Reiterating @diegomarangoni supporting dynamic keys in the template file would be very useful.

[kelseyhightower]

I've made the call here to reject the idea of dynamic keys in favor of forcing users to be explicit in configuration files. I know this limits a few things confd can do, but I really want to keep this tool simple. I've also added more context in the linked issues on why I don't want confd to go down this path.

In the future, once I clean up the docs, I want to outline the end game for confd, which is to provide a simple bridge between configuration stores and applications that need to read configs from disk.

I plan to add a few features that allow people to pick up template resources and templates dynamically which will make this like this a little easier, but not as streamlined. In the near future you will be able to do something like:

$ confd --template-resource=/myconfig.toml --template=myconfig.tmpl