Configuration as yaml/json

[warmfusion]

I've been trying to setup a FluentD puppet module that is able to build configuration from resource calls, however, the configuration syntax makes this very difficult.

Is it possible to use yaml or json for the configuration of this application as I would like to be able to

• Define sub-elements using a hierarchy of hashes
• Avoid having to declare 'footer' elements ( eg </match> )

Example Configuration

fluentd::match:
    pattern: 'nginx.*'
    priority: 10
    options:
      type: copy
      store:
       - type: 'file'
         path: '/var/log/fluent/myapp'
         time_slice_format: '%Y%m%d'
         time_slice_wait: '10m'
         time_format: '%Y%m%dT%H%M%S%z'
         compress: 'gzip'
       - type: 'mongo'
         host: 'fluentd'
         port: '27017'
         database: 'fluentd'
         collection: 'test'

[repeatedly]

Some users use fluentd with puppet: https://github.com/mmz-srf/puppet-fluentd
I'm not familiar with puppet but constructing yaml is easier? Dumping hash to yaml?

We don't have a plan to support these formats but
converting yaml to fluentd's configuration is one idea.

[warmfusion]

Thanks @repeatedly, I tried to use that puppet module however due to the limitations of the configuration format it also has the same problems I did when configuring matchers/sources etc.

For example, it is not possible to create a matcher which uses the forwarder plugin with 2 servers and a secondary (as described in the HA and Forwader documentation)

This is due to the way the matcher config is created from a template, which due to limitations in puppet, combined with limitations of the config format means sub-sections are hardcoded, duplicated and otherwise worked-around.

On the surface it appears that configuration is managed through yaml, however, this is only made possible by some complex coding under the hood in puppet which still isn't able to properly describe the configuration options available.

[repeatedly]

For example, it is not possible to create a matcher which uses the forwarder plugin with 2 servers and a secondary

From this spec, puppet-fluentd supports <server> sections: https://github.com/mmz-srf/puppet-fluentd/blob/develop/spec/defines/fluentd_match_spec.rb#L33
If secondary doesn't work, then adding secondary support to puppet-fluentd is good for other users.

Hm... supporting yaml instead of DSL may be a good idea for this problem.
But I'm not sure how many users want this feature and
we don't want to have multiple configuration formats.

[loveshell]

nice

[salekseev]

+1

[soylent]

Supporting commonly used configuration format will definitely improve product adoption. +1

[koman-maciej]

+1

[notslang]

It would be nice to switch to YAML and providing a tool to convert configurations to YAML would be simple since the mapping is basically 1:1. However, all the docs would need to be rewritten.

In the long run, YAML is preferable because there are already validation programs, fast parsers, YAML schemas, and tons of text editor integration / syntax highlighting.

[sandstrom]

YAML would be great, another option is TOML (https://github.com/toml-lang/toml/wiki#projects-using-toml).

Both would be better than the current configuration format.

[martinreber]

+1
our team has planned to us kustomize in the future and so YAML is the perfect format to apply patches to the fluend configuration.

[tace]

yaml please