Extending Configuration Schema

[edsiper]

One of the main reasons for the "schema" of Fluent Bit configuration is simplicity, by forcing the user to have a fixed indentation and avoiding sub-levels of nested elements.

There is no "perfect" configuration schema, all of them have pros and cons, and personally, I think we should focus on how to bring a good end-user experience. What works for us as a project, might not work for others.

The intention of this open discussion is to share the current context and challenges to extend Fluent Bit capabilities from a config file perspective.

Disclaimer: this is not about if we should move to Yaml, TOML, or another type of schema (despite we consider we should have some compatibility or translation from those).

Current Schema

ref: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/format-schema

We have 3 main concepts: sections and key/value pairs.

• Schemas: they start on the left by using braces and a name on it, e.g: [SERVICE], [INPUT], etc.
• Key / Values: Inside a section, multiple key/value pairs are allowed, but they must be indented respecting some offset position from left to right.
• Comments: a line can be commented with a # on the left side of the content. Note this don't support inline comments, only applicable for the while line.

Example:

[SERVICE]
    flush     1
    log_level info

[INPUT]
    name   tail
    path  /var/log/messages
    
[OUTPUT]
    name stdout
    match * 

Extension needs

There are cases where inside a section were would be ideal to have multiple "logical contexts", e.g:

[INPUT]
    name   something
    rule1.type    abc
    rule1.eval    def
    rule1.check true
    rule2.type    ttt
    rule2.eval    ghi
    rule3.check false

at this point, the config get's hard to read. In addition, the plugin developer must implement the logic to parse dynamic keys that start with rule. followed by a number.

Proposal for extension

I'd like to propose the concept of groups. So multiple contexts inside the same configuration can be grouped together but with good readability.

To split the context I propose to use a simple separator prefix followed by a name, e.g:

--- some separator name

by using that approach the configuration explained above will be represented as:

[INPUT]
    name   something
    --- rule for type abc
    type    abc
    eval    def
    check   true
    --- rule for type ttt
    type    ttt
    eval    ghi
    --- rule default
    check   false

Also, we might consider indentation inside the group too:

[INPUT]
    name   something
    --- rule for type abc
        type    abc
        eval    def
        check   true
    --- rule for type ttt
        type    ttt
        eval    ghi
    --- rule default
        check   false

Groups might be optional for the plugin that needs that kind of more complex configuration.

Open for Comments

I would like to get your input on this proposal, again, this is about extending the current format and not switching to another one (it's not the time for that)

[patrick-stephens]

Seems a reasonable shout to me, I'm thinking it might help with reviewing diffs. We would need to report config errors for the group which looks to be similar to an alias.

Do they have to be unique just in the section? I hope so rather than unique for the whole file once you start using includes.

Presumably you can just @include groups? Say you had a common one across sections that would be useful.

Maybe the existing square brackets and indentation rather than a new format?

[edsiper]

definitely error handling must print out the failed group +1 .

about @include inside a section not so sure honestly, do you see some use cases for that ?

note:

[INPUT]                       ----> section definition
    name   something          ----> normal key/value pair
    --- rule for type abc     ----> group separator / definition
    type    abc               ----> key/value pair for the group in question
    eval    def
    check   true
    --- rule for type ttt
    type    ttt
    eval    ghi
    --- rule default
    check   false

[patrick-stephens]

Possibly, I've got a lot of input plugins that share config I currently have to repeat. It would be nice to have a common approach to easily update things, e.g. as new config is added you might want to use a common value and I had to do something like this for the recent Skip_empty_lines by adding.it to all my config as an example.
For example these have a lot of common config:

• https://github.com/couchbase/couchbase-fluent-bit/blob/main/conf/couchbase/in-erlang-multiline-log.conf
• https://github.com/couchbase/couchbase-fluent-bit/blob/main/conf/couchbase/in-eventing-fts-log.conf
• etc for each input

Could we do that common stuff? I'm not sure your proposal is about that though. Thinking about it I probably can do it now with an @include, I just haven't!

It would also mean I could reuse a test configuration as well rather than my current approach of include followed by custom config in a loop:
https://github.com/couchbase/couchbase-fluent-bit/blob/569c0babb96741effa74d46683f44a973ff765b5/test/run-tests.sh#L180

[edsiper]

ahh I get it, so this is mostly an ask for templates-style. What would be a good design for those ? (from a config perspective ?).

I am open to ideas.

[patrick-stephens]

Yeah I'll have a think. Also had some other suggestions for explicit enable field and another that escapes my memory but I'll raise that separately.

[patrick-stephens]

Some of the other formats might already do this, e.g. with YAML anchors.

[nokute78]

https://docs.fluentbit.io/manual/administration/security
TLS supports tls and tls.some_key (e.g. tls.ca_file).
in_tail also supports DB and DB.sync.
How should we describe ?

[edsiper]

The proposal is mostly about "extending" to groups. No replacement of the current design, just an extra way to configure to split in groups "only when needed"

[nokute78]

It may be confusing if some properties support extra way and the others don't.
Should tls support extra way ?

[edsiper]

something like ?

[INPUT]                  
    name   something   
    abdc   efg
    --- tls
    validate    off
    ca_cert    /path/to/something

hmm not sure really (but this is an open discussion), I think the main property and sub-components with dots . works fine, but groups should be used for cases where groups are needed. Having more than one way to configure one specific stuff might be confusing

[nokute78]

Thank you.
Some properties may need an alias if it supports extra way.
e.g. DB expects a database file path , so we may need to support like DB.path as an alias of DB.

groups should be used for cases where groups are needed

What is the real use case ? Which plugin ?

[patrick-stephens]

Having had a bit of time to reflect on this, I'm not sure extending what is a slightly bespoke format is the right way to go. Now I know we're not suggesting changing the format entirely but in my opinion it is something worth considering: rather than putting the effort in to extend the current schema would it be better to adopt a schema that already has the concepts we need along with a full ecosystem?

Playing devil's advocate, if we had a common format like YAML or JSON as input then there are also a load of validation tools available for almost everyone to use. To maintain backwards compatibility we can have a legacy processing stage that converts the current config into the new one then consumes it: there are precedents already for this approach, e.g. YAML is converted into JSON for REST API calls.

The effort spent on extending, testing and maintaining the current schema could then be redirected into the conversion process.
We could potentially look at allowing config injection via an HTTP endpoint in the future with dynamic validation & reload then automatically.
You could autogenerate validation and other rules as well as documentation for fields too as we already can do with YAML or similar.

I guess my main point is that changing the config might be a better option all round so we should not discount it out of hand. I'm not sure what the killer feature is we currently need, if it's plugin development then there are a myriad of tools to deal with YAML, JSON, a.n.other language lookup in C, Go, etc.

Adding this syntax and keeping the dot notation would be confusing, I agree with @nokute78 on this. I think you should pick one and maybe allow aliases or similar for backwards compatibility but only for existing options. The goal being to migrate to a common syntax.

[mickeypash]

I tend to agree with Pat here. I think if you conformed to a base JSON schema then it's much easier to expose the config as YAML, TOML or JSON providing flexibility for many and also avoiding the need to maintain validation logic.

[PettitWesley]

I agree... Using some existing standard would make it easier for example for other tools/systems/services that want to integrate with or build on top of Fluent Bit, since then you can use existing libraries to validate/generate/parse Fluent Bit configuration.

[cosmo0920]

This is just my two cents but I'm one of the wondering how to extends Fluent Bit configuration format people.

Similar to Fluentd configuration style, it would be nice to handle TOML like format that uses indentations for representing groups on Fluent Bit configuration file.

[INPUT]
   name   something
   [INPUT.abc]
     type    abc
     eval    def
     check   true
   [INPUT.ttt]
     type    ttt
     eval    ghi
   [INPUT.DB]
     path /path/to/sqlitedb
   # Default rule needn't to use section notation, just use default level of indentations.
   check   false

[edsiper]

FYI:

A new config format that also supports YAML is under development, you can track the progress in the draft PR #4621

[edsiper]

This has been implemented in v1.9.