config_format: new config reader with YAML support (#4331)

[edsiper]

The following patch introduces a new interface called 'config_format'. This new interface implements a new layer to read and compose a configuration 'context' for Fluent Bit.

The idea started on discussion #4331 where initially looking for feedback to extend the format and support 'groups' (or sub section) many needs were expressed like other formats.

The actual implementation which comes from lib/monkey, is not flexible enough and adds complexity when trying to extend it.

The new 'config_format' is a 'layer' that generate configuration contexts:

      +-------------+     +---------------+     +----------------------------+
      | Config File | <-- | Config Format | --> | Fluent Bit Readers & Setup |
      +-------------+     +---------------+     +----------------------------+

The advantage of this is that 'config_format' supports different 'backends' to read data from: this implementation supports 'fluentbit' (current) format but also 'YAML' format.

                          +---------------+
                          | Config Format |
                          +------+-+------+
                                 | |
              +------------------+ +-------------+
              | fluentbit format | | YAML format |
              +------------------+ +-------------+

'fluentbit' format is a backport format from lib/monkey with API changes, the YAML format is parsed by using libyaml when available.

The concepts of configuration keeps being the same:

 - sections: define a block
 - properties: key/value pairs that belongs to a section

The following examples are identical configurations in different formats:

--- fluentbit format ---

[SERVICE]
    flush     1
    log_level info
    
[INPUT]
    name      tail
    path      /var/log/containers/*.log
    parser    docker
   
[INPUT]
    name      forward
    listen    0.0.0.0
    
[OUTPUT]
    name      stdout
    match     *

Now the identical representation in YAML:

service:
    flush:       1
    log_level:   info

pipeline:    
    inputs:
        tail:
            path: "/var/log/containers/*.log"
            parser: docker
    
        forward:
            listen: 0.0.0.0
    
    outputs:
        stdout:
            match: "*"

another YAML example:

# setting up a local environment variable
env:
    flush_interval: 1

# service configuration
service:
    flush:       ${flush_interval}
    log_level:   info
    http_server: on

# pipeline 1
pipeline:
    inputs:
        dummy:
            samples: 100
            tag: example
    outputs:
        stdout:
            match: example

# pipeline 2
pipeline:
    inputs:
        tail:
            path: /var/log/containers/coredns*.log
            read_from_head: true
            tag: kube.*

    filters:
        kubernetes:
            match: kube.*

    outputs:
        stdout:
            match: kube.*

To Do

• config format API
• fluentbit format reader
• YAML
  • format reader
  • support environment variables (aka @set A=B)
• Migrate service conf reader to config format (WIP)
• Migrate 'static config' reader to config format (is there anybody using this feature ?)
• Migrate command-line options to use config format
• Migrate library API (internal) to config format (double-check)
• Migrate parsers reader to config format
• migrate upstream servers readers to config format
• migrate stream processor reader to config format
• migrate external plugins reader to config format
• groups (sub-sections support)
  • API
  • fluentbit format
  • YAML format

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

YAML + Environment vars:

https://asciinema.org/a/Zz6ZAS3D6ZJgBtH3rjYwNmzqx

note: YAML schema will change a bit now

[patrick-stephens]

@edsiper do we have a docs PR to document the YAML format we're using?