out_loki: add support for structured metadata

[jtackaberry]

This PR adds support for structured metadata in the loki output. As of Loki 3.0, structured metadata has been made GA.

Implementation wise, labels and structured metadata are very similar: both are key/value pairs, and both support record accessors, consequently this PR does some light refactoring to avoid nontrivial code duplication between labels and structured metadata.

This PR only implements the analog of the labels config parameter, which is called structured_metadata. Analogs for label_keys and label_map_path were not implemented to keep the change both simple and immediately useful, though they could be added later.

Fixes #8080.

Testing

Because the code paths are slightly different when labels and/or structured metadata employ record accessors, I have two simple configurations: one that uses structured metadata with record accessors, and one that only uses static values.

Our Loki deployment uses mTLS for auth so I've kept that in the config to represent testing, although the Loki endpoint is obviously redacted.

All tests include a log containing combined output with Fluent Bit run under Valgrind.

Test 0: baseline without structured metadata

pipeline:
  inputs:
    - name: tail
      alias: tail
      path: /var/log/syslog
      path_key: filename

  outputs:
    - name: loki
      alias: loki
      match: "*"
      host: loki.example.com
      port: 443
      tls: true
      tls.crt_file: local/client.crt
      tls.key_file: local/client.key
      compress: gzip
      line_format: key_value
      drop_single_key: true
      remove_keys: filename
      labels: service=test, filename=$filename

0-baseline-without-structured-metadata.log

Test 1: structured metadata with record accessors

pipeline:
  inputs:
    - name: tail
      alias: tail
      path: /var/log/syslog
      path_key: filename

  outputs:
    - name: loki
      alias: loki
      match: "*"
      host: loki.example.com
      port: 443
      tls: true
      tls.crt_file: local/client.crt
      tls.key_file: local/client.key
      compress: gzip
      line_format: key_value
      drop_single_key: true
      remove_keys: filename
      labels: service=test
      structured_metadata: filename=$filename

1-with-record-accessor.log

Test 2: structured metadata (and labels) without record accessors

pipeline:
  inputs:
    - name: tail
      alias: tail
      path: /var/log/syslog
      path_key: filename

  outputs:
    - name: loki
      alias: loki
      match: "*"
      host: loki.example.com
      port: 443
      tls: true
      tls.crt_file: local/client.crt
      tls.key_file: local/client.key
      compress: gzip
      line_format: key_value
      drop_single_key: true
      remove_keys: filename
      labels: service=test
      structured_metadata: filename=dummy

2-without-record-accessor.log

Documentation

• Documentation required for this feature

Will submit a docs PR after this PR gets some initial feedback or indication it's likely to be accepted.

Backporting

N/A. It would be nice to make this in time for Fluent Bit 3.1 though. :)

Licensing

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

Does this add any breaking changes to Loki 2.x users ?

[jtackaberry]

Does this add any breaking changes to Loki 2.x users ?

@edsiper possibly if they define the new structured_metadata field, but definitely not if they don't. The payload delivered to Loki is unchanged without structured_metadata being set.

So this would be a matter of the docs to caution users not to configure structured_metadata unless they are using Loki 3.x.

[jtackaberry]

Thanks @edsiper! I submitted fluent/fluent-bit-docs#1406 for the docs part.