filter_nest: add wildcard_exclude config

[0x006EA1E5]

Implements #10111 by adding wildcard_exclude config to nest filter

#10111

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[0x006EA1E5]

Example config:

pipeline:
  inputs:
    - name: dummy
      tag: logs
      dummy: |
        {
          "message": "simple log generated",
          "logger": "my.logger",
          "level": "INFO",
          "hostname": "localhost",
          "my_map_of_attributes_1": {
            "key_1": "hello, world!",
            "key_2": "goodbye, world!"
          },
          "my_map_of_maps_1": {
            "root_key": {
              "sub_key_1": "hello, world!",
              "sub_key_2": "goodbye, world!"
            }
          },
          "an_unknown_key": "hello, world!",
          "another_unknown_key": "goodbye, world!"
        }

  filters:
    - name: nest
      match: logs
      nest_under: nested
      wildcard: '*'
      wildcard_exclude:
        - message
        - logger
        - hostname
        - level
        - 'my_map_of_*'

  outputs:
    - name: stdout
      match: logs

The above will nest keys an_unknown_key and another_unknown_key under nested

[0x006EA1E5]

I have added this config to the docker compose for structured_metadata.

Here is a screenshot showing that the end result is as expected:

(an_unknown_key and another_unknown_key are ultimately used as structured_metadata)

[0x006EA1E5]

Here is some console output, showing how the above example config produces the correct output:

fluentbit  | [0] logs: [[1742483011.135625671, {}], {"message"=>"simple log generated", "logger"=>"my.logger", "level"=>"INFO", "hostname"=>"localhost", "my_map_of_attributes_1"=>{"key_1"=>"hello, world!", "key_2"=>"goodbye, world!"}, "my_map_of_maps_1"=>{"root_key"=>{"sub_key_1"=>"hello, world!", "sub_key_2"=>"goodbye, world!"}}, "nested"=>{"an_unknown_key"=>"hello, world!", "another_unknown_key"=>"goodbye, world!"}}]

[edsiper]

key is not a null terminated string and this comparison can potentially fail. Since klen is set, I recommend comparing first klen == wildcard_exclude->key_len and if it passes then do the strncmp()

[0x006EA1E5]

FYI, I copy and pasted this bloc of code from wildcards below, and adapted to wildcards_excludes.

As I read it, this part is for "dynamic" matches (where there is a wildcard), so we don't expect the lengths to match. For "non-dynamic" (which follows), it does do a length check already.

However, we could do a check here, something like klen >= wildcard_exclude->key_len, right?

This is to check that ABC123 is a viable match for wildcard ABC*, so we want to check that ABC123 is at least three chars long...

I will add some more unit tests to verify all this 🤞

[0x006EA1E5]

I added the klen >= wildcard_exclude->key_len check, and some tests, and it all looks good

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.