[Filebeat] Fix conditions to decode_json_fields and which pipeline to run

[crespocarlos]

What does this PR do?

This PR fixes a problem with the ingest pipeline failing to use the correct ingest pipeline when json.keys_under_root is set to true

Why is it important?

The Kibana logs on ECS deployments constantly show field [json] doesn't exist message in error.message field. That happens because cloud deployments automatically set json.keys_under_root to true for versions > 8.0.0 and the pipeline didn't properly cover such scenario after this change

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

• Start local Kibana and add logging config to kibana.yml

logging:
  appenders:
    console:
      type: console
      layout:
        type: pattern
        highlight: true
    file:
      type: file
      fileName: ./logs/kibana-json.log
      layout:
        type: json
  root:
    appenders: [default, console, file]
    level: debug

Pull this branch and start filebeat from the source https://github.com/elastic/kibana/blob/main/x-pack/plugins/monitoring/dev_docs/how_to/running_components_from_source.md#filebeat

• On filebeat.yml, enable Kibana module.

- module: kibana
    log:
      enabled: true
      var.paths:
        - PATH_TO_KIBANA_LOG
      input:
        fields:
          ecs.version: "9.0.0" # existing field
          log.level: "TEST" # existing field
          service.name: "Kibana" # new field
          cloud.availability_zone: "danger-zone" # new field
        fields_under_root: true
        json.fields_under_root: true

The ingest pipeline has to properly override the original log content when there are custom field values on filebeat.yml input.fields settings. Therefore, these test scenarios have to work as following :

• fields_under_root: true and json.fields_under_root: true -> input.fields override original log content
• fields_under_root: true and json.fields_under_root: false -> input.fields override original log content
• fields_under_root: false and json.fields_under_root: true -> input.fields override original log content
• fields_under_root: false and json.fields_under_root: false -> input.fields override original log content
• Remove input object altogether -> original log content is kept

The fact that json.fields_under_root is true, shouldn't cause error.message field to be present.

Related issues

closes #34210

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @crespocarlos? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[crespocarlos]

Ensure this only runs if it's a json log

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-05-03T22:55:08.436+0000

• Duration: 69 min 12 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7783
Skipped	749
Total	8532

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[crespocarlos]

/test

[klacabane]

It seems that we don't set the message property anymore when json.fields_under_root is set (either to true or false). Message appears to be ingested without this change, even if error.message exists

[crespocarlos]

It seems that we don't set the message property anymore when json.fields_under_root is set (either to true or false). Message appears to be ingested without this change, even if error.message exists

Thanks for catching this. I've managed to fix it and also updated the description with all scenarios that this change covers.

[crespocarlos]

This way, the pipeline can properly merge the ECS log content with some fields the Filebeat adds by default (e.g log.offset)

[klacabane]

@crespocarlos walked me through the change offline, looks great thanks!