Adopt `parsers` in all Filebeat inputs

[kvch]

Goal

Parsers are the successors of readers of the log input. They provide a more flexible message parsing. At the moment they are only available in the filestream input. But the functionality could be useful in other inputs.

We have to inspect all available inputs in Filebeat to see if it makes sense to add parsers. If the answer is yes, then the feature should be added.

Parsers

• ndjson
• multiline
• container Add new parser to filestream input: container #26115
• syslog [libbeat] Add Syslog parser and processor #30541

How to add parsers to an input

1. Add parsers code to the input

The parser.Config object has its own Unpack function, so it is enough for you to add it as an attribute to your configuration.

type myInputConfiguration struct {
   // other attributes
    parsers parser.Config `config:",inline"`
}

Then create the parser based on the configuration. The Create function expects a reader.Reader as a parameter it can read from.

var reader reader.Reader
p = inp.config.parsers.Create(reader)

Example configuration accepted by the code above

parsers:
  - multiline:
      type: count
      lines_count: 3

2. Add parsers to the reference configuration

Add the reference configuration to either

• filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl
• x-pack/filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl

3. Add parsers to the documentation of the input

There is no common parsers section at the moment because it might not make sense to document all parsers for every input.

Inputs

• AWS CloudWatch Add json.* config support in the aws-cloudwatch input #26429
• AWS S3 [Filebeat] change multiline configuration in awss3 input to parsers #25873
• Azure Event Hub
• Cloud Foundry
  - [ ] Container must use filestream input in the future
  - [ ] Docker must use filestream input in the future
• filestream
• GCP Pub/Sub
• HTTP Endpoint
• HTTP JSON
• journald
• Kafka [Filebeat] kafka v2 using parsers #27335
  - [ ] Log must use filestream input in the future
• MQTT
• NetFlow
• Office 365 Management Activity API
• Redis
  - [ ] Stdin will be deprecated
  - [ ] Syslog syslog will be a parser
• TCP input with parsers #31023
• UDP input with parsers #31024
• Unix Add parsers to UNIX input #27858

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[ascendant512]

May I vote for multiline in the unix input? It is the closest thing Filebeat has to a fifo. Immensely useful for injecting arbitrary logs with netcat without messing with log directories or restarting Filebeat.