Support multiple technologies/logformats on one K8s node

[marckamerbeek]

Hi,
We run Kubernetes 1.8.1 on premise and start using fluent-bit as log forwarder to elastic search. There are running 600+ docker containers with all kinds of technologies. Java, Spring boot, Python, NodeJS etc...

How should my config look like? Do I need all kind of different tail inputs? Or just *.log tail input and go through all the filters with all kind of different parsers with regexes (performance wise not a good idea I think)?

Am I the only one with this question or I really have no clue how fluent-bit should work.

[edsiper]

hi @marckamerbeek

there are two approaches;

1. Fluent Bit 0.12: this is the actual stable version and the filter_kubernetes only allows to take the raw log message (without parsing) or parse it when the message comes as a JSON map. Of course this is not so ideal since not all applications generate logs in JSON format.

2. Fluent Bit 0.13: this is the actual development version which we aim to release on January 31 if everything is fine with testing and QA. The major feature here is that in your application Pod definition you can "suggest" a parser for the log processor, e.g: if you are going to deploy a Pod which is Apache web server, in the annotations you specify "logging.parser: apache", so the log processor will lookup that parser definition and apply it for the specific log entries for that container.

If you want to experiment with #2 and also help us with testing, please refer to the instructions sent to the Google group this week:

https://groups.google.com/forum/?hl=en#!topic/fluent-bit/Va2gsviVaRE

please let me know how that goes.

[marckamerbeek]

Wow...thats exactly what we had in mind. We have a story on the backlog to build a new plugin that does that. That saves us a lot of work! We are not familiar with building plugins in C.

Ofcourse I want to experiment with that. I'll keep you posted.

And what about multi format log files? Like kubernetes nginx-ingress-controller logs. That has 3 different formats in one log file. Like Access logs, Go and another one. Or am I asking to much now...:-)

[edsiper]

Ofcourse I want to experiment with that. I'll keep you posted.

great :)

And what about multi format log files? Like kubernetes nginx-ingress-controller logs. That has 3 different formats in one log file. Like Access logs, Go and another one. Or am I asking to much now...:-)

oh, I did not think about that specific use case, which can happen on any multi-container Pod. Likely the solutions seems to be around let the Pod suggest multiple parsers and let Fluent Bit process that. It's not that good in terms of performance but something that needs to be solved in some way.

[marckamerbeek]

Well, actually its not a multicontainer pod. Its just one container spitting out all different formats to stdout. We also got some python containers doing the same. Fluentd got that multi format plugin (https://github.com/repeatedly/fluent-plugin-multi-format-parser) which does the thing. But I can imagine that will have some performance issues running through those regexes.

I discussed your solution with some colleague's mentioned in 2. It's nice to have an annotation to define the parser that is needed. But wouldn't it be nice to define your own regex so development teams can determine their own format? Like an adding a regex with an annotation. Or two options. One custom and one predefined.

Like:

apiVersion: v1
kind: Pod
metadata:
name: apache-logs
namespace: default
annotations:
logging.parser: apache
logging.parser.regex: /...your regexp pattern.../
spec:
containers:
name: apache-logs
image: edsiper/apache_logs
imagePullPolicy: Always
restartPolicy: Always

[edsiper]

Actually I hear that idea recently from a conference attendee, initially I discarded that approach because of performance reasons, but as you said, the benefit of that solutions pays-off the cost of solving the log processing problem for multiple formats.

The cost associated is to compile the regular expression when Fluent Bit filter gather the metadata, of course this is just one time procedure while the regex keeps in the metadata cache.

I will think more about how to implement it properly..

[derekperkins]

Likely the solutions seems to be around let the Pod suggest multiple parsers and let Fluent Bit process that. It's not that good in terms of performance but something that needs to be solved in some way.

Can you just allow for annotations at the container level rather than at the pod level?

[edsiper]

@derekperkins would you please provide an example of a Pod with annotations at container level ?, I cannot find a reference.

[derekperkins]

@edsiper My bad, I thought they were supported. I think there's an opportunity to do the same thing at the pod annotation level. It's valid to nest json inside of an annotation:
kubernetes/kubernetes#12226 (comment)

What if the current implementation of a string stayed the same for the common use case, but also supported a json mapping?

    logging.parser: >
        { 
          "container1": "apache",
          "container2": "nginx",
          "container3": "mysql-slow"
          ...
        }

I'm not sure if it would be preferable to do it all with the same logging.parser key or whether to support a separate key like logging.parsers or logging.containerParsers, but that shouldn't impact the usability.

[derekperkins]

@edsiper any thoughts on that proposal?