syslog input has no support for RFC 5424

[huntc]

I note that there is no direct support for RFC 5424 by fluentd, only RFC 3164, the latter being inconsistent across Unix variants. A summary of the differences between the two: https://tools.ietf.org/html/rfc5424#page-34

Given the popularity of Ubuntu and RHEL, and the RFC 5424 compliant rsyslog, I recommend supporting it. Perhaps a new format type of syslog-5424...

A work-around for RFC 5424 version 1 style producers of syslog:

<source>
  type syslog
  format syslog
  time_format 1 %Y-%m-%dT%H:%M:%S.%L%z
  port 5140
  protocol_type tcp
  tag system
</source>

Related:

• in_syslog cannot handle custom-formatted syslogs that do not start with <pri> #363
• http://stackoverflow.com/questions/24398756/ietf-syslog-input-to-fluentd
• https://groups.google.com/d/msg/fluentd/k0HU1Dkbazs/qVRipdLsIpwJ

[repeatedly]

Could you show me the summery of differences between 5424 nad 3164 or pointer to the article?

[huntc]

@repeatedly Description has been updated to refer to the summary of differences. HTH.

[repeatedly]

Thanks! I will check it later.

[huntc]

If I were to create a PR with changes in support of RFC 5424 and RFC 5425/6587, would there be interest? Also what is the release cadence typically i.e. when would we expect to see a new version including the changes given a PR to your liking?

[repeatedly]

I don't have a enough time to create RFC5424 parser.
So if you can create a PR, it would be awesome.

[huntc]

OK, I'll give it a go. I don't have time either - but I think that the world needs it - I sort of do right now! ;-)

[repeatedly]

Thanks!

[huntc]

@repeatedly #640 is a start on supporting modern syslog. Please review that and let me know if it is to your liking (it is a WIP, but quite complete as a PR).

[lwh]

Using tcp input instead of syslog, this should match them : /^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>.*?) (?<message>.+)$/

[mkorkalo]

Just a note. Structured Data part is also important with RFC5424 format. You can add key-value pairs of data to the message (i.e. [myformat@12345 sessionid="1234" clientip="10.10.10.10"]) which can be quite handly when filtering data. i.e. automatically creating dynamic clickable tags from these fields.
Too bad so many software is missing support for that.

[kawanet]

I've tested the following regexp for syslog packets sent by logger and nginx.

/^(?<time>\w+\s+\d+\s+[\d\:\.]+)\s+(?:(?<host>\S+)\s+)?(?<ident>\S+?)(?:\[(?<pid>\d+)\])?\:\s+(?<message>.*?)(\x00|$)/

[mkorkalo]

Btw, found this:
https://github.com/athenahealth/fluent-plugin-newsyslog

That also doesn't support RFC5424 structured data.

[gemyago]

Major issue for me with rfc3164 it it's timestamp that is always in local time. I have some sources in a different timezone and other servers have date in utc. Fluentd will recognize all cases as local time on the server it is running on, which will obviously be wrong. The rfc5424 has support of timezone handling so would be good to have it supported natively by fluentd.

[cosmo0920]

I investigated this issue shortly, I've found that a regex for rfc5424 based on @lwh 's:

/\A^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$\z/ is suitable to parse a record which has extradata field.