in_syslog support for RFC5424 Octet Counting / MSG-LEN header for tcp connections

[guedressel]

While testing the in_syslog plugin to receive syslog messages generated by syslog-ng via a TCP connection I stuck with "that stupid integer at beginning of each message".

Example log message

119 <139>1 2017-08-30T14:09:39+00:00 my-host app-name 34 - [meta sequenceId=\"1\"] [autoindex:error] Some logging message...

After some reading I discovered that the leading integer is described here: https://tools.ietf.org/id/draft-gerhards-syslog-plain-tcp-12.html#rfc.section.3.4.1

What do you think about adding this (optional?) detail to the syslog parser?

[guedressel]

Does this Octet-Counting information relate to multi-line messages?

[guedressel]

There's another discussion about that integer: Graylog2/graylog2-server#159

[repeatedly]

Is this format supported by only syslog-ng?

[guedressel]

Not sure which systems support that format. We are using syslog-ng exclusively in our infrastructure.
But since it is also part of the older RFC3164 I suppose there are more systems out there using it...

It's main benefit might be to have multiple messages being sent over a single TCP connection without relying on "newlines" as message delimiter; hence allowing multi-line messages to be sent as stream.

[repeatedly]

Yeah, using Octet Counting seems better than \n delimited message frame, so supporting it is no problem for me.

Patches are welcome :)

[salsa-dev]

@guedressel how did you solve this issue?

I resolved this issue for myself by using network driver in syslog-ng:
destination d_fluentd { network("10.1.1.7" port(5140) flags(syslog-protocol));};
and using message_format rfc5424 option to configure fluentd.

P.S.: for anyone bumping this issue

[sepich]

We've investigated switching from rsyslog to fluentd,
and lack of support of messages with newlines for in_syslog is a showstopper for us.
Whole this famous java multiscreen stacktrace strips to just first line, when sent via syslog to fluentd.
Please note that "ancient" rsyslog even had $EscapeControlCharactersOnReceive off and sending multiline messages via syslog is correctly supported by "new" apps like graylog and splunk without any special configuring.
Please prioritise this, as for now in_syslog is sorta broken.
Also related: #1270

[gmile]

I'm working on draining logs from heroku to fluentd. Just discovered that Heroku frames all log messages according to syslog TCP protocol octet counting framing method.

Examples (see "HTTPS Drains"):

83 <40>1 2012-11-30T06:45:29+00:00 host app web.3 - State changed from starting to up
119 <40>1 2012-11-30T06:45:26+00:00 host app web.3 - Starting process with command `bundle exec rackup config.ru -p 24405`

Would love to see support for this from fluentd.

[repeatedly]

Patch is here: #2147