GELF messages to Graylog server doesn't contain mandatory field - short_message #243

eladtamary · 2018-12-16T17:20:22Z

Hi,
We are using the daemonset to send logs to centralized Graylog server using the following image: fluent/fluentd-kubernetes-daemonset:v1.2-debian-graylog.

The logs are sent to Graylog and we are able to filter them properly.
However, we saw repetitive errors in Graylog server logs about missing mandatory field - short_message.
We understood that this is a mandatory field in GELF protocol that must be sent from any client.

How do I make the daemonset send this field in the GELF message.

Thanks,
Elad Tamary

shinebayar-g · 2019-04-02T17:35:46Z

Hi, I've been using fluent/fluentd-kubernetes-daemonset:v1.3-debian-graylog image and I believe I didn't get this issue. I'd suggest try updated image.

de1m · 2019-08-19T11:26:45Z

I've tested with both images fluent/fluentd-kubernetes-daemonset:v1.4-debian-graylog-1 and fluent/fluentd-kubernetes-daemonset:v1.4.2-debian-graylog-1.1. But I get the same error.
Ps. I use the graylog 3.1 from this docker file

myspotontheweb · 2019-08-20T11:22:43Z

This is issue is being continually closed as a docker error. I'm wondering if it's actually an issue with the handling of the GELF message as reported here (logging an empty line):

Graylog2/graylog2-server#4842

shinebayar-g · 2019-08-20T13:23:54Z

Update: I just noticed I'm getting this error on graylog server console as well. So is there any side effects besides this error messages?

2019-08-20 13:22:06,229 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message RawMessage{id=8164dd48-c34d-11e9-b7e7-0242ac11000e, journalOffset=720106792, codec=gelf, payloadSize=554, timestamp=2019-08-20T13:22:06.228Z, remoteAddress=/XX.XX.XX.XX:36090} on input <5d3ec6aa6b2f07000fb685da>.
2019-08-20 13:22:06,229 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=8164dd48-c34d-11e9-b7e7-0242ac11000e, journalOffset=720106792, codec=gelf, payloadSize=554, timestamp=2019-08-20T13:22:06.228Z, remoteAddress=/XX.XX.XX.XX:36090}
java.lang.IllegalArgumentException: GELF message <8164dd48-c34d-11e9-b7e7-0242ac11000e> (received from <XX.XX.XX.XX:36090>) has empty mandatory "short_message" field.
	at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:252) ~[graylog.jar:?]
	at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
	at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:45) [graylog.jar:?]
	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

chuegel · 2019-09-24T11:14:42Z

I have the same issue. Rancher -> fluentd -> Graylog GELF TCP input

Rancher 2.2.4
Graylog 3.1.

chuegel · 2019-10-08T06:17:49Z

Exporting logs from Rancher to Graylog via fluentd is not supported yet. See: rancher/rancher#23052

repeatedly · 2019-10-08T09:33:39Z

Exporting logs from Rancher to Graylog via fluentd is not supported yet. See:

This issue says "Rancher can't export data to Graylog directly". fluentd seems not related.

robermar23 · 2020-03-13T14:31:05Z

I am seeing the same error.
using image fluent/fluentd-kubernetes-daemonset:v1.7.4-debian-graylog-2.2

fluentd daemonset running on every node, using gelf, sending to graylog 3.2.2.

ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=8d22033b-6535-11ea-b2aa-0a580a8102bb, journalOffset=-9223372036854775808, codec=gelf, payloadSize=1168, timestamp=2020-03-13T14:18:46.371Z, remoteAddress=****}
  | java.lang.IllegalArgumentException: GELF message <8d22033b-6535-11ea-b2aa-0a580a8102bb> (received from ****) has empty mandatory "short_message" field.

HaveFun83 · 2020-03-19T17:15:47Z

same here

ismailyenigul · 2020-05-23T19:47:52Z

and I have the same problem on Graylog 3.3 with ES 6.8 using fluentd-daemonset-graylog-rbac.yaml

aylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020-05-23T19:40:31.771Z ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=8e0e95a2-9d26-11ea-b135-1a3420eca63d, journalOffset=119788096, codec=gelf, payloadSize=561, timestamp=2020-05-23T18:52:30.586Z, remoteAddress=/10.135.210.216:35821} on input <5ec5b96fabdddd32c54deee6>.
2020-05-23T19:40:31.771Z ERROR [DecodingProcessor] Error processing message RawMessage{id=8e0e95a2-9d26-11ea-b135-1a3420eca63d, journalOffset=119788096, codec=gelf, payloadSize=561, timestamp=2020-05-23T18:52:30.586Z, remoteAddress=/10.135.210.216:35821}
java.lang.IllegalArgumentException: GELF message <8e0e95a2-9d26-11ea-b135-1a3420eca63d> (received from <10.135.210.216:35821>) has empty mandatory "short_message" field.
        at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
@

If I restart graylog server, log flow starts again.

shizacat · 2020-08-04T08:58:19Z

1.11
I have the same problem on Graylog 3.3

graylog_1  | 2020-08-04 11:53:34,187 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=fa72b590-d62f-11ea-9db4-3ace9f95a535, journalOffset=1974010089, codec=gelf, payloadSize=545, timestamp=2020-08-04T08:53:34.185Z, remoteAddress=/172.19.103.133:38921}
graylog_1  | java.lang.IllegalArgumentException: GELF message <fa72b590-d62f-11ea-9db4-3ace9f95a535> (received from <172.19.103.133:38921>) has empty mandatory "short_message" field.
graylog_1  | 	at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:258) ~[graylog.jar:?]
graylog_1  | 	at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?]
graylog_1  | 	at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
graylog_1  | 	at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
graylog_1  | 	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
graylog_1  | 	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
graylog_1  | 	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
graylog_1  | 	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
graylog_1  | 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]

ediezh · 2020-12-14T07:24:57Z

I have the same problem despite I added the below filter in the fluentd config.

    <filter **>
      @type grep
      <exclude>
        key log
        pattern ^\n$
      </exclude>
    </filter>

nix-power · 2021-05-08T15:02:16Z

Hi,
We are using the daemonset to send logs to centralized Graylog server using the following image: fluent/fluentd-kubernetes-daemonset:v1.2-debian-graylog.

The logs are sent to Graylog and we are able to filter them properly.
However, we saw repetitive errors in Graylog server logs about missing mandatory field - short_message.
We understood that this is a mandatory field in GELF protocol that must be sent from any client.

How do I make the daemonset send this field in the GELF message.

Thanks,
Elad Tamary

I am working on a workaround to resolve it.

danielfm · 2021-12-03T14:02:12Z

Did anyone find any workaround for eliminating these errors?

zolech · 2022-01-05T14:13:31Z

@nix-power Did you find any workaround ?

vasrem mentioned this issue Aug 12, 2019

Improve empty short_message identification hotschedules/fluent-plugin-gelf-hs#6

Open

repeatedly added the help wanted We need your help! label Oct 8, 2019

cosmo0920 added the enhancement label May 8, 2021

sebastien-helbert mentioned this issue Oct 24, 2023

Stop throwing errors on blank log messages Graylog2/graylog2-server#17058

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GELF messages to Graylog server doesn't contain mandatory field - short_message #243

GELF messages to Graylog server doesn't contain mandatory field - short_message #243

eladtamary commented Dec 16, 2018

shinebayar-g commented Apr 2, 2019

de1m commented Aug 19, 2019

myspotontheweb commented Aug 20, 2019 •

edited

Loading

shinebayar-g commented Aug 20, 2019 •

edited

Loading

chuegel commented Sep 24, 2019 •

edited

Loading

chuegel commented Oct 8, 2019 •

edited

Loading

repeatedly commented Oct 8, 2019

robermar23 commented Mar 13, 2020 •

edited

Loading

HaveFun83 commented Mar 19, 2020

ismailyenigul commented May 23, 2020 •

edited

Loading

shizacat commented Aug 4, 2020

ediezh commented Dec 14, 2020

nix-power commented May 8, 2021

danielfm commented Dec 3, 2021

zolech commented Jan 5, 2022

GELF messages to Graylog server doesn't contain mandatory field - short_message #243

GELF messages to Graylog server doesn't contain mandatory field - short_message #243

Comments

eladtamary commented Dec 16, 2018

shinebayar-g commented Apr 2, 2019

de1m commented Aug 19, 2019

myspotontheweb commented Aug 20, 2019 • edited Loading

shinebayar-g commented Aug 20, 2019 • edited Loading

chuegel commented Sep 24, 2019 • edited Loading

chuegel commented Oct 8, 2019 • edited Loading

repeatedly commented Oct 8, 2019

robermar23 commented Mar 13, 2020 • edited Loading

HaveFun83 commented Mar 19, 2020

ismailyenigul commented May 23, 2020 • edited Loading

shizacat commented Aug 4, 2020

ediezh commented Dec 14, 2020

nix-power commented May 8, 2021

danielfm commented Dec 3, 2021

zolech commented Jan 5, 2022

myspotontheweb commented Aug 20, 2019 •

edited

Loading

shinebayar-g commented Aug 20, 2019 •

edited

Loading

chuegel commented Sep 24, 2019 •

edited

Loading

chuegel commented Oct 8, 2019 •

edited

Loading

robermar23 commented Mar 13, 2020 •

edited

Loading

ismailyenigul commented May 23, 2020 •

edited

Loading