ERROR [NettyContainer] Uncaught exception during jersey resource handling

[razvanphp]

After update from 1.1.1 to 1.1.2 we get lots of this:

2015-06-22T16:59:37.641+02:00 ERROR [NettyContainer] Uncaught exception during jersey resource handling
java.lang.NullPointerException
    at org.graylog2.jersey.container.netty.ChunkedRequestAssembler.addChunk(ChunkedRequestAssembler.java:53)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:275)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
    at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
    at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

server is up, slow, caches locally but output is 0, so it's not saving into ES.

Any idea why?

[joschi]

Please upgrade to Graylog 1.1.3 and verify that the problem still exists.

Are there any other exceptions before or after the NullPointerException you've posted?

[razvanphp]

I did upgrade to 1.1.3, but I was still receiving this error. To fix it I had to delete my journal folder and restart the ES cluster.

There were sometimes some errors that it could not connect to ES, but the status API directly on the cluster said the cluster was yellow, with no hot threads, so it should've saved messages there.

Also after 5 minutes of getting this error the web interface could not connect to the server anymore and I had to restart it every time.

[joschi]

@razvanphp Could you please post the complete log file of Graylog server when the error occurs?

[razvanphp]

This is mostly it, repeating more times:

2015-06-23T11:37:35.520+02:00 ERROR [Messages] Failed to index [1155] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[357]: index [graylog2_254], type [message], id [79aa9880-198b-11e5-9de5-00163e7a938a], message [IllegalArgumentException[Document contains at least one immense term in field="req_body" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[123, 92, 120, 50, 50, 110, 97, 109, 101, 92, 120, 50, 50, 58, 92, 120, 50, 50, 65, 114, 105, 122, 111, 110, 97, 32, 71, 101, 114, 97]...', original message: bytes can be at most 32766 in length; got 63104]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 63104]; ]
[493]: index [graylog2_254], type [message], id [79d8ad60-198b-11e5-9de5-00163e7a938a], message [IllegalArgumentException[Document contains at least one immense term in field="req_body" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[123, 92, 120, 50, 50, 110, 97, 109, 101, 92, 120, 50, 50, 58, 92, 120, 50, 50, 77, 65, 67, 32, 53, 45, 80, 111, 99, 107, 101, 116]...', original message: bytes can be at most 32766 in length; got 63806]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 63806]; ]
2015-06-23T11:37:35.916+02:00 ERROR [NettyContainer] Uncaught exception during jersey resource handling
java.lang.NullPointerException
        at org.graylog2.jersey.container.netty.ChunkedRequestAssembler.addChunk(ChunkedRequestAssembler.java:53)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:275)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2015-06-23T11:37:37.537+02:00 ERROR [Messages] Failed to index [1179] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[80]: index [graylog2_254], type [message], id [7a967f20-198b-11e5-9de5-00163e7a938a], message [IllegalArgumentException[Document contains at least one immense term in field="req_body" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[123, 92, 120, 50, 50, 110, 97, 109, 101, 92, 120, 50, 50, 58, 92, 120, 50, 50, 77, 65, 67, 32, 53, 45, 80, 111, 99, 107, 101, 116]...', original message: bytes can be at most 32766 in length; got 63806]; nested: MaxBytesLengthExceededException[bytes can be at most 32766 in length; got 63806]; ]

[dangmocrang]

I once turn of journal feature while have still have unprocessed message and then turn back on. Now I have same error as you. Maybe it relates to journal handling ?

[dangmocrang]

hmm.. I may know when I have this error. Anytime I switch to Streams page this error pop out and my streams page still showing the stream list normal but about 2 or 3 seconds it like... trigger a refresh, at that time I can not see number of msg throughput.
When I switch to another pages, this error stop.

[kroepke]

This is an elasticsearch error message.
Your log message contains a field that is a single long string, in this case it looks like a request body of some sort. It apparently contains no useful information and cannot be broken down into words by elasticsearch.
Thus it cannot be indexed either.
Unfortunately without inspecting the field ourselves (in the same way elasticsearch will do it) we cannot find out whether indexing will be successful or not.

For a workaround indexing could be turned off for that field, for example using an index template.
See http://stackoverflow.com/questions/24019868/utf8-encoding-is-longer-than-the-max-length-32766 for a related discussion.

[joschi]

I couldn't reproduce the issue (the NullPointerException in ChunkedRequestAssembler) and I think it's not related to the MaxBytesLengthExceededException that was mentioned in those logs (also see #873 for this particular issue).

Please reopen this issue if the problem still exists with Graylog 1.1.6 or 1.2.0.