QueueingConsumer.nextDelivery can hang forever, even if the underlying connection dies

[swenson]

When calling nextDelivery to grab the next message from QueueingConsumer, it is calling the blocking method on the queue, take(), which will block forever.

https://github.com/rabbitmq/rabbitmq-java-client/blob/master/src/com/rabbitmq/client/QueueingConsumer.java#L215

The problem occurs then if the underlying channel dies, then the thread that called nextDelivery will now be waiting forever.

The solution we had to in our code that uses this was to use the timeout version, and manually check if the channel had closed while we were waiting every time.

Possible solution: make both versions of nextDelivery use a timeout (and spin in the case of the no-timeout version), check if the channel died, and throw a ShutdownSignalException in that case.

[buchireddy]

How can i submit a patch for this issue? I'm interested in contributing to this library.

[hyperthunk]

Hi @swenson / @buchireddy - I'm not at all convinced this is true. If/when a channel dies, the handleShutdownSignal callback will be invoked on a separate thread pool thread, and that code adds a poison pill to the blocking queue, thereby preventing the permanent blocking behaviour in client/application threads calling handleNext which you describe. See https://github.com/rabbitmq/rabbitmq-java-client/blob/master/src/com/rabbitmq/client/QueueingConsumer.java#L110.

Can you produce an isolated test case demonstrating that this mechanism doesn't work as advertised?

[swenson]

@hyperthunk I will see if I can get a reproducer.

[hyperthunk]

Ok thanks @swenson - if there is an issue here, we'd like to track it down and fix it.

[swenson]

@hyperthunk I was unable to come up with a reproducer. The logic around the poison pill looks pretty solid. I'm not sure what is happening in our code then.

Thanks. Closing.

[irufus]

I think we may have found a way to reproduce this issue in a multi-threaded process in our environment. It seems to happen on high utilization (94% CPU 24% memory) Which has us investigating if the object may have been touched by the garbage collector. I will update this post as we find out more information.