Handle exception in NIO loop to avoid abrupt termination

[hailin84]

rabbitmq-nio thread terminated unexpectedly

I am using rabbitmq in my java system. When I try to use nio, I got rabbitmq-nio thread terminated unexpectedly. I review the code, there must be two problems(NioLoop.run):

• NIO epoll bug, not handled；
• exception should be caught in the main while loop, that's why rabbitmq-nio thread terminated unexpectedly. SocketChannel to rabbitmq server may be closed, but nio selector thread should not exit.

Here is my exception stacktrace log:

2019-05-15 14:37:28.379 ERROR [rabbitmq-nio] nio.NioLoop(NioLoop.run:254) - Error in NIO loop
java.nio.channels.ClosedChannelException
    at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:197) ~[?:1.8.0_181]
    at com.rabbitmq.client.impl.nio.NioLoop.run(NioLoop.java:118) ~[amqp-client-5.5.1.jar:5.5.1]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

Thanks for your help.

[acogoluegnes]

Thanks for reporting this. Could you please upgrade to the latest release (5.7.0), provide steps to reproduce, and logs? Thanks!

[acogoluegnes]

OK, found it, there's no error handling here, so an exception can indeed make the whole NIO thread stops:

registration.state.getChannel().register(selector, operations, registration.state);

This is actually already handled in the write part:

try {
    if (writeRegistration.state.getChannel().isOpen()) {
        writeRegistration.state.getChannel().register(writeSelector, operations, writeRegistration.state);
        writeRegistered = true;
    }
} catch (Exception e) {
    // can happen if the channel has been closed since the operation has been enqueued
    LOGGER.info("Error while registering socket channel for write: {}", e.getMessage());
}

We just have to do the same. This fixes the second item you mentioned, what exactly do you mean by NIO epoll bug, not handled in the first item?

[hailin84]

Thanks, man, good job. @acogoluegnes
Here is some thing about NIO epoll bug:
bug id 640393
bug id 2147719
Hope it's helpful to you.
Thanks again.

[hailin84]

Netty framework has a solution for nio epoll bug. When it detects that bug happen, a new selector will be created. All channels registered to the old selector will register to the new one.

[acogoluegnes]

I pushed a fix. The NIO epoll bug is marked to have been solved more than a decade ago in the JDK, so my advice would be to upgrade to a recent JVM to avoid it.

[hailin84]

I'm using JDK 1.8.0_181, I didn't encounter NIO epoll bug, and yes, It was marked as fixed, but it's not actually. I found a few cases under jdk1.8 through google. Doesn't matter, I think I am lucky enough.
Thanks.

[acogoluegnes]

@hailin84 There's a 5.7.1 snapshot available with the fix: https://oss.sonatype.org/content/repositories/snapshots/com/rabbitmq/amqp-client/5.7.1-SNAPSHOT/

Can you give it a try?