Implement network failure detection

[carlhoerberg]

If i let a bunny connection sitting idle in an irb session it will eventually start throwing this endlessly:

AMQ::Protocol::EmptyResponseError
Empty response received from the server.
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/amq-protocol-1.0.1/lib/amq/protocol/frame.rb:45:in `decode_header'
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/bunny-0.9.0.pre3/lib/bunny/transport.rb:110:in `read_next_frame'
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/bunny-0.9.0.pre3/lib/bunny/main_loop.rb:26:in `block in run_loop'
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/bunny-0.9.0.pre3/lib/bunny/main_loop.rb:22:in `loop'
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/bunny-0.9.0.pre3/lib/bunny/main_loop.rb:22:in `run_loop'

Server is RabbitMQ 3.0.0, so maybe something with the heartbeat/connection timeout?

[michaelklishin]

Carl,

Your understanding is correct: this suggests a TCP connection failure which Bunny 0.9 does not detect yet. This is the next big thing I am going to work on.

[michaelklishin]

Using heartbeats should help. Can you be using load balancers/proxies by any chance? Those may be closing connections they consider idle. Bunny 0.9 should set SO_KEEPALIVE on the socket it uses.

[carlhoerberg]

do you want me to test with a LB or are you asking if i used one when i got this error? yes and no..

[celldee]

Some observations on a long-running Bunny/RabbitMQ connection in irb that you may be interested in -

1. Bunny 0.9 does not seem to set keepalive on a socket by default, however, you can explicitly set it with Bunny.new(:keepalive => true)
2. RabbitMQ 3.0.0 has a default heartbeat interval of 600 secs. After using Wireshark, the server did indeed send heartbeats to the client and kills the connection some time after 2 heartbeats have been sent without reply. This results in the loop described above.
3. Bunny::Session#negotiate_value method doesn't allow the heartbeat interval to be set to 0 if the server setting is > 0. Looking at the AMQP 0.9.1 spec (http://www.rabbitmq.com/amqp-0-9-1-reference.html), I think that Bunny should allow the heartbeat interval to be set to 0 regardless of the server setting.

[carlhoerberg]

Similar thing happens when a connection is forcefully closed:

NoMethodError
undefined method `read_fully' for nil:NilClass
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/bunny-0.9.0.pre3/lib/bunny/transport.rb:109:in `read_next_frame'
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/bunny-0.9.0.pre3/lib/bunny/main_loop.rb:26:in `block in run_loop'
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/bunny-0.9.0.pre3/lib/bunny/main_loop.rb:22:in `loop'
/Users/carl/.rvm/gems/ruby-1.9.3-p194/gems/bunny-0.9.0.pre3/lib/bunny/main_loop.rb:22:in `run_loop'

[michaelklishin]

I agree with that we probably should allow setting heartbeats to 0. I will look into that tomorrow. And we should make :keepalive => true the default, it makes much more sense that way.

[michaelklishin]

Done, 0.9.0.pre6 includes this feature.