Heap out of memory error if consumer is slow and auto ack is used

[nukemberg]

Because WorkPool uses an unbounded LinkedList, when a consumer is not fast enough the list will be filled by the connection thread until OOM error. This is aggravated by GC pressure making consumers even slower.

I suggest using a bounded blocking queue instead.

[michaelklishin]

The right thing to do is to use basic.qos. Then the number of deliveries will be limited. See tutorial 2: http://www.rabbitmq.com/getstarted.html.

[michaelklishin]

To be more specific: if we use a bounded queue, it will result in deliveries being dropped (effectively). This is both dangerous and not how things have been working for 7 years, so we cannot afford to do this change, nor we consider it to be a solution.

Use manual acknowledgements if your consumer can't keep up with the delivery rate.

[nukemberg]

a. OOM happens even if basic.qos is used.
b. you risk loosing message even with an unbounded list. you will loose messages when GC pressure starts and certainly when OOM happens.
By using an unbounded list you are merely postponing the inevitable while risking application stability even when the programmer is willing to lose messages.

Also, perhaps tcp backpressure can be used to make Rabbit stop sending messages, if the os buffer feels up tcp should signal that the window is full afaik. not sure how rabbit handles this though.

[michaelklishin]

I don't see how the internal queue can grow out of bounds if basic.qos is used. Set prefetch to N and you will have no more than N messages being processed at a time, per channel.

I also don't see how GC pressure can lead to the linked queue items being lost.

By using an unbounded list you are merely postponing the inevitable while risking application stability even when the programmer is willing to lose messages

The thing is, most users aren't willing to do this. As for "the inevitable", see above. When basic.qos is used, the linked deliveries queue will not grow indefinitely.

[michaelklishin]

perhaps tcp backpressure can be used to make Rabbit stop sending messages, if the os buffer feels up tcp should signal that the window is full afaik

But it will not be full with automatic acknowledgements. And with manual/basic.qos, RabbitMQ's internal limiter will prevent queues from sending messages to consumers, so again, they will not be full.

I'm sorry, you seem to be suggesting things at random. Use manual acknowledgements with a (relatively) low prefetch value. Plan your consumer capacity accordingly. Read up on TCP backpressure applied to publishers (to prevent messages piling up forever in the Rabbit itself). See what features can prevent unbounded buffers.

At the end of the day, you have to either consume fast enough, or drop data, but dropping data silently in the client is about the worst possible solution.

[michaelklishin]

I now have a sample app that is supposed to reproduce the problem. We'll investigate. One issue that may be causing the observed behaviour is #20.

[michaelklishin]

Here's the sample app.

When running a consumer in manual acknowledgement mode, I see a very stable heap size:

• Consumer
• Producer
• queue stats

Note that the consumer is slow and will ack a batch of ~ 500 messages roughly every 25 seconds.

With automatic acknowledgements, consumer heap size keeps growing and memory profiler suggests most of the data is just raw data and parsed content. This is expected: the consumer
only handles 20 deliveries per second, but RabbitMQ keeps delivering new messages to it,
which are being read and parsed by the I/O loop, and then wait for being dispatched.

All of this is expected.

[michaelklishin]

We may have figured out a way to ensure TCP back pressure in the automatic acknowledgements case. If it works well, it will be in 3.4.0.

[simonmacmullen]

avishai-ish-shalom: yes, a bounded blocking queue is the right idea.

"Also, perhaps tcp backpressure can be used to make Rabbit stop sending messages, if the os buffer feels up tcp should signal that the window is full afaik. not sure how rabbit handles this though." --> Yes, TCP backpressure will cause the RabbitMQ server to stop sending messages.

[michaelklishin]

@avishai-ish-shalom resolved in 0e1806f. Works really well in practice: with a consumer consuming 1 msg/s and a fast publisher (thousands of msg/s) we've been able to run a sample test for 20 minutes with very stable used heap size. With a fast consumer, we now can publish over 15M messages with used heap size staying < 35 MB the whole time.

Thank you for pushing us to find a solution. It makes more difference than I expected it would. The change will be in 3.4.0.

[nukemberg]

awesome. i'll stay tuned for the release.

On Tue, Sep 23, 2014 at 5:23 PM, Michael Klishin notifications@github.com
wrote:

@avishai-ish-shalom https://github.com/avishai-ish-shalom resolved in
0e1806f
0e1806f.
Works really well in practice: with a consumer consuming 1 msg/s and a fast
publisher (thousands of msg/s) we've been able to run a sample test for 20
minutes with very stable used heap size. With a fast consumer, we now can
publish over 15M messages with used heap size staying < 35 MB the whole
time.

Thank you for pushing us to find a solution. It makes more difference than
I expected it would. The change will be in 3.4.0.

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

[michaelklishin]

Unfortunately, we've discovered an issue with this: when MainLoop runs into TCP back pressure, it can't read responses for synchronous protocol methods (those that expect a response), e.g. basic.cancel. There are some options but they are all opinionated and not entirely straightforward one way or another.