DMLC: JMS resources are never cleared under certain conditions [SPR-10092]

[spring-projects-issues]

Raul Kripalani opened SPR-10092 and commented

JMS Resources (Sessions & MessageConsumers) associated with a AsyncMessageListenerInvoker are never closed and disposed if:

• you're using a defined maxMessagesPerTask, i.e. >= 0
• you first stop the DMLC and then destroy it

This is what happens:

• the call to stop() sets running = false
• invokers whose receiveTimeout elapses will notice this, but because AbstractJmsListeningContainer.active is still = true, they will pause (see logic in AbstractJmsListeningContainer#rescheduleTaskIfNecessary)
• when you call destroy(), the pausedTasks are never dealt with, so they remain holding the JMS resources forever.

To reproduce easily, set a very low receiveTimeout, e.g. 10 ms.

This is a dangerous bug. Not only does it provoke a memory leak at both the client and the broker, but the broker can keep delivering messages to the client because the underlying MessageConsumer is left open forever.

---

Affects: 3.0.7

Referenced from: commits cc11b81, 6b3284f

Backported to: 3.1.4

[spring-projects-issues]

Raul Kripalani commented

This is funny, I can't edit the ticket to populate the "Affects version" field?
Anyway, this is it: 3.0.7.RELEASE. Tested against Apache ActiveMQ 5.7.0.

[spring-projects-issues]

Juergen Hoeller commented

Fixed for 3.2 GA and 3.1.4, with DMLC clearing resources of paused tasks on shutdown if some happen to be left over.

Note that those resources would have been released by the garbage collector if you do not hold on to the DMLC instance after shutting it down. And in case of a shared connection, it would have been stopped - not delivering any messages - and closed on DMLC shutdown, which usually implies closing all of its sessions and consumers as well. So the actual negative effects should have been limited, I would argue. In any case, early cleanup is of course desirable, and there may be scenarios where it really matters, which is why we're glad to have heard about this problem and are fixing it in both of our active branches now.

Juergen

[spring-projects-issues]

Raul Kripalani commented

Thanks for the quick turnaround, Juergen!

You are right about closing the shared connection on DMLC shutdown. On deeper investigation, I found that AMQ's PooledConnectionFactory doesn't clean up the Sessions attached to a specific Connection checkout, which also adds up to this issue.

In my opinion, activemq-pool should track Sessions created from a each Connection checkout instance, and should explicitly close them when the Connection is returned by the user code (in this case, when DMLC shuts down), even if the physical connection is kept alive if it's in use by other areas or threads.

See https://issues.apache.org/jira/browse/AMQ-4225.