Improve reconnect attemps option in DefaultMessageListenerContainer [SPR-11746]

[spring-projects-issues]

Karthik Prasad opened SPR-11746 and commented

DefaultMessageListenerContainer doesn't have number of reconnect attempts( after which listener ceases and shutdown ) and reconnect interval multipler( multiplier to reconnect interval) which both are found in connection factory of HornetQ. Due to this when hornetq server is not started the reconnect attempt goes for infite time, resulting the log file bombarded with error logs for failed connection.

---

Affects: 3.2.3

Reference URL: http://stackoverflow.com/questions/23270113/hornetq-reconnect-attempts-not-working-with-defaultmessagelistenercontainer/23271326#23271326

[spring-projects-issues]

Karthik Prasad commented

Tried to Extend the DefaultMessageListenerContainer and used the invokerExceptionListener

@Override
protected void invokeExceptionListener(JMSException ex) {

	ExceptionListener exceptionListener = getExceptionListener();
	if (exceptionListener != null) {
		exceptionListener.onException(ex);
	}

	retryCounter++;

	if (retryCounter < reconnectAttempts) {
		LOG.warn(
						"Will Try to reconnect "
								+ (reconnectAttempts - retryCounter)
								+ " times/time after which reconnect attempt will be left and listener container will be shutdown");
	}
	if (retryIntervalMultiplier == 0) {
		retryInterval = retryInterval * 1;
	} else {
		final Double interval = retryInterval * retryIntervalMultiplier;
		retryInterval = interval.longValue();
	}
	this.setRecoveryInterval(retryInterval);
	if (reconnectAttempts != -1 && retryCounter >= reconnectAttempts) {
		LOG.error("Reconnect Attempts of "
						+ reconnectAttempts

						+ " reached, Now shutting down listener container");
	    
		this.destroy();
	}
}

But the thread does stop. Getting message Still waiting for shutdown of 1 message listener invokers. Since invoker parameter is private, there is no option to stop invokers

[spring-projects-issues]

Stéphane Nicoll commented

Can you explain what you are looking at exactly? I see two requirements here:

1. Smarter wait interval (time that increase over time)
2. Maximal number of attemps

Ideally this should be exposed so that a subclass can easily tune this even further.

[spring-projects-issues]

Karthik Prasad commented

Yes I'm looking for those two requirement. Currently connection factory has these features but DefaultMessageListenerContainer does not have these features. As an workaround till the fix is done, I tried extended DefaultMessageListenerContainer and two properties to increase wait interval and maximum number of attempts and using overridden invokeExceptionListener, I implemented the above requirement. When the attempts reaches max, I shutdown the listener container, However i was not able to successfully shutdown the container as still an invoker is alive. Since AsyncMessageListenerInvoker is a inner class with private access specifier. There is no means to stop the invoker by extending the DefaultMessageListenerContainer class and add our own feature.

[spring-projects-issues]

Stéphane Nicoll commented

looking at implementing a BackOff with a FixedBackOff and an ExponentialBackOff and replacing the simple interval by an instance of the policy.

[spring-projects-issues]

Stéphane Nicoll commented

The latest 4.1.0.BUILD-SNAPSHOT has a new BackOff facility with two implementations:

• FixedBackOff
• ExponentialBackOff

You can create one of these or implement your own and configure it to your DMLC (see setBackOff on DefaultMesageListenerContainer).

And I just realized that I forgot to update the xml namespace. Updating this now.

[spring-projects-issues]

Stéphane Nicoll commented

Tested a few scenario and this seems to work just fine. If you could try a SNAPSHOT and report here if that works for you that would be awesome. Thanks!

[spring-projects-issues]

Karthik Prasad commented

I could not understand the difference between maxElapsedTime and maxInterval, could you please explain. And also I do not see a log which says container is being shutdown. Since I use logback to parse the log to find errors based on text to raise notification a warn or error log intimating the shutdown of container will be of great help.

This configuration did no work for me

<bean id="backOffPlan" class="org.springframework.util.ExponentialBackOff">
<property name="initialInterval" value="2000" />
<property name="maxElapsedTime" value="40000"/>
<property name="maxInterval" value="40000" />
<property name="multiplier" value="1.5" ></property>
</bean>

It stopped after 15ms

014-05-13 16:51:42,363 [listener-2-1] ERROR o.s.j.l.DefaultMessageListenerContainer - Could not refresh JMS Connection for destination 'kpInBoundQueue' - retrying using ExponentialBackOff{currentInterval=15187ms, multiplier=1.5}. Cause: Failed to create session factory; nested exception is HornetQException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.]
javax.jms.JMSException: Failed to create session factory
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:587) ~[hornetq-jms-client-2.3.0.Final.jar:na]
at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:107) ~[hornetq-jms-client-2.3.0.Final.jar:na]
at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:102) ~[hornetq-jms-client-2.3.0.Final.jar:na]
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:185) ~[spring-jms-4.1.0.BUILD-SNAPSHOT.jar:4.1.0.BUILD-SNAPSHOT]
at org.springframework.jms.listener.AbstractJmsListeningContainer.createSharedConnection(AbstractJmsListeningContainer.java:413) ~[spring-jms-4.1.0.BUILD-SNAPSHOT.jar:4.1.0.BUILD-SNAPSHOT]
at org.springframework.jms.listener.AbstractJmsListeningContainer.refreshSharedConnection(AbstractJmsListeningContainer.java:398) ~[spring-jms-4.1.0.BUILD-SNAPSHOT.jar:4.1.0.BUILD-SNAPSHOT]
at org.springframework.jms.listener.DefaultMessageListenerContainer.refreshConnectionUntilSuccessful(DefaultMessageListenerContainer.java:905) [spring-jms-4.1.0.BUILD-SNAPSHOT.jar:4.1.0.BUILD-SNAPSHOT]
at org.springframework.jms.listener.DefaultMessageListenerContainer.recoverAfterListenerSetupFailure(DefaultMessageListenerContainer.java:880) [spring-jms-4.1.0.BUILD-SNAPSHOT.jar:4.1.0.BUILD-SNAPSHOT]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1041) [spring-jms-4.1.0.BUILD-SNAPSHOT.jar:4.1.0.BUILD-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
Caused by: org.hornetq.api.core.HornetQNotConnectedException: HQ119007: Cannot connect to server(s). Tried with all available servers.
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:852) ~[hornetq-core-client-2.3.0.Final.jar:na]
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:583) ~[hornetq-jms-client-2.3.0.Final.jar:na]
... 9 common frames omitted

[spring-projects-issues]

Stéphane Nicoll commented

You get a point with the log statement. It has been added and will be available in the next snapshot.

Could you please read the Javadoc of ExponentialBackOff and report back here if that's still not clear? Thanks.