Avoid full singleton lock for DefaultSingletonBeanRegistry.getSingleton(beanName, false)

[jhoeller]

Following up on the latest comments in #13117, DefaultSingletonBeanRegistry.getSingleton(beanName, false) seems to have a specific issue when called from isSingleton and isTypeMatch checks in AbstractBeanFactory: When isSingletonCurrentlyInCreation returns true, we're trying to enter the global lock for an early singleton exposure check... even if we're not allowed to create early references, so only able to check for an existing early reference. By turning earlySingletonObjects into a ConcurrentHashMap and separating that early reference existence check from the creation of early references (which needs to happen within the global lock), we could potentially address a whole range of issues for lookup attempts in concurrent singleton creation scenarios.

[CODESTHN]

Thanks @jhoeller for opening this ticket. This is an application startup issue. Please find below thread dump :-

Thread 1:-

   java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
- waiting to lock <0x000008001202c048> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.isSingleton(AbstractBeanFactory.java:399)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:431)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:395)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:515)
at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1197)
at org.springframework.amqp.rabbit.core.RabbitAdmin.initialize(RabbitAdmin.java:457)
at org.springframework.amqp.rabbit.core.RabbitAdmin$11.onCreate(RabbitAdmin.java:419)
at org.springframework.amqp.rabbit.connection.CompositeConnectionListener.onCreate(CompositeConnectionListener.java:33)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:553)
- locked <0x000009fff2a3a620> (a java.lang.Object)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:90)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:140)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:76)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:496)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1331)
at java.lang.Thread.run(Thread.java:748)

Thread 2:-

 java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:544)
- waiting to lock <0x000009fff2a3a620> (a java.lang.Object)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1431)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1412)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1388)
at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueue(RabbitAdmin.java:207)
<some call stack lines removed from here for privacy>
- locked <0x000008001202c048> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

I am trying to establish connection to rabbit in the @PostConstruct method something like below:-

BeanA{

@PostConstruct
public void init(){
  <call to CachingConnectionFactory.createConnection is made using some code)>
}
}

BeanB{

@PostConstruct
public void init(){
  <call to CachingConnectionFactory.createConnection is made using some code)>
}
}


stackoverflow -- https://stackoverflow.com/questions/63670363/deadlock-while-creating-bean-in-defaultsingletonbeanregistry-and-cachingconnecti

[jhoeller]

This will be available in the upcoming 5.2.9.BUILD-SNAPSHOT from https://repo.spring.io/snapshot. Please give it an early try and let me know whether it makes a difference for you...

[CODESTHN]

Thanks for fixing this issue quickly, I will surely let you know the results once the snapshot version is released.

[snicoll]

Could you please try the snapshot version? If there is an issue with that, that gives us some time to review the fix based on your feedback.

[jhoeller]

To be clear, the latest snapshot build has the change already: https://repo.spring.io/snapshot/org/springframework/spring-framework-bom/5.2.9.BUILD-SNAPSHOT/spring-framework-bom-5.2.9.BUILD-20200831.160538-32.pom

[CODESTHN]

Sure, I will update you guys with results. This issue came in prod environment therefore need some time to replicate this into local development environment, please expect few days delay in this. Thanks for fix 👍