SpringIterableConfigurationPropertySource fails when underlying source throws ConcurrentModificationException

[ddcruver]

We have not been able to reproduce it reliably but we have notice instances of Spring Boot not initializing correctly and each time throwing a java.Util.ConcurrentModificationException exception when trying to bind properties for the Spring Application.

This seems to happen in a AWS instance were the I/O and CPU is more restrained and does not happen as much when we deploy to larger instances. There is no documented case of this happening in our local development environments.

Caused by: java.util.ConcurrentModificationException
    at java.util.Hashtable$Enumerator.next(Hashtable.java:1387) [rt.jar:1.8.0_212]
    at java.util.AbstractCollection.addAll(AbstractCollection.java:343) [rt.jar:1.8.0_212]
    at java.util.HashSet.<init>(HashSet.java:120) [rt.jar:1.8.0_212]
    at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource$CacheKey.copyKey(SpringIterableConfigurationPropertySource.java:189)
    at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource$CacheKey.copy(SpringIterableConfigurationPropertySource.java:184)
    at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.getCache(SpringIterableConfigurationPropertySource.java:142)
    at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.getConfigurationPropertyNames(SpringIterableConfigurationPropertySource.java:97)
    at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.iterator(SpringIterableConfigurationPropertySource.java:87)
    at org.springframework.boot.context.properties.source.ConfigurationPropertyState.search(ConfigurationPropertyState.java:62)
    at org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.containsDescendantOf(SpringIterableConfigurationPropertySource.java:93)
    at org.springframework.boot.context.properties.bind.Binder.containsNoDescendantOf(Binder.java:367)
    at org.springframework.boot.context.properties.bind.Binder.bindObject(Binder.java:256)
    at org.springframework.boot.context.properties.bind.Binder.bind(Binder.java:220)
    ... 65 more

I will attach more complete stack traces to it being called from different points in the initialization of our application.

concurrent-modification-instance-1.txt
concurrent-modification-instance-2.txt

[philwebb]

There appears to be a few issues with the way we're storing the cache and cache key which should be fixed, but I'm not sure those updates will totally solve your issue. Although I can protect against a ConcurrentModificationException when copying the key, it's likely to just pop up again in getPropertyMappings when we call getPropertySource().getPropertyNames().

The real problem appears to be that multiple threads are trying to access a property source that isn't thread safe. With most Spring Boot applications, the contents of the property sources aren't modified once they're created so I'm curious if there's something specific to your environment? Do you have any custom property sources? Or do you change property sources on the fly?

If you're able to create a sample project that replicates the issue, or share privately your existing project we can dig a little more.

[philwebb]

I've raise #17017 for the obvious concurrency issues. We'll leave this one to track the root cause of ConcurrentModificationException.

[dewthefifth]

There appears to be a few issues with the way we're storing the cache and cache key which should be fixed, but I'm not sure those updates will totally solve your issue. Although I can protect against a ConcurrentModificationException when copying the key, it's likely to just pop up again in getPropertyMappings when we call getPropertySource().getPropertyNames().

The real problem appears to be that multiple threads are trying to access a property source that isn't thread safe. With most Spring Boot applications, the contents of the property sources aren't modified once they're created so I'm curious if there's something specific to your environment? Do you have any custom property sources? Or do you change property sources on the fly?

If you're able to create a sample project that replicates the issue, or share privately your existing project we can dig a little more.

I work with ddcruver, and can provide a little bit more detail about the deployment environment. Our application is being deployed along side many other applications on a shared wildfly 12 application server (as WARs). My suspicion is that the SystemProperties are being modified by one of the other applications during startup, and as a result of that we're getting the concurrent modification exception. I suspect that the problem lies in how System.getProperties values are accessed and not in other property sources (necessarily)