Deadlock

[adamretter]

When running XQTS (exist-xqts-runner) against eXist-db 6.1.0-SNAPSHOT we can encounter a deadlock in eXist-db.

Running jstack reveals:

Found one Java-level deadlock:
========================
"io-compute-42":
  waiting to lock monitor 0x00007f85207a0e08 (object 0x00000007b2918688, a org.exist.storage.BrokerPool),
  which is held by "io-compute-32"
"io-compute-32":
  waiting for ownable synchronizer 0x00000007b0839710, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "global.BrokerPools.ShutdownHook"
"global.BrokerPools.ShutdownHook":
  waiting to lock monitor 0x00007f85207a0e08 (object 0x00000007b2918688, a org.exist.storage.BrokerPool),
  which is held by "io-compute-32"

Java stack information for the threads listed above:
=======================================
"io-compute-42":
	at org.exist.storage.BrokerPool.get(BrokerPool.java:1258)
	- waiting to lock <0x00000007b2918688> (a org.exist.storage.BrokerPool)
	at org.exist.storage.BrokerPool.getBroker(BrokerPool.java:1200)
	at org.exist.xqts.runner.ExistServer.$anonfun$getConnection$1(ExistServer.scala:112)
	at org.exist.xqts.runner.ExistServer$$Lambda$5145/2103542716.apply(Unknown Source)
	at cats.effect.unsafe.WorkerThread.blockOn(WorkerThread.scala:624)
	at scala.concurrent.package$.blocking(package.scala:124)
	at cats.effect.IOFiber.runLoop(IOFiber.scala:967)
	at cats.effect.IOFiber.execR(IOFiber.scala:1332)
	at cats.effect.IOFiber.run(IOFiber.scala:139)
	at cats.effect.unsafe.WorkerThread.run(WorkerThread.scala:425)
"io-compute-32":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007b0839710> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
	at org.exist.storage.lock.ManagedLock.acquire(ManagedLock.java:129)
	at org.exist.storage.BrokerPools.getInstance(BrokerPools.java:242)
	at org.exist.storage.BrokerPools.getInstance(BrokerPools.java:229)
	at org.exist.security.Session.<init>(Session.java:48)
	at org.exist.security.AbstractSubject.<init>(AbstractSubject.java:42)
	at org.exist.security.internal.SubjectAccreditedImpl.<init>(SubjectAccreditedImpl.java:41)
	at org.exist.security.internal.SecurityManagerImpl.lambda$1(SecurityManagerImpl.java:143)
	at org.exist.security.internal.SecurityManagerImpl$$Lambda$4781/341827876.get(Unknown Source)
	at com.evolvedbinary.j8fu.lazy.AtomicLazyVal.lambda$get$0(AtomicLazyVal.java:61)
	at com.evolvedbinary.j8fu.lazy.AtomicLazyVal$$Lambda$4812/1170159101.apply(Unknown Source)
	at java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:179)
	at com.evolvedbinary.j8fu.lazy.AtomicLazyVal.get(AtomicLazyVal.java:61)
	at org.exist.security.internal.SecurityManagerImpl.getGuestSubject(SecurityManagerImpl.java:443)
	at org.exist.storage.BrokerPool$$Lambda$4847/304425307.get(Unknown Source)
	at java.util.Optional.orElseGet(Optional.java:267)
	at org.exist.storage.BrokerPool.get(BrokerPool.java:1291)
	- locked <0x00000007b2918688> (a org.exist.storage.BrokerPool)
	at org.exist.storage.BrokerPool.getBroker(BrokerPool.java:1200)
	at org.exist.xqts.runner.ExistServer.$anonfun$getConnection$1(ExistServer.scala:112)
	at org.exist.xqts.runner.ExistServer$$Lambda$5145/2103542716.apply(Unknown Source)
	at cats.effect.unsafe.WorkerThread.blockOn(WorkerThread.scala:624)
	at scala.concurrent.package$.blocking(package.scala:124)
	at cats.effect.IOFiber.runLoop(IOFiber.scala:967)
	at cats.effect.IOFiber.execR(IOFiber.scala:1332)
	at cats.effect.IOFiber.run(IOFiber.scala:139)
	at cats.effect.unsafe.WorkerThread.run(WorkerThread.scala:523)
"global.BrokerPools.ShutdownHook":
	at org.exist.storage.BrokerPool.shutdown(BrokerPool.java:1636)
	- waiting to lock <0x00000007b2918688> (a org.exist.storage.BrokerPool)
	at org.exist.storage.BrokerPools.stopAll(BrokerPools.java:285)
	at org.exist.storage.BrokerPools.lambda$0(BrokerPools.java:71)
	at org.exist.storage.BrokerPools$$Lambda$4740/449969267.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

Can be reproduced by running sbt "run --xqts-version HEAD --test-set fn-round" from the exist-xqts-runner.