fix: resolve database deadlock:

[seelabs]

The rotateWithLock function holds a lock while it calls a callback function that's passed in by the caller. This is a problematic design that needs to be used very carefully. In this case, at least one caller passed in a callback that eventually relocks the mutex on the same thread, causing UB (a deadlock was observed). The caller was from SHAMapStoreImpl, and it called clearCaches. This clearCaches can potentially call fetchNodeObject, which tried to relock the mutex.

This patch resolves the issue by changing the mutex type to a recursive_mutex. Ideally, the code should be rewritten so it doesn't hold the mutex during the callback and the mutex should be changed back to a regular mutex.

Type of Change

• [x ] Bug fix (non-breaking change which fixes an issue)

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.0%. Comparing base (24a275b) to head (e6e26d4).
Report is 1 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff           @@
##           develop   #4989   +/-   ##
=======================================
  Coverage     70.9%   71.0%           
=======================================
  Files          796     796           
  Lines        66727   66727           
  Branches     10981   10973    -8     
=======================================
+ Hits         47333   47345   +12     
+ Misses       19394   19382   -12     

Files	Coverage Δ
src/ripple/nodestore/impl/DatabaseRotatingImp.h	50.0% <ø> (ø)

... and 3 files with indirect coverage changes

[scottschurr]

I wish that the lock handling did not require a recursive_mutex. But we are where we are, and this is a pragmatic solution. 👍

[HowardHinnant]

I do not see a low-risk alternative at this time.