Backport 1.5: fix deadlock on core state lock (#10456)

[ncabatoff]

• fix race that can cause deadlock on core state lock

The bug is in the grabLockOrStop function. For specific concurrent
executions the grabLockOrStop function can return stopped=true when
the lock is still held. A comment in grabLockOrStop indicates that the
function is only used when the stateLock is held, but grabLockOrStop is
being used to acquire the stateLock. If there are concurrent goroutines
using grabLockOrStop then some concurrent executions result in
stopped=true being returned when the lock is acquired.

The fix is to add a lock and some state around which the parent and
child goroutine in the grabLockOrStop function can coordinate so that
the different concurrent executions can be handled.

This change includes a non-deterministic unit test which reliably
reproduces the problem before the fix.

• use rand instead of time for random test stopCh close

Using time.Now().UnixNano()%2 ends up being system dependent because
different operating systems and hardware have different clock
resolution. A lower resolution will return the same unix time for a
longer period of time.

It is better to avoid this issue by using a random number generator.
This change uses the rand package default random number generator. It's
generally good to avoid using the default random number generator,
because it creates extra lock contention. For a test it should be fine.

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployments, click below or on the icon next to each commit.

vault – ./website

🔍 Inspect: https://vercel.com/hashicorp/vault/8zZJnJwS6u112CtsRPPiy2eLEHHW
✅ Preview: Canceled

[Deployment for 45d6e29 failed]

vault-storybook – ./ui

🔍 Inspect: https://vercel.com/hashicorp/vault-storybook/Ek82zAXPznCT7X51Pizu5Pfv17q8
✅ Preview: Canceled

[Deployment for 45d6e29 canceled]