Prevent concurrent health check race causing leadership lock corruption

[dmytro-pryvedeniuk]

This PR started as an attempt to fix flaky RavenDbTests.LeaderElection.leadership_election_compliance.take_over_leader_ship_if_leader_becomes_stale test, but turned out to be a fix for the issue that may occur in non-testing environment. Relevant for the storages using lease-based locks (RavenDB or CosmosDB).

Problem

DoHealthChecksAsync can be called from two concurrent paths:

• CheckAgentHealth -> DoHealthChecksAsync -> TryAttainLockAsync (index = 0)
• heartbeat -> DoHealthChecksAsync -> TryAttainLockAsync (index = 0)

One wins and updates the index (_lastLockIndex for RavenDB), another one uses the same initial (stale now) index, fails and calls stepDownAsync -> ReleaseLeadershipLockAsync.
Result: Leader kills itself. Election stalls until the lease expires.

Fix

Non-blocking re-entrancy guard is added to NodeAgentController.DoHealthChecksAsync. concurrent_DoHealthChecksAsync_guard_prevents_spurious_stepdown test demonstrates the leader survives the concurrent calls.

The tests leader_switchover_between_nodes and take_over_leader_ship_if_leader_becomes_stale were hardened.
The default heartbeat in tests is 1 second so all hosts compete for the leadership. Using combination of default fast (1s) and custom slow (10mins) heartbeats gives more control.

Also, added a new test - take_over_leader_ship_if_leader_becomes_stale_with_racing_nodes - which relies only on the heartbeats without explicit CheckAgentHealth message.

[jeremydmiller]

@dmytro-pryvedeniuk Good catch!