Remove node from cluster when node locks are broken

[DaveCTurner]

In #52680 we are introducing a mechanism that will allow nodes to remove themselves from the cluster if they locally determine themselves to be unhealthy. The only check today is that their data paths are all empirically writeable. We could also check NodeEnvironment#assertEnvIsLocked() here; indeed we already call this method during the health check but do not consider a failure to be fatal (see #52680 (comment)).

A broken node lock today blocks things like allocating new shards to the node, but I think it does not block indexing or searching on existing shards since these are protected by shard-level locks instead. On the other hand there's something very wrong with your environment if the node lock is broken and it seems reasonable to treat it pretty seriously.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)

[Bukhtawar]

In #52860 we are introducing a mechanism that will allow nodes to remove themselves from the cluster if they locally determine themselves to be unhealthy

nit : #52680 :)

Thanks for raising this.

[DaveCTurner]

We discussed this today and agreed to proceed.

[Bukhtawar]

Thanks @DaveCTurner should we work on the existing PR, or else we can start a new one

[DaveCTurner]

I'd prefer a new one once #52680 is merged.