Add timeouts to `refresh_health` in VC

[michaelsproul]

Description

While working on:

• Broadcast VC requests in parallel and fix subscription error #6223

I noticed that our implementation of refresh_status could be optimised in several ways:

lighthouse/validator_client/src/beacon_node_fallback.rs

Lines 179 to 207 in 9e12c21

	/// Perform some queries against the node to determine if it is a good candidate, updating
	/// `self.status` and returning that result.
	pub async fn refresh_status<T: SlotClock>(
	&self,
	slot_clock: Option<&T>,
	spec: &ChainSpec,
	log: &Logger,
	) -> Result<(), CandidateError> {
	let previous_status = self.status(RequireSynced::Yes).await;
	let was_offline = matches!(previous_status, Err(CandidateError::Offline));
	let new_status = if let Err(e) = self.is_online(was_offline, log).await {
	Err(e)
	} else if let Err(e) = self.is_compatible(spec, log).await {
	Err(e)
	} else if let Err(e) = self.is_synced(slot_clock, log).await {
	Err(e)
	} else {
	Ok(())
	};
	// In case of concurrent use, the latest value will always be used. It's possible that a
	// long time out might over-ride a recent successful response, leading to a falsely-offline
	// status. I deem this edge-case acceptable in return for the concurrency benefits of not
	// holding a write-lock whilst we check the online status of the node.
	*self.status.write().await = new_status;
	new_status
	}

The issues are:

• There's no timeout for any of the HTTP methods we call, so this function could run indefinitely in case of a very unresponsive BN.
• We make 3 calls when 2 would probably be sufficient. We could use the response from is_synced to infer online status, rather than making a separate request for the version.

Version

Lighthouse v5.3.0

[macladson]

Just want to point out with the merge of #4393:

• This function is now called refresh_health.
• Issue 2 is resolved, it now only makes 2 calls to the BN.
• Issue 1 is still present.

[Gua00va]

@michaelsproul how much timeout should be added here and where should the timeout value be configured? And should it be added on individual await calls or the whole method?