client: Add retry logic on Pubsub 429 error during connect

[joncinque]

Problem

The pubsub client can give a 429 (TOO_MANY_REQUESTS) when trying to connect, causing avoidable failures to users.

Summary of Changes

Respect the backoff from a 429, and retry the connection after waiting. This mimics the send retry logic at

solana/client/src/http_sender.rs

Lines 96 to 117 in 74cc2e5

	if response.status() == StatusCode::TOO_MANY_REQUESTS
	&& too_many_requests_retries > 0
	{
	let mut duration = Duration::from_millis(500);
	if let Some(retry_after) = response.headers().get(RETRY_AFTER) {
	if let Ok(retry_after) = retry_after.to_str() {
	if let Ok(retry_after) = retry_after.parse::<u64>() {
	if retry_after < 120 {
	duration = Duration::from_secs(retry_after);
	}
	}
	}
	}
	too_many_requests_retries -= 1;
	debug!(
	"Too many requests: server responded with {:?}, {} retries left, pausing for {:?}",
	response, too_many_requests_retries, duration
	);
	sleep(duration);
	continue;

Fixes #

[mvines]

sweet!

[codecov]

Codecov Report

Merging #19990 (b2304c9) into master (4675fa6) will increase coverage by 0.0%.
The diff coverage is 80.2%.

@@           Coverage Diff            @@
##           master   #19990    +/-   ##
========================================
  Coverage    82.6%    82.6%            
========================================
  Files         478      480     +2     
  Lines      133547   133674   +127     
========================================
+ Hits       110369   110514   +145     
+ Misses      23178    23160    -18