Evict oldest vote on vote refresh after restart

[carllin]

Problem

If a validator votes on slot X, saves the vote into their tower, then dies, there's the potential that due to hardware failure or a bad gossip peer, that vote never made it onto the network.

This means that this existing assumption that we can always get the slot X to refresh from gossip

agave/gossip/src/cluster_info.rs

Lines 1128 to 1135 in 228413c

	// If you don't see a vote with the same slot yet, this means you probably
	// restarted, and need to wait for your oldest vote to propagate back to you.
	//
	// We don't write to an arbitrary index, because it may replace one of this validator's
	// existing votes on the network.
	if let Some(vote_index) = vote_index {
	self.push_vote_at_index(vote, vote_index);
	}

is not necessarily true, and we may never refresh

Summary of Changes

If you cannot find the vote to be refreshed AND there are more than 32 votes in gossip just evict the oldest vote X. This is fine with the new vote state update since only the latest vote matters anyways, so there should always be at least one vote newer than X if X is evicted.

Fixes #

[AshwinSekar]

Not sure if there's a good way to address this, but after a restart we could attempt to refresh a vote before our gossip crds is fully populated. in this case we could be evicting a later vote.

I think it's fine anyway, since with VoteStateUpdate since most of the time only the last vote matters, but just wanted to point out that we might not always evict the "oldest" vote here.

[carllin]

yeah, it will be some vote "older" than the newest vote because to evict, you have to have >=32 votes in your gossip table

[AshwinSekar]

yea I think that's true as long as the tower file isn't corrupt which we shouldn't worry about.

[AshwinSekar]

hmm should this be:
if num_crds_votes < MAX_LOCKOUT_HISTORY { num_crds_votes } else { vote_index}.
that way if the crds table isn't full we prioritize not evicting a vote?

[carllin]

I thought about this, but wanted to keep the logic as similar as possible to the existing logic from push_vote where we evict a vote if it doesn't exist in tower, even if there's room in the crds table. Maybe thats slightly more efficient in gosisp than keeping around an unnecessary vote