Complete validator set replacement halts block building

[ceyonur]

After introduction of Durango and Warp, some older subnets wanted to update their validator set those with BLS keys registered in P-Chain; so that they can start using warp messages in their subnets. This can be done by (in order) adding new validators, let them sit for a while, and then remove old validators.

We have seen this was causing troubles for some Fuji subnets and halts the block building in those subnet chains. This is because Proposer VM works with delayed P-Chain heights for wrapped chains and chain's proposer slots are not updated according to new validator sets right aways. Eventually making proposer slots to be reserved for old validators and thus new validators skip block building.

Networks that do not have high load (like Fuji) can have less frequent blocks (in both subnet chains and P-chain). For those networks hours even days might not be enough. We should investigate if we can optimize Proposer VM behaviour by relaxing the Durango rule (that removed non-validator block builders).

[ceyonur]

from @StephenButtolph's notes:

Ideas for how to prevent / unblock subnets who have run into a situation where the intersection of their current validator set on the P-chain and their current validator set on their chain is empty:

1. Re-introduce the any can proposer after some amount of time
   a. Can filter based on nodeID hash
2. Add CLI flag to skip verification at a specific height
   b. Could be specified by the VM
   c. Is essentially a mandatory network upgrade ;)
   d. Definitely the easiest way to unblock people running into this now
3. Always produce blocks
4. After a sufficient amount of time, use the proposed P-chain height rather than the parent P-chain height.
   e. Makes the block building logic a bit more complex

Result:
5. Add a CLI flag to skip verification at a specific height (Easiest path to unblock subnets).
6. Consider providing improved visibility of the current P-chain height to provide feedback for subnet owners changing their validator sets.
7. Monitor how much of an ongoing issue this is. If this continues to be a pain point for subnet owners, consider implementing one of the alternatives 1, 4, or 3 (in order of expected complexity).

[github-actions]

This issue has become stale because it has been open 60 days with no activity. Adding the lifecycle/frozen label will cause this issue to ignore lifecycle events.