BEEFY validators skip finality proof when they are all closed at the same time

[serban300]

The root cause of #2285 was that no validator had a BEEFY finality proof for block #7829924 . They just skipped the justification for #7829924 and moved on. More details in this comment: #2285 (comment)

Stumbled upon this issue also while working on #2787

Prerequisites:

1. A zombienet config:

[relaychain]
default_image = "docker.io/parity/polkadot:latest"
default_command = "/home/serban/workplace/sources/polkadot-sdk/target/release/polkadot"

# refer to ./README.md for more details on how to create snapshot and spec
chain = "rococo-local"
chain_spec_path = "/home/serban/workplace/snapshots/db-test-gen/rococo-local.json"

  [[relaychain.nodes]]
  name = "alice"
  validator = true

  [[relaychain.nodes]]
  name = "bob"
  validator = true

2. Adjust rococo-local epoch duration from:

	frame_support::parameter_types! {
		pub EpochDurationInBlocks: BlockNumber =
			prod_or_fast!(1 * HOURS, 1 * MINUTES, "ROCOCO_EPOCH_DURATION");
	}

to

	frame_support::parameter_types! {
		pub EpochDurationInBlocks: BlockNumber =
			prod_or_fast!(1 * HOURS, 1 * MINUTES / 6, "ROCOCO_EPOCH_DURATION");
	}

Reproduction steps:

1. Start the zombienet. Let it run for ~60s.
2. Close the zombienet. I think it needs to be closed right when a finality proof is generated, but it can also be close randomly.
3. Start the zombienet again and also start a 3rd node separately:

./target/release/polkadot --chain=/home/serban/workplace/snapshots/db-test-gen/rococo-local.json --dave --tmp --enable-offchain-indexing=true --sync warp --reserved-nodes "/ip4/0.0.0.0/tcp/30310/p2p/12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm"

4. Check the logs of the 3rd node. See if it prints something like 🥩 ran out of peers to request justif #43 from. If not, repeat steps 2-3

The problem seems related to closing all validators at the same time. Maybe saving the AUX DB on drop as suggested here would fix this.

[acatangiu]

Maybe saving the AUX DB on drop as suggested #2378 (comment) would fix this.

drop() handlers are not run when node client crashes or is force killed (which I think are the two main reasons a client ever stops) so it won't actually help here IMO

The problem seems related to closing all validators at the same time.

On decentralized networks this should never be the case so I think this issue is low priority.

[stakeworld]

@serban300 On my westend rpc nodes I get a lot of "🥩 ran out of peers to request justif #19153161 from". Always the same block number. I suppose something similar?

[serban300]

Thanks for the report ! Yes, probably related to #3003

Published a fix which should prevent this from happening in the future: #3074

But it won't fix the missing BEEFY proofs on older blocks. Will have to investigate how to handle this.

[stakeworld]

But it won't fix the missing BEEFY proofs on older blocks. Will have to investigate how to handle this.

Yes its a little spammy because the errors comes at every block, so around 10 per minute. I'm now filtering them out but that is of course not ideal.

[serban300]

Merged the fix into master. Waiting for it to be deployed on Westend, and after that maybe we should restart BEEFY on Westend in order to fix the warning.

[serban300]

Restarted BEEFY on Westend: https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fwestend-rpc.polkadot.io#/explorer/query/0x131c054695e41a8c44a9c1cf698e747454c8166cd6d60accec72334b6ed720ba

This should be fixed now. Closing.