Crash on start (liveness issue)

[nalepae]

Sometimes, just after the start, we get this:

HTTPError: 400 Client Error: Bad Request for url: http://localhost:5052/lighthouse/liveness

Command line:

eth-validator-watcher --beacon-url http://localhost:5052 --web3signer-url http://localhost:9000 --beacon-type=lighthouse

Additional info:
It seems this error is more likely to happen when the watcher is started towards the end of an epoch.

[nalepae]

Analysis:

• When the watcher start, it takes the current slot/epoch and begin to process all slots from it.
• The first slot processing takes a long time, since it does more processing. All the is_new_epoch stuff.
• When starting the watcher at the end epoch n, it is possible that a new epoch n+1 has already started when the watcher will effectively handle the lasts slots of epoch n.
• When executing process_missed_attestations for epoch n, the watcher will retrieve the liveness for epoch n-1.

==> Issue:

• current epoch is now n+1
• the watcher retrieves liveness for epoch n-1

The Beacon API specification states:

Requests the beacon node to indicate if a validator has been observed to be live in a given epoch. The beacon node might detect liveness by observing messages from the validator on the network, in the beacon chain, from its API or from any other source. A beacon node SHOULD support the current and previous epoch, however it MAY support earlier epoch. It is important to note that the values returned by the beacon node are not canonical; they are best-effort and based upon a subjective view of the network. A beacon node that was recently started or suffered a network partition may indicate that a validator is not live when it actually is.

==> Here, we don't retrieve liveness for neither current nor previous epoch, but for previous-previous epoch, and it seems all BN do not support it.

[nalepae]

Fixed, will be deployed in v0.7.0