Bug: Restarting tenderduty while alert is active causes that alert to struck forever and never produce any new alert (Critical)

[upnodedev]

Problem

Restarting tenderduty while alert is active causes that alert to struck forever and never produce any new alert

Above picture clearly state that xxx has missed 10 blocks on mocha is already resolved. In contrast, tenderduty still showing that alert is not resolved and neither pagerduty nor telegram receive any resolve event.

Moreover, it never produce missed 10 blocks alert anymore in the future.

Impact

Alert struck forever and never trigger any new alert. Causes our validator node to be at risk of being jailed and slashed. (Critical)

Step to reproduce

1. Run both tenderduty and validator normally
2. Stop validator and wait for it to trigger alert "ALERT: - xxx has missed xx blocks on xxx"
3. Stop tenderduty
4. Restart tenderduty
5. Restart validator
6. Notice that alert doesn't get resolved
7. Repeat these steps and notice that new alert never get sent