feat(unrecoverable-error): implement halting of full node execution

[glevco]

Motivation

Currently, when an exception happens during a consensus update, it marks the tx as voided with a custom marker and continues to operate. This operation may be faulty, though, as the database is likely in an undesired state. For example, if such exception happens when a block is received, no following block will be accepted and the full node will not be able to sync anymore. If the full node is manually stopped and restarted, it starts up but continues to be unable to sync.

This PR's goal is to, instead, completely halt and exit the full node in those cases, forcing manual intervention. When some exception happens during a consensus update, the full node process exits with a non-zero exit code. This also guarantees that the database is marked as corrupted, so the full node cannot be restarted normally, only by a full verification or new database.

This PR is only more restrictive than what's currently implemented, and will be necessary for the Feature Activation for Transactions.

Acceptance Criteria

• Create new ExecutionManager with crash_and_exit() method.
• Create new related methods in EventManager and TransactionStorage for dealing with a full node crash.
• Change handling of exceptions during consensus so it halts and exit the full node, in addition to voiding the tx.

Checklist

• If you are requesting a merge into master, confirm this code is production-ready and can be included in future releases as soon as it gets merged

[codecov]

Codecov Report

Attention: Patch coverage is 86.76471% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 85.08%. Comparing base (da4a89c) to head (452b16b).

Files	Patch %	Lines
hathor/event/event_manager.py	66.66%	3 Missing ⚠️
hathor/manager.py	57.14%	2 Missing and 1 partial ⚠️
hathor/execution_manager.py	92.30%	2 Missing ⚠️
hathor/transaction/storage/transaction_storage.py	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #809      +/-   ##
==========================================
- Coverage   85.17%   85.08%   -0.09%     
==========================================
  Files         292      293       +1     
  Lines       22667    22719      +52     
  Branches     3415     3418       +3     
==========================================
+ Hits        19306    19331      +25     
- Misses       2687     2706      +19     
- Partials      674      682       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[glevco]

@msbrogli the code changed a lot, so I just rebased it. Could you please re-review all files?

[jansegre]

We could start using this now when a storage write fails. Do you think it'd be worth to include this in this PR?

[glevco]

We could start using this now when a storage write fails. Do you think it'd be worth to include this in this PR?

@jansegre I think we can do it in a separate PR