fix: allow users to optionally configure node startup even if index reconciliation fails

[aarshkshah1992]

For #12897.

There chain state has some missing messages and this PR enables them to start the node even if index reconciliation fails.

[rvagg]

This is pretty blunt. @emmanuelm41 can you consider this change and comment on whether it makes your life any better? This gives you a flag that lets you start the node up, log an error, but the reconciliation is aborted entirely. It doesn't address the messages being missing (that's still a mystery for a full archival node, we really need to get to the bottom of that but it's separate to the chainindexer tripping over it). It means there may be a small gap in what's been indexed, although it can always manually run a backfill over epochs you're concerned about with lotus index validate-backfill.

[aarshkshah1992]

@emmanuelm41

• Based on the error message at Node crashed, and stop syncing with EOF #12897, that node's chain state is missing some messages at epoch 4702087.

• This PR enables you to start your node with the ChainIndexerConfig .AllowIndexReconciliationFailure flag enabled (false by default, you need to set it to true) which means that your node will boot up even if it fails to reconcile the Index with the chain state. However, note that this means that your Index will not have entries for epoch 4702087 and also for epochs before it if you have NOT backfilled them already.

• Note the validate-backfill CLI will also fail for epoch 4702087.

• Ideally, what you want to do is validate-backfill from the latest epoch upto epoch 4702087 and then validate-backfill from epoch 4702086 upto genesis in the background while the node is running so you don't need the reconciliation at all to backfill the Index.

Let me know if you have any questions.