interop: Reset Derivation and Backfill Supervisor when Too Far Behind

[axelKingsley]

What

Forces the op-node to reset its Derivation Pipeline when an error is returned from the Supervisor regarding being "too far behind"

Why

Currently, there exists no persisted "L1 <> L2" Derivation Relationship. In the future, we would like to use a Safety Index to track the derivation increments the Node has processed.

Because there is no persistence of DerivedFrom information, if the Supervisor is ever not able to subscribe to new Safety information (eg because it is offline), there is no way to replay that call. As a result, if the Supervisor is offline for any duration, it comes back up with a gap in its data which cannot be filled by calls from the node.

So now in this PR, when this gap scenario is encountered, an error is emitted back to the Node. The Node in turn notices that the Supervisor is "too far behind", and emits a Derivation Reset Event.

How

The phrase "too far behind" is used in error checking failures when calling UpdateLocalSafe. I did it this way because error type-checking seems to not survive when the error transports over RPC (please LMK if that's incorrect, but it was my experience during development).

When the AddDerived function of the Supervisor finds that the new data is more than one ahead of the existing derivation data, it returns the error:

		return fmt.Errorf("cannot add block (%s derived from %s), last block (%s derived from %s) is too far behind: (%w)",
			derived, derivedFrom,
			lastDerived, lastDerivedFrom,
			types.ErrOutOfOrder)

Testing

I tested this using a local-devnet, which was not very efficient, but was the most obvious reproduction I came up with. To test, I started a fresh network, stopped the Supervisor for some time, and then restarted it.

• Before this fix, "out of order" errors hit the Supervisor indefinitely until nodes are restarted.
• After this fix, the Node resets the Derivation pipeline and walks back, re-deriving and backfilling the supervisor.

Future

We should really consider architectural changes that eliminate this required coordination in favor of stronger consistency ownership by the Supervisor. See this document for more for more words about that.

Notes

I also had to disable Batcher Throttling, which seems to prevent the devnet from coming up right now. I think it makes sense to disable for devnets generally, but we should also investigate this. CC @sebastianst