op-node: EngineController: Proper locking

[pcw109550]

Description

There are multiple control flows which invokes Engine Controller logic. Some subroutines will be invoked starting from events, namely onEvent, and it properly acquires RW mutex which is embedded in EngineController struct.

The current codebase does not properly acquire mutexes while modifying the state of the EngineController. This may lead to potential race conditions.

The root of the current mis-synchronization stems from two reasons.

1. While we were refactoring op-node, proper synchronization was missed. When we replace the events with synchronous function calls, the function call itself loses locking. Therefore we need an explicit locking while removing the events.
   • ex) op-node: remove PromoteFinalizedEvent #17165: PromoteFinalizedEvent was replaced with function, and mutex locking got removed. The Finalizer struct directly called engine controller subroutine.
   • ex) Remove event for Promote Pending Safe / Promote Local Safe / Promote Unsafe #16941: PromotePendingSafeEvent, PromoteLocalSafeEvent was replaced with function, and mutex locking got removed. AttributeHandler struct directly called engine controller subroutine.
   • ex) op-node: Remove TryUpdateEngine event #17096: Some direct struct embeddings at s.Engine.TryUpdateEngine at SyncDeriver.
   • ex) ForceReset(ctx context.Context, localUnsafe, crossUnsafe, localSafe, crossSafe, finalized eth.L2BlockRef)
2. Existing subroutines which were not handled correctly.
   • ex) EngineController.IsEngineSyncing() was invoked outside the engine controller, at SyncDeriver without read lock.

Upper lists are non-exhaustive.

The easy way to check the engine controller subroutines entrypoint is to search type EngineController interface throughout the codebase. This PR attempts to apply proper mutex locking.

If possible, the exposure of these methods should be minimized.

Will follow up on the other subroutines.

Possibly related with the race conditions reported at #17564