EVM Gateway Hardening

[j1010001]

Why (Objective)

• We need to stabilize the EVM GW - it intermittently stops ingesting blocks and need to be restarted manually, after oncall is alerted.
• The demands on EVM GW are growing as more partners are onboarding to FLOW. The EVM GW is relying on Access and Execution nodes to get execution state data, executing trace calls and getting debug traces, which is inefficient (was implemented as short-term work-around) and pushes ANs and ENs to their limits. We need to enable partners to run their own EVM GW with customs config, so we can take the load from the ANs and ENs by enabling dry-run on the EVM GW itself. EVM GW implementing dry-run also enables partners to customize the GW to provide only those debug tracers they require for their workflow.

How will we measure success (Key Results) ?

• EVM GW runs without block ingestion stalls for 4 weeks.
• Dry-run - ANs and ENs are no longer loaded by dry-run & other queries / calls.
• Enables adding custom debug_trace to the EVM GW when re-executing Txs.
• Alexar is able to deploy [30] their own GWs with minimal support from FF.
• Alchemy endpoints are unblocked after the dry-run is deployed on FF-run GW.

Effort Estimate

• Stabilizing: unknown
• Dry-run: ~ 8 dev/weeks (@ramtinms , @janezpodhostnik)
• Ops: 3d/weeks (@vishalchangrani, @franklywatson)

DACI

EVM GW stabilization

Role	Assigned
Driver	Technical: @janezpodhostnik , EM: @j1010001
Approver	@Kay-Zee
Consulted	@ramtinms, @zhangchiqing, @peterargue
Informed	Flow Protocol engineering

Dry-run feature

Role	Assigned
Driver	Technical: @ramtinms, EM: @j1010001
Approver	@dete
Consulted	@zhangchiqing, @peterargue, @janezpodhostnik
Informed	Flow Protocol Engineering

Stabilization

Beta Give feedback

1. identify root-cause for the block ingestion stalls, implement fix
2. Review EVM GW code to identify & fix edge-cases that could lead to stability problems
3. Gateway stops indexing new blocks flow-evm-gateway#590
   Bug +1
   

Dry-run (reducing the GW dependency on the AN)

Beta Give feedback

1. move replaying blocks logic (Chain-follower receiving EVM events) to flow-go
2. Ephemeral view: pass-in storage which provides what dry-run calls need
3. Gateway - Integrate with the flow-go to index data and store locally (StorageProvider interface) Incorporate & implement interface from offchain package in flow-go flow-evm-gateway#629
4. pre-requisite: Sort EVM related events by their EventIndex flow-evm-gateway#621
5. pre-requisite: Aggregate the parsed EVM.BlockExecuted & EVM.TransactionExecuted event payloads flow-evm-gateway#627
6. Add support for debug_traceCall to EVM GW API flow-evm-gateway#530
   Compatibility EVM Feature +3
   
7. Validate transactions using local state flow-evm-gateway#586
   Feature Feedback +2

Operationalizing the EVM GW for partners

Beta Give feedback

1. Update & consolidate existing docs to operations guide to enable partners to deploy and configure their own EVM GW. #6551
   +0