fix: return MemoryLimitOOG when memory_limit is enabled

[wojtek0x]

Summary

This PR reintroduces memory memory limit enforcement when memory_limit feature is used.

Currently, EVM configuration allows specifying memory_limit here, but the limit is not actually enforced anywhere. It appears that enforcement logic was unintentionally dropped in this major refactor. On top of that, in another refactor, MemoryOOG was replaced with MemoryLimitOOG. This PR restores the original semantics of these two instruction results.

Context

We run Reth with gas cap set to uint64::MAX to allow for complex EVM computation. Without memory limit enforcement, this configuration risks out-of-memory crashes if a buggy contract triggers invalid allocations (e.g., MSTORE with an invalid offset). The included unit test demonstrates this scenario.

[codspeed-hq]

CodSpeed Performance Report

Merging #3109 will improve performances by 3.73%

_{Comparing wojtek0x:fix_memory_limit (e9a6fbd) with main (9613d61)}

Summary

⚡ 1 improvement
✅ 172 untouched

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	ecrecover precompile	204 µs	196.7 µs	+3.73%

[rakita]

good catch