ENG-33 chore(sequencer): add cache for immutable check-tx results

[Fraser999]

Summary

Added an in-memory cache of check-tx results.

Background

Currently if check_tx is called repeatedly for the same tx, we recheck everything every time. There is a portion of that work which will always yield the same result, and hence would benefit from being cached, meaning rechecks only need to actually execute the changeable portion of the checks if we get a cache hit.

We also needlessly call mempool.remove for txs which fail the immutable portion of the parse/check (briefly write-locking the mempool). Such txs can never have made their way into the app mempool, hence mempool.remove doesn't need to be called for these cases.

Changes

• split the bulk of handle_check_tx into two new functions: parse_tx_and_run_immutable_checks with a result which can be cached, and run_mutable_checks for the checks which can vary depending upon state.
• Added a cache of 10,000 immutable check-tx results which is queried near the start of handle_check_tx. If we get a cache hit and the cached result is Ok we continue with the mutable checks. If we get a cache hit where the cached result is Err we return the cached error response. If we get a cache miss, we run the immutable checks and cache the result before proceeding with the mutable checks.
• removed all calls to mempool.remove from the tx parsing and immutable checking stage.
• added metrics for the cache (see below for details). If we find that we never get any cache hits, we can remove the cache in the future
• changed the app mempool to receive Arc<SignedTransaction> to avoid expensive clones since these are now potentially also being held in the new cache.
• changed transaction::checks::check_nonce_mempool to get_current_nonce_if_tx_nonce_valid where it now returns the nonce. This is to avoid having to fetch the nonce twice (we fetch it again when adding the tx to the mempool if all checks pass), but also to avoid potential TOCTOU issues.

Testing

Added unit tests to ensure the cache is populated. Once metrics become testable (when #1192 is merged) we can extend these tests to ensure we get cache hits.

Metrics

• Added astria_sequencer_check_tx_cache_hit counter to record total cache hits.
• Added astria_sequencer_check_tx_cache_miss counter to record total cache misses where the request was of type CheckTxKind::Recheck.

Related Issues

Closes #1253.

[noot]

is there a reason for caching the tx instead of just its hash? since you're sent the tx again in CheckTx, this seems unnecessary

[Fraser999]

It's to avoid parsing the protobuf tx from bytes and then converting the protobuf tx to a SignedTransaction. It's mainly the second part which is worth avoiding, since we're doing signature checking in SignedTransaction::try_from_raw.

[SuperFluffy]

Requesting @Lilyjjo review for this because she is owning the mempool bits of sequencer at the moment.

[Fraser999]

Closed in favour of #1515.