Split Status Cache into one cache that holds ‘transaction seenedness’ for the validator, and one that holds transaction results for the RPC

[steveluscher]

Problem

Today, the status cache does two things

• It acts as a set of message hashes the validator has seen, to prevent replay attacks
• Maps Signatures to TransactionResults for RPC lookups

Never the less, when we take a snapshot of the validator, we get both of these things. My claim is that we only need the set of seen messages, and not the transaction statuses. The reason being: the cache only holds the latest 300 slots of statuses, and there's no way that you can start up an RPC node from a snapshot in <300 slots. By the time the RPC is ready to respond to requests (ie. is ‘healthy’) the snapshotted transaction statuses will be too old to be useful and will have since been replaced.

Summary of Changes

• Create two caches on Bank; one into which message hashes are written, and one into which Signatures are written
• Only snapshot the one that holds message hashes

A follow up PR could now make status_cache an Option<BankStatusCache> and disable it when the node is not running in RPC mode.

How to review this PR

• The first commit is exclusively a rename of the existing status cache to seen_transaction_cache
• The second commit introduces a second cache, with the original name of status_cache, and makes it so that it only deals with Signatures and TransactionResults.

FAQ

Q: Aren't we doing double the writes now?
A: Yes and no. Only message Hashes are being written to seen_transaction_cache and only Signatures are being written to status_cache. Only seen_transaction_cache participates in snapshots now. All in all we're not doubling the writes but rather splitting them (and splitting the locks too) but we are duplicating whatever work that StatusCache::insert() does.

Q: Why don't you stop writing TransactionStatus to seen_transaction_cache
A: We do that in #6460 because it changes how snapshots are written and I didn't want to bother you with that code here.

[mergify]

If this PR represents a change to the public RPC API:

1. Make sure it includes a complementary update to rpc-client/ (example)
2. Open a follow-up PR to update the JavaScript client @solana/kit (example)

Thank you for keeping the RPC clients in sync with the server API @steveluscher.

[steveluscher]

Now that this cache is distinct, a follow up PR could contemplate disabling this write when the node is not in RPC mode.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 80.45113% with 26 lines in your changes missing coverage. Please review.

Project coverage is 82.8%. Comparing base (347b25d) to head (8df6799).
Report is 15 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #6459     +/-   ##
=========================================
- Coverage    82.8%    82.8%   -0.1%     
=========================================
  Files         848      848             
  Lines      379479   379528     +49     
=========================================
- Hits       314588   314536     -52     
- Misses      64891    64992    +101     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[steveluscher]

Abandoning this with the belief that #6546 is the way forward.