v2.3: increase max ingest (backport of #6958)

[mergify]

Problem

We are over-throttling tx ingest and could potentially land more, higher value txs.

QUIC fetch and overall TPU pipeline has been IBRL'ed and can handle significantly more traffic. Old value was set based on ability to ingest/allocate for 250k pps. In testing, we have observed handling more than 2x this (was seeing sustained 600k pps in 10-node cluster testing) with latest code.

Even when we are not hitting the global cap, many individual connections are having their traffic throttled by the EMA throttling window. It is also extremely unlikely that all connections are using their full quota all the time, so this upper bound probably shouldn't be overly pessimistic.

Summary of Changes

Double the current global limit from 250k pps to 500k. This value then gets used for managing individual connection limits.

Note:

• We can likely increase this even more. Doubling is conservative.
• Longer term, we would like a more sophisticated connection heuristic. But this stopgap should provide some nice relief.

  ---

  This is an automatic backport of pull request increase max ingest #6958 done by Mergify.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.8%. Comparing base (668c135) to head (c0e13db).
⚠️ Report is 34 commits behind head on v2.3.

Additional details and impacted files

@@           Coverage Diff           @@
##             v2.3    #6992   +/-   ##
=======================================
  Coverage    82.8%    82.8%           
=======================================
  Files         850      850           
  Lines      379962   379964    +2     
=======================================
+ Hits       314779   314852   +73     
+ Misses      65183    65112   -71     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bw-solana]

Adding @lijunwangs and @alessandrod as SME reviewers.

BP Justification

This is expected enable larger, higher value blocks to be packed on mainnet because some senders are getting throttled under the current limits.

While we don't usually see the global limit (currently 250k pps and moving to 500k w/ this change) being hit, we base individual staked connection limits off of this number, and those individual connections do get throttled.

We ran mds4 w/ a 1M pps limit and confirmed we were seeing more traffic coming in and no adverse effects observed in the form of skipped slots. This was the counter-balance in the past that caused us to pick 250k (ingesting too many meant too many allocations and would cause disruptions to the rest of the pipeline and ultimately delayed replay and skipped leader slots)

Note: Alessandro has proposed further increases (up to 1M pps) to further desconstrain things.