increase max ingest

[bw-solana]

Problem

We are over-throttling tx ingest and could potentially land more, higher value txs.

QUIC fetch and overall TPU pipeline has been IBRL'ed and can handle significantly more traffic. Old value was set based on ability to ingest/allocate for 250k pps. In testing, we have observed handling more than 2x this (was seeing sustained 600k pps in 10-node cluster testing) with latest code.

Even when we are not hitting the global cap, many individual connections are having their traffic throttled by the EMA throttling window. It is also extremely unlikely that all connections are using their full quota all the time, so this upper bound probably shouldn't be overly pessimistic.

Summary of Changes

Double the current global limit from 250k pps to 500k. This value then gets used for managing individual connection limits.

Note:

• We can likely increase this even more. Doubling is conservative.
• Longer term, we would like a more sophisticated connection heuristic. But this stopgap should provide some nice relief.

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.

[bw-solana]

Pushed fixes to tests + comments.

I opted to double the current load for the tests to keep the same load % in place.

Also note that these changes now expose a loss of precision that comes with the available_load_capacity_in_throttling_duration function as a result of computing the capacity for 50ms window and then doubling it (i.e. we can overstrain by ~2 packets per 100ms in the worst case)

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.2%. Comparing base (fe1ff00) to head (cc8c78b).
Report is 4 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6958   +/-   ##
=======================================
  Coverage    83.2%    83.2%           
=======================================
  Files         856      856           
  Lines      376845   376847    +2     
=======================================
+ Hits       313710   313734   +24     
+ Misses      63135    63113   -22     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.