v3.1: Streamer: add BDP scaling (backport of #10144)

[mergify]

Problem

Streamer ignores RTT in all its calculations. This means that connections with high latency are heavily rate limited, much more than the stake amount would suggest. This happens before the actual stake-based stream throttling even has a change to kick in, and is thus very counterintuitive to the client.

A more principled version of #7954.

Some preliminary concepts:

• Sender can not have more than max_concurrent_streams worth of open streams at any time
• These together limit how many TXs can be “in flight” between client and server and not yet ACKd. Currently, these limits are computed based on stake. We need to compute them based on stake and RTT (as longer RTT means you need to have more things on the wire before you see an ACK).
• Beyond the limit mentioned above, there is an overall stream limit imposed by the code in stream_throttle.rs. That is out of scope of this PR, we are not modifying SWQOS here.

This mechanism is not intended as the actual rate limiter for complaint clients, just as a limit on network buffers. This will never allocate less bandwidth than what you'd get today.

Summary of Changes

• Assume that in the original code the RTT was expected to be ~50ms
• Scale max_concurrent_uni_streams with RTT if it is above 50ms

50 ms was chosen as a compromise between allowing more TPS and keeping impact small.

Impact

Before:

After:

Related: #8948 (which was reverted)

---

This is an automatic backport of pull request #10144 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 25a96c6 has failed:

On branch mergify/bp/v3.1/pr-10144
Your branch is up to date with 'origin/v3.1'.

You are currently cherry-picking commit 25a96c61a.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   streamer/src/nonblocking/swqos.rs

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.54839% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.2%. Comparing base (ec3d409) to head (2ab319c).

Additional details and impacted files

@@            Coverage Diff            @@
##             v3.1   #10151     +/-   ##
=========================================
- Coverage    83.2%    83.2%   -0.1%     
=========================================
  Files         865      865             
  Lines      376518   376546     +28     
=========================================
+ Hits       313533   313549     +16     
- Misses      62985    62997     +12     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[alexpyattaev]

Odd that it had "conflicts" still. LGTM.

[t-nelson]

as discussed during the bp call yesterday, we'll let this bake until mb is fully on 3.1