Streamer: swap governor crate with TokenBucket by alexpyattaev · Pull Request #8740 · anza-xyz/agave

alexpyattaev · 2025-10-28T22:04:06Z

Problem

re-try of TPU: Replace governor crate with TokenBucket #8154 which was reverted due to side-effects on local-cluster

Summary of Changes

Use TokenBucket instead of governor crate
Add a patch to prevent localhost connections from getting ratelimited

CONTEXT

https://discord.com/channels/428295358100013066/560503042458517505/1430348086042951722

codecov-commenter · 2025-10-29T02:49:51Z

Codecov Report

❌ Patch coverage is 72.88136% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.2%. Comparing base (b4cce97) to head (d0874c8).

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #8740     +/-   ##
=========================================
- Coverage    83.2%    83.2%   -0.1%     
=========================================
  Files         863      863             
  Lines      373947   373921     -26     
=========================================
- Hits       311236   311184     -52     
- Misses      62711    62737     +26

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

alexpyattaev · 2025-10-29T06:47:31Z

Extra context:
Previous version of this PR was reverted due to regression in local-cluster

This PR addresses the regression by bumping number of connections per IP for test-validator setups

 cargo nextest run --profile ci --test-threads=1 --package solana-local-cluster -- test_run_test_load_program_accounts_root
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.28s
────────────
 Nextest run ID e175959f-0a95-4bca-8c3f-a87f3e5e308c with nextest profile: ci
    Starting 1 test across 2 binaries (51 tests skipped)
        PASS [  42.601s] solana-local-cluster::local_cluster test_run_test_load_program_accounts_root
────────────
     Summary [  42.604s] 1 test run: 1 passed, 51 skipped

Full partition:

cargo nextest run --profile ci --package solana-local-cluster --test local_cluster --partition hash:9/10 --test-threads=1 --no-tests=warn 
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.28s
────────────
 Nextest run ID be8af19b-3b78-4641-8f65-293b324a072d with nextest profile: ci
    Starting 7 tests across 1 binary (45 tests skipped)
        PASS [  18.627s] solana-local-cluster::local_cluster test_local_cluster_signature_subscribe
        PASS [  40.884s] solana-local-cluster::local_cluster test_no_lockout_violation_with_tower
        PASS [  17.204s] solana-local-cluster::local_cluster test_no_voting
  TRY 1 SLOW [> 60.000s] solana-local-cluster::local_cluster test_run_test_load_program_accounts_partition_root
        PASS [  67.404s] solana-local-cluster::local_cluster test_run_test_load_program_accounts_partition_root
        PASS [  49.464s] solana-local-cluster::local_cluster test_run_test_load_program_accounts_root
        PASS [  33.753s] solana-local-cluster::local_cluster test_spend_and_verify_all_nodes_2
  TRY 1 SLOW [> 60.000s] solana-local-cluster::local_cluster test_wait_for_max_stake
        PASS [ 116.574s] solana-local-cluster::local_cluster test_wait_for_max_stake
────────────
     Summary [ 343.917s] 7 tests run: 7 passed (2 slow), 45 skipped

All CI partitions of local-cluster complete in <10 min.

alexpyattaev · 2025-10-29T06:48:39Z

Bumping the max_connections_per_ipaddr_per_min way up to avoid tests ever flaking in the future due to ratelimits.

apfitzge · 2025-10-29T12:58:19Z

Have no context beyond the previously slow CI test. Will defer to @lijunwangs for the review.
Sorry for the trouble 🫡

lijunwangs · 2025-10-29T17:56:05Z

Have no context beyond the previously slow CI test. Will defer to @lijunwangs for the review. Sorry for the trouble 🫡

@apfitzge can you point to the slow CI test link? Thanks

alexpyattaev · 2025-10-29T19:35:07Z

Have no context beyond the previously slow CI test. Will defer to @lijunwangs for the review. Sorry for the trouble 🫡

@apfitzge can you point to the slow CI test link? Thanks

https://discord.com/channels/428295358100013066/560503042458517505/1430348086042951722

lijunwangs · 2025-11-04T00:25:27Z

    let quic_server_params = Arc::new(quic_server_params);
    let rate_limiter = Arc::new(ConnectionRateLimiter::new(
        quic_server_params.max_connections_per_ipaddr_per_min,
+        // allow for 10x burst to make sure we can accommodate legitimate


Should we do this only to test? Would this make our rate limiting ineffective against abusive clients?

It will not. If someone is truly abusive they will exhaust this very quickly, and sustained attack will be stopped, as intended.

KirillLykov · 2025-11-04T07:05:39Z

    pub fn new_for_tests(tpu_enable_udp: bool) -> Self {
        let tpu_quic_server_config = SwQosQuicStreamerConfig {
            quic_streamer_config: QuicStreamerConfig {
-                max_connections_per_ipaddr_per_min: 32,


was this test trying to create > 32 connections per minute? Seems suspicious. I think this should be investigated as a spin-off in a new issue to avoid blocking progress on the current change

I did not dive too deeply into test_run_test_load_program_accounts_root. However now that we allow a larger initial burst it should not be necessary to patch new_for_tests.

add burst handling to the per-IP rate limiter This should allow legit container users to connect without triggering ratelimits more easily

alexpyattaev · 2025-11-05T14:42:02Z

Had to rebase due to conflicts + clean up commit messages.

swap governor crate with TokenBucket add burst handling to the per-IP rate limiter This should allow legit container users to connect without triggering ratelimits more easily

alexpyattaev force-pushed the rm_governor_2 branch 2 times, most recently from d6cac0d to eba17f2 Compare October 29, 2025 02:03

alexpyattaev requested a review from lijunwangs October 29, 2025 06:47

alexpyattaev commented Oct 29, 2025

View reviewed changes

alexpyattaev force-pushed the rm_governor_2 branch from eba17f2 to 19dd910 Compare October 29, 2025 06:51

alexpyattaev marked this pull request as ready for review October 29, 2025 07:11

alexpyattaev requested a review from apfitzge October 29, 2025 07:11

alexpyattaev changed the title ~~swap governor crate with TokenBucket~~ Streamer: swap governor crate with TokenBucket Oct 29, 2025

alexpyattaev requested a review from KirillLykov November 3, 2025 16:30

alexpyattaev force-pushed the rm_governor_2 branch from 19dd910 to d1941ff Compare November 3, 2025 16:34

lijunwangs reviewed Nov 4, 2025

View reviewed changes

KirillLykov reviewed Nov 4, 2025

View reviewed changes

alexpyattaev force-pushed the rm_governor_2 branch from d1941ff to 98f53cf Compare November 4, 2025 21:30

alexpyattaev requested a review from KirillLykov November 4, 2025 21:30

lijunwangs previously approved these changes Nov 5, 2025

View reviewed changes

swap governor crate with TokenBucket

d0874c8

add burst handling to the per-IP rate limiter This should allow legit container users to connect without triggering ratelimits more easily

alexpyattaev dismissed lijunwangs’s stale review via d0874c8 November 5, 2025 14:41

alexpyattaev force-pushed the rm_governor_2 branch from 98f53cf to d0874c8 Compare November 5, 2025 14:41

KirillLykov approved these changes Nov 5, 2025

View reviewed changes

alexpyattaev added the automerge automerge Merge this Pull Request automatically once CI passes label Nov 5, 2025

alexpyattaev added this pull request to the merge queue Nov 5, 2025

Merged via the queue into anza-xyz:master with commit a28400f Nov 5, 2025
56 checks passed

alexpyattaev deleted the rm_governor_2 branch November 5, 2025 15:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streamer: swap governor crate with TokenBucket#8740

Streamer: swap governor crate with TokenBucket#8740
alexpyattaev merged 1 commit intoanza-xyz:masterfrom
alexpyattaev:rm_governor_2

alexpyattaev commented Oct 28, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 29, 2025 •

edited

Loading

Uh oh!

alexpyattaev commented Oct 29, 2025 •

edited

Loading

Uh oh!

alexpyattaev Oct 29, 2025

Uh oh!

apfitzge commented Oct 29, 2025

Uh oh!

lijunwangs commented Oct 29, 2025

Uh oh!

alexpyattaev commented Oct 29, 2025

Uh oh!

lijunwangs Nov 4, 2025

Uh oh!

alexpyattaev Nov 4, 2025

Uh oh!

KirillLykov Nov 4, 2025 •

edited

Loading

Uh oh!

alexpyattaev Nov 4, 2025

Uh oh!

alexpyattaev commented Nov 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

alexpyattaev commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Summary of Changes

CONTEXT

Uh oh!

codecov-commenter commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alexpyattaev commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexpyattaev Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

apfitzge commented Oct 29, 2025

Uh oh!

lijunwangs commented Oct 29, 2025

Uh oh!

alexpyattaev commented Oct 29, 2025

Uh oh!

lijunwangs Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

alexpyattaev Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

KirillLykov Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexpyattaev Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

alexpyattaev commented Nov 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alexpyattaev commented Oct 28, 2025 •

edited

Loading

codecov-commenter commented Oct 29, 2025 •

edited

Loading

alexpyattaev commented Oct 29, 2025 •

edited

Loading

KirillLykov Nov 4, 2025 •

edited

Loading