interface: Optimize the `StakeHistory::get` function by vadorovsky · Pull Request #81 · solana-program/stake

vadorovsky · 2025-08-13T09:51:53Z

StakeHistory::get was performing a binary search for every requested epoch. That was a visible bottleneck in the redeem_rewards function in Agave, taking 47% of the whole process:

Given that the stake history is contigous and ordered starting from the latest epochs, we don't have to perform a binary search. We can derive the index by looking up the latest element's epoch and subtracting the requested epoch from it. That reduces the StakeHistory::get operation to taking just 3.1% of the entire redeem_rewards process.

`StakeHistory::get` was performing a binary search for every requested epoch. Given that the stake history is contigous and ordered starting from the latest epochs, we don't to binary search. We can derive the index by looking up the latest element's epoch and subtracting the requested epoch from it.

vadorovsky · 2025-08-19T08:41:00Z

@2501babe it seems like the workflows need to be manually enabled for me and I cannot mark you as a reviewer. I would appreciate your review here.

2501babe · 2025-08-21T11:45:47Z

nice find!

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ```

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ``` (cherry picked from commit e752ae6) # Conflicts: # Cargo.toml # programs/sbf/Cargo.toml

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ```

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1.

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ```

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1.

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ```

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1.

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1. (cherry picked from commit 8c21276) # Conflicts: # Cargo.toml # programs/sbf/Cargo.toml

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1.

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1. (cherry picked from commit 8c21276)

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ```

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ``` (cherry picked from commit 8aa41ea) # Conflicts: # runtime/src/bank/partitioned_epoch_rewards/calculation.rs

…z#7742) `calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ``` (cherry picked from commit 8aa41ea)

) (#7958) bump solana-stake-interface from 2.0.0 to 2.0.1 (#7919) `StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1. (cherry picked from commit 8c21276) Co-authored-by: Michal R <vad.sol@proton.me>

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ``` (cherry picked from commit 8aa41ea)

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1.

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ```

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1.

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ```

…z#7742) `calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ``` (cherry picked from commit 8aa41ea) Conflicts: runtime/src/bank/partitioned_epoch_rewards/calculation.rs

…ackport of #7742) (#8012) `calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ``` (cherry picked from commit 8aa41ea) Co-authored-by: Michal R <vad.sol@proton.me>

`calculate_stake_vote_rewards` was storing accumulated rewards per vote account in a `DashMap`, which then was used in a parallel iterator over all stake delegations. There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processes one of the stake delegations and tries to acquire the lock on a `DashMap` shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution results in high contention, with some threads spending the most of their time on waiting for lock. The time spent on these calculations was ~208.47ms: ``` redeem_rewards_us=208475i ``` Fix that by: * Removing the `DashMap` and instead using `fold` and `reduce` operations to build a regular `HashMap`. * Pre-allocating the `stake_rewards` vector and passing `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool. * Pulling the optimization of `StakeHistory::get` in `solana-stake-interface`. solana-program/stake#81 ``` redeem_rewards_us=48781i ```

`StakeHistory::get` in solana-stake-interface was performing a binary search for every requested epoch, which was a visible performance bottleneck. solana-program/stake#81 fixed that by subtracting the indices. The fix was released in 2.0.1.

vadorovsky force-pushed the optimize-stake-history branch from 1b64360 to 6a2c5af Compare August 13, 2025 10:50

vadorovsky force-pushed the optimize-stake-history branch from 6a2c5af to 116f055 Compare August 13, 2025 12:45

vadorovsky marked this pull request as ready for review August 15, 2025 16:46

2501babe self-requested a review August 20, 2025 06:43

2501babe approved these changes Aug 21, 2025

View reviewed changes

2501babe merged commit a9c19ee into solana-program:main Aug 21, 2025
9 checks passed

2501babe mentioned this pull request Aug 21, 2025

interface: test both sides of stake history #84

Merged

vadorovsky mentioned this pull request Aug 25, 2025

runtime: Avoid locking during stake vote rewards calculation anza-xyz/agave#6900

Merged

mergify Bot mentioned this pull request Aug 26, 2025

v3.0: runtime: Avoid locking during stake vote rewards calculation (backport of #6900) anza-xyz/agave#7725

Closed

vadorovsky mentioned this pull request Aug 27, 2025

runtime: Avoid locking during stake vote rewards calculation anza-xyz/agave#7742

Merged

vadorovsky mentioned this pull request Sep 5, 2025

bump solana-stake-interface from 2.0.0 to 2.0.1 anza-xyz/agave#7919

Merged

mergify Bot mentioned this pull request Sep 9, 2025

v3.0: bump solana-stake-interface from 2.0.0 to 2.0.1 (backport of #7919) anza-xyz/agave#7958

Merged

mergify Bot mentioned this pull request Sep 11, 2025

v3.0: runtime: Avoid locking during stake vote rewards calculation (backport of #7742) anza-xyz/agave#8012

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

interface: Optimize the `StakeHistory::get` function#81

interface: Optimize the `StakeHistory::get` function#81
2501babe merged 1 commit intosolana-program:mainfrom
vadorovsky:optimize-stake-history

vadorovsky commented Aug 13, 2025 •

edited

Loading

Uh oh!

vadorovsky commented Aug 19, 2025

Uh oh!

Uh oh!

2501babe commented Aug 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vadorovsky commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vadorovsky commented Aug 19, 2025

Uh oh!

Uh oh!

2501babe commented Aug 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vadorovsky commented Aug 13, 2025 •

edited

Loading