Skip to content

v3.0: runtime: Avoid locking during stake vote rewards calculation (backport of #6900)#7725

Closed
mergify[bot] wants to merge 1 commit intov3.0from
mergify/bp/v3.0/pr-6900
Closed

v3.0: runtime: Avoid locking during stake vote rewards calculation (backport of #6900)#7725
mergify[bot] wants to merge 1 commit intov3.0from
mergify/bp/v3.0/pr-6900

Conversation

@mergify
Copy link
Copy Markdown

@mergify mergify Bot commented Aug 26, 2025

Problem

calculate_stake_vote_rewards was storing accumulated rewards per vote account in a DashMap, which then was used in a parallel iterator over all stake delegations.

There are over 1,000,000 stake delegations and around 1,000 validators. Each thread processed one of the stake delegations and tried to acquire a lock on a DashMap shard corresponding to a validator. Given that the number of validators is disproportionally small and they have thousands of delegations, such solution resulted in high contention, with some threads spending the most of their time on waiting for lock.

The time spent on these calculations was ~232.21ms:

redeem_rewards_us=232210i

Threads spent 65% of their time on waiting for locks:

reward-calculation-before-1

Summary of Changes

Fix that by:

The time spent on reward calculations goes down to ~48.78ms:

redeem_rewards_us=48781i

Threads spend the most of time doing actual calculations:

epoch-rewards-final

Fixes #6899


This is an automatic backport of pull request #6900 done by [Mergify](https://mergify.com).

`calculate_stake_vote_rewards` was storing accumulated rewards per vote
account in a `DashMap`, which then was used in a parallel iterator over
all stake delegations.

There are over 1,000,000 stake delegations and around 1,000 validators.
Each thread processes one of the stake delegations and tries to acquire
the lock on a `DashMap` shard corresponding to a validator. Given that
the number of validators is disproportionally small and they have
thousands of delegations, such solution results in high contention,
with some threads spending the most of their time on waiting for lock.

The time spent on these calculations was ~208.47ms:

```
redeem_rewards_us=208475i
```

Fix that by:

* Removing the `DashMap` and instead using `fold` and `reduce`
  operations to build a regular `HashMap`.
* Pre-allocating the `stake_rewards` vector and passing
  `&mut [MaybeUninit<PartitionedStakeReward>]` to the thread pool.
* Pulling the optimization of `StakeHistory::get` in
  `solana-stake-interface`. solana-program/stake#81

```
redeem_rewards_us=48781i
```

(cherry picked from commit e752ae6)

# Conflicts:
#	Cargo.toml
#	programs/sbf/Cargo.toml
@mergify mergify Bot added the conflicts label Aug 26, 2025
@mergify mergify Bot requested a review from a team as a code owner August 26, 2025 16:57
@mergify
Copy link
Copy Markdown
Author

mergify Bot commented Aug 26, 2025

Cherry-pick of e752ae6 has failed:

On branch mergify/bp/v3.0/pr-6900
Your branch is up to date with 'origin/v3.0'.

You are currently cherry-picking commit e752ae6d8.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   Cargo.lock
	modified:   cli/Cargo.toml
	modified:   genesis/Cargo.toml
	modified:   programs/sbf/Cargo.lock
	modified:   runtime/src/bank.rs
	modified:   runtime/src/bank/partitioned_epoch_rewards/calculation.rs
	modified:   runtime/src/bank/partitioned_epoch_rewards/distribution.rs
	modified:   runtime/src/bank/partitioned_epoch_rewards/epoch_rewards_hasher.rs
	modified:   runtime/src/bank/partitioned_epoch_rewards/mod.rs
	modified:   runtime/src/bank/tests.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   Cargo.toml
	both modified:   programs/sbf/Cargo.toml

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@mergify mergify Bot added the conflicts label Aug 26, 2025
@vadorovsky vadorovsky closed this Aug 27, 2025
@jstarry jstarry deleted the mergify/bp/v3.0/pr-6900 branch August 27, 2025 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant