Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validator set persistence #200

Merged
merged 38 commits into from
Feb 17, 2025
Merged

Validator set persistence #200

merged 38 commits into from
Feb 17, 2025

Conversation

pk910
Copy link
Member

@pk910 pk910 commented Dec 18, 2024

This PR adds a validators table to the db and persists the finalized validator set there.

The validator cache is responsible for writing and updating validator entries.
It uses a batched approach for mass updates to keep the db load manageable.

This PR touches a lot of code as the validator set accessors changed.
Instead of fetching the full validator set and doing filtering/aggregation logic in the page handlers, the page handlers will now only fetch a filtered subset of the validator set.

This PR should significantly lower the memory consumption of dora when running in a network with large validator sets (holesky, mainnet)

@pk910 pk910 added the build-docker-image Automatically build docker image for PR branch label Jan 24, 2025
@pk910
Copy link
Member Author

pk910 commented Jan 25, 2025

This branch is now running on https://dora-holesky.pk910.de/ for the holesky testnet.
The first results look very promising - dora runs with a total memory consumption of ~2-3GB :)
Compared to a consumption of 10GB before the update, that's quite an improvement.

Issues found:

  • Active Validators, Staked Ether & Average Balance stats on index page show 0 during the first few slots of a new epoch (latest epoch stats are still loading)
  • Validators list status filter does not work

@pk910 pk910 linked an issue Feb 2, 2025 that may be closed by this pull request
@pk910 pk910 changed the title WIP: validator set persistence Validator set persistence Feb 7, 2025
@pk910 pk910 marked this pull request as ready for review February 7, 2025 20:12
@pk910 pk910 mentioned this pull request Feb 9, 2025
db/validators.go Show resolved Hide resolved
db/validators.go Show resolved Hide resolved
indexer/beacon/pubkeycache.go Show resolved Hide resolved
indexer/beacon/validatoractivity.go Outdated Show resolved Hide resolved
@pk910 pk910 merged commit 83f33f7 into master Feb 17, 2025
14 checks passed
@pk910 pk910 removed the build-docker-image Automatically build docker image for PR branch label Feb 17, 2025
@pk910 pk910 deleted the pk910/validators-in-db branch February 17, 2025 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Duty calculation is wrong with unbalanced validator set
2 participants