Skip to content

config: move crash and stateproof DB defaults to hot dir#5817

Merged
algorandskiy merged 8 commits intoalgorand:masterfrom
cce:crashdbhotdir
Nov 16, 2023
Merged

config: move crash and stateproof DB defaults to hot dir#5817
algorandskiy merged 8 commits intoalgorand:masterfrom
cce:crashdbhotdir

Conversation

@cce
Copy link
Copy Markdown
Contributor

@cce cce commented Nov 1, 2023

Summary

This moves the crash and stateproof DBs to default to using the hot data directory. Like the ledger DB, they are frequently written while processing consensus and block updates, and are also fairly small in size, so they would benefit from being located in the hot tier.

Resolves #5831

Test Plan

Updated TestEnsureAndResolveGenesisDirs_hierarchy and TestConfiguredDataDirs.
Added TestEnsureAndResolveGenesisDirs_migrate and TestEnsureAndResolveGenesisDirs_migrateFail.

@winder
Copy link
Copy Markdown
Contributor

winder commented Nov 2, 2023

Why?

@cce cce added the Enhancement label Nov 3, 2023
@codecov
Copy link
Copy Markdown

codecov Bot commented Nov 3, 2023

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (8c54c22) 55.63% compared to head (1d8c990) 55.61%.
Report is 14 commits behind head on master.

Files Patch % Lines
config/localTemplate.go 90.47% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5817      +/-   ##
==========================================
- Coverage   55.63%   55.61%   -0.02%     
==========================================
  Files         475      475              
  Lines       66842    66860      +18     
==========================================
- Hits        37187    37184       -3     
- Misses      27141    27156      +15     
- Partials     2514     2520       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Copy Markdown
Contributor

@algorandskiy algorandskiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree on the crash db move but do we really write stateproof db that often?

@cce
Copy link
Copy Markdown
Contributor Author

cce commented Nov 6, 2023

The stateproof DB is

  • updated every time you observe a stateproof signature message — these start coming in 128 rounds into the interval, each participant issues them staggered over the remaining 128 rounds, and you have to update the DB for each observation.
  • You also need to query the state proof DB to sign the state proof transaction.
  • The voters tracker in ledger calls through OnPrepareVoterCommit to stateproof.Worker.OnPrepareVoterCommit to do gets & writes to the state proof DB as part of the tracker prepareCommit, so you can think of state proof DB as an extension to the tracker DB, since it is read/written/flushed at regular intervals along with tracker DB..

I guess it depends on how cold you think cold will be..

Copy link
Copy Markdown
Contributor

@algorandskiy algorandskiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but use existing file name constants instead of literals

Comment thread config/localTemplate.go Outdated
Comment thread config/localTemplate.go Outdated
algorandskiy
algorandskiy previously approved these changes Nov 16, 2023
Comment thread config/config_test.go
Comment thread config/localTemplate.go
Comment thread config/localTemplate.go
Copy link
Copy Markdown
Contributor

@winder winder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good. Left some suggestions that I think should be adopted, but don't need to block the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

put agreement DB in hot storage tier

3 participants