consensus/ethash: improve cache/dataset handling#15864
Merged
Merged
Conversation
There are two fixes in this commit: Unmap the memory through a finalizer like the libethash wrapper did. The release logic was incorrect and freed the memory while it was being used, leading to crashes like in ethereum#14495 or ethereum#14943. Track caches and datasets using simplelru instead of reinventing LRU logic. This should make it easier to see whether it's correct.
fjl
force-pushed
the
ethash-release-fix
branch
from
7216825 to
253747a
Compare
karalabe
suggested changes
Jan 12, 2018
Member
karalabe
left a comment
karalabe
left a comment
There was a problem hiding this comment.
Minor nitpick, otherwise seems good to me.
| return calcCacheSize(epoch) | ||
| } | ||
|
|
||
| func calcCacheSize(epoch int) uint64 { |
Member
There was a problem hiding this comment.
Please document this method. I know it's private, but all methods in ethash are fully documented and I'd like to keep it so.
| return calcDatasetSize(epoch) | ||
| } | ||
|
|
||
| func calcDatasetSize(epoch int) uint64 { |
Member
There was a problem hiding this comment.
Please document this method. I know it's private, but all methods in ethash are fully documented and I'd like to keep it so.
| } | ||
| } | ||
|
|
||
| func TestCacheFileEvict(t *testing.T) { |
Member
There was a problem hiding this comment.
Please add a small doc describing what this tests.
This makes it possible to shorten the time taken for TestCacheFileEvict.
Contributor
Author
|
@karalabe PTAL |
karalabe
approved these changes
Jan 23, 2018
Member
karalabe
left a comment
karalabe
left a comment
There was a problem hiding this comment.
Fixed some tiny nitpicks, otherwise LGTM.
| func datasetSize(block uint64) uint64 { | ||
| // If we have a pre-generated value, use that | ||
| epoch := int(block / epochLength) | ||
| if epoch < len(datasetSizes) { |
Member
There was a problem hiding this comment.
len(datasetSized) -> maxEpoch (you changed this everywhere else).
| futureItem interface{} | ||
| } | ||
|
|
||
| func newlru(what string, maxItems int, new func(epoch uint64) interface{}) *lru { |
Member
There was a problem hiding this comment.
// newlru create a new least-recently-used cache for ither the verification caches
// or the mining datasets.
| used time.Time // Timestamp of the last use for smarter eviction | ||
| once sync.Once // Ensures the cache is generated only once | ||
| lock sync.Mutex // Ensures thread safety for updating the usage time | ||
| func newCache(epoch uint64) interface{} { |
Member
There was a problem hiding this comment.
// newCache creates a new ethash verification cache and returns it as a plain Go
// interface to be usable in an LRU cache.
| used time.Time // Timestamp of the last use for smarter eviction | ||
| once sync.Once // Ensures the cache is generated only once | ||
| lock sync.Mutex // Ensures thread safety for updating the usage time | ||
| func newDataset(epoch uint64) interface{} { |
Member
There was a problem hiding this comment.
// newDataset creates a new ethash mining dataset and returns it as a plain Go
// interface to be usable in an LRU cache.
prestonvanloon
pushed a commit
to prestonvanloon/go-ethereum
that referenced
this pull request
Apr 2, 2018
* consensus/ethash: add maxEpoch constant * consensus/ethash: improve cache/dataset handling There are two fixes in this commit: Unmap the memory through a finalizer like the libethash wrapper did. The release logic was incorrect and freed the memory while it was being used, leading to crashes like in ethereum#14495 or ethereum#14943. Track caches and datasets using simplelru instead of reinventing LRU logic. This should make it easier to see whether it's correct. * consensus/ethash: restore 'future item' logic in lru * consensus/ethash: use mmap even in test mode This makes it possible to shorten the time taken for TestCacheFileEvict. * consensus/ethash: shuffle func calc*Size comments around * consensus/ethash: ensure future cache/dataset is in the lru cache * consensus/ethash: add issue link to the new test * consensus/ethash: fix vet * consensus/ethash: fix test * consensus: tiny issue + nitpick fixes
mariameda
pushed a commit
to NiluPlatform/go-nilu
that referenced
this pull request
Aug 23, 2018
* consensus/ethash: add maxEpoch constant * consensus/ethash: improve cache/dataset handling There are two fixes in this commit: Unmap the memory through a finalizer like the libethash wrapper did. The release logic was incorrect and freed the memory while it was being used, leading to crashes like in ethereum#14495 or ethereum#14943. Track caches and datasets using simplelru instead of reinventing LRU logic. This should make it easier to see whether it's correct. * consensus/ethash: restore 'future item' logic in lru * consensus/ethash: use mmap even in test mode This makes it possible to shorten the time taken for TestCacheFileEvict. * consensus/ethash: shuffle func calc*Size comments around * consensus/ethash: ensure future cache/dataset is in the lru cache * consensus/ethash: add issue link to the new test * consensus/ethash: fix vet * consensus/ethash: fix test * consensus: tiny issue + nitpick fixes
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There are two fixes in this PR:
Unmap the memory through a finalizer like the libethash wrapper did. The
release logic was incorrect and freed the memory while it was being
used, leading to crashes like in #14495 or #14943.
Track caches and datasets using simplelru instead of reinventing LRU
logic. This should make it easier to see whether it's correct.
Fixes #14495, #14943