SearchableSnapshotDirectory should not evict cache files when closed#66264
Merged
tlrx merged 3 commits intoelastic:7.xfrom Dec 14, 2020
Merged
Conversation
…lastic#66173) This commit changes the SearchableSnapshotDirectory so that it does not evict all its cache files at closing time, but instead delegates this work to the CacheService. This change is motivated by the fact that Lucene directories are closed as the consequence of applying a new cluster state and as such the closing is executed within the cluster state applier thread; and we want to minimize disk IO operations in such thread (like deleting a lot of evicted cache files). It is also motivated by the future of the searchable snapshot cache which should become persistent. This change is built on top of the existing SearchableSnapshotIndexEventListener and a new SearchableSnapshotIndexFoldersDeletionListener (see elastic#65926) that are used to detect when a searchable snapshot index (or searchable snapshot shard) is removed from a data node. When such a thing happens, the listeners notify the CacheService that maintains an internal list of removed shards. This list is used to evict the cache files associated to these shards as soon as possible (but not in the cluster state applier thread) or right before the same searchable snapshot shard is being built again on the same node. In other situations like opening/closing a searchable snapshot shard then the cache files are not evicted anymore and should be reused.
tlrx
added
>enhancement
:Distributed/Snapshot/Restore
Collaborator
|
Pinging @elastic/es-distributed (Team:Distributed) |
tlrx
deleted the
use-listeners-to-manage-cache-files-for-removed-shards-7.x
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit changes the SearchableSnapshotDirectory so
that it does not evict all its cache files at closing time, but
instead delegates this work to the CacheService.
This change is motivated by the fact that Lucene directories
are closed as the consequence of applying a new cluster
state and as such the closing is executed within the cluster
state applier thread; and we want to minimize disk IO
operations in such thread (like deleting a lot of evicted
cache files). It is also motivated by the future of the
searchable snapshot cache which should become persistent.
This change is built on top of the existing
SearchableSnapshotIndexEventListener and a new
SearchableSnapshotIndexFoldersDeletionListener
(see #65926) that are used to detect when a searchable
snapshot index (or searchable snapshot shard) is
removed from a data node.
When such a thing happens, the listeners notify the
CacheService that maintains an internal list of removed
shards. This list is used to evict the cache files associated
to these shards as soon as possible (but not in the cluster
state applier thread) or right before the same searchable
snapshot shard is being built again on the same node.
In other situations like opening/closing a searchable
snapshot shard then the cache files are not evicted
anymore and should be reused.
Backport of #66173 for 7.11